_LargeFileCopy UDF

[llewxam]

Good results! And of course it will be a drop-in replacement for most folks, though I had to make some modifications since I don't use your internal $sFunction, I prefer an AdlibRegister to track speed, progress etc.

Anyway, what I did was found 2 folders on my NAS with roughly the same GB of data but with very different fie sizes to get a good metric on very large VS very small files. The files were transferred to my PC, mid-range performing NAS, gigabit connection all the way through.

Code folder, 6,934 files, 4.80GB, vast majority under 100KB

LFC November 2011: date preservation on -> date preservation off, speed loss

7:58 -> 6:15, 22%

LFC September 2012: date preservation on -> date preservation off, speed loss

5:08 -> 5:07, margin of error

TV show folder, 28 files, 4.87GB, all over 100MB

LFC November 2011: date preservation on -> date preservation off, speed loss

5:38 -> 5:31, 3%

LFC September 2012: date preservation on -> date preservation off, speed loss

4:33 -> 4:35, margin of error

I was frankly thinking that your date preservation wasn't working since it took such little time, but it is working fine. VERY nice! And speed without updating the timestamp has improved a lot as well showing lots of optimizations!!

Now for the questions. I dug through the code a bit but still have a few grey areas:

• Are there any file attributes a file could have that would cause the copy to fail?
• Can the timestamps be applied to the creation of a folder, or just the files?
• Is it moot to include the "accessed" timestamp, since it will always be at the moment it is copied? (The source's "accessed" is changed as well)
• The code has a comment "disable NTFS specific features" and goes in to _LFC_PathGetDrive, but I don't see anything I recognize as disabling an NTFS feature, what is that referring to?
• _LFC_CheckDestination looks to be specifically removing UNC path prefixes via StringRegExpReplace functions, is that going to cause a problem for long path lengths? Or am I reading that wrong, I still haven't gotten to the point of using regular expressions.......

Great stuff, looking forward to more testing, I didn't have that much time tonight. I only had time to test the date functions, MD5 at the least will happen soon, I use Ward's so I expect that to be a good deal faster.

Ian

[llewxam]

Will hopefully get time to play with the one you just posted tomorrow some time........ Unless you update it!

[wraithdu]

Awesome results I'm glad to see the speed improvements. So to answer some questions.

• There aren't any attributes to my knowledge that would fail a copy operation. If a file exists with the System or Hidden attribute, then calling CreateFile with CREATE_ALWAYS will fail unless those attributes are specified in the call. However LFC pre-deletes existing files if $fOverwrite is true, after first setting FILE_ATTRIBUTE_NORMAL to prevent any read-only failures.
• Yes timestamps can be applied to folders with the same functions (FILE_FLAG_BACKUP_SEMANTICS is supplied to get folder handles). LFC does not handle folders internally however.
• Accessed times are weird (do some googling). They are actually disabled by default in Vista+. Even so they are updated somehow, you can look and see that fact, but I don't know exactly what triggers it. And even then there is a 1 hour resolution on NTFS, and a 1 day resolution on FAT. Honestly I wouldn't worry either way. If you're setting one timestamp, you might as well do them all.
• If either the source or destination is not NTFS, then checking / setting Encryption, Compression, and ACLs are all disabled, since they are NTFS specific features. This is controlled with the $IsNTFS and $fACL flags.
• Paths are manipulated in two functions. _LFC_FixPath only operates on paths without a : (colon). In that case, it ensures a path is in one of the following formats: .. (relative), . (rooted in working dir), or (unc). _LFC_CheckDestination checks whether the specified destination is a directory or a file (it is assumed to be a directory if it does not exist and ends in a ). If it is a directory then the source file name is appended and used as the destination file name. If the destination file exists, $fOverwrite is checked and the file is deleted if true, otherwise the copy fails. Otherwise $fCreate is checked and the destination parent directory is created if true and it doesn't exist. If the destination parent does not exist and $fCreate is false, the copy will fail.

And one last note on file verification. I highly recommend using the bit by bit comparison. If you think about it, hashing MUST read all the bits anyway, and has the extra expense of the hashing operation. So unless you are storing a hash for checking later on, in this case it is much more efficient to do the bit comparison.

Edited September 29, 2012 by wraithdu

[llewxam]

I spent some more time crushing this UDF today, what I did differently this time was I didn't add it to my Sync tool, I just put together a very simple script to do a 3-lap run of one of my folders and took the average. I decided to bite the bullet and do this for all 3 builds of LFC I had, as well as with and without date preservation. That amounted to copying my code folder from my NAS to my hard drive 18 times!

Below, I refer to LFC1, LFC2, and LFC3. 1 was from November 2011, 2 was from September 26 2012, and 3 was September 28 2012. The folder has 6,947 files 4.80GB.

LFC1 with date preservation off, average time 370.466 seconds

LFC1 with date preservation on, average time 525.550 seconds (yeah, my method sucked!! haha)

LFC2 with date preservation off, average time 344.128 seconds

LFC2 with date preservation on, average time 344.574 seconds

LFC3 with date preservation off, average time 329.003 seconds

LFC3 with date preservation on, average time 342.791 seconds

I was surprised at LFC3 with date preservation on. LFC2 with and without barely changed, LFC3 had a huge penalty. The without times were a lot better though.

I hope to start beating on some of the other abilities of the UDF tomorrow a bit, I want to look at penalties for attribute preservation as I can see that being useful as well as testing removal of all attributes.

Ian

[wraithdu]

Nice testing! I appreciate all the HDD cycles However LFC3-1 seems like a fluke to me. There weren't any changes between your builds 2 and 3 that should have made a difference unless you were using hash verification or copying ACLs. Odd.

I realize that I need to make a few more changes for UNC paths in my PathGetDrive function. It won't affect your testing though, so carry on. It would be interesting to see at this point how LFC stacks up against a few other methods like AutoIt's FileCopy, cmd's copy, or even robocopy. I'm not suggesting you have to do that though

[llewxam]

I already had planned on throwing AutoIt's FileCopy in there for comparison, as well as just a good-ole Windows copy and paste with a stopwatch for kicks. And your UDF was already faster than RoboCopy I think!

As for the LFC3-1, I know it seems "flukey", but here are the raw numbers from the 3 runs I had it do:

LFC3 with time preservation off

339.427585039488 seconds

LFC3 with time preservation off

323.300680584697 seconds

LFC3 with time preservation off

324.281295648437 seconds

average time 329.003187090874

Don't read in to the first run being the slowest, in many tests I ran the first was the fastest, there was no sort of caching going on. I was also not using any verification, ACL, or attribute copying, just the files.

LFC1, LFC2&3 with no date preservation "$CopyTheFile = _LargeFileCopy($FileList[$a], $CopyDestination, BitOR(1, 2))"

LFC2&3 with date preservation "$CopyTheFile = _LargeFileCopy($FileList[$a], $CopyDestination, BitOR(1, 2, 128, 256, 512))"

Ian

PS - Don't worry about the HDD cycles, it's an SSD, no cycles to speak of!!

Edited September 30, 2012 by llewxam

[wraithdu]

Next update, few changes in this one. I've updated some of the path routines to fully support all the UNC type paths that were introduced for long path support. This was mainly for the cluster size calculation (which was still using an old function that only supported drives with letters) for unbuffered copying. I also swapped out the CRT functions for the Rtl* functions (the Rtl* functions use kernel32.dll instead of msvcrt.dll). These changes affect the bit by bit comparison, and the _LargeRawCopyUnbuffered function. Lastly I stripped out the functions that I had originally copied from my _FileEx UDF and just included the FileEx UDF. I suppose at the start I wasn't using much of it, which is why I just copied them in, but now I'm using practically the whole thing, so I'd rather just maintain one copy of those functions. None of these changes should have any impact on performance.

I think I might actually be done messing with this thing now, unless you find any bugs or have any more additions.

*snip*

Edited October 4, 2012 by wraithdu

[llewxam]

Only thing I'd really like to ask is that the created folder timestamps match up to the original timestamps (I guess yet another flag...), then I'd say it's pretty much perfect! I haven't gotten around to testing the unbuffered copy yet, what do you see comparing the speed of buffered VS unbuffered?

I just did one of the silly tests I was planning on doing over the weekend - your UDF took my code folder from my NAS in 5:55 seconds, FileCopy took 8:10

Also, I am currently building a new version of my Sync tool around the new UDF and it's going to be sweeeeeet

Ian

[wraithdu]

According to xcopy, unbuffered copying is supposed to be faster for very large files. I don't know if I buy it though.

So regarding folder timestamps... how do you envision LFC handling these? LFC only gets a source file and a destination (folder or file). The folders it creates are the parent folders to the destination file, but it knows nothing about how these parents relate to the source. So, what do you expect the logic to be, and what destination folder(s) do you expect LFC to time stamp?

Programmatically it is trivial, the function calls to read a set timestamps are exactly the same as the files, and you could use the functions from _FileEx to do so. But I'm not sure how to sanely implement this given the fact LFC has no way to relate source to destination since it only operates file by file.

[llewxam]

Gotcha. Without parsing the code to get the answer to this myself, I would imagine that if the UDF is sending a file to "C:testdirnew foldercodesync" and the only folder in that path that exists is "testdir", then the UDF would know that it has to create "new folder", "code", and "sync" individually, not all in one line. Therefore, any time the UDF has to create a folder, grab the timestamp from the folder in $sSrc.

I suppose the logic would be:

1) Realize a folder has to be created

2) Parse $sSrc up to the location of the missing folder in $sDest

3) Grab timestamp from the parsed $sSrc

4) Apply timestamp to created folder

5) Repeat until all folders have been created

If that is oversimplifying the issue sorry, just thinking of the steps..........

Also, I will find time to run some speed comparisons of buffered VS unbuffered, it would be easy to divert a file copy according to size to the appropriate routine.

Ian

Edited October 2, 2012 by llewxam

[wraithdu]

That's what I thought you meant... but realize that in terms of individual file copies and LFC, the source folder structure has nothing to do with the destination folder structure and final location of the copied file. Indiscriminately copying source folder time stamps with no frame of reference is irresponsible.

This is really the place where a tool built around LFC comes into play, like my A3COPY project (which needs some major modification based on the new LFC UDF). A3COPY works just like xcopy, and will preserve folder as well as file timestamps where appropriate and instructed to do so. That only works, of course, when the source is a folder. Now you can reasonably say, OK, we're cloning a whole directory tree and I know where the time stamps are coming from.

So, I guess the answer is no to your request, folder time stamping is additional functionality outside the scope of LFC. I loathe the idea of building a full directory copy into LFC, as that is practically the entire scope of a full project like A3COPY(all the handling for attributes, especially compression and encryption is really laborious). I think you should be able to put something more targeted together for your sync tool using the file time functions in _FileEx fairly easily.

[llewxam]

Agreed about making LFC's job dealing with whole folders being outside the scope of the UDF. No prob, I'll shoehorn that in to my app as it gets closer to completion.

Thanks!

Ian

[wraithdu]

Major sweet awesome updates in OP. Run don't walk!

@llewxam

Take note, additional flags for Encryption and Compression have been separated from the Attributes flag. This will be script breaking for you.

I've also optimized some more. This could be valuable for your copy tool, but you'll see more of an improvement when copying to a network destination versus local file system. Based on the changes, the fastest code path is copying to a file name which does not exist with the creation flag off. This of course assumes you create your destination files' parent directory first.

Edited October 4, 2012 by wraithdu

[llewxam]

The new flags don't hurt what I have been testing and building SyncV4 with, BitOR(1, 2, 128, 256, 512). I probably won't have time to integrate it in to my current build to test it out tonight, but am slightly confused by your last sentence. Are you suggesting it would be better for me to script in the logic for folder creation rather than using your 2 flag? Overall I mean, would I see a net gain by using the built-in DirCreate as needed and leaving the 2 off? I could do that, and it would give me a different way of handling the timestamp preservation of the newly created folders than I was thinking of...

Ian

[wraithdu]

I was testing today on the work network for an app I wrote. The reason the code path I mentioned is fastest, is because otherwise the destination existence (file or folder) is checked on every file copy. It is unavoidable to do this at least once. I use the _FileEx_GetAttributes function to both check for existence and get the required info to check if the destination is a directory. However this function returns much faster when the destination doesn't exist. Additionally, if you pass flag 2, then _FileEx_CreateDirectory first checks the full destination path, then each segment from the root to determine what needs to be created.

As I mentioned, on a local file system this is negligible at best. However over a network, it dropped copy time for 25 jpgs ~35 K in size from 9 seconds to 7. You can extrapolate from there for a huge number of files.

One thing to note on setting directory times (and I had to account for this in A3COPY), if you copy any new files into the folder then those times are updated. So you have to set the times as the very last thing you do to the folder.

[llewxam]

Makes sense. It almost makes me think that I should build an array of all created directories as the script runs, and at the end of the script have it set the timestamps for those folders. And since my script would know the relative paths (from and to), it would not be a big deal to track the source, get it's timestamp, and the created folder path to apply those timestamps at the end.

On the other hand, it makes me wonder if it is worth the effort. I am possibly being more picky about this than it is worth! Relatively small overhead, but really necessary?? Perhaps not.

And thanks for the thoughts on the path creation. It's a challenging concept to overcome - I would hate to have my script parse the entire list of files up front to find folders needing creation and create them prior to the files being copies to avoid the 2 flag, because what if the copy was stopped before completion, because then there would be a ton of empty folders in the destination. I'll have to stew on that.

Ian

[wraithdu]

It depends on how your tool parses the source. A3COPY for example uses an iterative search. For each source directory, it pre-creates the destination and calls _LargeFileCopy without the 2 flag. If more folders are found, it stores the source and destination for time stamps, attributes, etc for later, and adds the new folder to the search array. Rinse and repeat.

Really you're only going to see a difference if you are copying a lot of files into one directory. If you have 1 file in 1000 directories, not so much.

If you're interested in keeping a running list of folders and such, take a look at the A3COPY source. I implemented an algorithm to process stored directories as soon as possible, thereby keeping the storage array as small as possible, instead of storing everything until the end.

As far as being picky... depends on your backup / sync scenario I suppose. I like to keep file times and attributes, including for folders. I would provide it as an option to your users.

Edited October 5, 2012 by wraithdu

[wraithdu]

Update to fix some x86 bugs due to missing function exports. Sorry.

[Syed23]

I can't help without an example of the problem.

Hi Wraithdu,

Here i have an example which will help me to create a directory and copy 58MB file. but the copy AVI progress hangs, is that wrong using avi on GUI during copy? sorry if i have some stupid question.

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6
#include <_LargeFileCopy.au3>
#include "RecFileListToArray.au3"
#include"Array.au3"
#include <GUIConstantsEx.au3>
#include <AVIConstants.au3>

Global $msg, $timer, $ret
Global $src, $SrcBut, $SrcLab, $Srcres
Global $Dest, $DestLab, $DestBut, $DestRes, $GetList, $go,$ani1
_Main()

Func _Main()

GUICreate("File Copy", 550, 300)
$SrcLab = GUICtrlCreateLabel("Select Source Folder", 5, 10, 110, 40)
$src = GUICtrlCreateInput("", 115, 5, 400, 20)
$SrcBut = GUICtrlCreateButton("...", 520, 5, 25, 20)
$DestLab = GUICtrlCreateLabel("Select Destination Folder", 5, 60, 110, 40)
$Dest = GUICtrlCreateInput("", 115, 55, 400, 20)
$DestBut = GUICtrlCreateButton("...", 520, 55, 25, 20)
$go = GUICtrlCreateButton("Copy Large File", 200, 150, 150, 50)
$ani1 = GUICtrlCreateAvi(@SystemDir & "shell32.dll", 165, 10, 230,BitAND($ACS_AUTOPLAY,$ACS_CENTER))
GUICtrlSetState(-1,$GUI_HIDE)


GUISetState()



While 1
$msg = GUIGetMsg()

If $msg = $SrcBut Then
$Srcres = FileSelectFolder("Select Folder", @DesktopDir, 1)
GUICtrlSetData($src, $Srcres)

EndIf

If $msg = $GUI_EVENT_CLOSE Then ExitLoop

If $msg = $DestBut Then
$DestRes = FileSelectFolder("Select Folder", @DesktopDir, 1)
GUICtrlSetData($Dest, $DestRes)

EndIf

If $msg = $go Then
If GUICtrlRead($src) = "" Then
MsgBox(48, "Source Path - Warning", "Source path can not be blank")
ElseIf GUICtrlRead($Dest) = "" Then
MsgBox(48, "Destination Path - Warning", "Destination path can not be blank")
ElseIf GUICtrlRead($src) <> "" And GUICtrlRead($Dest) <> "" Then
GUICtrlSetState($ani1,$GUI_SHOW)
GUICtrlSetState($ani1, 1)
$timer = TimerInit()
$GetList = _RecFileListToArray(GUICtrlRead($src) & "", "*", 0, 1, 1)

;Exit
If IsArray($GetList) Then
For $i = 0 To UBound($GetList) -1
$ret = _LargeFileCopy(GUICtrlRead($src) & "" & $GetList[$i], GUICtrlRead($Dest) & "" & $GetList[$i], BitOR(1, 2, 8))
Next
EndIf
EndIf
EndIf
WEnd
EndFunc   ;==>_Main

[wraithdu]

Not a stupid question, but your script is not right. Your parameters to GUICtrlCreateAvi are wrong (see the help file). And you need to use BitOR to combine styles, not BitAND. I did notice though that AutoIt seems to have repainting issues on the AVI if you minimize the window, the AVI no longer seems to play until it loops and starts over again (but this has nothing to do with my UDF). Here's a fixed version of your script with the addition of a filename label and a few fixes, to show that it's working.

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6
#include <_LargeFileCopy.au3>
#include "RecFileListToArray.au3"
#include"Array.au3"
#include <GUIConstantsEx.au3>
#include <AVIConstants.au3>

Global $msg, $timer, $ret
Global $src, $SrcBut, $SrcLab, $Srcres
Global $Dest, $DestLab, $DestBut, $DestRes, $GetList, $go, $ani1, $label
_Main()

Func _Main()

    GUICreate("File Copy", 550, 300)
    $SrcLab = GUICtrlCreateLabel("Select Source Folder", 5, 10, 110, 40)
    $src = GUICtrlCreateInput("", 115, 5, 400, 20)
    $SrcBut = GUICtrlCreateButton("...", 520, 5, 25, 20)
    $DestLab = GUICtrlCreateLabel("Select Destination Folder", 5, 60, 110, 40)
    $Dest = GUICtrlCreateInput("", 115, 55, 400, 20)
    $DestBut = GUICtrlCreateButton("...", 520, 55, 25, 20)
    $go = GUICtrlCreateButton("Copy Large File", 200, 150, 150, 50)
    ; ACS_AUTOPLAY is not doing anything, playback stops as soon as you hide the control
    $ani1 = GUICtrlCreateAvi(@SystemDir & "shell32.dll", 165, 10, 230, 275, 75, BitOR($ACS_CENTER, $ACS_TRANSPARENT))
    GUICtrlSetState(-1, $GUI_HIDE)
    $label = GUICtrlCreateLabel("", 10, 210, 540, 20)

    GUISetState()

    While 1
        $msg = GUIGetMsg()

        If $msg = $SrcBut Then
            $Srcres = FileSelectFolder("Select Folder", @DesktopDir, 1)
            GUICtrlSetData($src, $Srcres)

        EndIf

        If $msg = $GUI_EVENT_CLOSE Then ExitLoop

        If $msg = $DestBut Then
            $DestRes = FileSelectFolder("Select Folder", @DesktopDir, 1)
            GUICtrlSetData($Dest, $DestRes)

        EndIf

        If $msg = $go Then
            If GUICtrlRead($src) = "" Then
                MsgBox(48, "Source Path - Warning", "Source path can not be blank")
            ElseIf GUICtrlRead($Dest) = "" Then
                MsgBox(48, "Destination Path - Warning", "Destination path can not be blank")
            ElseIf GUICtrlRead($src) <> "" And GUICtrlRead($Dest) <> "" Then
                GUICtrlSetState($ani1, $GUI_SHOW)
                GUICtrlSetState($ani1, 1) ; apparently this is needed, as $GUI_HIDE stops playback
                $timer = TimerInit()
                $GetList = _RecFileListToArray(GUICtrlRead($src) & "", "*", 0, 1, 1)

                ;Exit
                If IsArray($GetList) Then
                    For $i = 0 To UBound($GetList) - 1
                        GUICtrlSetData($label, $GetList[$i])
                        $ret = _LargeFileCopy(GUICtrlRead($src) & "" & $GetList[$i], GUICtrlRead($Dest) & "" & $GetList[$i], BitOR(1, 2, 8))
                    Next
                EndIf
                GUICtrlSetData($label, "")
                GUICtrlSetState($ani1, $GUI_HIDE)
            EndIf
        EndIf
    WEnd
EndFunc   ;==>_Main

Edited November 12, 2012 by wraithdu