_LargeFileCopy UDF

[wraithdu]

Several people have had problems with their GUIs freezing when they try to copy large files using FileCopy(). This is because FileCopy() is a blocking function, and the GUI stops responding to messages from the OS during the operation. If that goes on long enough, the OS thinks it is hung and you get the (Not Responding) thing going on.

Here's a function that copies large files using Windows API calls in 8MB 2 MB chunks. This keeps the GUI alive and responsive. I've added additional copy verification functionality as well (disabled by default).

Take a look at my A3COPY project for a great usage example of this UDF.

Update: 1/31/13

- Moved a few more functions into _FileEx.

- Fixed a bug in GetDiskClusterSize function.

Update: 10/8/12

- Fixed bug due to missing RtlCopyMemory and RtlCompareMemory exports in x86 kernel32.dll. Now uses RtlMoveMemory and RtlCompareMemory from ntdll.dll (sorry).

Update: 10/4/12

Read NOTES at top of UDF

- Reduced default buffer size to 2 MB

- Added flags for copying source file attributes, all three file times individually, ACL security descriptors and ownership, and NTFS specific compression and encryption

- Optimized buffer creation

- Enhanced path handling to fully support UNC and network conventions ( C:... , ?C:... , srvshare... , ?UNCsrvshare... )

- Removed C runtime functions (msvcrt.dll) in favor of native kernel32 functions

- Code cleanup and many optimizations in LFC and the accompanying _FileEx UDF

Update: 4/15/11

- Added additional verification methods - bit by bit, and size only

- The default verification method (ie iFlag 8) is now bit by bit - it is faster and less computationally intensive than MD5 hashing, it also allows the function to bail out early in the case of a failed copy operation

Update: 3/9/11

SCRIPT BREAKING CHANGES

- Combined some parameters into one iFlags parameter

- Re-ordered some parameters based on priority

- Added a parameter to specify a different read buffer size

- Destination file is now opened with GENERIC_READ | GENERIC_WRITE per MSDN for better compatibility copying files across a network

- No longer create destination directory structure unless specified in iFlags parameter

Update: 9/20/10

- Removed tracexxx's hashing functions (sorry!) in favor of a different method which is not RAM limited

- Hashing algorithm can be specified (those supported by Crypt.au3: CALG_MD2, CALG_MD4, CALG_MD5, CALG_SHA1)

Update: 8/26/10

- Fix for failed verification of 0 byte files

- Added check for 0 byte source in RAW functions

Update: 8/22/10

- Small change to CreateFile function to avoid errors when opening SYSTEM or HIDDEN files when specifying CREATE_ALWAYS.

Update: 8/10/10

- Added _LargeFileCopyUnbuffered and _LargeRawCopyUnbuffered which will copy files using unbuffered I/O.

- MD5 hashing of the source file / memory block is now done on the fly from the memory buffer. This avoids an additional read of the source file later on and saves time.

- Destination file MD5 hashing is now done with trancexxx's nice hashing UDF. It uses the Windows API and is really fast.

Update: 7/9/2010

- Added _LargeRawCopy which will copy large memory blocks (usually from an embedded resource) to a file in the same manner as _LargeFileCopy. This function also has the verify functionality via MD5 hash.

Update: 7/8/2010

- Added optional user function and variable parameters. This function, if given, will be called after each write operation with the total bytes written, total file size in bytes, and the user given variable. This is perfect for a copy-with-progess type of function call.

Example

expandcollapse popup

#NoTrayIcon
#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6
#include <_LargeFileCopy.au3>

_Main()

Func _Main()
    Local $msg, $timer, $ret
    Local $src = "ubuntu-10.04-desktop-i386.iso" ; <---- path to a really large file goes here, recommended 600MB+
    Local $destpath = @DesktopDir & "testdir" ; <---- destination path goes here
    GUICreate("File Copy Test", 250, 150)
    GUICtrlCreateLabel("Try to interact with the GUI during file copy...", 20, 15)
    Local $go = GUICtrlCreateButton("Copy Large File", 50, 50, 150, 50)
    Local $doing = GUICtrlCreateLabel("Ready...", 20, 120, 200)
    GUISetState()

    Do
        $msg = GUIGetMsg()
        If $msg = $go Then
            ; create destination path
            DirCreate($destpath)
            ; internal FileCopy() function
            GUICtrlSetData($doing, "FileCopy()...")
            $timer = TimerInit()
            ConsoleWrite("FileCopy:" & @CRLF)
            $ret = FileCopy($src, $destpath, 1)
            ConsoleWrite("return: " & $ret & @CRLF & "error: " & @error & @CRLF & "time: " & TimerDiff($timer) & " ms" & @CRLF)
            ConsoleWrite("====================" & @CRLF)
            ; _LargeFileCopy() UDF function
            GUICtrlSetData($doing, "_LargeFileCopy()...")
            $timer = TimerInit()
            ConsoleWrite("_LargeFileCopy:" & @CRLF)
            $ret = _LargeFileCopy($src, $destpath, 1)
            ConsoleWrite("return: " & $ret & @CRLF & "error: " & @error & @CRLF & "time: " & TimerDiff($timer) & " ms" & @CRLF)
            ConsoleWrite("====================" & @CRLF)
            GUICtrlSetData($doing, "Ready...")
        EndIf
    Until $msg = -3
EndFunc

LargeFileCopy.zip

Edited January 31, 2013 by wraithdu

[KaFu]

Looks good, will for sure someday come in handy and goes right to the snippet folder ...

[wraithdu]

Small update in 1st post.

[Fran]

Looks good, will for sure someday come in handy

Came in handy today!

Thanx wraithdu!!

[Fran]

Just one more question...

How would I apply this to FileInstall()?

I'm creating a script that will run an update for a program on the user's computer. And copying the update file to a temp directory (using FileInstall), is what causes my script to run.

(I've already applied the _LargeFileCopy function in another gui and it works 100%)

F

[James]

Fran,

You could use Zednas Resource UDF to include the file in the EXE. I don't actually think you can merge them since it seems like _LargeFileCopy already requires the file source to be readily available - unless you used one of Zednas function to feed it pure binary data?

James

[wraithdu]

Hmm, this is just like FileCopy, in that a physical source file must exist. FileInstall decompresses an included resource from the EXE into a physical file. If that is what's hanging your GUI, then you need an alternative to FileInstall. Is there a reason you can't include the update file as a separate file with your updater? Or setup a network download location or something like that? Once you have a physical file you can use my function to copy it around if you need to.

A more intensive option would be to include the update as a resource, as James suggested, then read the binary data in chunks and pass it to _WinAPI_WriteFile. It's just as I've done, except the source data would be directly from the resource section. I suppose UPX could compress that data somewhat to save space as well.

Edited July 9, 2010 by wraithdu

[wraithdu]

See the latest update. I wrote a _LargeRawCopy to copy large memory blocks in the same way. So you should be able to embed your update as a resource, and use Zedna's UDF to get the resource pointer and size.

Be careful with this function. It checks for read access to the memory range pointed to by $pSrc and $iSrcSize, but it obviously cannot guarantee what that memory actually contains.

[gcue]

ive been SumTingWong's script for copying

http://www.autoitscript.com/forum/index.php?showtopic=11888&view=findpost&p=82020

but as you mentioned, I get the freezing gui. so your UDF caught my interest-the verifying is really cool too!

I do wish the process was cancelable though - possible?

[wraithdu]

It can be. You'd have to add a global flag in the copy loop and a cancel button in your GUI. Something like an 'If $flag Then close handles and return an error' in the 'Do... Until' loop. That's not a feature I want to add to the UDF, but it should be very easy for you to modify.

[gcue]

so if i specify a directory it doesnt copy subdirs?

i tried both

$src = @DesktopDir

$src = @DesktopDir & "\*.*"

[wraithdu]

No. If you bother to read the headers, the function takes a single source file. You may specify a file name or existing directory for the destination. If you specify a path that doesn't exist for the destination, it will assume it's a file name.

I don't see the need to implement a recursive copy at this point. It's easy enough for you to design whatever kind of loop you need and plug in my function where appropriate.

[Fran]

Wraithdu, Zedna, James,

Thank you all for your comments and suggestions.

It's all a bit greek to me at this stage, but I'm sure it will start making sense as soon as I begin applying it.

I hope I can push on your buttons again if I get stuck somewhere

Fran

[jfcby]

Whew, this UDF save me time while copying my large files!

Edited July 11, 2010 by jfcby

[supersonic]

wraithdu,

thank you for sharing!

Your UDF works pretty well...

Greets,

-supersonic.

Edited July 22, 2010 by supersonic

[wraithdu]

Update in first post with a link to my A3COPY project.

[wraithdu]

Small update, see first post.

[wraithdu]

Updated, see first post.

[llewxam]

Just a quick observation, this UDF is FAST FAST FAST with dealing with copying lots of little files, as well as dealing with larger files without causing the dreaded GUI stutter!! My previous method was to use

$PID = Run(@ComSpec & " /c copy /y " & Chr(34) & $copySource & Chr(34) & " " & Chr(34) & $copyDestination & Chr(34), @ScriptDir, @SW_HIDE)

With a Do....Until loop to check the status with ProcessGetStats which I thought was doing pretty well. I included a files copied per second function in an app I have been working on, and just for giggles switched to this UDF, when dealing with lots of little files it shot up from 12-17 per second to 50-60! Very low overhead!!

I also tinkered with the buffer size, wondering if I would get better performance in those smaller files with a smaller buffer, no difference.

So thanks a lot for posting wraithdu!!!

Ian

[wraithdu]

Updated, see first post.

The reason I removed trancexxx's hashing functions and rewrote them myself, is that the CreateFileMapping / MapViewOfFile method is RAM limited. Meaning if you try to hash a file larger than the available RAM, the MapViewOfFile function fails. This is, unfortunately, unacceptable.