Popular Post wraithdu Posted July 8, 2010 Popular Post Share Posted July 8, 2010 (edited) Several people have had problems with their GUIs freezing when they try to copy large files using FileCopy(). This is because FileCopy() is a blocking function, and the GUI stops responding to messages from the OS during the operation. If that goes on long enough, the OS thinks it is hung and you get the (Not Responding) thing going on.Here's a function that copies large files using Windows API calls in 8MB 2 MB chunks. This keeps the GUI alive and responsive. I've added additional copy verification functionality as well (disabled by default).Take a look at my A3COPY project for a great usage example of this UDF.Update: 1/31/13- Moved a few more functions into _FileEx.- Fixed a bug in GetDiskClusterSize function.Update: 10/8/12- Fixed bug due to missing RtlCopyMemory and RtlCompareMemory exports in x86 kernel32.dll. Now uses RtlMoveMemory and RtlCompareMemory from ntdll.dll (sorry).Update: 10/4/12 Read NOTES at top of UDF- Reduced default buffer size to 2 MB- Added flags for copying source file attributes, all three file times individually, ACL security descriptors and ownership, and NTFS specific compression and encryption- Optimized buffer creation- Enhanced path handling to fully support UNC and network conventions ( C:... , ?C:... , srvshare... , ?UNCsrvshare... )- Removed C runtime functions (msvcrt.dll) in favor of native kernel32 functions- Code cleanup and many optimizations in LFC and the accompanying _FileEx UDFUpdate: 4/15/11- Added additional verification methods - bit by bit, and size only- The default verification method (ie iFlag 8) is now bit by bit - it is faster and less computationally intensive than MD5 hashing, it also allows the function to bail out early in the case of a failed copy operationUpdate: 3/9/11SCRIPT BREAKING CHANGES- Combined some parameters into one iFlags parameter- Re-ordered some parameters based on priority- Added a parameter to specify a different read buffer size- Destination file is now opened with GENERIC_READ | GENERIC_WRITE per MSDN for better compatibility copying files across a network- No longer create destination directory structure unless specified in iFlags parameterUpdate: 9/20/10- Removed tracexxx's hashing functions (sorry!) in favor of a different method which is not RAM limited- Hashing algorithm can be specified (those supported by Crypt.au3: CALG_MD2, CALG_MD4, CALG_MD5, CALG_SHA1)Update: 8/26/10- Fix for failed verification of 0 byte files- Added check for 0 byte source in RAW functionsUpdate: 8/22/10- Small change to CreateFile function to avoid errors when opening SYSTEM or HIDDEN files when specifying CREATE_ALWAYS.Update: 8/10/10- Added _LargeFileCopyUnbuffered and _LargeRawCopyUnbuffered which will copy files using unbuffered I/O.- MD5 hashing of the source file / memory block is now done on the fly from the memory buffer. This avoids an additional read of the source file later on and saves time.- Destination file MD5 hashing is now done with trancexxx's nice hashing UDF. It uses the Windows API and is really fast.Update: 7/9/2010- Added _LargeRawCopy which will copy large memory blocks (usually from an embedded resource) to a file in the same manner as _LargeFileCopy. This function also has the verify functionality via MD5 hash.Update: 7/8/2010- Added optional user function and variable parameters. This function, if given, will be called after each write operation with the total bytes written, total file size in bytes, and the user given variable. This is perfect for a copy-with-progess type of function call.Exampleexpandcollapse popup#NoTrayIcon #AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 #include <_LargeFileCopy.au3> _Main() Func _Main() Local $msg, $timer, $ret Local $src = "ubuntu-10.04-desktop-i386.iso" ; <---- path to a really large file goes here, recommended 600MB+ Local $destpath = @DesktopDir & "testdir" ; <---- destination path goes here GUICreate("File Copy Test", 250, 150) GUICtrlCreateLabel("Try to interact with the GUI during file copy...", 20, 15) Local $go = GUICtrlCreateButton("Copy Large File", 50, 50, 150, 50) Local $doing = GUICtrlCreateLabel("Ready...", 20, 120, 200) GUISetState() Do $msg = GUIGetMsg() If $msg = $go Then ; create destination path DirCreate($destpath) ; internal FileCopy() function GUICtrlSetData($doing, "FileCopy()...") $timer = TimerInit() ConsoleWrite("FileCopy:" & @CRLF) $ret = FileCopy($src, $destpath, 1) ConsoleWrite("return: " & $ret & @CRLF & "error: " & @error & @CRLF & "time: " & TimerDiff($timer) & " ms" & @CRLF) ConsoleWrite("====================" & @CRLF) ; _LargeFileCopy() UDF function GUICtrlSetData($doing, "_LargeFileCopy()...") $timer = TimerInit() ConsoleWrite("_LargeFileCopy:" & @CRLF) $ret = _LargeFileCopy($src, $destpath, 1) ConsoleWrite("return: " & $ret & @CRLF & "error: " & @error & @CRLF & "time: " & TimerDiff($timer) & " ms" & @CRLF) ConsoleWrite("====================" & @CRLF) GUICtrlSetData($doing, "Ready...") EndIf Until $msg = -3 EndFuncLargeFileCopy.zip Edited January 31, 2013 by wraithdu abberration, garbb, KaFu and 2 others 4 1 Link to comment Share on other sites More sharing options...
KaFu Posted July 8, 2010 Share Posted July 8, 2010 Looks good, will for sure someday come in handy and goes right to the snippet folder ... Â OS: Win10-22H2 - 64bit - German, AutoIt Version: 3.3.16.1, AutoIt Editor: SciTE, Website: https://funk.eu AMT - Auto-Movie-Thumbnailer (2024-Oct-13)Â BIC - Batch-Image-Cropper (2023-Apr-01) COP - Color Picker (2009-May-21) DCS - Dynamic Cursor Selector (2024-Oct-13) HMW - Hide my Windows (2024-Oct-19) HRC - HotKey Resolution Changer (2012-May-16)Â ICU - Icon Configuration Utility (2018-Sep-16) SMF - Search my Files (2024-Oct-20) - THE file info and duplicates search tool SSD - Set Sound Device (2017-Sep-16) Link to comment Share on other sites More sharing options...
wraithdu Posted July 8, 2010 Author Share Posted July 8, 2010 Small update in 1st post. Link to comment Share on other sites More sharing options...
Fran Posted July 9, 2010 Share Posted July 9, 2010 Looks good, will for sure someday come in handyCame in handy today! Thanx wraithdu!! Link to comment Share on other sites More sharing options...
Fran Posted July 9, 2010 Share Posted July 9, 2010 Just one more question... How would I apply this to FileInstall()? I'm creating a script that will run an update for a program on the user's computer. And copying the update file to a temp directory (using FileInstall), is what causes my script to run. (I've already applied the _LargeFileCopy function in another gui and it works 100%) F Link to comment Share on other sites More sharing options...
James Posted July 9, 2010 Share Posted July 9, 2010 Fran,You could use Zednas Resource UDF to include the file in the EXE. I don't actually think you can merge them since it seems like _LargeFileCopy already requires the file source to be readily available - unless you used one of Zednas function to feed it pure binary data?James Blog - Seriously epic web hosting - Twitter - GitHub - Cachet HQ Link to comment Share on other sites More sharing options...
wraithdu Posted July 9, 2010 Author Share Posted July 9, 2010 (edited) Hmm, this is just like FileCopy, in that a physical source file must exist. FileInstall decompresses an included resource from the EXE into a physical file. If that is what's hanging your GUI, then you need an alternative to FileInstall. Is there a reason you can't include the update file as a separate file with your updater? Or setup a network download location or something like that? Once you have a physical file you can use my function to copy it around if you need to. A more intensive option would be to include the update as a resource, as James suggested, then read the binary data in chunks and pass it to _WinAPI_WriteFile. It's just as I've done, except the source data would be directly from the resource section. I suppose UPX could compress that data somewhat to save space as well. Edited July 9, 2010 by wraithdu Link to comment Share on other sites More sharing options...
wraithdu Posted July 9, 2010 Author Share Posted July 9, 2010 See the latest update. I wrote a _LargeRawCopy to copy large memory blocks in the same way. So you should be able to embed your update as a resource, and use Zedna's UDF to get the resource pointer and size. Be careful with this function. It checks for read access to the memory range pointed to by $pSrc and $iSrcSize, but it obviously cannot guarantee what that memory actually contains. Link to comment Share on other sites More sharing options...
gcue Posted July 9, 2010 Share Posted July 9, 2010 ive been SumTingWong's script for copying http://www.autoitscript.com/forum/index.php?showtopic=11888&view=findpost&p=82020 but as you mentioned, I get the freezing gui. so your UDF caught my interest-the verifying is really cool too! I do wish the process was cancelable though - possible? Link to comment Share on other sites More sharing options...
wraithdu Posted July 9, 2010 Author Share Posted July 9, 2010 It can be. You'd have to add a global flag in the copy loop and a cancel button in your GUI. Something like an 'If $flag Then close handles and return an error' in the 'Do... Until' loop. That's not a feature I want to add to the UDF, but it should be very easy for you to modify. Link to comment Share on other sites More sharing options...
gcue Posted July 9, 2010 Share Posted July 9, 2010 so if i specify a directory it doesnt copy subdirs? i tried both $src = @DesktopDir $src = @DesktopDir & "\*.*" Link to comment Share on other sites More sharing options...
wraithdu Posted July 9, 2010 Author Share Posted July 9, 2010 No. If you bother to read the headers, the function takes a single source file. You may specify a file name or existing directory for the destination. If you specify a path that doesn't exist for the destination, it will assume it's a file name. I don't see the need to implement a recursive copy at this point. It's easy enough for you to design whatever kind of loop you need and plug in my function where appropriate. Link to comment Share on other sites More sharing options...
Fran Posted July 11, 2010 Share Posted July 11, 2010 Wraithdu, Zedna, James, Thank you all for your comments and suggestions. It's all a bit greek to me at this stage, but I'm sure it will start making sense as soon as I begin applying it. I hope I can push on your buttons again if I get stuck somewhere Fran Link to comment Share on other sites More sharing options...
jfcby Posted July 11, 2010 Share Posted July 11, 2010 (edited) Whew, this UDF save me time while copying my large files! Edited July 11, 2010 by jfcby Determined -- Devoted -- Delivered Make your mind up -- to seriously apply yourself -- accomplishing the desired results. **** A soft answer turneth away wrath: but grievous words stir up anger. Proverbs 15:1 KJB **** Link to comment Share on other sites More sharing options...
supersonic Posted July 21, 2010 Share Posted July 21, 2010 (edited) wraithdu, thank you for sharing! Your UDF works pretty well... Greets, -supersonic. Edited July 22, 2010 by supersonic Link to comment Share on other sites More sharing options...
wraithdu Posted August 10, 2010 Author Share Posted August 10, 2010 Update in first post with a link to my A3COPY project. Link to comment Share on other sites More sharing options...
wraithdu Posted August 22, 2010 Author Share Posted August 22, 2010 Small update, see first post. Link to comment Share on other sites More sharing options...
wraithdu Posted August 26, 2010 Author Share Posted August 26, 2010 Updated, see first post. Link to comment Share on other sites More sharing options...
llewxam Posted September 13, 2010 Share Posted September 13, 2010 Just a quick observation, this UDF is FAST FAST FAST with dealing with copying lots of little files, as well as dealing with larger files without causing the dreaded GUI stutter!! My previous method was to use $PID = Run(@ComSpec & " /c copy /y " & Chr(34) & $copySource & Chr(34) & " " & Chr(34) & $copyDestination & Chr(34), @ScriptDir, @SW_HIDE) With a Do....Until loop to check the status with ProcessGetStats which I thought was doing pretty well. I included a files copied per second function in an app I have been working on, and just for giggles switched to this UDF, when dealing with lots of little files it shot up from 12-17 per second to 50-60! Very low overhead!! I also tinkered with the buffer size, wondering if I would get better performance in those smaller files with a smaller buffer, no difference. So thanks a lot for posting wraithdu!!! Ian My projects: IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged. INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them. PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses. Sync Tool - Folder sync tool with lots of real time information and several checking methods. USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions. Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent. CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction. MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app. 2048 Game - My version of 2048, fun tile game. Juice Lab - Ecigarette liquid making calculator. Data Protector - Secure notes to save sensitive information. VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive. Find in File - Searches files containing a specified phrase. Link to comment Share on other sites More sharing options...
wraithdu Posted September 20, 2010 Author Share Posted September 20, 2010 Updated, see first post. The reason I removed trancexxx's hashing functions and rewrote them myself, is that the CreateFileMapping / MapViewOfFile method is RAM limited. Meaning if you try to hash a file larger than the available RAM, the MapViewOfFile function fails. This is, unfortunately, unacceptable. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now