_LargeFileCopy UDF

[JScript]

Very nice UDF, thanks 4 shared

JS

[Syed23]

Not a stupid question, but your script is not right. Your parameters to GUICtrlCreateAvi are wrong (see the help file). And you need to use BitOR to combine styles, not BitAND. I did notice though that AutoIt seems to have repainting issues on the AVI if you minimize the window, the AVI no longer seems to play until it loops and starts over again (but this has nothing to do with my UDF). Here's a fixed version of your script with the addition of a filename label and a few fixes, to show that it's working.

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6
#include <_LargeFileCopy.au3>
#include "RecFileListToArray.au3"
#include"Array.au3"
#include <GUIConstantsEx.au3>
#include <AVIConstants.au3>

Global $msg, $timer, $ret
Global $src, $SrcBut, $SrcLab, $Srcres
Global $Dest, $DestLab, $DestBut, $DestRes, $GetList, $go, $ani1, $label
_Main()

Func _Main()

GUICreate("File Copy", 550, 300)
$SrcLab = GUICtrlCreateLabel("Select Source Folder", 5, 10, 110, 40)
$src = GUICtrlCreateInput("", 115, 5, 400, 20)
$SrcBut = GUICtrlCreateButton("...", 520, 5, 25, 20)
$DestLab = GUICtrlCreateLabel("Select Destination Folder", 5, 60, 110, 40)
$Dest = GUICtrlCreateInput("", 115, 55, 400, 20)
$DestBut = GUICtrlCreateButton("...", 520, 55, 25, 20)
$go = GUICtrlCreateButton("Copy Large File", 200, 150, 150, 50)
; ACS_AUTOPLAY is not doing anything, playback stops as soon as you hide the control
$ani1 = GUICtrlCreateAvi(@SystemDir & "shell32.dll", 165, 10, 230, 275, 75, BitOR($ACS_CENTER, $ACS_TRANSPARENT))
GUICtrlSetState(-1, $GUI_HIDE)
$label = GUICtrlCreateLabel("", 10, 210, 540, 20)

GUISetState()

While 1
$msg = GUIGetMsg()

If $msg = $SrcBut Then
$Srcres = FileSelectFolder("Select Folder", @DesktopDir, 1)
GUICtrlSetData($src, $Srcres)

EndIf

If $msg = $GUI_EVENT_CLOSE Then ExitLoop

If $msg = $DestBut Then
$DestRes = FileSelectFolder("Select Folder", @DesktopDir, 1)
GUICtrlSetData($Dest, $DestRes)

EndIf

If $msg = $go Then
If GUICtrlRead($src) = "" Then
MsgBox(48, "Source Path - Warning", "Source path can not be blank")
ElseIf GUICtrlRead($Dest) = "" Then
MsgBox(48, "Destination Path - Warning", "Destination path can not be blank")
ElseIf GUICtrlRead($src) <> "" And GUICtrlRead($Dest) <> "" Then
GUICtrlSetState($ani1, $GUI_SHOW)
GUICtrlSetState($ani1, 1) ; apparently this is needed, as $GUI_HIDE stops playback
$timer = TimerInit()
$GetList = _RecFileListToArray(GUICtrlRead($src) & "", "*", 0, 1, 1)

;Exit
If IsArray($GetList) Then
For $i = 0 To UBound($GetList) - 1
GUICtrlSetData($label, $GetList[$i])
$ret = _LargeFileCopy(GUICtrlRead($src) & "" & $GetList[$i], GUICtrlRead($Dest) & "" & $GetList[$i], BitOR(1, 2, 8))
Next
EndIf
GUICtrlSetData($label, "")
GUICtrlSetState($ani1, $GUI_HIDE)
EndIf
EndIf
WEnd
EndFunc ;==>_Main

I am sorry for bothering you.. still that hangs when the copy process goes in background

[wraithdu]

You need to be more specific.

"still that hangs" - What hangs? The AVI issue I mentioned is an AutoIt problem and not related to the UDF.

"goes in background" - What goes where, how and when?

Unless you get the classic 'this application is not responding' dialog with the offer to forcefully kill the process, then whatever 'hanging' you're experiencing is not related to the UDF. It could even be your AV program actively scanning a large file which will hang up even my UDF if it's aggressive enough.

[ilko]

First off- thanks for this great UDF.

I've tested and used it only with file copy to USB sticks, here are some results you may find interesting.

Legend:

LFC 0.25, XP , 379.41

LFC- LargeFileCopy buffered

0.25- buffer used for LFC- _LargeFileCopy($src, $dest, BitOR(1, 2, 8, 128, 512), $buffer * 1024 * 1024, $CALG_MD5)

379.41- seconds to finish

XP- I386 folder from XP SP3 Setup CD, being copied from the internal hard disk, 5878 small files, 375MB total.

ISO- two ISO files, 414MB and 495MB

FileCopy- AutoIt internal FileCopy function

WFileCopy- Windows

xcopy - Windows xcopy with /q and /h switches. Xcopy and LFC unbuffered took a lot more time so weren't tested at all.

YCopy- Yashied's

Tested in FAT16, FAT32, NTFS with a fast USB 2.0 stick- A and quite slow one-B, under XP SP3 and 7 x64, tests were repeated several times with format before each test, default cluster size for 4GB partition, AV turned off.

A, under XP

===============NTFS=============

LFC 0.25, XP , 124.96

LFC 0.25, ISO , 57.11

LFC 0.5, XP , 154.68

LFC 0.5, ISO , 173.81

LFC 1, XP , 143.39

LFC 1, ISO , 120.71

LFC 2, XP , 133.38

LFC 2, ISO , 87.61

LFC 4, XP , 133.78

LFC 4, ISO , 71.54

LFC 8, XP , 129.07

LFC 8, ISO , 60.94

LFC 16, XP , 127.24

LFC 16, ISO , 56.5

xcopy, XP , 135.18

xcopy,ISO , 46.31

FileCopy , XP , 128.31

FileCopy , ISO , 46.33

WFileCopy , XP , 0.32

WFileCopy , ISO , 46.32

YCopy , XP , 154.1

YCopy , ISO , 44.28

===============Fat 16============

LFC 0.25, XP , 160.53

LFC 0.25, ISO , 58.06

LFC 0.5, XP , 159.38

LFC 0.5, ISO , 55.52

LFC 1, XP , 159.9

LFC 1, ISO , 54.06

LFC 2, XP , 157.06

LFC 2, ISO , 52.34

LFC 4, XP , 154.45

LFC 4, ISO , 52.76

LFC 8, XP , 159.14

LFC 8, ISO , 52.79

LFC 16, XP , 150.89

LFC 16, ISO , 52.88

xcopy, XP , 185.33

xcopy,ISO , 51.21

FileCopy , XP , 155.35

FileCopy , ISO , 50.03

WFileCopy , XP , 0.05

WFileCopy , ISO , 51.33

YCopy , XP , 178.05

YCopy , ISO , 50.26

===============Fat 32============

LFC 0.25, XP , 177.03

LFC 0.25, ISO , 59.61

LFC 0.5, XP , 173.95

LFC 0.5, ISO , 55.82

LFC 1, XP , 167.27

LFC 1, ISO , 54.53

LFC 2, XP , 168.64

LFC 2, ISO , 54.25

LFC 4, XP , 172.99

LFC 4, ISO , 51.37

LFC 8, XP , 152.86

LFC 8, ISO , 52.84

LFC 16, XP , 170.97

LFC 16, ISO , 53.84

xcopy, XP , 206.25

xcopy,ISO , 50.92

FileCopy , XP , 169.8

FileCopy , ISO , 51.15

WFileCopy , XP , 0.04

WFileCopy , ISO , 51.62

YCopy , XP , 181.01

YCopy , ISO , 50.57

A, under Windows 7

===============NTFS=============

LFC 0.25, XP , 379.41

LFC 0.25, ISO , 49.57

LFC 0.5, XP , 401.51

LFC 0.5, ISO , 48.83

LFC 1, XP , 405.99

LFC 1, ISO , 48.59

LFC 2, XP , 400.09

LFC 2, ISO , 47.65

LFC 4, XP , 399.18

LFC 4, ISO , 49.7

LFC 8, XP , 397.1

LFC 8, ISO , 48.07

LFC 16, XP , 397.45

LFC 16, ISO , 48.28

xcopy, XP , 569.33

xcopy,ISO , 49.7

FileCopy , XP , 377.08

FileCopy , ISO , 49.91

WFileCopy , XP , 404.86

WFileCopy , ISO , 50.28

YCopy , XP , 406.69

YCopy , ISO , 50.18

===============Fat 16============

LFC 0.25, XP , 117.95

LFC 0.25, ISO , 44.34

LFC 0.5, XP , 156.83

LFC 0.5, ISO , 42.33

LFC 1, XP , 156.99

LFC 1, ISO , 41.47

LFC 2, XP , 158.42

LFC 2, ISO , 41.35

LFC 4, XP , 157

LFC 4, ISO , 41.21

LFC 8, XP , 157.85

LFC 8, ISO , 41.25

LFC 16, XP , 160.97

LFC 16, ISO , 61.21

xcopy, XP , 178.43

xcopy,ISO , 41.35

FileCopy , XP , 149.71

FileCopy , ISO , 40.32

WFileCopy , XP , 163.25

WFileCopy , ISO , 40.59

YCopy , XP , 148.88

YCopy , ISO , 40.5

===============Fat 32============

LFC 0.25, XP , 114.1

LFC 0.25, ISO , 45.84

LFC 0.5, XP , 152.42

LFC 0.5, ISO , 42.75

LFC 1, XP , 153.22

LFC 1, ISO , 42.01

LFC 2, XP , 169.16

LFC 2, ISO , 41.85

LFC 4, XP , 153.72

LFC 4, ISO , 41.74

LFC 8, XP , 152.91

LFC 8, ISO , 41.83

LFC 16, XP , 156.06

LFC 16, ISO , 60.91

xcopy, XP , 173.69

xcopy,ISO , 43.07

FileCopy , XP , 142.32

FileCopy , ISO , 40.94

WFileCopy , XP , 115.86

WFileCopy , ISO , 41.3

YCopy , XP , 143.08

YCopy , ISO , 41.03

B, under Windows 7

===============NTFS=============

LFC 0.25, XP , 747.8

LFC 0.25, ISO , 165.11

LFC 0.5, XP , 705.83

LFC 0.5, ISO , 163.25

LFC 1, XP , 724.65

LFC 1, ISO , 163.6

LFC 2, XP , 668.09

LFC 2, ISO , 164.18

LFC 4, XP , 685.93

LFC 4, ISO , 163.48

LFC 8, XP , 657.66

LFC 8, ISO , 164.7

LFC 16, XP , 686.28

LFC 16, ISO , 162.81

xcopy, XP , 715.94

xcopy,ISO , 172.56

FileCopy , XP , 626.53

FileCopy , ISO , 162.05

WFileCopy , XP , 712.84

WFileCopy , ISO , 159.22

YCopy , XP , 666.32

YCopy , ISO , 160.87

===============Fat 16============

LFC 0.25, XP , 742.78

LFC 0.25, ISO , 170.15

LFC 0.5, XP , 791.88

LFC 0.5, ISO , 160.17

LFC 1, XP , 754.53

LFC 1, ISO , 156.01

LFC 2, XP , 778.32

LFC 2, ISO , 154.28

LFC 4, XP , 784.73

LFC 4, ISO , 153.31

LFC 8, XP , 778.35

LFC 8, ISO , 153.1

LFC 16, XP , 784.75

LFC 16, ISO , 168.92

xcopy, XP , 832.92

xcopy,ISO , 152.01

FileCopy , XP , 778.39

FileCopy , ISO , 151.43

WFileCopy , XP , 838.63

WFileCopy , ISO , 152.23

YCopy , XP , 798.12

YCopy , ISO , 151.12

===============Fat 32============

LFC 0.25, XP , 814.16

LFC 0.25, ISO , 178.34

LFC 0.5, XP , 862.87

LFC 0.5, ISO , 170.46

LFC 1, XP , 846.75

LFC 1, ISO , 167.16

LFC 2, XP , 852.09

LFC 2, ISO , 164.55

LFC 4, XP , 866.86

LFC 4, ISO , 166.16

LFC 8, XP , 862.91

LFC 8, ISO , 164.63

LFC 16, XP , 833.4

LFC 16, ISO , 186.34

xcopy, XP , 917.72

xcopy,ISO , 158.97

FileCopy , XP , 851.13

FileCopy , ISO , 162.71

WFileCopy , XP , 859.61

WFileCopy , ISO , 162.83

YCopy , XP , 890.42

YCopy , ISO , 163

Did not have patience to repeat the test with the slow stick under XP, but resluts were quite informative for me- LFC flies with small files when the proper buffer is used.

For large files, above say 100-200MB I couldn't get it not the freeze the GUI- it does not freeze during copying, but once finished, until windows file system buffer/cache is emptied, it gets frozen. For files above, say 500MB it's quite noticable, can't recall if optimize for performace/quick removal policy helped or not, but wasn't a path to take anyway. Ended up using copy.exe and ProcessGetStats, like

As a side note- _LFC_GetDiskClusterSize function seems to need a correction, return should be $aRet[3]*$aRet[2], BytesPerSector*SectorsPerCluster, unless you had other intentions.

Edited January 29, 2013 by ilko

[Syed23]

Hi ,

Thank you so much for this wonderful script. with help of this tool i created a tool which made my job so easy. But i have a doubt when i copy the file from source location to destination location date is getting changed. could you please help me how to avoid changing the date? sorry for bothering you.

[ilko]

@Sayed23

Use the appropriate flag(s):

; #FUNCTION# ====================================================================================================
; Name...........: _LargeFileCopy
; Description....: Copy large files in such a way as to keep AutoIt GUIs responsive
; Syntax.........: _LargeFileCopy($sSrc, $sDest[, $iFlags = 0[, $iToRead = 2097152[, $iAlg = $CALG_MD5[, $sFunction = ""[, $vUserVar = Default]]]]])
; Parameters.....: $sSrc     - Source file name
;                $sDest  - Destination: may be a file name or directory
;                $iFlags     - [Optional] Combine flags with BitOR
;                            | 1 - Overwrite existing file
;                            | 2 - Create destination directory structure
;                            | 4 - Flush the destination file buffer before returning
;                            | 8 - Verify source and destination are identical via bit by bit comparison
;                            | 16 - Verify source and destination are identical via MD5 hash
;                            | 32 - Verify source and destination file size only
;                            | 64 - Copy source file attributes (NOT including Compression or Encryption)
;                            | 128 - Copy source file Creation time
;                            | 256 - Copy source file Last Accessed time
;                            | 512 - Copy source file Modified time
;                            |1024 - Copy source file Security Descriptors and Ownership
;                            |2048 - Copy source compression state
;                            |4096 - Copy source encryption state

[Syed23]

Thank you @iLko.. will be the below syntax correct?

$mds = _LargeFileCopy(GUICtrlRead($DrivePath) & $Docs & "\" & $DocsAr[$i], "C:\Users\" & @UserName & "\Documents" & "\" & $DocsAr[$i], BitAND(BitOR(1, 2, 8),(128,256,512)))

[ilko]

Nope. Combine all parameters needed in a single BitOR- BitOR(1, 2, 8, 128, 512) or whatever you need inside.

[wraithdu]

As a side note- _LFC_GetDiskClusterSize function seems to need a correction, return should be $aRet[3]*$aRet[2], BytesPerSector*SectorsPerCluster, unless you had other intentions.

I actually caught this bug, and I thought I uploaded a fixed version. My local sources are fixed, I had moved some more functions into FileEx, so not sure what happened. Anyway, I'll update shortly.

[VenusProject2]

Thank you, your script is still being appreciated...

Edited April 17, 2015 by VenusProject2

[JibsMan]

I have used _LargeFileCopy before and it worked great! Thanks for the script! However with a new script I'm having problems.

Wraithdu, (or anyone!) do you know why FileCopy and _LargeFileCopy fails? DirCopy does work.

I get a Zero (0) returned when I run my code using either FileCopy or _LargeFileCopy

_LargeFileCopy.au3 is in the C:\Program Files (x86)\AutoIt3\Include directory.

My actual script does a lot more than this, so I toned it down to only do the copy. I used _LargeFileCopy in a script last year and it worked perfectly.

#include <MsgBoxConstants.au3>
#include <FileConstants.au3>
#include <_LargeFileCopy.au3>
#include <File.au3>


Local $sSourceTestDir = "C:\Validation\214" ;Source is 3 Dirs Deep, over 12,000 files, 400MB
Local $sDestTestDir = "C:\FileCopyTest\214" ;Empty Dir

MsgBox($MB_OK,"","Source Dir is "&$sSourceTestDir&" and Dest Dir is "&$sDestTestDir&". Copy Files Next!",5)
;$iRes = FileCopy($sSourceTestDir,$sDestTestDir,$FC_OVERWRITE + $FC_CREATEPATH)
;$iRes = _LargeFileCopy($sSourceTestDir,$sDestTestDir,1)
$iRes = DirCopy($sSourceTestDir, $sDestTestDir,$FC_OVERWRITE)
MsgBox($MB_OK,"","$iRes is "&$iRes,10)

I have created the Sink directories C:\FileCopyTest\214 while trying to get _LargeFileCopy working. However DirCopy creates these if they are not there.

I have tried adding backslashes ( C:\Validation\214\ ), asterisks, ( C:\Validation\214\* ) & ( C:\Validation\214\*.* ) to the code.

I have made both the Source and Sink directories Shared, Full Control for Everyone and turned off Read Only attributes.

I would prefer to use the _LargeFileCopy because of the progress bar showing the darned script is actually doing something instead of waiting, not knowing if the script is stuck or working.

I have tried this on multiple systems; Windows 10, Windows 8.1 and get the same results. I'm pulling out my hair trying to figure out what I'm doing wrong!

Any help is certainly appreciated!

JibsMan

[GhostLine]

Back from the deads ...

I've the same issue here, but it's something I think I can manage.

I've only two regrets :

- it miss a post copy control to see if it is about to rewrite the exact same file

- it miss lot of return (for example, the MD5 key, or the detailled output when there's an error while copying, etc.)

wraithdu , have you ever considered to extend the functionalities of this tool ?