Folder Sync Tool

[llewxam]

I have put more features in to this thing than I thought I would when it started, and have been tooling around with it for months years, on and off. Features include:

• Handles large files with no GUI stuttering/pausing
• Will check if a file needs to be copied by If Exist, size, date, and MD5 check
• Optional file Create, Access, and Modify timestamp preservation
• Optional Source file deletion after sync
• Optional deletion of Destination files no longer present in Source (ala Robocopy /MIR)
• Percent of total job and bytes remaining is shown live, not just updated between transfers
• Optional system shut down after completion
• Job can be cancelled even in the middle of a transfer
• Full Command Line configuration to make automation easy (thanks darthwhatever)
• Now uses UNC paths internally to avoid >260 character path length errors. Thanks wraithdu for the pointer!

I have put in a LOT of on-the-fly feedback via AdlibRegister, which has become one of my favorite commands

As stated at the top of the code, I owe a lot of thanks for ideas and code snippets to a LOT of forum members for this project especially:

Spiff59 for multiple suggestions and improvements, wraithdu for _LargeFileCopy, BaKaMu and others for _FileListToArrayXT, Prog@ndy for GDIpProgress, Ward for his machine code MD5, and Melba23 for general guidance on innumerable issues over the last 2 years, many of your suggestions make their way in to each of my scripts!

Enjoy

Ian

Previous downloads 145

Sync.au3

Edited November 12, 2011 by llewxam

[Spiff59]

You have an older buggy version of _StringAddThousandsSep(), it's slow and will leave a "," as the first character returned if the integer portion of the number string is evenly divisable by 3. This one is twice as fast and doesn't have the bug:

Func _StringAddThousandsSep($sText, $sThousands = ',', $sDecimal = '.')
    If Not StringIsInt($sText) And Not StringIsFloat($sText) Then Return SetError(1)
    Local $rKey = "HKCU\Control Panel\International"
    If $sDecimal = -1 Then $sDecimal = RegRead($rKey, "sDecimal")
    If $sThousands = -1 Then $sThousands = RegRead($rKey, "sThousand")
    Local $aSplit = StringSplit($sText, "-" & $sDecimal )
    Local $iInt = 1, $iMod
    If Not $aSplit[1] Then
        $aSplit[1] = "-"
        $iInt = 2
    EndIf
    If $aSplit[0] > $iInt Then
        $aSplit[$aSplit[0]] = "." & $aSplit[$aSplit[0]]
    EndIf
    $iMod = Mod(StringLen($aSplit[$iInt]), 3)
    If Not $iMod Then $iMod = 3
    $aSplit[$iInt] = StringRegExpReplace($aSplit[$iInt], '(?<=\d{' & $iMod & '})(\d{3})', $sThousands & '\1')
    For $i = 2 to $aSplit[0]
        $aSplit[1] &= $aSplit[$i]
    Next
    Return $aSplit[1]
EndFunc

This is way fast, but working with the locale is a little screwy:

Func _StringAddThousandsSep($amt)
    $locale = 4105;  = canada.  default locale = 2048 or null - see http://msdn.microsoft.com/en-us/library/0h88fahh.aspx
    $CurrencyFmt = 0 ; null pointer to CURRENCTFMT structure (use default values for locale)
    $len = 80
    $dll=DllOpen("kernel32.dll")
    $GCF=DllCall($dll,"int","GetCurrencyFormat","int",$locale,"int",0,"str",$Amt,"int",$CurrencyFmt,"str","", "int",$len)
    DllClose($dll)
    Return StringRegExpReplace($GCF[5], "\$|0\z", "")
EndFunc

Func _ByteSuffix($bytes)
    Local $x, $bytes_suffix[6] = [" B"," KB"," MB"," GB"," TB"," PB"]
    While $bytes > 1023
        $x += 1
        $bytes /= 1024
    WEnd
    Return Round($bytes, 2) & $bytes_suffix[$x]
EndFunc

I believe there are also much faster versions of a "recursive" _FileListToArray() in this thread (ones that also don't commit the structured programming sin of a function calling itself):

Improvement of included _FileListToArray function.

[llewxam]

GOOD GRIEF MAN!! hahaha, thanks a lot for the suggestions!

I read a lot of the thread you linked on the recursive file/folder search a long time ago and settled on the one I use now because of the feedback I can have it give. I will definitely spend some time looking at one or two of them harder to try to achieve the same effect, but may just keep this one, I'd rather have it look a bit cooler than go faster at that stage Unless it is a ridiculously slow machine or enumerating a network source the current routine is fast enough.

As for the other bits though, when _StringAddThousandsSep was removed somebody posted the routine I was using, I guess that person needs to update as well! Thanks for that, I use it often. I chopped this one just a bit and put the declarations in the top of the script rather than doing it with every call to the routine.

I also replaced my much-loved and short-lived _FileSizeSuffix with the _ByteSuffix you offered, that's nice! I made a very slight alteration so it will always have 2 decimal places, I just prefer to see it that way rather than watch the value keep bouncing around...

First post updated

Thanks again!

Ian

Edited July 9, 2010 by llewxam

[Spiff59]

I made a very slight alteration so it will always have 2 decimal places

You can force the 2 decimals places by just changed the last line from a Round() to a StringFormat():

Func _ByteSuffix($bytes)
    Local $x, $bytes_suffix[6] = [" B"," KB"," MB"," GB"," TB"," PB"]
    While $bytes > 1023
        $x += 1
        $bytes /= 1024
    WEnd
    Return StringFormat('%.2f', $bytes) & $bytes_suffix[$x]
EndFunc

[llewxam]

Thanks again

Some day I'm just going to start reading the function reference from start to finish So many great functions already built in.....

Ian

[llewxam]

First post updated with new code. I had a few issues changing the file recursion routine but it was for the best, for a couple of reasons. Yes, it is faster which is nice, but it also showed me a few really bad bugs!! The check code for making sure the destination matches the source was all fouled up before, so that has been cleared up. I also found during this last round of testing that some people may want to throw a #RequireAdmin in - any files with attributes that cause the copy to fail that are located in any Windows or Program Files folders will require admin rights to change, and without it there will be a lot of failures. I choose to not put it in, because copying from those locations isn't really the goal of this script and I'd rather not be annoyed by an "Are You Sure?" every time I run this. YMMV.

Ian

[Spiff59]

Looks pretty good.

I think out of that 280 post thread, you found the largest example of _FileListToArray() there is!

(That version received some ridicule as suffering from "bloat")

But if it's working for you...

I do see a large block of identical code starting with "$attrib =" and ending with "until not" that's repeated 3 times. That code might make a good candidate for an internal function.

Edited July 10, 2010 by Spiff59

[llewxam]

hahaha, I was thinking about hacking out the chunks of _FileListToArrayXT that I wouldn't be using, but felt that it would be unfair to the authors, and I really like its flexibility with excludes so will be using it in other projects such as my Windows user profile backup tool. Might as well just leave it in tact.

Ian

[sliceofpie]

Hello,

I get the following error when running the sync tool

C:\synctool.au3 (965) : ==> Duplicate function name.:

Func _copy($copySource, $copySize, $copyDestination)

[llewxam]

Hmm, can't imagine why... What version of AutoIt are you using? Was the script compiled or not when you ran it? If compiuled, with what options? I personally always compile without UPX but that's about it.

Ian

[sliceofpie]

Thanks for your feedback llewxam. I had not compiled it.

[llewxam]

I have updated the _FileListToArrayXT routine due to a bug fix by BaKaMu, and heavily stripped it down while I was at it so it only has the options I am using. If you want to use this highly versatile and fast routine, make sure you get the original, not this one!

I also corrected an unrelated issue that was not deleting the source folder if it was supposed to, and a few other teeny tiny little things that I didn't think justified an update by themselves.

As usual, questions - ask.

Thanks!

Ian

[llewxam]

Cool new stuff:

• User can either browse for paths, type paths, or drag and drop
• Nice GUI color added
• some code restructuring

Enjoy

Ian

[llewxam]

First post updated, improvements including now you can sync by date.

Ian

[nitekram]

I tried your tool on a bunch of txt files and made a change to one of them from the source - I then synced again and it said that it synced one file, but looking at the file after the synce - there was no change (like the file was not copied over)

Any ideas?

I did not compile the code - which I assume would not make a difference.

[llewxam]

I tried your tool on a bunch of txt files and made a change to one of them from the source - I then synced again and it said that it synced one file, but looking at the file after the synce - there was no change (like the file was not copied over)

Any ideas?

I did not compile the code - which I assume would not make a difference.

My only question is which Compare method did you use - if you left it on "If Exist" than nothing would have happened, with a text file it is likely that the Size method would also not have worked if the amount of characters did not change (changed an "r" to an "s"), but the Date and MD5 functions are designed to deal with your situation so set it to one of those.

Let me know if that helps!

Ian

[gregnottage]

Cheers for the date update Ian. Will test when I get a sec!

Cheers,

Greg.

[nitekram]

Sorry - forgot about this one during xmas break

Date and MD5 functions are designed to deal with your situation so set it to one of those.

Let me know if that helps!

Ian

I tried to change it to DATE - crashed the application.

Just to make sure it was working, I tried IFEXIST and the application did not crash.

Tried MD5 and it appears to have worked, but 5 items had trouble? the file in question did get synced though.

[Emiel Wieldraaijer]

Hi Ian,

Thanks for this example..

There is a little bug -> Select a source .. press the source again and again .. it will recount the folder and update the label

Best regards,

Emiel

[llewxam]

Hi Ian,

Thanks for this example..

There is a little bug -> Select a source .. press the source again and again .. it will recount the folder and update the label

Best regards,

Emiel

True, the button could be disabled or replaced with an Input set to $ES_READONLY, I'll do that for my next release. I do have a worse glitch in it where sometimes the percent complete goes beyond 100%, so some day soon I hope to address both items.

Thanks!

Ian