Combining multiple arrays...round 2

[Cravin]

Good evening - so here's my predicament:  I'm attempting to create a network backup script.  I'm using Melba23's great UDF, RecFileListToArray.au3, and plan to actually perform the copy process with Yashied's Copy UDF.  I'm running into some trouble at this point... I'm able to get this script as it is to create multiple arrays depending on what options are selected, but the trouble seems to come with the function I created called "_ConcatenateFileArrays" - What I want this to do, is take each array generated by _RecFileListToArray and add it to one big array so that I can call _Copy_CopyFile on each item..

This is very rough code so bear with me.  Any thoughts and suggestions would be greatly appreciated.  

Thanks for all the help!

NOTE: For testing purposes, run the script, and in the username field, type in your username, and in the UNC path field, type in the name of your computer in this form, pointing to the admin share:  yourpcnamec$, then choose "Selective Copy" and check "C: Variances" and "User Desktop" and then "Begin Transfer".  This should give you the general idea of what I'm wanting to do.

EDIT: Added the copy of Melba23's RecFileListToArray.au3 for ease

expandcollapse popup

#RequireAdmin
#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Res_requestedExecutionLevel=requireAdministrator
#AutoIt3Wrapper_Run_Debug_Mode
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
#include <Array.au3>
#include <ButtonConstants.au3>
#include <ComboConstants.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <TabConstants.au3>
#include <WindowsConstants.au3>
#include <GuiStatusBar.au3>
#include <ProgressConstants.au3>
#include "RecFileListToArray.au3"

Global $FileMenuExit, $HelpMenuAbout, $HelpMenuHelp, $TargetNameTest, $FileArrays[10], $CVarianceFiles, $UserSendToFiles
Global $Radio1, $Radio2, $Radio3, $Radio4, $Checkbox[11][2], $Checkbox1, $CheckBox2, $Checkbox3, $DataTabImportPSTsBtn, $DataTabXferBtn, $ExternalTabXferBtn, $ServerTabXferBtn, $UNCTabXferBtn, $WinVer
Global $Checkbox4, $Checkbox5, $Checkbox6, $Checkbox7, $Checkbox8, $Checkbox9, $Checkbox10, $DataTabImportNtwkDrvsBtn, $DataTabImportNtwkPrntrsBtn, $TermTabXferBtn, $DataTabUserNameInput, $DataTabTargetNameInput
Global $Main, $Radio1Status, $Radio2Status, $LastRadio1Status, $LastRadio2Status, $rDrive, $ramDrive, $cDrive, $nDrive, $fDrive, $uDrive, $aDrive, $Radio3Status, $Radio4Status, $DataTabUserName, $DataTabTargetName
Global $InputClear = 0

Global $FilesToTransfer[1]

_MainGUI()

Func _MainGUI()

Local $GUIWidth = 432, $GUIHeight = 540
Local $posX = (@DesktopWidth - $GUIWidth) / 2
Local $posY = (@DesktopHeight - $GUIHeight) / 2

$Main = GUICreate("Backup Tool", $GUIWidth, $GUIHeight, $posX, $posY)
$FileMenu = GUICtrlCreateMenu("&File")
$FileMenuExit = GUICtrlCreateMenuItem("E&xit", $FileMenu)
$HelpMenu = GUICtrlCreateMenu("&Help")
$HelpMenuAbout = GUICtrlCreateMenuItem("&About", $HelpMenu)

;Create a placeholder label
GUICtrlCreateLabel("", 0, 0, 0, 0)

$Tab1 = GUICtrlCreateTab(8, 16, 417, 473)
GUICtrlSetResizing(-1, $GUI_DOCKAUTO + $GUI_DOCKLEFT + $GUI_DOCKRIGHT + $GUI_DOCKHCENTER + $GUI_DOCKWIDTH + $GUI_DOCKHEIGHT)

$DataTabSheet = GUICtrlCreateTabItem("Data Backup")

$Group1 = GUICtrlCreateGroup("Step 1", 24, 72, 377, 65)
$Label1 = GUICtrlCreateLabel("Enter username:", 32, 88, 141, 17)

;Data tab employee username
$aInput1 = "e.g. lejenkins"
$DataTabUserName = GUICtrlCreateInput("", 32, 104, 137, 21)
GUICtrlSendMsg(-1, $EM_SETCUEBANNER, False, $aInput1)
$Label2 = GUICtrlCreateLabel("UNC path of computer to transfer from:", 208, 88, 186, 17)

;Data tab employee old computer name
$aInput2 = "e.g. \\leeroy-jen-lap\c$"
$DataTabTargetName = GUICtrlCreateInput("", 208, 104, 177, 21)
GUICtrlSendMsg(-1, $EM_SETCUEBANNER, False, $aInput2)
GUICtrlCreateGroup("", -99, -99, 1, 1, 0, 2)

$Group2 = GUICtrlCreateGroup("Step 2", 24, 160, 377, 65)

GUIStartGroup()
$Radio1 = GUICtrlCreateRadio("Transfer Everything", 56, 176, 129, 17)
GUICtrlSetFont(-1, 8, 800, 0, "MS Sans Serif")
$Radio2 = GUICtrlCreateRadio("Selective Copy", 248, 176, 105, 17)
GUICtrlSetFont(-1, 8, 800, 0, "MS Sans Serif")

GUIStartGroup()
$Radio3 = GUICtrlCreateRadio("w/ PST", 56, 200, 57, 17)
$Radio4 = GUICtrlCreateRadio("w/o PST", 120, 200, 65, 17)

$Label4 = GUICtrlCreateLabel("-OR-", 200, 176, 32, 20)
GUICtrlSetFont(-1, 10, 400, 0, "MS Sans Serif")
GUICtrlCreateGroup("", -99, -99, 1, 1, 0, 2)

$Group3 = GUICtrlCreateGroup("Selective Copy - choose options below", 24, 248, 377, 177)

$Checkbox[1][0] = GUICtrlCreateCheckbox("C: Variances", 72, 272, 89, 17)
$Checkbox[1][1] = "CVariance"
$Checkbox[2][0] = GUICtrlCreateCheckbox("User Desktop", 168, 272, 89, 17)
$Checkbox[2][1] = "UserDesktop"
$Checkbox[3][0] = GUICtrlCreateCheckbox("Templates", 272, 272, 97, 17)
$Checkbox[3][1] = "Templates"
$Checkbox[4][0] = GUICtrlCreateCheckbox("Outlook Data", 72, 296, 89, 17)
$Checkbox[4][1] = "OutlookData"
$Checkbox[5][0] = GUICtrlCreateCheckbox("User Profile", 168, 296, 89, 17)
$Checkbox[5][1] = "UserProfile"
$Checkbox[6][0] = GUICtrlCreateCheckbox("System Certificates", 272, 296, 105, 17)
$Checkbox[6][1] = "SysCerts"
$Checkbox[7][0] = GUICtrlCreateCheckbox("Outlook Sigs", 72, 320, 89, 17)
$Checkbox[7][1] = "OutlookSigs"
$Checkbox[8][0] = GUICtrlCreateCheckbox("Favorites", 168, 320, 89, 17)
$Checkbox[8][1] = "Favorites"
$Checkbox[9][0] = GUICtrlCreateCheckbox("PST Files", 72, 344, 89, 17)
$Checkbox[9][1] = "PSTFiles"
$Checkbox[10][0] = GUICtrlCreateCheckbox("Send To items", 168, 344, 89, 17)
$Checkbox[10][1] = "SendToItems"

;Data tab transfer button
$DataTabXferBtn = GUICtrlCreateButton("Begin Transfer", 140, 384, 139, 33)
GUICtrlSetFont(-1, 10, 400, 0, "MS Sans Serif")
GUICtrlSetResizing(-1, $GUI_DOCKAUTO + $GUI_DOCKHCENTER + $GUI_DOCKWIDTH + $GUI_DOCKHEIGHT)

;GUICtrlCreateGroup("", -99, -99, 1, 1, 0, 2)

GUICtrlCreateTabItem("")
$StatusBar1 = _GUICtrlStatusBar_Create($Main)

Dim $Main_AccelTable[1][2] = [["+{F5}", $HelpMenuAbout]]
GUISetAccelerators($Main_AccelTable)
GUISetState()
EndFunc ; Func MainGUI()

While 1
    $msg = GUIGetMsg()
    Switch $msg
        Case $GUI_EVENT_CLOSE, $FileMenuExit
            Local $var
            $ans = MsgBox(8228, "Exit", "Are you sure you wish to exit this program?")
            If $ans = 6 Then Exit

        Case $DataTabXferBtn
            Local $numChecked = 0
            Local $DataTabUserNameInput = GUICtrlRead($DataTabUserName)
            Local $DataTabTargetNameInput = GUICtrlRead($DataTabTargetName)

            ; UNC Regular Expression Test - if UNC path doesn't = \\[a-z][A-Z]\[a-z]$ this test will fail and returns an error message
            ; RegEx that will test for proper folder paths and UNC paths also - ^((\\\\[a-zA-Z0-9-]+\\[a-zA-Z0-9`~!@#$%^&(){}'._-]+([ ]+[a-zA-Z0-9`~!@#$%^&(){}'._-]+)*)|([a-zA-Z]:))(\\[^ \\/:*?""<>|]+([ ]+[^ \\/:*?""<>|]+)*)*\\?$
            Local $TargetNameTest = StringRegExp($DataTabTargetNameInput, '^(\\\\[^/\\\]\[":;|<>+=,?@\(\)* _]{1,15}+\\[a-z|A-Z|$]+)', 0)
            $WinVer = ""
            _OSCheck()
            If $DataTabUserNameInput = "" Then
            MsgBox(8240, "Step 1", "The 'Username' field can't be blank.")
            ;EndIf

            ElseIf $DataTabTargetNameInput = "" Then
                MsgBox(8240, "Step 1", "The 'Target Computername' field can't be blank.")
            ;EndIf

            ElseIf $TargetNameTest = 0 Then
                MsgBox(8240, "Step 1", "You have entered an invalid UNC path.  Please try again.")

            ElseIf $WinVer = "" Then
            MsgBox(8240, "ERROR", "User profile was not found.  Please verify that the employee's username and computer name are correct and try again.  Also, check that the target computer is on the network and that you are set as an administrator.")

            ElseIf $Radio1Status = 4 And $Radio2Status = 4 Then
                MsgBox(8240, "Step 2", "You have not selected any options. Please choose what to copy and try again.")
            ;EndIf

        ElseIf $Radio2Status = 1 Then
            For $1 = 1 to UBound($Checkbox) - 1
                If BitAnd(GUICtrlRead($Checkbox[$1][0]),$GUI_CHECKED) = $GUI_CHECKED Then
                    $numChecked += 1
                    _Checkbox_Procedures($Checkbox[$1][1])
                EndIf
            Next
            If $numchecked = 0 Then MsgBox(8240, "ERROR", "You have chosen to perform a selective copy, but no options were checked.  Please choose what to copy and try again.")
        EndIf

    EndSwitch

        $Radio1Status = GUICtrlRead($Radio1)
    $Radio2Status = GUICtrlRead($Radio2)
    $Radio3Status = GUICtrlRead($Radio3)
    $Radio4Status = GUICtrlRead($Radio4)
    If $Radio1Status = $GUI_CHECKED Then
        If $Radio1Status <> $LastRadio1Status Then
            $LastRadio1Status = $Radio1Status
            For $I = 1 To 10
                GUICtrlSetState($Checkbox[$I][0], BitOR($GUI_DISABLE, $GUI_UNCHECKED))
            Next
            GUICtrlSetState($Radio3, BitOR($GUI_ENABLE, $GUI_CHECKED))
            GUICtrlSetState($Radio4, $GUI_ENABLE)
        EndIf
    Else
        If $Radio1Status <> $LastRadio1Status Then
            $LastRadio1Status = $Radio1Status
        EndIf
    EndIf
    If $Radio2Status = $GUI_CHECKED Then
        If $Radio2Status <> $LastRadio2Status Then
            $LastRadio2Status = $Radio2Status
            For $I = 1 To 10
                GUICtrlSetState($Checkbox[$I][0], $GUI_ENABLE)
            Next
            GUICtrlSetState($Radio3, BitOR($GUI_DISABLE, $GUI_UNCHECKED))
            GUICtrlSetState($Radio4, BitOR($GUI_DISABLE, $GUI_UNCHECKED))
        EndIf
    Else
        If $Radio2Status <> $LastRadio2Status Then
            $LastRadio2Status = $Radio2Status
        EndIf
    EndIf

WEnd

Func _Checkbox_Procedures($parm1)

    Switch $parm1

        Case 'CVariance' ; Should be complete
            ; Returns Files only, not folders
            $CVarianceFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "", "*|swapfile.sys;ntuser.dat;ntuser.dat.LOG1;ntuser.dat.LOG2;*.regtrans-ms;ntuser.pol;ntuser.ini;*.blf;*.pst;*.ost;*.obi;atlog.txt;autoexec.bat;boot.ini;config.sys;io.sys;logfile.txt;msdos.sys;ntdetect.com;ntldr;pagefile.sys;hiberfile.sys;hiberfil.sys;normal.dot;bootmgr;bootsect.bak;desktop.ini|Temporary Internet Files;programdata;Appdata;Application Data;roaming;Local Settings;drivers;intel;winnt;Program Files (x86);msocache;program files;windows;$Recycle.Bin;system volume information;i386;spoolerlogs;Documents and Settings;Recovery;boot;nvid;nvidia;PerfLogs;Config.Msi", 1, 1, 0, 2)
            If IsArray($CVarianceFiles) Then _ConcatenateFileArrays()

        Case 'UserDesktop' ; Should be complete

            ; Returns Files only, not folders
            If $WinVer = "new" Then
                $UserDesktopFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\Desktop", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserDesktopFiles) Then _ConcatenateFileArrays()
            ElseIf $WinVer = "old" Then
                $UserDesktopFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Documents and Settings\" & $DataTabUserNameInput & "\Desktop", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserDesktopFiles) Then _ConcatenateFileArrays()
            Else
                MsgBox(4096, "ERROR", "Unable to determine location of the user desktop folder")
            EndIf
        Case 'Templates' ; Should be complete

            ; Returns Files only, not folders
            If $WinVer = "new" Then
                $UserTemplateFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Roaming\Microsoft\Windows\Templates", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserTemplateFiles) Then _ConcatenateFileArrays()
            ElseIf $WinVer = "old" Then
                $UserTemplateFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Documents and Settings\" & $DataTabUserNameInput & "Application Data\Microsoft\Templates", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserTemplateFiles) Then _ConcatenateFileArrays()
            Else
                MsgBox(4096, "ERROR", "Unable to determine location of the user templates folder")
            EndIf
        Case 'OutlookData' ; Should be complete

            ; Returns Files only, not folders
            If $WinVer = "new" Then
                $UserOutlookFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Local\Microsoft\Outlook", "*|desktop.ini;*.obi;*.ost;*.kfl;*.pst", 1, 1, 0, 2)
            If IsArray($UserOutlookFiles) Then _ConcatenateFileArrays()
            ElseIf $WinVer = "old" Then
                $UserOutlookFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Documents and Settings\" & $DataTabUserNameInput & "Local Settings\Application Data\Microsoft\Outlook", "*|desktop.ini;*.obi;*.ost;*.kfl;*.pst", 1, 1, 0, 2)
            If IsArray($UserOutlookFiles) Then _ConcatenateFileArrays()
            Else
                MsgBox(4096, "ERROR", "Unable to determine location of the user outlook folder")
            EndIf
        Case 'UserProfile' ;Should be complete

            ; Returns Files only, not folders
            If $WinVer = "new" Then
                $UserProfileFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput, "*|ntuser.dat;desktop.ini;*.dat.log1;*.dat.log2;*.blf;*.regtrans-ms;ntuser.ini;ntuser.pol|.thumbnails;AppData;Application Data;Cookies;Local Settings;NetHood;PrintHood;Searches", 1, 1, 0, 2)
            If IsArray($UserProfileFiles) Then _ConcatenateFileArrays()
            ElseIf $WinVer = "old" Then
                $UserProfileFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Documents and Settings\" & $DataTabUserNameInput, "*|ntuser.dat;desktop.ini*.dat.log1;*.dat.log2;*.blf;*.regtrans-ms;ntuser.ini;ntuser.pol|.thumbnails;AppData;Application Data;Cookies;Local Settings;NetHood;PrintHood;Searches", 1, 1, 0, 2)
            If IsArray($UserProfileFiles) Then _ConcatenateFileArrays()
            Else
                MsgBox(4096, "ERROR", "Unable to determine location of the user profile folder")
            EndIf
        Case 'SysCerts' ; Should be complete

                        ; Returns Files only, not folders
            If $WinVer = "new" Then
                $UserSystemCertFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Roaming\Microsoft\Systemcertificates", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserSystemCertFiles) Then _ConcatenateFileArrays()
            ElseIf $WinVer = "old" Then
                $UserSystemCertFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Documents and Settings\" & $DataTabUserNameInput & "\Application Data\Microsoft\SystemCertificates", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserSystemCertFiles) Then _ConcatenateFileArrays()
            Else
                MsgBox(4096, "ERROR", "Unable to determine location of the user system certs folder")
            EndIf
        Case 'OutlookSigs' ; Should be complete

                        ; Returns Files only, not folders
            If $WinVer = "new" Then
                $UserOutlookSigFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Roaming\Microsoft\signatures", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserOutlookSigFiles) Then _ConcatenateFileArrays()
            ElseIf $WinVer = "old" Then
                $UserOutlookSigFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Documents and Settings\" & $DataTabUserNameInput & "\Application Data\Microsoft\signatures", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserOutlookSigFiles) Then _ConcatenateFileArrays()
            Else
                MsgBox(4096, "ERROR", "Unable to determine location of the user signatures folder")
            EndIf
        Case 'Favorites' ;Should be complete

                        ; Returns Files only, not folders
            If $WinVer = "new" Then
                $UserFavoriteFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\Favorites", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserFavoriteFiles) Then _ConcatenateFileArrays()
            ElseIf $WinVer = "old" Then
                $UserFavoriteFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Documents and Settings\" & $DataTabUserNameInput & "\Favorites", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserFavoriteFiles) Then _ConcatenateFileArrays()
            Else
                MsgBox(4096, "ERROR", "Unable to determine location of the user favorites folder")
            EndIf
        Case 'PSTFiles' ;Should be complete

                        ; Returns Files only, not folders
            If $WinVer = "new" Then
                $UserOutlook07PSTFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Local\Microsoft\Outlook", "*.pst|desktop.ini", 1, 1, 0, 2)
                $UserOutlook10PSTFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\Documents\Outlook Files", "*.pst|desktop.ini", 1, 1, 0, 2)
                _ArrayConcatenate($UserOutlook07PSTFiles, $UserOutlook10PSTFiles, 1)
                $UserOutlookPSTFiles = _ArrayUnique($UserOutlook07PSTFiles, 1, 1, 0)
            If IsArray($UserOutlookPSTFiles) Then _ConcatenateFileArrays()
            ElseIf $WinVer = "old" Then
                $UserOutlook07PSTFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Documents and Settings\" & $DataTabUserNameInput & "\Local Settings\Application Data\Microsoft\Outlook", "*.pst|desktop.ini", 1, 1, 0, 2)
                $UserOutlook10PSTFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Documents and Settings\" & $DataTabUserNameInput & "\My Documents\Outlook Files", "*.pst|desktop.ini", 1, 1, 0, 2)
                _ArrayConcatenate($UserOutlook07PSTFiles, $UserOutlook10PSTFiles, 1)
                $UserOutlookPSTFiles = _ArrayUnique($UserOutlook07PSTFiles, 1, 1, 0)
            If IsArray($UserOutlookPSTFiles) Then _ConcatenateFileArrays()
            Else
                MsgBox(4096, "ERROR", "Unable to determine location of the user PSTs folder")
            EndIf
        Case 'SendToItems' ; Should be complete

                        ; Returns Files only, not folders
            If $WinVer = "new" Then
                $UserSendToFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Roaming\Microsoft\Windows\SendTo", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserSendToFiles) Then _ConcatenateFileArrays()
            ElseIf $WinVer = "old" Then
                $UserSendToFiles = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Documents and Settings\" & $DataTabUserNameInput & "\SendTo", "*|desktop.ini", 1, 1, 0, 2)
            If IsArray($UserSendToFiles) Then _ConcatenateFileArrays()
            Else
                MsgBox(4096, "ERROR", "Unable to determine location of the user SendTo folder")
            EndIf
        Case Else
    EndSwitch

EndFunc

Func _ConcatenateFileArrays()
    Local $FileArrays[10] = ["$CVarianceFiles", "$UserDesktopFiles", "$UserTemplateFiles", "$UserOutlookFiles", "$UserProfileFiles", "$UserSystemCertFiles", "$UserOutlookSigFiles", "$UserFavoriteFiles", "$UserOutlookPSTFiles", "$UserSendToFiles"]

    For $i = 0 To UBound($FileArrays) - 1
        _ArrayConcatenate($FilesToTransfer, $FileArrays[$i])
    Next
    $FilesToTransfer = _ArrayUnique($FilesToTransfer)
    _ArrayDisplay($FilesToTransfer)

EndFunc

Func _OSCheck()
    If FileExists($DataTabTargetNameInput & "\users\" & $DataTabUserNameInput) Then
        $WinVer = "new"
    ElseIf FileExists($DataTabTargetNameInput & "\Documents and Settings\" & $DataTabUserNameInput) Then
        $WinVer = "old"
    EndIf
EndFunc; _OSCheck()

expandcollapse popup

#include-once

;#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w- 4 -w 5 -w 6 -w- 7

; #INDEX# =======================================================================================================================
; Title .........: _RecFileListToArray
; AutoIt Version : v3.3.1.1 or higher
; Language ......: English
; Description ...: Lists files and\or folders in specified path with optional recursion to defined level and result sorting
; Note ..........:
; Author(s) .....: Melba23
; Remarks .......: - Modified Array.au3 functions - credit: Jos van der Zande, LazyCoder, Tylo, Ultima, SolidSnake and gcriaco
;                  - SRE patterns - credit: various forum members and Spiff59 in particular
;                  - DllCall code suggestion - credit: guinness
;                  - Despite the name, this UDF is iterative, not recursive
; ===============================================================================================================================

; #CURRENT# =====================================================================================================================
; _RecFileListToArray: Lists files and\or folders in a specified path with optional recursion to defined level and result sorting
; ===============================================================================================================================

; #INTERNAL_USE_ONLY#============================================================================================================
; _RFLTA_ListToMask ......; Convert include/exclude lists to SRE format
; _RFLTA_AddToList .......; Add element to list which is resized if necessary
; _RFLTA_AddFileLists ....; Add internal lists after resizing and optional sorting
; _RFLTA_FileListSearch ..; Search file match list for files associated with a folder
; _RFLTA_ArraySort .......; Wrapper for QuickSort function
; _RFLTA_QuickSort .......: Recursive array sort
; _RFLTA_ArrayConcatenate : Join 2 arrays
; ===============================================================================================================================

; #FUNCTION# ====================================================================================================================
; Name...........: _RecFileListToArray
; Description ...: Lists files and\or folders in a specified path with optional recursion to defined level and result sorting
; Syntax.........: _RecFileListToArray($sPath[, $sMask = "*"[, $iReturn = 0[, $iRecur = 0[, $iSort = 0[, $iReturnPath = 1[, $sExclude_List = ""[, $sExclude_List_Folder]]]]]]])
; Parameters ....: $sPath - Initial path used to generate filelist
;                               If path ends in \ then folders will be returned with an ending \
;                               If path lengths > 260 chars, prefix path with "\\?\" - return paths are not affected
;                  $sMask - Optional: Filter for result. Multiple filters must be separated by ";"
;                           Use "|" to separate 3 possible sets of filters: "Include|Exclude|Exclude_Folders"
;                               Include = Files/Folders to include (default = "*" [all])
;                               Exclude = Files/Folders to exclude (default = "" [none])
;                               Exclude_Folders = only used if basic $iReturn = 0 AND $iRecur = 1 to exclude defined folders (default = "" [none])
;                  $iReturn - Optional: specifies whether to return files, folders or both and omits those with certain attributes
;                              0 - Return both files and folders (Default)
;                                   If non-recursive Include/Exclude_List applies to files and folders
;                                   If recursive Include/Exclude_List applies to files only, all folders are searched unless excluded using $sExclude_List_Folder
;                              1 - Return files only    - Include/Exclude_List applies to files only, all folders searched if recursive
;                              2 - Return folders only  - Include/Exclude_List applies to folders only for searching and return
;                                   Add one or more of the following to $iReturn to omit files/folders with that attribute
;                                   + 4  - Hidden files and folders
;                                   + 8  - System files and folders
;                                   + 16 - Link/junction folders
;                                   Note:  Omitting files/folders uses a different search algorithm and takes approx 50% longer
;                  $iRecur - Optional: specifies whether to search recursively in subfolders and to what level
;                             1 - Search in all subfolders (unlimited recursion)
;                             0 - Do not search in subfolders (Default)
;                             Negative integer - Search in subfolders to specified depth
;                  $iSort - Optional: sort ordered in alphabetical and depth order
;                             0 - Not sorted (Default)
;                             1 - Sorted
;                             2 - Sorted with faster algorithm (assumes files sorted within each folder - requires NTFS drive)
;                  $iReturnPath - Optional: specifies displayed path of results
;                             0 - File/folder name only
;                             1 - Relative to initial path (Default)
;                             2 - Full path included
;
; [These parameters are now deprecated as they should be included within the $sMask parameter]
; [If $sMask contains sections for Exclude and Exclude_Folders (i.e "|" characters are used) then these parameters will be ignored]
;                  $sExclude_List - Optional: filter for excluded results (default ""). Multiple filters must be separated by ";"
;                  $sExclude_List_Folder - Optional: only used if $iReturn = 0 AND $iRecur = 1 to exclude folders matching the filter
;
; Requirement(s).: v3.3.1.1 or higher
; Return values .: Success: One-dimensional array made up as follows:
;                            [0] = Number of Files\Folders returned
;                            [1] = 1st File\Folder
;                            [2] = 2nd File\Folder
;                            ...
;                            [n] = nth File\Folder
;                   Failure: Null string and @error = 1 with @extended set as follows:
;                            1 = Path not found or invalid
;                            2 = Invalid $sInclude_List
;                            3 = Invalid $iReturn
;                            4 = Invalid $iRecur
;                            5 = Invalid $iSort
;                            6 = Invalid $iReturnPath
;                            7 = Invalid $sExclude_List
;                            8 = Invalid $sExclude_List_Folder
;                            9 = No files/folders found
; Author ........: Melba23
; Remarks .......: Compatible with existing _FileListToArray syntax
; Related .......:
; Link ..........;
; Example .......; Yes
; ===============================================================================================================================
Func _RecFileListToArray($sInitialPath, $sMask = "*", $iReturn = 0, $iRecur = 0, $iSort = 0, $iReturnPath = 1, $sExclude_List = "", $sExclude_List_Folder = "")

    Local $asReturnList[100] = [0], $asFileMatchList[100] = [0], $asRootFileMatchList[100] = [0], $asFolderMatchList[100] = [0], $asFolderSearchList[100] = [1]
    Local $sFolderSlash = "", $iMaxLevel, $sInclude_List, $sInclude_File_Mask, $sExclude_File_Mask, $sInclude_Folder_Mask = ".+", $sExclude_Folder_Mask = ":"
    Local $hSearch, $fFolder, $sRetPath = "", $sCurrentPath, $sName, $iAttribs, $iHide_HS = 0, $iHide_Link = 0, $fLongPath = False
    Local $asFolderFileSectionList[100][2] = [[0, 0]], $sFolderToFind, $iFileSectionStartIndex, $iFileSectionEndIndex

    ; Check for valid path
    If StringLeft($sInitialPath, 4) == "\\?\" Then
        $fLongPath = True
    EndIf
    If Not FileExists($sInitialPath) Then Return SetError(1, 1, "")
    ; Check if folders should have trailing \ and ensure that initial path does have one
    If StringRight($sInitialPath, 1) = "\" Then
        $sFolderSlash = "\"
    Else
        $sInitialPath = $sInitialPath & "\"
    EndIf
    ; Add path to folder search list
    $asFolderSearchList[1] = $sInitialPath

    ; Check for H or S omitted
    If BitAND($iReturn, 4) Then
        $iHide_HS += 2
        $iReturn -= 4
    EndIf
    If BitAND($iReturn, 8) Then
        $iHide_HS += 4
        $iReturn -= 8
    EndIf

    ; Check for link/junction omitted
    If BitAND($iReturn, 16) Then
        $iHide_HS = 0x406
        $iReturn -= 16
    EndIf

    ; Check for valid recur value
    If $iRecur > 1 Or Not IsInt($iRecur) Then Return SetError(1, 4, "")
    ; If required, determine \ count for max recursive level setting
    If $iRecur < 0 Then
        StringReplace($sInitialPath, "\", "", 0, 2)
        $iMaxLevel = @extended - $iRecur
    EndIf

    ; Check mask parameter
    Local $aMaskSplit = StringSplit($sMask, "|")
    ; Check for multiple sections and set values
    Switch $aMaskSplit[0]
        Case 3
            $sExclude_List_Folder = $aMaskSplit[3]
            ContinueCase
        Case 2
            $sExclude_List = $aMaskSplit[2]
            ContinueCase
        Case 1
            $sInclude_List = $aMaskSplit[1]
    EndSwitch

    ; Create Include mask for files
    If $sInclude_List = "*" Then
        $sInclude_File_Mask = ".+"
    Else
        If Not _RFLTA_ListToMask($sInclude_File_Mask, $sInclude_List) Then Return SetError(1, 2, "")
    EndIf
    ; Set Include mask for folders
    Switch $iReturn
        Case 0
            ; Folders affected by mask if not recursive
            Switch $iRecur
                Case 0
                    ; Folders match mask for compatibility
                    $sInclude_Folder_Mask = $sInclude_File_Mask
            EndSwitch
        Case 2
            ; Folders affected by mask
            $sInclude_Folder_Mask = $sInclude_File_Mask
    EndSwitch

    ; Create Exclude List mask for files
    If $sExclude_List = "" Then
        $sExclude_File_Mask = ":" ; Set unmatchable mask
    Else
        If Not _RFLTA_ListToMask($sExclude_File_Mask, $sExclude_List) Then Return SetError(1, 7, "")
    EndIf

    ; Create Exclude mask for folders
    If $iRecur Then
        If $sExclude_List_Folder Then
            If Not _RFLTA_ListToMask($sExclude_Folder_Mask, $sExclude_List_Folder) Then Return SetError(1, 8, "")
        EndIf
        ; If folders only
        If $iReturn = 2 Then
            ; Folders affected by normal mask
            $sExclude_Folder_Mask = $sExclude_File_Mask
        EndIf
    Else
        ; Folders affected by normal mask
        $sExclude_Folder_Mask = $sExclude_File_Mask
    EndIf

    ; Verify other parameters
    If Not ($iReturn = 0 Or $iReturn = 1 Or $iReturn = 2) Then Return SetError(1, 3, "")
    If Not ($iSort = 0 Or $iSort = 1 Or $iSort = 2) Then Return SetError(1, 5, "")
    If Not ($iReturnPath = 0 Or $iReturnPath = 1 Or $iReturnPath = 2) Then Return SetError(1, 6, "")

    ; Prepare for DllCall if required
    If $iHide_HS Or $iHide_Link Then
        Local $tFile_Data = DllStructCreate("struct;align 4;dword FileAttributes;uint64 CreationTime;uint64 LastAccessTime;uint64 LastWriteTime;" & _
                "dword FileSizeHigh;dword FileSizeLow;dword Reserved0;dword Reserved1;wchar FileName[260];wchar AlternateFileName[14];endstruct")
        Local $pFile_Data = DllStructGetPtr($tFile_Data), $hDLL = DllOpen('kernel32.dll'), $aDLL_Ret
    EndIf

    ; Search within listed folders
    While $asFolderSearchList[0] > 0

        ; Set path to search
        $sCurrentPath = $asFolderSearchList[$asFolderSearchList[0]]

        ; Reduce folder search list count
        $asFolderSearchList[0] -= 1

        ; Determine return path to add to file/folder name
        Switch $iReturnPath
            ; Case 0 ; Name only
            ; Leave as ""
            Case 1 ;Relative to initial path
                $sRetPath = StringReplace($sCurrentPath, $sInitialPath, "")
            Case 2 ; Full path
                If $fLongPath Then
                    $sRetPath = StringTrimLeft($sCurrentPath, 4)
                Else
                    $sRetPath = $sCurrentPath
                EndIf
        EndSwitch

        ; Get search handle - use code matched to required listing
        If $iHide_HS Or $iHide_Link Then
            ; Use DLL code
            $aDLL_Ret = DllCall($hDLL, 'ptr', 'FindFirstFileW', 'wstr', $sCurrentPath & "*", 'ptr', $pFile_Data)
            If @error Or Not $aDLL_Ret[0] Then
                ContinueLoop
            EndIf
            $hSearch = $aDLL_Ret[0]
        Else
            ; Use native code
            $hSearch = FileFindFirstFile($sCurrentPath & "*")
            ; If folder empty move to next in list
            If $hSearch = -1 Then
                ContinueLoop
            EndIf
        EndIf

        ; If sorting files and folders with paths then store folder name and position of associated files in list
        If $iReturn = 0 And $iSort And $iReturnPath Then
            _RFLTA_AddToList($asFolderFileSectionList, $sRetPath, $asFileMatchList[0] + 1)
        EndIf

        ; Search folder - use code matched to required listing
        While 1
            ; Use DLL code
            If $iHide_HS Or $iHide_Link Then
                ; Use DLL code
                $aDLL_Ret = DllCall($hDLL, 'int', 'FindNextFileW', 'ptr', $hSearch, 'ptr', $pFile_Data)
                ; Check for end of folder
                If @error Or Not $aDLL_Ret[0] Then
                    ExitLoop
                EndIf

                ; Extract data
                $sName = DllStructGetData($tFile_Data, "FileName")
                ; Check for .. return - only returned by the DllCall
                If $sName = ".." Then
                    ContinueLoop
                EndIf
                $iAttribs = DllStructGetData($tFile_Data, "FileAttributes")
                ; Check for hidden/system attributes and skip if found
                If $iHide_HS And BitAND($iAttribs, $iHide_HS) Then
                    ContinueLoop
                EndIf
                ; Check for link attribute and skip if found
                If $iHide_Link And BitAND($iAttribs, $iHide_Link) Then
                    ContinueLoop
                EndIf
                ; Set subfolder flag
                $fFolder = 0
                If BitAND($iAttribs, 16) Then
                    $fFolder = 1
                EndIf
            Else
                ; Use native code
                $sName = FileFindNextFile($hSearch)
                ; Check for end of folder
                If @error Then
                    ExitLoop
                EndIf
                ; Set subfolder flag - @extended set in 3.3.1.1 +
                $fFolder = @extended
            EndIf

            ; If folder then check whether to add to search list
            If $fFolder Then
                Select
                    Case $iRecur < 0 ; Check recur depth
                        StringReplace($sCurrentPath, "\", "", 0, 2)
                        If @extended < $iMaxLevel Then
                            ContinueCase ; Check if matched to masks
                        EndIf
                    Case $iRecur = 1 ; Full recur
                        If Not StringRegExp($sName, $sExclude_Folder_Mask) Then ; Add folder unless excluded
                            _RFLTA_AddToList($asFolderSearchList, $sCurrentPath & $sName & "\")
                        EndIf
                        ; Case $iRecur = 0 ; Never add
                        ; Do nothing
                EndSelect
            EndIf

            If $iSort Then ; Save in relevant folders for later sorting

                If $fFolder Then
                    If StringRegExp($sName, $sInclude_Folder_Mask) And Not StringRegExp($sName, $sExclude_Folder_Mask) Then
                        _RFLTA_AddToList($asFolderMatchList, $sRetPath & $sName & $sFolderSlash)
                    EndIf
                Else
                    If StringRegExp($sName, $sInclude_File_Mask) And Not StringRegExp($sName, $sExclude_File_Mask) Then
                        ; Select required list for files
                        If $sCurrentPath = $sInitialPath Then
                            _RFLTA_AddToList($asRootFileMatchList, $sRetPath & $sName)
                        Else
                            _RFLTA_AddToList($asFileMatchList, $sRetPath & $sName)
                        EndIf
                    EndIf
                EndIf

            Else ; Save directly in return list
                If $fFolder Then
                    If $iReturn <> 1 And StringRegExp($sName, $sInclude_Folder_Mask) And Not StringRegExp($sName, $sExclude_Folder_Mask) Then
                        _RFLTA_AddToList($asReturnList, $sRetPath & $sName & $sFolderSlash)
                    EndIf
                Else
                    If $iReturn <> 2 And StringRegExp($sName, $sInclude_File_Mask) And Not StringRegExp($sName, $sExclude_File_Mask) Then
                        _RFLTA_AddToList($asReturnList, $sRetPath & $sName)
                    EndIf
                EndIf
            EndIf

        WEnd

        ; Close current search
        FileClose($hSearch)

    WEnd

    ; Close the DLL if needed
    If $iHide_HS Then
        DllClose($hDLL)
    EndIf

    If $iSort Then
        ; Check if any file/folders have been added depending on required return
        Switch $iReturn
            Case 0 ; If no folders then number of files is immaterial
                If $asRootFileMatchList[0] = 0 And $asFolderMatchList[0] = 0 Then Return SetError(1, 9, "")
            Case 1
                If $asRootFileMatchList[0] = 0 And $asFileMatchList[0] = 0 Then Return SetError(1, 9, "")
            Case 2
                If $asFolderMatchList[0] = 0 Then Return SetError(1, 9, "")
        EndSwitch

        Switch $iReturn
            Case 2 ; Folders only
                ; Correctly size folder match list
                ReDim $asFolderMatchList[$asFolderMatchList[0] + 1]
                ; Copy size folder match array
                $asReturnList = $asFolderMatchList
                ; Simple sort list
                _RFLTA_ArraySort($asReturnList)
            Case 1 ; Files only
                If $iReturnPath = 0 Then ; names only so simple sort suffices
                    ; Combine file match lists
                    _RFLTA_AddFileLists($asReturnList, $asRootFileMatchList, $asFileMatchList)
                    ; Simple sort combined file list
                    _RFLTA_ArraySort($asReturnList)
                Else
                    ; Combine sorted file match lists
                    _RFLTA_AddFileLists($asReturnList, $asRootFileMatchList, $asFileMatchList, 1)
                EndIf
            Case 0 ; Both files and folders
                If $iReturnPath = 0 Then ; names only so simple sort suffices
                    ; Combine file match lists
                    _RFLTA_AddFileLists($asReturnList, $asRootFileMatchList, $asFileMatchList)
                    ; Set correct count for folder add
                    $asReturnList[0] += $asFolderMatchList[0]
                    ; Resize and add file match array
                    ReDim $asFolderMatchList[$asFolderMatchList[0] + 1]
                    _RFLTA_ArrayConcatenate($asReturnList, $asFolderMatchList)
                    ; Simple sort final list
                    _RFLTA_ArraySort($asReturnList)
                Else
                    ; Size return list
                    Local $asReturnList[$asFileMatchList[0] + $asRootFileMatchList[0] + $asFolderMatchList[0] + 1]
                    $asReturnList[0] = $asFileMatchList[0] + $asRootFileMatchList[0] + $asFolderMatchList[0]
                    ; Sort root file list
                    _RFLTA_ArraySort($asRootFileMatchList, 1, $asRootFileMatchList[0])
                    ; Add the sorted root files at the top
                    For $i = 1 To $asRootFileMatchList[0]
                        $asReturnList[$i] = $asRootFileMatchList[$i]
                    Next
                    ; Set next insertion index
                    Local $iNextInsertionIndex = $asRootFileMatchList[0] + 1
                    ; Sort folder list
                    _RFLTA_ArraySort($asFolderMatchList, 1, $asFolderMatchList[0])
                    ; Work through folder list
                    For $i = 1 To $asFolderMatchList[0]
                        ; Add folder to return list
                        $asReturnList[$iNextInsertionIndex] = $asFolderMatchList[$i]
                        $iNextInsertionIndex += 1
                        ; Format folder name for search
                        If $sFolderSlash Then
                            $sFolderToFind = $asFolderMatchList[$i]
                        Else
                            $sFolderToFind = $asFolderMatchList[$i] & "\"
                        EndIf
                        ; Find folder in FolderFileSectionList
                        For $j = 1 To $asFolderFileSectionList[0][0]
                            ; If found then deal with files
                            If $sFolderToFind = $asFolderFileSectionList[$j][0] Then
                                ; Set file list indexes
                                $iFileSectionStartIndex = $asFolderFileSectionList[$j][1]
                                If $j = $asFolderFileSectionList[0][0] Then
                                    $iFileSectionEndIndex = $asFileMatchList[0]
                                Else
                                    $iFileSectionEndIndex = $asFolderFileSectionList[$j + 1][1] - 1
                                EndIf
                                ; Sort files
                                _RFLTA_ArraySort($asFileMatchList, $iFileSectionStartIndex, $iFileSectionEndIndex)
                                ; Add files to return list
                                For $k = $iFileSectionStartIndex To $iFileSectionEndIndex
                                    $asReturnList[$iNextInsertionIndex] = $asFileMatchList[$k]
                                    $iNextInsertionIndex += 1
                                Next
                                ExitLoop
                            EndIf
                        Next
                    Next
                EndIf
        EndSwitch

    Else ; No sort

        ; Check if any file/folders have been added
        If $asReturnList[0] = 0 Then Return SetError(1, 9, "")

        ; Remove any unused return list elements from last ReDim
        ReDim $asReturnList[$asReturnList[0] + 1]

    EndIf

    Return $asReturnList


EndFunc   ;==>_RecFileListToArray

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_ListToMask
; Description ...: Convert include/exclude lists to SRE format
; Syntax ........: _RFLTA_ListToMask(ByRef $sMask, $sList)
; Parameters ....: $asMask - Include/Exclude mask to create
;                  $asList - Include/Exclude list to convert
; Return values .: Success: 1
;                  Failure: 0
; Author ........: SRE patterns developed from those posted by various forum members and Spiff59 in particular
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_ListToMask(ByRef $sMask, $sList)

    ; Check for invalid characters within list
    If StringRegExp($sList, "\\|/|:|\<|\>|\|") Then Return 0
    ; Strip WS and insert | for ;
    $sList = StringReplace(StringStripWS(StringRegExpReplace($sList, "\s*;\s*", ";"), 3), ";", "|")
    ; Convert to SRE pattern
    $sList = StringReplace(StringReplace(StringRegExpReplace($sList, "[][$^.{}()+\-]", "\\$0"), "?", "."), "*", ".*?")
    ; Add prefix and suffix
    $sMask = "(?i)^(" & $sList & ")\z"
    Return 1

EndFunc   ;==>_RFLTA_ListToMask

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_AddToList
; Description ...: Add element to [?] or [?][2] list which is resized if necessary
; Syntax ........: _RFLTA_AddToList(ByRef $asList, $vValue_0, [$vValue_1])
; Parameters ....: $aList - List to be added to
;                  $vValue_0 - Value to add (to [0] element in [?][2] array if $vValue_1 exists)
;                  $vValue_1 - Value to add to [1] element in [?][2] array (optional)
; Return values .: None - array modified ByRef
; Author ........: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_AddToList(ByRef $aList, $vValue_0, $vValue_1 = -1)

    If $vValue_1 = -1 Then ; [?] array
        ; Increase list count
        $aList[0] += 1
        ; Double list size if too small (fewer ReDim needed)
        If UBound($aList) <= $aList[0] Then ReDim $aList[UBound($aList) * 2]
        ; Add value
        $aList[$aList[0]] = $vValue_0
    Else ; [?][2] array
        $aList[0][0] += 1
        If UBound($aList) <= $aList[0][0] Then ReDim $aList[UBound($aList) * 2][2]
        $aList[$aList[0][0]][0] = $vValue_0
        $aList[$aList[0][0]][1] = $vValue_1
    EndIf

EndFunc   ;==>_RFLTA_AddToList

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_AddFileLists
; Description ...: Add internal lists after resizing and optional sorting
; Syntax ........: _RFLTA_AddFileLists(ByRef $asTarget, $asSource_1, $asSource_2[, $iSort = 0])
; Parameters ....: $asReturnList - Base list
;                  $asRootFileMatchList - First list to add
;                  $asFileMatchList - Second list to add
;                  $iSort - (Optional) Whether to sort lists before adding
;                  |$iSort = 0 (Default) No sort
;                  |$iSort = 1 Sort in descending alphabetical order
; Return values .: None - array modified ByRef
; Author ........: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_AddFileLists(ByRef $asTarget, $asSource_1, $asSource_2, $iSort = 0)

    ; Correctly size root file match array
    ReDim $asSource_1[$asSource_1[0] + 1]
    ; Simple sort root file match array if required
    If $iSort = 1 Then _RFLTA_ArraySort($asSource_1)
    ; Copy root file match array
    $asTarget = $asSource_1
    ; Add file match count
    $asTarget[0] += $asSource_2[0]
    ; Correctly size file match array
    ReDim $asSource_2[$asSource_2[0] + 1]
    ; Simple sort file match array if required
    If $iSort = 1 Then _RFLTA_ArraySort($asSource_2)
    ; Add file match array
    _RFLTA_ArrayConcatenate($asTarget, $asSource_2)

EndFunc   ;==>_RFLTA_AddFileLists

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_FileListSearch
; Description ...: Search file array for beginning and end indices of folder associated files
; Syntax ........: _RFLTA_FileListSearch(Const ByRef $avArray, $vValue)
; Parameters ....: $avArray - Array to search ($asFileMatchList)
;                  $vValue  - Value to search for (Folder name from $asFolderMatchList)
;                  $iIndex  - Index to begin search (search down from here - and then from here to top if not found)
;                  $sSlash  - \ if folder names end in \ - else empty string
; Return values .: Success: Array holding top and bottom indices of folder associated files
;                  Failure: Returns -1
; Author ........: Melba23
; Modified.......:
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_FileListSearch(Const ByRef $avArray, $vValue, $iIndex, $sSlash)

    Local $aRet[2]

    ; Add final \ if required
    If Not $sSlash Then $vValue &= "\"
    ; Start by getting top match - search down from start index
    For $i = $iIndex To $avArray[0]
        ; SRE gives path less filename
        If StringRegExpReplace($avArray[$i], "(^.*\\)(.*)", "\1") = $vValue Then ExitLoop
    Next
    If $i > $avArray[0] Then
        ; No match found so look from start index upwards
        If $iIndex = $avArray[0] Then $iIndex -= 1
        For $i = $iIndex + 1 To 1 Step -1
            If StringRegExpReplace($avArray[$i], "(^.*\\)(.*)", "\1") = $vValue Then ExitLoop
        Next
        ; If still no match - return " nothing found" for empty folder
        If $i = 0 Then Return SetError(1, 0, "")
        ; Set index of bottom file
        $aRet[1] = $i
        ; Now look for top match
        For $i = $aRet[1] To 1 Step -1
            If StringRegExpReplace($avArray[$i], "(^.*\\)(.*)", "\1") <> $vValue Then ExitLoop
        Next
        ; Set top match
        $aRet[0] = $i + 1
    Else
        ; Set index of top associated file
        $aRet[0] = $i
        ; Now look for bottom match - find first file which does not match
        For $i = $aRet[0] To $avArray[0]
            If StringRegExpReplace($avArray[$i], "(^.*\\)(.*)", "\1") <> $vValue Then ExitLoop
        Next
        ; Set bottom match
        $aRet[1] = $i - 1
    EndIf

    Return $aRet

EndFunc   ;==>_RFLTA_FileListSearch

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_ArraySort
; Description ...: Wrapper for QuickSort function
; Syntax ........: _RFLTA_ArraySort(ByRef $avArray)
; Parameters ....: $avArray - Array to sort
;                  $iStart  - Index to start sort
;                  $iEnd    - Index to end sort
; Return values .: None - array modified ByRef
; Author ........: Jos van der Zande, LazyCoder, Tylo, Ultima
; Modified.......: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_ArraySort(ByRef $avArray, $iStart = 1, $iEnd = -99)

    If $iEnd = -99 Then $iEnd = UBound($avArray) - 1

    _RFLTA_QuickSort($avArray, $iStart, $iEnd)

EndFunc   ;==>_RFLTA_ArraySort

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_QuickSort
; Description ...: Recursive array sort
; Syntax ........: _RFLTA_QuickSort(ByRef $avArray, ByRef $iStart, ByRef $iEnd)
; Parameters ....: $avArray - Array to sort in descending alphabetical order
;                  $iStart - Start index
;                  $iEnd - End index
; Return values .: None - array modified ByRef
; Author ........: Jos van der Zande, LazyCoder, Tylo, Ultima
; Modified.......: Melba23
; Remarks .......: This function is used internally by _RFLTA_ArraySort
; ===============================================================================================================================
Func _RFLTA_QuickSort(ByRef $avArray, ByRef $iStart, ByRef $iEnd)

    Local $vTmp
    If ($iEnd - $iStart) < 15 Then
        Local $i, $j, $vCur
        For $i = $iStart + 1 To $iEnd
            $vTmp = $avArray[$i]
            If IsNumber($vTmp) Then
                For $j = $i - 1 To $iStart Step -1
                    $vCur = $avArray[$j]
                    If ($vTmp >= $vCur And IsNumber($vCur)) Or (Not IsNumber($vCur) And StringCompare($vTmp, $vCur) >= 0) Then ExitLoop
                    $avArray[$j + 1] = $vCur
                Next
            Else
                For $j = $i - 1 To $iStart Step -1
                    If (StringCompare($vTmp, $avArray[$j]) >= 0) Then ExitLoop
                    $avArray[$j + 1] = $avArray[$j]
                Next
            EndIf
            $avArray[$j + 1] = $vTmp
        Next
        Return
    EndIf
    Local $L = $iStart, $R = $iEnd, $vPivot = $avArray[Int(($iStart + $iEnd) / 2)], $fNum = IsNumber($vPivot)
    Do
        If $fNum Then
            While ($avArray[$L] < $vPivot And IsNumber($avArray[$L])) Or (Not IsNumber($avArray[$L]) And StringCompare($avArray[$L], $vPivot) < 0)
                $L += 1
            WEnd
            While ($avArray[$R] > $vPivot And IsNumber($avArray[$R])) Or (Not IsNumber($avArray[$R]) And StringCompare($avArray[$R], $vPivot) > 0)
                $R -= 1
            WEnd
        Else
            While (StringCompare($avArray[$L], $vPivot) < 0)
                $L += 1
            WEnd
            While (StringCompare($avArray[$R], $vPivot) > 0)
                $R -= 1
            WEnd
        EndIf
        If $L <= $R Then
            $vTmp = $avArray[$L]
            $avArray[$L] = $avArray[$R]
            $avArray[$R] = $vTmp
            $L += 1
            $R -= 1
        EndIf
    Until $L > $R
    _RFLTA_QuickSort($avArray, $iStart, $R)
    _RFLTA_QuickSort($avArray, $L, $iEnd)

EndFunc   ;==>_RFLTA_QuickSort

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_ArrayConcatenate
; Description ...: Joins 2 arrays
; Syntax ........: _RFLTA_ArrayConcatenate(ByRef $avArrayTarget, Const ByRef $avArraySource)
; Parameters ....: $avArrayTarget - Base array
;                  $avArraySource - Array to add from element 1 onwards
; Return values .: None - array modified ByRef
; Author ........: Ultima
; Modified.......: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_ArrayConcatenate(ByRef $avArrayTarget, Const ByRef $avArraySource)

    Local $iUBoundTarget = UBound($avArrayTarget) - 1, $iUBoundSource = UBound($avArraySource)
    ReDim $avArrayTarget[$iUBoundTarget + $iUBoundSource]
    For $i = 1 To $iUBoundSource - 1
        $avArrayTarget[$iUBoundTarget + $i] = $avArraySource[$i]
    Next

EndFunc   ;==>_RFLTA_ArrayConcatenate

Edited September 4, 2013 by Cravin

[BrewManNH]

What you're doing in the _ConcatentateArray function will never work. All you're sending to _ArrayConcatenate is the variable name of your arrays, not the arrays themselves. Why are you using a function just to call a function? It would be far easier to just send _arrayconcatenate the arrays you want to merge rather than going to the extremes of trying to loop through an array that isn't doing what you want it to. This way, you don't need to make the arrays Global, you can send them from inside the function and keep them local. Saves on memory and time.

[Cravin]

I see what you mean, but in that respect, how would I do this in such a way that it only attempts to concatenate the file list arrays that are checked in the "Selective copy" section?

[BrewManNH]

The same way you're doing it now, just without the extraneous function that doesn't work.

You'd have to run the same check you do in the function to delete the duplicates, even though there shouldn't be any if they're in different folders. But, you'd only have to run _ArrayUnique once on the end result instead of up to 10 times to do the same thing.

[Cravin]

Okay, I'll give it a shot and see where it gets me, thanks!  Got headed down the wrong path on this one I suppose

[Cravin]

So going back through the code, I guess I'm a little unsure as to where I need to place the final _ArrayUnique portion at so that it runs after all the selected "Cases" have been executed.  What's the most logical place?

EDIT: I figured this out, but wow, using _ArrayUnique on an array with 50k+ elements is just not going to cut it performance wise....I've left this script running for literally 30 minutes so far and it's not finished yet... Is there absolutely any other way to eliminate duplicates from an array that is like, 100x faster?  Eesh.

Edited September 4, 2013 by Cravin

[czardas]

Meh! Yes you can speed this up but you will have to approach things differently. The best way is to avoid adding any duplicates in the first place. The problem here is the amount of data being processed. You need to process it in smaller chunks. I don't know exactly the best chunk size for optimal performance. By using _ArrayConcatenate you can't exclude duplicates. M23's UDF does not appear to have such an option niether, although it does have an exclude option (I doubt it will solve the speed problem, though you could try it).

Writing a special unique recursive file list function is the more complicated option and represents a challenge, plus it would require lots of testing for optimization. Anyway, here's an idea you might be able to use. It tests array entries against a string filter (which is faster than testing two arrays against each other) and finally concatenates all unique entries. This method would replace _ArrayConcatenate in your code, but first needs turning into a proper function.

;

#include <Array.au3>

Local $sStringFormat = "|" ; Start with a delimited string for checking duplicates

$sStringFormat &= "data_1|data_2|data_3|" ; Snapshot from previous searches ==> "|data_1|data_2|data_3|"
Local $aMainArray[4] = [3, "data_1", "data_2", "data_3"] ; This array is always unique
Local $iBound = 4 ; The size of the array so far

; Get data from the next search
Local $aArrayFormat[8] = [7, "data1", "data_2", "data_3", "data123", "data_3", "data_4", "data_5"] ; New Search

$iBound += $aArrayFormat[0] ; The main array needs to be at least this size to accomodate all the new items
ReDim $aMainArray[$iBound] ; Resize the main array

Local $iAdded = 0 ; Nothing added yet
For $i = 1 To $aArrayFormat[0] ; Loop through the new items
    ; Check to see if anything found so far already exists ==> Searching strings is faster
    If StringInStr($sStringFormat, "|" & $aArrayFormat[$i] & "|") Then ContinueLoop ; Do not add duplicates to the main array

    $iAdded += 1 ; Count the number of items added
    $aMainArray[$aMainArray[0] + $iAdded] = $aArrayFormat[$i] ; Add each unique new item
    $sStringFormat &= $aArrayFormat[$i] & "|" ; Add each unique new item to the delimited string for checking duplicates
Next

$iBound = $aMainArray[0] + $iAdded +1 ; The new size of the array
ReDim $aMainArray[$iBound]
$aMainArray[0] = $iBound -1 ; Update the new item count

$aArrayFormat = 0 ; Free up memory before you continue

_ArrayDisplay($aMainArray)

Edited September 4, 2013 by czardas

[Cravin]

Czardas,

Interesting thoughts, I'll play around with the concept and get back with you on some feedback.  Thanks for the idea!

[czardas]

If you have any problems, post a small reproducer which demonstrates the issues you encounter. The processing is still going to take time however you approach it. You can use the functions TimerInit() and TimerDiff() to test speed on smaller samples, so you don't have to wait so long on every test. You must free memory as you go and I also suggest you pass large arrays to functions using ByRef (unless they are global) to avoid duplicating all the data in memory. The code I posted was intended to illustrate one method which hopefully inspires you. Perhaps someone else can improve on it. Good luck!

Edited September 4, 2013 by czardas

[czardas]

I have had a bit of time to look into this. The following function still needs rigorous testing. There are parameters which allow you to optimize the function for performance. This will require a second post. In testing so far I discovered that with two arrays of 8267 elements it is approximately 27 times faster than using the Array UDF functions to do the same task. The difference may be greater with larger arrays. I tried testing two arrays of 100000 elements and _ArrayUniqueConcatenate() took 1 minute and 38 seconds. I gave up waiting for the second test after 45 minutes.

$avTarget - The array to concatenate
$avSource - The array to concatenate from
$iTargetBase - Is the Array 0-base or 1-base index? 0-base by default
$iSourceBase - Is the Array 0-base or 1-base index? 0-base by default
$iUniqueTarget - Is the target Array already unique? 0 by default
$iCaseSense - Case Sensitivity. 0 by default
$sDelimiter - Choose a different delimiter. '|' by default
$iDelimCheck - Set to 0 ONLY if you are certain the delimiter does not exist in any Array element

;

expandcollapse popup

#include <Array.au3>

Global $iBound = 8267 ; The larger this number, the more repeats will occur
Global $aOriginal[$iBound]
$aOriginal[0] = $iBound -1
For $i = 1 To $aOriginal[0]
    $aOriginal[$i] = _RandomHexStr(3) ; 16534 possible patterns
Next
ConsoleWrite("Array Created" & @LF)

Global $aArray_1 = $aOriginal, $aArray_2 = $aOriginal
_ArrayReverse($aArray_2, 1) ; Simulation of a more general case

ConsoleWrite("Starting Timer" & @LF)
Local $iTimer
$iTimer = TimerInit()
_ArrayUniqueConcatenate($aArray_1, $aArray_2, 1, 1) ; First Test
ConsoleWrite(TimerDiff($iTimer) & @LF & "Item Count = " & $aArray_1[0] & @LF)
;_ArrayDisplay($aArray_1)

$aArray_1 = $aOriginal ; To run exactly the same test using Array UDF functions

ConsoleWrite("Starting Timer" & @LF)
$iTimer = TimerInit()
_ArrayConcatenate($aArray_1, $aArray_2, 1) ; Second Test
$aArray_1 = _ArrayUnique($aArray_1, 1, 1) ; This is the slow part
ConsoleWrite(TimerDiff($iTimer) & @LF & "Item Count = " & $aArray_1[0] & @LF)
;_ArrayDisplay($aArray_1)

Func _ArrayUniqueConcatenate(ByRef $avTarget, ByRef $avSource, $iTargetBase = 0, $iSourceBase = 0, $iUniqueTarget = 0, $iCaseSense = 0, $sDelim = "|", $iDelimCheck = 1)
    If Not IsArray($avTarget) Or UBound($avTarget, 0) <> 1 Then Return SetError(1)
    If Not IsArray($avSource) Or UBound($avSource, 0) <> 1 Then Return SetError(2)

    Local $iBoundTarget = UBound($avTarget), $iBoundSource = UBound($avSource)
    If $iTargetBase Then $iTargetBase = 1
    If $iSourceBase Then $iSourceBase = 1

    If $iDelimCheck Then
        Local $s2ndDelim
        If $sDelim = ";" Then
            $s2ndDelim = "|"
        Else
            $s2ndDelim = ";"
        EndIf

        For $i = $iTargetBase To $iBoundTarget - 1
            While StringInStr($avTarget[$i], $sDelim)
                $sDelim &= $s2ndDelim
            WEnd
        Next

        For $i = $iSourceBase To $iBoundSource - 1
            While StringInStr($avSource[$i], $sDelim)
                $sDelim &= $s2ndDelim
            WEnd
        Next
    EndIf

    Local $sStringArray = $sDelim
    If $iUniqueTarget Then
        For $i = $iTargetBase To $iBoundTarget - 1
            $sStringArray &= $avTarget[$i] & $sDelim
        Next
    Else
        For $i = $iTargetBase To $iBoundTarget - 1
            If StringInStr($sStringArray, $sDelim & $avTarget[$i] & $sDelim, $iCaseSense) Then ContinueLoop
            $sStringArray &= $avTarget[$i] & $sDelim
        Next
    EndIf
    $avTarget = 0

    For $i = $iSourceBase To $iBoundSource - 1
        If StringInStr($sStringArray, $sDelim & $avSource[$i] & $sDelim, $iCaseSense) Then ContinueLoop
        $sStringArray &= $avSource[$i] & $sDelim
    Next

    Local $iLen = StringLen($sDelim)
    $sStringArray = StringTrimLeft(StringTrimRight($sStringArray, $iLen), $iLen)

    $avTarget = StringSplit($sStringArray, $sDelim, 1)

    Return $avTarget[0] +1 ; The size of the new array
EndFunc ;==> _ArrayUniqueConcatenate


Func _RandomHexStr($sLen) ; Unoptimized function
    Local $sHexString = ""
    For $i = 1 To $sLen
        $sHexString &= StringRight(Hex(Random(0, 15, 1)), 1)
    Next
    Return $sHexString
EndFunc ;==> _RandomHexStr

Edited September 5, 2013 by czardas

[czardas]

Doubling the size of the arrays in the above test gave a further increase in performance: from 27 to 45 times faster. Here's an example showing how best to use the function to optimise for speed.

;

expandcollapse popup

#include <Array.au3>

Local $aArray_1[5] = [4, "data_1", "data_2", "data_2", "data_3"]
Local $aArray_2[8] = [7, "data1", "data_2", "data_3", "data123", "data_3", "data_4", "data_5"]
Local $aArray_3[8] = [7, "data_3", "data123", "data_3", "data_6", "data_1", "data_7", "data_8"]
Local $aArray_4[6] = [5, ";|data_3", "data123;||", "da;|||ta_3", "data_9", "data_0"]

Local $aAfter[1] = [0] ; Start with a unique array to optimize speed

; We know the delim does not exist in $aArray_1, $aArray_2, $aArray_3
_ArrayUniqueConcatenate($aAfter, $aArray_1, 1, 1, 1, 0, "|", 0)
$aArray_1 = 0 ; We are finnished with this array
_ArrayUniqueConcatenate($aAfter, $aArray_2, 1, 1, 1, 0, "|", 0)
$aArray_2 = 0
_ArrayUniqueConcatenate($aAfter, $aArray_3, 1, 1, 1, 0, "|", 0)
$aArray_3 = 0

; We do not know if the delim appears in $aArray_4 so allow the check to run
_ArrayUniqueConcatenate($aAfter, $aArray_4, 1, 1, 1, 0, ";") ; Change the delimiter
$aArray_4 = 0

_ArrayDisplay($aAfter)

Func _ArrayUniqueConcatenate(ByRef $avTarget, ByRef $avSource, $iTargetBase = 0, $iSourceBase = 0, $iUniqueTarget = 0, $iCaseSense = 0, $sDelim = "|", $iDelimCheck = 1)
    If Not IsArray($avTarget) Or UBound($avTarget, 0) <> 1 Then Return SetError(1)
    If Not IsArray($avSource) Or UBound($avSource, 0) <> 1 Then Return SetError(2)

    Local $iBoundTarget = UBound($avTarget), $iBoundSource = UBound($avSource)
    If $iTargetBase Then $iTargetBase = 1
    If $iSourceBase Then $iSourceBase = 1

    If $iDelimCheck Then
        Local $s2ndDelim
        If $sDelim = ";" Then
            $s2ndDelim = "|"
        Else
            $s2ndDelim = ";"
        EndIf

        For $i = $iTargetBase To $iBoundTarget - 1
            While StringInStr($avTarget[$i], $sDelim)
                $sDelim &= $s2ndDelim
            WEnd
        Next

        For $i = $iSourceBase To $iBoundSource - 1
            While StringInStr($avSource[$i], $sDelim)
                $sDelim &= $s2ndDelim
            WEnd
        Next
    EndIf

    Local $sStringArray = $sDelim
    If $iUniqueTarget Then
        For $i = $iTargetBase To $iBoundTarget - 1
            $sStringArray &= $avTarget[$i] & $sDelim
        Next
    Else
        For $i = $iTargetBase To $iBoundTarget - 1
            If StringInStr($sStringArray, $sDelim & $avTarget[$i] & $sDelim, $iCaseSense) Then ContinueLoop
            $sStringArray &= $avTarget[$i] & $sDelim
        Next
    EndIf
    $avTarget = 0

    For $i = $iSourceBase To $iBoundSource - 1
        If StringInStr($sStringArray, $sDelim & $avSource[$i] & $sDelim, $iCaseSense) Then ContinueLoop
        $sStringArray &= $avSource[$i] & $sDelim
    Next

    Local $iLen = StringLen($sDelim)
    $sStringArray = StringTrimLeft(StringTrimRight($sStringArray, $iLen), $iLen)

    $avTarget = StringSplit($sStringArray, $sDelim, 1)

    Return $avTarget[0] +1 ; The size of the new array
EndFunc ;==> _ArrayUniqueConcatenate

[Cravin]

Czardas,

Thank you greatly for your insightfulness.  I'll run some of my arrays through it and see what kind of difference it makes, looking much more hopeful!

[kylomas]

Cravin,

Another alternative.  Comments and run times are in the code...

expandcollapse popup

#include <array.au3>
#include "RecFileListToArray.au3"

global $DataTabTargetNameInput = 'C:'
global $DataTabUserNameInput = 'ADMIN010'

global $tDups = 0, $aFinal[1], $tUniqueFiles = 0, $ttFiles = 0

; The following file lists were setup to mimic your code using my PC
;
;   The Results for processing these arrays was
;
;       Total Files = 04880   Total Unique Files = 01659   Total Dups = 03221
;       Total time to process 004880 files = 0.1598 seconds
;
;   This did not adequately stress the code so I also run the 4 directories at the bottom.

$a02 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\Desktop", "*|desktop.ini", 1, 1, 0, 2)
$a04 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Roaming\Microsoft\Windows\Templates", "*|desktop.ini", 1, 1, 0, 2)
$a06 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Local\Microsoft\Outlook", "*|desktop.ini;*.obi;*.ost;*.kfl;*.pst", 1, 1, 0, 2)
$a08 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput, "*|ntuser.dat;desktop.ini;*.dat.log1;*.dat.log2;*.blf;*.regtrans-ms;ntuser.ini;ntuser.pol|.thumbnails;AppData;Application Data;Cookies;Local Settings;NetHood;PrintHood;Searches", 1, 1, 0, 2)
$a10 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Roaming\Microsoft\Systemcertificates", "*|desktop.ini", 1, 1, 0, 2)
$a12 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Roaming\Microsoft\signatures", "*|desktop.ini", 1, 1, 0, 2)
$a14 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\Favorites", "*|desktop.ini", 1, 1, 0, 2)
$a16 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Local\Microsoft\Outlook", "*.pst|desktop.ini", 1, 1, 0, 2)
$a17 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\Documents\Outlook Files", "*.pst|desktop.ini", 1, 1, 0, 2)
$a20 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Roaming\Microsoft\Windows\SendTo", "*|desktop.ini", 1, 1, 0, 2)

$a38 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput, "*|ntuser.dat;desktop.ini;*.dat.log1;*.dat.log2;*.blf;*.regtrans-ms;ntuser.ini;ntuser.pol|.thumbnails;AppData;Application Data;Cookies;Local Settings;NetHood;PrintHood;Searches", 1, 1, 0, 2)
$a31 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Roaming\Microsoft\Systemcertificates", "*|desktop.ini", 1, 1, 0, 2)
$a32 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Roaming\Microsoft\signatures", "*|desktop.ini", 1, 1, 0, 2)
$a34 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\Favorites", "*|desktop.ini", 1, 1, 0, 2)
$a36 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Local\Microsoft\Outlook", "*.pst|desktop.ini", 1, 1, 0, 2)
$a37 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\Documents\Outlook Files", "*.pst|desktop.ini", 1, 1, 0, 2)
$a30 = _RecFileListToArray("" & $DataTabTargetNameInput & "" & "\Users\" & $DataTabUserNameInput & "\AppData\Roaming\Microsoft\Windows\SendTo", "*|desktop.ini", 1, 1, 0, 2)
;
;   The Results for processing the following 4 directories
;
;       Total Files = 60444   Total Unique Files = 45910   Total Dups = 14534
;       Total time to process 060444 files = 1.7127 seconds

;~ $a40 = _RecFileListToArray("k:\sd\sd0100\mlb\boxes","*",1, 1, 0, 2)
;~ $a41 = _RecFileListToArray("k:\sd\sd0100\ncb\boxes","*",1, 1, 0, 2)
;~ $a42 = _RecFileListToArray("k:\sd\sd0100\nba\boxes","*",1, 1, 0, 2)
;~ $a43 = _RecFileListToArray("k:\sd\sd0100\mlb\boxes","*",1, 1, 0, 2)

local $st = timerinit(), $sta = timerinit()

for $1 = 0 to 50
    if isarray(eval( 'a' & stringformat('%02s',$1) ) ) then _process_array( eval( 'a' & stringformat('%02s',$1) ))
next

ConsoleWrite(stringformat('Total Files = %05i   Total Unique Files = %05i   Total Dups = %05i',$ttFiles, $tUniqueFiles, $tDups) & @LF)

redim $aFinal[$tUniqueFiles]

ConsoleWrite(stringformat('Total time to process %06i files = %2.4f seconds',$tDups+$tUniqueFiles,timerdiff($st)/1000) & @LF)

_arraydisplay($aFinal)

func _process_array($array)

    local static $aFinal_idx = 0
    local $tname, $tFiles = $array[0]
    $ttFiles += $array[0]

    redim $aFinal[ ubound($aFinal) + $tFiles ]

    for $1 = 1 to ubound($array) - 1
        $tname = stringregexpreplace($array[$1],'[\.\:\\ ]','_')
        if isdeclared( eval('s' & $tname) ) = 0 then
            assign('s' & $tname,1,2)
            $aFinal[$aFinal_idx] = $array[$1]
            $aFinal_idx += 1
            $tUniqueFiles += 1
        Else
            $tDups += 1
        endif
    next

endfunc

kylomas

Edited September 6, 2013 by kylomas

[czardas]

kylomas - very interesting: those are pretty impressive results. I was thinking that my method could be modified to accomodate multiple arrays and only do one StringSplit at the end. This would improve speed, but it still wouldn't match the speed of your function because StringSplit can be slow on large strings.

Edited September 6, 2013 by czardas

[kylomas]

@czardas - Do you have some large dir's you can pump through this?  Use any name for the array from $a00-$a50.

[czardas]

I just tested it on some large arrays and it's amazing.

I spent most of yesterday on my example. Oh well!

Edited September 6, 2013 by czardas

[kylomas]

Thanks...between your technique and this Cravin should have enough to solve his problem.

[czardas]

If I'm not mistaken, it appears only to work with file names. I don't think it's an issue for Cravin, but looking at the regexp, will it not fail with the following unlikely entries?

folderfile
folder.file

folder_file

Because this line:

$tname = stringregexpreplace($array[$1],'[\.\:\\ ]','_')

;

Will produce folder_file in both all cases. Not being picky or anything. The method is pretty cool anyway.

Also what happens if one of the resulting names matches an existing variable declared in another part of the code?

There should be a solution for this.

Edits ==> in grey text

Edited September 6, 2013 by czardas

[kylomas]

Yes, it was designed for filenames.  The replacement is done because ".:" are invalid in variable names.  I'm sure that there is some DSN pattern that will produce an invalid duplicate but I can't see it right now (1:30AM).  This was more proof of concept than anything else.  Too be useful to Cravin it needs to be turned into a clean procedure (like the code you presented).  Cravin would know best how to fit it to his app.

Good Night,

kylomas

edit: 

Not being picky or anything.

 

That's not picky, it's observant....thanks

Edited September 6, 2013 by kylomas

[czardas]

Okay have a good sleep. Thanks for the great work.