[solved] copy file if found in array

[Tosyk]

I'm making a script to search items from one array (small one) inside another (big one).

item in small array - partial path to file (attached civ_female_act3.model_materials.txt)
item in big array - full path to file (attached TextureList.lst)

if item from small array is found in big one then file need to be copied from the full path into another folder. I'm searching for $find atm. Script:

#include <Array.au3>
#include <File.au3>

Global $aArrayInput, $aArrayOutput, $matArray
Global $sDir = @ScriptDir
Global $sFileName = '*'
Global $sFileExtension = 'texture.dds' ; or '*' for all files or 'pdf' for just PDFs
Global $sListPath = @ScriptDir & '\TextureList.lst'
Global $find = 'textures\cperrella\fabric\cloth_knits\wrinkles_01_n.texture' & '.dds'
Global $matInput = @ScriptDir & '\civ_female_act3.model_materials.txt'
Global $sExclude

$aArrayOutput = _FileListToArrayRec( $sDir, $sFileName & '.' & $sFileExtension, 1, 1, 0, 2 )
If IsArray($aArrayOutput) Then
   If FileExists($sListPath) Then
      _FileSearchInTXT()
   Else
      _FileWriteFromArray($sListPath, $aArrayOutput)
      _FileSearchInTXT()
;~       _ArrayDisplay( $aArrayOutput )
   EndIf
EndIf

_FileReadToArray($matInput, $matArray, $FRTA_COUNT)
_ArrayDelete($matArray, $matArray[1])

$sExclude = "===*|---*|shader*|material*| *"

$sExclude = StringReplace($sExclude, ".", "\.")
$sExclude = StringReplace($sExclude, "?", ".")
$sExclude = StringReplace($sExclude, "*", ".*?")

$iIndex = 0
For $i = 0 To UBound($matArray) - 1
    If NOT StringRegExp($matArray[$i], $sExclude) Then
        $matArray[$iIndex] = $matArray[$i]
        $iIndex += 1
    EndIf
Next
Redim $matArray[$iIndex]


For $i = UBound($matArray) - 1 To 0 Step -1
    If $matArray[$i] = "" Then
        _ArrayDelete($matArray, $i)
    EndIf
Next

_ArrayTrim($matArray, 17, 0)
_ArrayDisplay($matArray, "2D array - no count (trimmed)", Default, 8)


;~ ----------------------------------------------------------------------------------------------
Func _FileSearchInTXT()
If $sListPath <> '' And $find <> '' Then
    _FileReadToArray($sListPath, $aArrayInput)
    For $i = 1 To UBound($aArrayInput) - 1
        If StringInStr($aArrayInput[$i], $find) Then
           ConsoleWrite("Found file: " & $aArrayInput[$i] & @CRLF)
        EndIf
     Next
Else
    MsgBox(48, 'Error', 'A file was not picked or what to find was cancelled/empty!')
EndIf
EndFunc    ;==>_FileSearchInTXT

please help me to do this.

 

TextureList.zip

Edited December 6, 2020 by Tosyk

[water]

And your question is?

[Tosyk]

41 minutes ago, Tosyk said:

if item from small array is found in big one then file need to be copied from the full path into another folder

this is the idea, sorry for not been clear  right now it's just display the small array and saves big array as file.

here's where I get small array:

_ArrayDisplay($matArray, "2D array - no count (trimmed)", Default, 8)

after this I need to check all the items from small array ($matArray) inside big array ($aArrayOutput) and if string is found copy the file into another folder. I don't understand how to do it

[water]

We need more information:

• About how many records in the small and large array do we talk?
• Is one/both of the arrays sorted? If yes, we could do a binary search to enhance processing speed

[Sidley]

Basic comparison logic would be as follows (untested), but as @water said, there are better and more efficient ways of doing it.

For $i = 0 To UBound($smallArray - 1)
    For $j = 0 To UBound($bigArray - 1)
        If $smallArray[$i] = $bigArray[$j] Then
            FileCopy("Path\To\Source\" & $smallArray[$i], "Path\To\Destination\Filename.file", $FC_NOOVERWRITE)
        Else
            ConsoleWrite("No match found for " & $smallArray[$i] & @CRLF)
        EndIf
    Next
Next

 

[Tosyk]

small array is up to 1000 items, big one can be around 50000 items

also, I'm sure there's a cool and speed solution. is just I don't really a programmer person so I'm trying to do a clear code

Edited December 4, 2020 by Tosyk

[water]

If speed doesn't matter, then I would start with Sidleys solution.
You need to decide what to do when the copied file already exists in the target directory. Currently an existing file doesn't get overwritten.

[Tosyk]

I change the solution like this:

For $i = 0 To UBound($matArray - 1)
    For $j = 0 To UBound($aArrayOutput - 1)
        If $matArray[$i] = $aArrayOutput[$j] Then
            FileCopy($aArrayOutput[$j], @ScriptDir, $FC_NOOVERWRITE)
        Else
            ConsoleWrite("No match found for " & $matArray[$i] & @CRLF)
        EndIf
    Next
Next

but it returned:

...
No match found for 
No match found for 
No match found for 
"d:\MyTools\tex_collector.au3" (54) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
If $matArray[$i] = $aArrayOutput[$j] Then
If $matArray[$i] = ^ ERROR
>Exit code: 1    Time: 5.851

 

Edited December 4, 2020 by Tosyk

[water]

Can you please post an example of the small and large array (if you like as files) so we can play with your script?

[Tosyk]

already did in the first post:

item in small array - attached civ_female_act3.model_materials.txt
item in big array - attached TextureList.lst

[water]

What is this part of your scipt meant for?
Do you want to exclude all lines containing the specified pattern $sExclude?

$sExclude; Lines with certain strings should be excluded
$sExclude = "===*|---*|shader*|material*| *"
$sExclude = StringReplace($sExclude, ".", "\.")
$sExclude = StringReplace($sExclude, "?", ".")
$sExclude = StringReplace($sExclude, "*", ".*?")

; Remove all lines 
$iIndex = 1
For $i = 0 To UBound($matArray) - 1
    If Not StringRegExp($matArray[$i], $sExclude) Then
        $matArray[$iIndex] = $matArray[$i]
        $iIndex += 1
    EndIf
Next
ReDim $matArray[$iIndex]

 

[Tosyk]

@water

as input (and small array) I have $matArray, here's list of actions by which I clear this array of redundant lines:

;~ ----------------------------------------------------------------------------------------------
;~ CLEARING matArray START
;~ ----------------------------------------------------------------------------------------------
$sExclude = "===*|---*|shader*|material*| *"

$sExclude = StringReplace($sExclude, ".", "\.")
$sExclude = StringReplace($sExclude, "?", ".")
$sExclude = StringReplace($sExclude, "*", ".*?")

$iIndex = 0
For $i = 0 To UBound($matArray) - 1
    If NOT StringRegExp($matArray[$i], $sExclude) Then
        $matArray[$iIndex] = $matArray[$i]
        $iIndex += 1
    EndIf
Next
Redim $matArray[$iIndex]


For $i = UBound($matArray) - 1 To 0 Step -1
    If $matArray[$i] = "" Then
        _ArrayDelete($matArray, $i)
    EndIf
Next

_ArrayTrim($matArray, 17, 0)
;~ ----------------------------------------------------------------------------------------------
;~ CLEARING matArray END
;~ ----------------------------------------------------------------------------------------------

 

[Tosyk]

37 minutes ago, water said:

Do you want to exclude all lines containing the specified pattern $sExclude?

yes! also exclude empty lines

[Sidley]

Those UBounds should read UBound($Variable) - 1, and not UBound($Variable-1), sorry about that. Don't know if it'll make a difference.

[water]

You want to exclude just lines that START with the pattern or even lines that CONTAIN the patterns?
Right now all lines that CONTAIN "shader" (like "A87F9241FDCE1112textures\default_pink_shader\default_pink_shader_n.texture") get ignored.

[Tosyk]

14 minutes ago, water said:

You want to exclude just lines that START with the pattern or even lines that CONTAIN the patterns?

this 

14 minutes ago, water said:

Right now all lines that CONTAIN "shader" (like "A87F9241FDCE1112textures\default_pink_shader\default_pink_shader_n.texture") get ignored.

oh, I didn't realize this :/ thanks for point that out

[water]

As far as I can test this code does what you want it to do.

• I have modified the script so that all arrays hold the number of rows in the first row.
• I have modified the PCRE so the search pattern has to be found at column 1 of each record
• I have added some debugging code. Can easily be removed

expandcollapse popup

#include <Array.au3>
#include <File.au3>

; Documentation:
;   All arrays store the row/column count in row 0

Global $aArrayInput, $aArrayOutput, $aMat
Global $sDir = @ScriptDir
Global $sFileName = '*'
Global $sFileExtension = 'texture.dds' ; or '*' for all files or 'pdf' for just PDFs
Global $sListPath = @ScriptDir & '\TextureList.lst'
Global $sFind = 'textures\cperrella\fabric\cloth_knits\wrinkles_01_n.texture' & '.dds'
Global $sMatInput = @ScriptDir & '\civ_female_act3.model_materials.txt'

$aArrayOutput = _FileListToArrayRec($sDir, $sFileName & '.' & $sFileExtension, 1, 1, 0, 2)
; if @error then Exit ConsoleWrite("_FileListToArrayRec: @error=" & @error & ", IsArray($aArrayOutput): " & IsArray($aArrayOutput) & @CRLF)
If IsArray($aArrayOutput) Then
    If Not FileExists($sListPath) Then _FileWriteFromArray($sListPath, $aArrayOutput, 1) ; Do not write the record count to the file
    _FileSearchInTXT()
Else
    Global $aArrayOutput[1] = [0] ; Modified: $aArrayOutput needs to be defined as an array as _FileListToArray might not return/create the array
EndIf

_FileReadToArray($sMatInput, $aMat, $FRTA_COUNT)
_ArrayDisplay($aMat, "before Moving rows") ; ==> Just for debugging can be removed

; Lines with certain strings should be excluded
Global $sExclude = "(?i)^===*|(?i)^---*|(?i)^shader*|(?i)^material*|(?i)^template*|^\s|^ *"
$sExclude = StringReplace($sExclude, ".", "\.")
; $sExclude = StringReplace($sExclude, "?", ".") ; REMOVED as it would kill (?i) in the pattern
$sExclude = StringReplace($sExclude, "*", ".*?")

; Remove all lines holding on of the $sExclude patterns or empty lines
Global $iOutIndex = 1
For $iInIndex = 1 To $aMat[0]
    If Not StringRegExp($aMat[$iInIndex], $sExclude) And $aMat[$iInIndex] <> "" Then
        ConsoleWrite("Moved  : " & $iInIndex & " | " & $iOutIndex & " | " & $aMat[$iInIndex] & @CRLF)  ; ==> Just for debugging can be removed
        $aMat[$iOutIndex] = $aMat[$iInIndex]
        $iOutIndex += 1
    Else
        ConsoleWrite("Ignored: " & $iInIndex & " | " & $iOutIndex & " | " & $aMat[$iInIndex] & @CRLF) ; ==> Just for debugging can be removed
    EndIf
Next
_ArrayDisplay($aMat, "After moving rows") ; ==> Just for debugging can be removed
ReDim $aMat[$iOutIndex]
$aMat[0] = $iOutIndex - 1
_ArrayDisplay($aMat, "After ReDim") ; ==> Just for debugging can be removed

For $i = 1 To $aMat[0]
    For $j = 1 To $aArrayOutput[0]
        If $aMat[$i] = $aArrayOutput[$j] Then
            ConsoleWrite("Copying file " & $aArrayOutput[$j] & " to " & @ScriptDir & @CRLF) ; ==> Just for debugging can be removed
            FileCopy($aArrayOutput[$j], @ScriptDir, $FC_NOOVERWRITE)
        Else
            ConsoleWrite("No match found for " & $aMat[$i] & @CRLF)
        EndIf
    Next
Next

_ArrayTrim($aMat, 17, 0, 1)
_ArrayDisplay($aMat, "Array (trimmed)", Default, 8)
Exit

;~ ----------------------------------------------------------------------------------------------
Func _FileSearchInTXT()
    If $sListPath <> '' And $sFind <> '' Then
        _FileReadToArray($sListPath, $aArrayInput, $FRTA_COUNT)
        For $i = 1 To $aArrayInput[0]
            If StringInStr($aArrayInput[$i], $sFind) Then
                ConsoleWrite("Found file: " & $aArrayInput[$i] & @CRLF)
            EndIf
        Next
    Else
        MsgBox(48, 'Error', 'A file was not picked or what to find was cancelled/empty!')
    EndIf
EndFunc   ;==>_FileSearchInTXT

 

[Tosyk]

thanks! I'll try it right now. can you explain what's going on here(?):

ReDim $aMat[$iOutIndex]
$aMat[0] = $iOutIndex - 1
_ArrayDisplay($aMat, "After ReDim") ; ==> Just for debugging can be removed

after this operation I get array almost 2 times shorter

edit:

also, in a small array I have strings like:

characters/civilians/clothing/har_female_knittedcap/textures/har_female_knittedcap_n.texture

but in big array this should be:

D:\_r\to_con\_gm\characters\civilians\clothing\har_female_knittedcap\textures\har_female_knittedcap_n.texture.dds

and script couldn't find the string because of reversed "\". how it can be reversed back or ignored or something else so the string could be found in big array?

Edited December 4, 2020 by Tosyk

[water]

The code before ReDim removed all empty lines and lines matching the exclude pattern.
Let's say the array consists of 11 rows (row 0 holds the row count + 10 rows holding your data). 3 rows were removed (empty or matching the exclude pattern). Then the code is reduced to 8 rows (row 0 holds the row count + 7 rows holding your data). The I set the row count in row 0 to 7.

[Tosyk]

1 hour ago, water said:

Then the code is reduced to 8 rows

why the code doing this after the array is cleared pretty well?

also, script couldn't find the string because of reversed "\". how it can be reversed back or ignored or something else so the string could be found in big array?