[solved] copy file if found in array

[Tosyk]

also, after line 1024 the clearing part of the code seems do nothing:

 

[water]

I have taken the code to ReDim the array from your original script:

Redim $matArray[$iIndex]

I tested the code from my last post and it works as expected.

Edit:
I have tested with the files you provided and my code works perfect.
If you find errors please provide the data you are testing with so we can reproduce the problem.

Edit2:
To replace \ with / or vice versa should be easy with function StringReplace.

Edited December 4, 2020 by water

[Tosyk]

30 minutes ago, water said:

To replace \ with / or vice versa should be easy with function StringReplace.

Yeah, with this I was able to do the trick:

      For $i = 0 To Ubound($aMatArray) - 1
         $aMatArray[$i] = StringReplace($aMatArray[$i], "/", "\")
      Next

 

30 minutes ago, water said:

I have taken the code to ReDim the array from your original script

it seems like I copy this from another script and left it here by accident. I don't need to resize my array by removing elements, silly me.

although I still have issue with the code not removing all redundant rows after 1024 Row

Edited December 4, 2020 by Tosyk

[GokAy]

You said earlier 

Quote

small array is up to 1000 items, big one can be around 50000 items

Maybe you are using small array ubound for that part and it stops afterwards?

[Tosyk]

15 minutes ago, GokAy said:

Maybe you are using small array ubound for that part and it stops afterwards?

thanks for you reply. I'm not sure what to answer. How to define ubound for small array or for big array?

This is my code where I think the limitation is present:

;     Exlude mask
        $sExclude = "(?i)^===*|(?i)^---*|(?i)^shader*|(?i)^material*|(?i)^template*|^\s|^ *"
        $sExclude = StringReplace($sExclude, ".", "\.")
        $sExclude = StringReplace($sExclude, "*", ".*?")

;      Remove redundant lines by mask
        For $i = 1 To $aMatArray[0]
            If NOT StringRegExp($aMatArray[$i], $sExclude) And $aMatArray[$i] <> "" Then
                $aMatArray[$iIndex] = $aMatArray[$i]
                $iIndex += 1
            Else
                ConsoleWrite("Ignored lines: " & $i & " | " & $iIndex & " | " & $aMatArray[$i] & @CRLF)
                EndIf
        Next

 

[GokAy]

Is MatArray the large one?

And I wonder if $aMatArray[0] is correct, can you add this just before

; Remove redundant lines by mask

ConsoleWrite("MatArray[0]: " & $aMatArray[0] & " > Ubound: " & Ubound($aMatArray)-1 & @CRLF)

See if they are equal

[Tosyk]

1 hour ago, GokAy said:

Is MatArray the large one?

this is small one. I'm changing it by removing redundant parts from the Rows.

also I get this with you console output code:

MatArray[0]: 2818 > Ubound: 2818

 

Edited December 5, 2020 by Tosyk

[GokAy]

Hmm, when you process this part

$aMatArray[$iIndex] = $aMatArray[$i]
$iIndex += 1

You are not doing anything to the $aMatArray[$i], so it is left as it is.

If you are not redimming to crop the remaining (rows $iIndex +), then they are left there.

Perhaps you should assign empty string to those rows?

$aMatArray[$iIndex] = $aMatArray[$i]
$aMatArray[$i] = "" ; add this?
$iIndex += 1

 

[water]

Leave the ReDim in your script as it works perfectly - I already tested this.

Provide the files you are testing with (50000 and 1000 records) so we can play with them.

[Tosyk]

I don't understand what Redim do, reading the help doesn't help. Resizing? why do I need to resize my array tho? I have my Rows, all of them I need and the duplicates I remove with ArrayUnique.

I attached file on which I have issue

civ_female_act3_materials.txt

[water]

Example:
Array before removing unwanted lines (shader etc.):

5
1111
Shader
2222
3333
4444

After removing unwanted lines. Note that unwanted lines do not get deleted but overwitten with a copy of the next valid line. This means that you havwe to remove as many lines from the end of the array as you have found unwanted lines in the array.

5
1111
2222
3333
4444
4444

After Redim and setting the counter in row 0:

4
1111
2222
3333
4444

 

[water]

The PCRE pattern needs to be modified if you want to keep the following lines:

material\character\npc\test\test_npc_basic_skin_05_newsss_02\civ_female_head_lat_02.material    
material\characters\civilian\hair\civ_har_hair_straight_brown.material  
material\characters\civilian\t4\t5_head_teeth.material  
material\characters\civilian\t5\civ_t5_female_head_afr_02.material  
material\characters\civilian\t5\civ_t5_female_head_asn_01.material  
material\characters\civilian\t5\civ_t5_female_head_asn_02.material  
material\characters\civilian\t5\civ_t5_female_head_cau_01.material  
material\characters\civilian\t5\civ_t5_female_head_lat_01.material  
materialgraph\characters\shared\char_eye_darkbrown.material 
materials\characters\character_default_skin.material    
materials\hrm_bdya_raj_clothes_sneakers\har_female_pompom_01_common.material

Do you want to keep them?

Everything else works well.

[Tosyk]

45 minutes ago, water said:

After Redim and setting the counter in row 0:

so ReDim removes duplicates?

26 minutes ago, water said:

Do you want to keep them?

actually yes, is it possible to consider not only a few characters from begging of the line but also few characters at the end?

[water]

2 minutes ago, Tosyk said:

so ReDim removes duplicates?

Kind of. The algorithm used to remove lines matching the pattern creates supernumerous lines at the end of the array. By using ReDim you drop this lines.

 

4 minutes ago, Tosyk said:

actually yes, is it possible to consider not only a few characters from begging of the line but also few characters at the end?

Yes.

[Tosyk]

1 hour ago, water said:

Yes.

I can do it like this?

$sExclude = "(?i)^===*===|(?i)^---*---|(?i)^shader*.shader|(?i)^material*.material|(?i)^template*.template|^\s|^ *"

 

[water]

I tested with the following pattern and it worls great.

Global $sExclude = "(?i)^===*|(?i)^---*|(?i)^shader *|(?i)^material *|(?i)^template: *|(?i)^template *|^ *"

 

[GokAy]

2 hours ago, Tosyk said:

so ReDim removes duplicates?

To eliminate any misunderstanding, ReDim doesn't remove duplicates, but re-dimensions the array. In your case, and how Water wrote the algorithm it does, in a way, function like that. 

Edited December 5, 2020 by GokAy

[Tosyk]

I got it finished. Thanks guys for the help — I finally re-done it without regular expressions. It is slower but at least I understand almost all the lines. Here's it is:

expandcollapse popup

;~ spiderman 2018 ps4 texture collector
;~ Tosyk
;~ 5 dec 2020

#include <Array.au3>
#include <File.au3>
#Include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>

If Not $CmdLine[0] Then
    MsgBox(0, "Usage", "Drop file(s) on " & @ScriptName)
    ConsoleWrite("Usage: " & @ScriptName & " <file>" & @CRLF)
    Exit
 EndIf

AutoItSetOption("MustDeclareVars", 1)
HotKeySet("{ESC}", "_Terminate")

Global $sIniFile = @ScriptDir & '\' & 'settings.ini'
Global $sDirInput = IniRead ($sIniFile, 'Setting', 'Folder', @ScriptDir)
Global $sTexFolder = IniRead ($sIniFile, 'Setting', 'TexFolder', @ScriptDir)

Local $i ; input file
Local $sSourceFile, $sDestFile, $sDestPath, $sExePath, $hGUI, $ProgressBar1, $Label, $Progress, $Progress1
Local $sDrive, $sFolder, $sFileName, $sExt, $matInput
Local $aArrayInput, $aArrayOutput, $aMatArray, $aMatArrayUnique
Local $sDirOutput = @WorkingDir & $sTexFolder
Local $sFileNameTex = '*', $sFileExtension = 'texture.dds'
Local $sListPath = @ScriptDir & '\TextureList.lst'
Local $sExclude, $iIndex = 1, $sSearch, $sTexFile

If $CmdLine[0] <> 0 Then
    _GUIFunc()
    _MainLoop()
EndIf

Func _GUIFunc()
    #Region ### START Koda GUI section ### Form=
    $hGUI = GUICreate("Collecting textures", 300, 108, -1, -1, BitXOR($GUI_SS_DEFAULT_GUI, $WS_MINIMIZEBOX))
    $Label = GUICtrlCreateLabel("", 20, 25, 260, 40, $SS_LEFT)
    $Progress1 = _CreateProgress(20, 57, 260, 20)
    GUISetState(@SW_SHOW)
    #EndRegion ### END Koda GUI section ###
EndFunc   ;==>_GUIFunc

Func _MainLoop()
    For $i = 1 To $CmdLine[0]
        $sSourceFile = $CmdLine[$i]
        _PathSplit($sSourceFile, $sDrive, $sFolder, $sFileName, $sExt)
        GUICtrlSetData($ProgressBar1, ($i / $CmdLine[0]) * 100)

        _UpdateProgress($Progress1, ($i / $CmdLine[0]) * 100)
        GUICtrlSetData($Label, "Processing file: " & $sFileName & $sExt)
        Sleep(650)

        If FileExists($sSourceFile) Then
            If $sExt = '.txt' Then
            _FileProcessor()
            GUICtrlSetData($Label, "Now reducing '.texture'!")
            Sleep(200)
            ; Renaming result file extension from '.texture.dds' to '.dds' with function
            _RenameResultFiles()
            Else
                MsgBox(0, 'Wrong extension', 'Wrong extension detected.' & @CRLF & 'file: ' & $sFileName & $sExt & @CRLF & 'Supported extensions: TXT only.')
            EndIf
        Else
            MsgBox(0, '', "File [" & $sSourceFile & "] not found")
        EndIf
    Next

    GUICtrlSetData($Label, "Collecting complete!")
    Sleep(600)
 EndFunc   ;==>_MainLoop

Func _FileProcessor()

    $matInput = $sSourceFile
    ConsoleWrite("Input material file: " & $matInput & @CRLF)

    If FileExists($sListPath) Then
        _Settings()
    Else
        ;~----------------------------------------------------------------------------------------------
        ;~ If TextureList.lst doesn't exist creat it
        ;~----------------------------------------------------------------------------------------------
        $aArrayOutput = _FileListToArrayRec($sDirInput, $sFileNameTex & '.' & $sFileExtension, 1, 1, 0, 2)
        _FileWriteFromArray($sListPath, $aArrayOutput)
        ConsoleWrite('TextureList.lst created' & @LF)
        _Settings()
    EndIf
EndFunc   ;==>_FileProcessor

Func _Settings()
    _FileReadToArray($sListPath, $aArrayOutput)

    If IsArray($aArrayOutput) Then

        _FileReadToArray($matInput, $aMatArray)
;~      _ArrayDisplay($aMatArray, "Initial")

        ;~----------------------------------------------------------------------------------------------
        ;~ CLEARING matArray START
        ;~----------------------------------------------------------------------------------------------

        ; Exlude mask
        $sExclude = "(?i)^===*|(?i)^---*|(?i)^shader*|(?i)^material*|(?i)^template*|^\s|^ *"
        $sExclude = StringReplace($sExclude, ".", "\.")
        $sExclude = StringReplace($sExclude, "*", ".*?")

        ; Remove redundant lines by mask
        For $i = 1 To $aMatArray[0]
            If NOT StringRegExp($aMatArray[$i], $sExclude) And $aMatArray[$i] <> "" Then
                $aMatArray[$iIndex] = $aMatArray[$i]
                $aMatArray[$i] = "" ; add this?
                $iIndex += 1
            Else
                ConsoleWrite("Ignored lines: " & $i & " | " & $iIndex & " | " & $aMatArray[$i] & @CRLF)
                EndIf
        Next

;~      _ArrayDisplay($aMatArray, "After moving rows") ; ==> Just for debugging can be removed

      ReDim $aMatArray[$iIndex]
      $aMatArray[0] = $iIndex - 1
;~    _ArrayDisplay($aMatArray, "After ReDim") ; ==> Just for debugging can be removed

        ; Remove first 17 characters of hash
        _ArrayTrim($aMatArray, 17, 0)
;~      _ArrayDisplay($aMatArray, "After Remove Hashes")

;~    Replace '/' with '\'
        For $i = 0 To Ubound($aMatArray) - 1
         $aMatArray[$i] = StringReplace($aMatArray[$i], "/", "\")
        Next

;~    Remove duplicates
        $aMatArrayUnique = _ArrayUnique($aMatArray)

        ;~----------------------------------------------------------------------------------------------
        ;~ CLEARING matArray END
        ;~----------------------------------------------------------------------------------------------

;~    [DEBUG ONLY] Show 2 arrays before copy files
;~      _ArrayDisplay($aMatArrayUnique, "Material with unique lines only", Default, 8)          ; preview small array, material
;~    _ArrayDisplay($aArrayOutput, "Texture List", Default, 8)      ; preview big array, texture list

;     Copy files to Material folder

        For $iA = 0 To UBound($aMatArrayUnique) -1
         For $iB = 0 To UBound($aArrayOutput) -1
            If StringInStr($aArrayOutput[$iB], $aMatArrayUnique[$iA]) Then
                ;MsgBox(0, '', $sDirOutput)
                FileCopy($aArrayOutput[$iB], $sDirOutput, $FC_OVERWRITE + $FC_CREATEPATH)
                ; ConsoleWrite($aMatArrayUnique[$iA] & ' = ' & $aArrayOutput[$iB] & @LF)
                ConsoleWrite('String: ' & $aMatArrayUnique[$iA] & ' = ' & $aArrayOutput[$iB] & @LF)
                ExitLoop ;прервать поиск для текущего файла из маленького списка и начать поиск другого
            EndIf
         Next
        Next

    EndIf
EndFunc   ;==>_Settings

;---------------------------------------------------------------
;- Enhanced Progress widget
;---------------------------------------------------------------
Func _CreateProgress($x, $y, $w, $h, $Label="")
    Dim $Progress[2]
    $Progress[0] = GuiCtrlCreateProgress($x, $y, $w, $h) ; this is progress bar
    $Progress[1] = GuiCtrlCreateLabel($Label, $x+0, $y+21, '', '', $SS_RIGHT) ; this is percentage label
    GUICtrlSetFont($Progress[1], 11, 500, 0, "")
    GUICtrlSetBkColor($Progress[1], $GUI_BKCOLOR_TRANSPARENT)
    Return $Progress
EndFunc   ;==>_UpdateProgress

Func _UpdateProgress($ProgressID, $Percent, $Label="")
    GUICtrlSetData($ProgressID[0], $Percent)
    GUICtrlSetData($ProgressID[1], $Label & Round($Percent, 1) & "%")
EndFunc   ;==>_UpdateProgress

Func _RenameResultFiles()
    $sSearch = _FileListToArrayRec($sDirOutput, $sFileNameTex & '.' & $sFileExtension, 1, 1, 0, 2)
;~  _ArrayDisplay($sSearch, "Files to rename")

;~Rename files
    For $i = 1 to UBound($sSearch) - 1
        $sTexFile = StringTrimRight($sSearch[$i], 12)
        $sTexFile = $sTexFile & ".dds"
        FileMove($sSearch[$i], $sTexFile, 1)
        ConsoleWrite("Title: " & $sSearch[$i] & " | " & $sTexFile & @LF)
    Next

    FileClose($sSearch)
EndFunc  ;==>_RenameResultFiles

Func _Terminate()
    If WinGetHandle('') <> $hGUI Then
        HotKeySet('{ESC}')
        Send('{ESC}')
        HotKeySet('{ESC}', '_Terminate')
        Return
    EndIf
;   UpdateINI()
    GUIDelete()
    Exit
EndFunc   ;==>_Terminate

Exit (0)

the most hard part of it and reason why I created this topic is this function:

expandcollapse popup

Func _Settings()
    _FileReadToArray($sListPath, $aArrayOutput)
    If IsArray($aArrayOutput) Then
        _FileReadToArray($matInput, $aMatArray)

        ;~----------------------------------------------------------------------------------------------
        ;~ CLEARING matArray START
        ;~----------------------------------------------------------------------------------------------

        ; Exlude mask
        $sExclude = "(?i)^===*|(?i)^---*|(?i)^shader*|(?i)^material*|(?i)^template*|^\s|^ *"
        $sExclude = StringReplace($sExclude, ".", "\.")
        $sExclude = StringReplace($sExclude, "*", ".*?")

        ; Remove redundant lines by mask
        For $i = 1 To $aMatArray[0]
            If NOT StringRegExp($aMatArray[$i], $sExclude) And $aMatArray[$i] <> "" Then
                $aMatArray[$iIndex] = $aMatArray[$i]
                $aMatArray[$i] = "" ; add this?
                $iIndex += 1
            Else
                ConsoleWrite("Ignored lines: " & $i & " | " & $iIndex & " | " & $aMatArray[$i] & @CRLF)
                EndIf
        Next

        ReDim $aMatArray[$iIndex]
        $aMatArray[0] = $iIndex - 1

        ; Remove first 17 characters of hash
        _ArrayTrim($aMatArray, 17, 0)

;~      Replace '/' with '\'
        For $i = 0 To Ubound($aMatArray) - 1
         $aMatArray[$i] = StringReplace($aMatArray[$i], "/", "\")
        Next

;~      Remove duplicates
        $aMatArrayUnique = _ArrayUnique($aMatArray)

        ;~----------------------------------------------------------------------------------------------
        ;~ CLEARING matArray END
        ;~----------------------------------------------------------------------------------------------

;       Copy files to Material folder
        For $iA = 0 To UBound($aMatArrayUnique) -1
         For $iB = 0 To UBound($aArrayOutput) -1
            If StringInStr($aArrayOutput[$iB], $aMatArrayUnique[$iA]) Then
                FileCopy($aArrayOutput[$iB], $sDirOutput, $FC_OVERWRITE + $FC_CREATEPATH)
                ExitLoop
            EndIf
         Next
        Next
    EndIf
EndFunc   ;==>_Settings

 

[Tosyk]

1 hour ago, water said:

I tested with the following pattern and it worls great.

Global $sExclude = "(?i)^===*|(?i)^---*|(?i)^shader *|(?i)^material *|(?i)^template: *|(?i)^template *|^ *"

 

can you explain how this works?

[water]

(?i): ignore case. So the pattern matches "template" and "Template"
^: Only matches at the start of the line
|: matches multiple patterns

More details can be found in the help file for StringRegExp.

 

Edited December 5, 2020 by water