Another _FileListToArray()

[caozhiju]

Try it ,so fast!

expandcollapse popup

#cs
    High Speed 
    $sPath filepath
    $sSub include sub-folders? 0=N 1=Y annother = N
    $sFilter  Optional the filter to use, default is *.*
    $flag 0=output all files & folders，1=folders only ，2=files only ,default = 0 annother = 0
    $sOUT tempfile
#ce

Func _FileSearchToArray($sPath,  $sFilter = "*", $sSub = 0,$sFalg = 0, $sOUT = "")
    If $sOUT = "" Then $sOUT = @ScriptDir & "\filelist.txt"
    If StringRight($sPath,1)<>"\" Then $sPath=$sPath&"\"
    If $sSub = 1 Then
        $sSub = " /s"
    Else
        $sSub = ""
    EndIf
    If $sFilter = "" Then $sFilter = "*"
    If $sFalg = 1 Then
        $sFalg = "/a:d"
    ElseIf $sFalg = 2 Then
        $sFalg = "/a:-d"
    Else
        $sFalg = "/a"
    EndIf
    RunWait(@ComSpec & ' /c ' & 'dir "' & $sPath  & $sFilter & '" ' & $sFalg & ' /b' & $sSub & ' > "' & $sOUT & '"', '', @SW_HIDE)

    Dim $sfilelist
    _FileReadToArray($sOUT, $sfilelist)
    If Not IsArray($sfilelist) Then
        Dim $sfilelist[1]
        $sfilelist[0] = 0
    Else
        If $sfilelist[$sfilelist[0]] = "" Then
            _ArrayDelete($sfilelist, $sfilelist[0])
            $sfilelist[0] = $sfilelist[0] - 1
        EndIf
    EndIf
    FileDelete($sOUT)
    Return $sfilelist
EndFunc   ;==>_FilesearchToArray

Edited June 8, 2009 by caozhiju

[Ashalshaikh]

Nice Work ..

Thank You

[sheeva]

Hello,

Nice work with local drive it's working perfectly but when i try

with my server your function return 0 in array.

my path is this one "\\server\f\Movies" with this absolutly nothing append.

can you tell me why. do i make something wrong ?

Thank's

[Turkey77]

I ran into the same issue. Will DirGetSize work with UNC paths over the WAN? I get results for UNC path for my local server, but my not for a remote server. Thoughts?

My main goal is to get folder size, file & folder counts on a remote server share. Thanks, Turkey77

[caozhiju]

Hello,

Nice work with local drive it's working perfectly but when i try

with my server your function return 0 in array.

my path is this one "\\server\f\Movies" with this absolutly nothing append.

can you tell me why. do i make something wrong ?

Thank's

Do u logon server?

you can use "net use" command logon server first!

[caozhiju]

I ran into the same issue. Will DirGetSize work with UNC paths over the WAN? I get results for UNC path for my local server, but my not for a remote server. Thoughts?

My main goal is to get folder size, file & folder counts on a remote server share. Thanks, Turkey77

First you need logon Server(remote pc),you can use "net use" command

Then try it!

expandcollapse popup

#cs
    High Speed
    $sPath filepath
    $sSub include sub-folders? 0=N 1=Y annother = N
    $sFilter  Optional the filter to use, default is *.*
    $sflag 0=output all files & folders，1=folders only ，2=files only ,3=folders & size default = 0 annother = 0
    $sOUT tempfile
#ce
Func _FilesearchToArray($sPath, $sFilter = "*", $sSub = 0, $sFalg = 0, $sOUT = "")
    $sSize = 0
    If $sOUT = "" Then $sOUT = @ScriptDir & "\filelist.txt"
    If StringRight($sPath, 1) <> "\" Then $sPath = $sPath & "\"
    If $sSub = 1 Then
        $sSub = " /s"
    Else
        $sSub = ""
    EndIf
    If $sFilter = "" Then $sFilter = "*"
    Select
        Case $sFalg = 1
            $sFalg = "/a:d"
        Case $sFalg = 2
            $sFalg = "/a:-d"
        Case $sFalg = 3
            $sFalg = "/a:d"
            $sSize = 1
        Case Else
            $sFalg = "/a"
    EndSelect
    RunWait(@ComSpec & ' /c ' & 'dir "' & $sPath & $sFilter & '" ' & $sFalg & ' /b' & $sSub & ' > "' & $sOUT & '"', '', @SW_HIDE)
    Dim $sfilelist
    _FileReadToArray($sOUT, $sfilelist)
    If Not IsArray($sfilelist) Then
        Dim $sfilelist[1]
        $sfilelist[0] = 0
    Else
        If $sfilelist[$sfilelist[0]] = "" Then
            _ArrayDelete($sfilelist, $sfilelist[0])
            $sfilelist[0] = $sfilelist[0] - 1
        EndIf
    EndIf
    If $sSub = "" Then
        For $x = 1 To $sfilelist[0]
            $sfilelist[$x] = $sPath & $sfilelist[$x]
        Next
    EndIf
    If $sSize = 1 Then
        For $x = 1 To $sfilelist[0]
            $sfilelist[$x] = $sfilelist[$x] & ":size:" & DirGetSize($sfilelist[$x])
        Next
    EndIf
    FileDelete($sOUT)
    Return $sfilelist
EndFunc   ;==>_FilesearchToArray

Edited June 10, 2009 by caozhiju

[Tlem]

@SmOke_N

@caozhiju

I'm surprise that you don't care about what I said here about special characters and why you don't take the solution proposed by aec in this post !

Try your function with this directory :

"C:\My Directory N°1"

Edited June 14, 2009 by Tlem

[SmOke_N]

@SmOke_N

@caozhiju

I'm surprise that you don't care about what I said here about special characters and why you don't take the solution proposed by aec in this post !

Try your function with this directory :

"C:\My Directory N°1"

Not to be rude, but ...

@cazhiju, start your own thread, what you're doing really has nothing to do with this thread, and requires you to have two includes to even process the functions.

@Tlem, you're probably right... I could possibly incorporate aec's suggestion if I actually remembered it (I've slept once or twice in the last two years between those posts, and my files don't have special characters for me to remember to even bother with it).

Edited June 14, 2009 by SmOke_N

[Tlem]

I noticed another strange think (for me).

If I use the flag $f_full_path then the directory structure isn't respected and only filename and dir name is returned without the directory structure.

_FileListToArrayEx("C:\Dir1", "*", 0, -1, 1, 0)
C:\Dir1\Dir2\Dir3\File1.txt => File1.txt
                                Dir2
                                Dir3

Instead of : \Dir2\Dir3\File1.txt

Edited June 14, 2009 by Tlem

[SmOke_N]

I noticed another strange think (for me).

If I use the flag $f_full_path then the directory structure isn't respected and only filename and dir name is returned without the directory structure.

_FileListToArrayEx("C:\Dir1", "*", 0, -1, 1, 0)
C:\Dir1\Dir2\Dir3\File1.txt => File1.txt
                                Dir2
                                Dir3

Instead of : \Dir2\Dir3\File1.txt

Awesome, anything else that would make your world perfect?

[Tlem]

Nothing is perfect (I think).

But if your function ran like FileListToArray3.au3, it will be a fast and great function.

[SmOke_N]

Nothing is perfect (I think).

But if your function ran like FileListToArray3.au3, it will be a fast and great function.

I'm afraid I'm not familiar with that function (nor do I really want to go and find it lol).

See if this fits your needs:

expandcollapse popup

Func _FileListToArrayEx($s_path, $s_mask = "*.*", $i_flag = 0, $s_exclude = -1, $f_recurse = True, $f_full_path = True)

    If FileExists($s_path) = 0 Then Return SetError(1, 1, 0)

    ; Strip trailing backslash, and add one after to make sure there's only one
    $s_path = StringRegExpReplace($s_path, "[\\/]+\z", "") & "\"

    ; Set all defaults
    If $s_mask = -1 Or $s_mask = Default Then $s_mask = "*.*"
    If $i_flag = -1 Or $i_flag = Default Then $i_flag = 0
    If $s_exclude = -1 Or $s_exclude = Default Then $s_exclude = ""

    ; Look for bad chars
    If StringRegExp($s_mask, "[\\/ :> <\|]") Or StringRegExp($s_exclude, "[\\/ :> <\|]") Then
        Return SetError(2, 2, 0)
    EndIf

    ; Strip leading spaces between semi colon delimiter
    $s_mask = StringRegExpReplace($s_mask, "\s*;\s*", ";")
    If $s_exclude Then $s_exclude = StringRegExpReplace($s_exclude, "\s*;\s*", ";")

    ; Confirm mask has something in it
    If StringStripWS($s_mask, 8) = "" Then Return SetError(2, 2, 0)
    If $i_flag < 0 Or $i_flag > 2 Then Return SetError(3, 3, 0)

    ; Validate and create path + mask params
    Local $a_split = StringSplit($s_mask, ";"), $s_hold_split = ""
    For $i = 1 To $a_split[0]
        If StringStripWS($a_split[$i], 8) = "" Then ContinueLoop
        If StringRegExp($a_split[$i], "^\..*?\..*?\z") Then
            $a_split[$i] &= "*" & $a_split[$i]
        EndIf
        $s_hold_split &= '"' & $s_path & $a_split[$i] & '" '
    Next
    $s_hold_split = StringTrimRight($s_hold_split, 1)
    If $s_hold_split = "" Then $s_hold_split = '"' & $s_path & '*.*"'

    Local $i_pid, $s_stdout, $s_hold_out, $s_dir_file_only = "", $s_recurse = "/s "
    If $i_flag = 1 Then $s_dir_file_only = ":-d"
    If $i_flag = 2 Then $s_dir_file_only = " :D "
    If Not $f_recurse Then $s_recurse = ""

    $i_pid = Run(@ComSpec & " /c dir /b " & _
                $s_recurse & "/a" & $s_dir_file_only & " " & _
                $s_hold_split, "", @SW_HIDE, 4 + 2)
    
    ; Collect data from std i/o stream
    While 1
        $s_stdout = StdoutRead($i_pid)
        If @error Then ExitLoop
        $s_hold_out &= $s_stdout
    WEnd
    
    ; Replace any extending verticle spaces from the end of the string
    $s_hold_out = StringRegExpReplace($s_hold_out, "\v+\z", "")
    If Not $s_hold_out Then Return SetError(4, 4, 0)
    
    ; For special characters, reset the entire string to _
    ;    char base (so we're not constantly polling dllcall unecessarily)
    Local $t_buffer = DllStructCreate("char[" & StringLen($s_hold_out) * 2 & "]")
    Local $p_buffer = DllStructGetPtr($t_buffer)

    DllCall("User32.dll", "byte", "OemToChar", "str", $s_hold_out, "ptr", $p_buffer)
    Local $s_buffer = DllStructGetData($t_buffer, 1)
    ; If for some reason we fail to get a valid string back, don't fail
    If $s_buffer Then $s_hold_out = $s_buffer
    
    ; Parse data and find matches based on flags
    Local $a_fsplit = StringSplit(StringStripCR($s_hold_out), @LF), $s_hold_ret
    $s_hold_out = ""

    If $s_exclude Then $s_exclude = StringReplace(StringReplace($s_exclude, "*", ".*?"), ";", "|")
    Local $i_path_len = StringLen($s_path)
    
    For $i = 1 To $a_fsplit[0]
        If $s_exclude And StringRegExp(StringRegExpReplace( _
            $a_fsplit[$i], "(.*?[\\/]+)*(.*?\z)", "\2"), "(?i)" & $s_exclude) Then ContinueLoop
        If StringRegExp($a_fsplit[$i], "^\w:[\\/]+") = 0 Then $a_fsplit[$i] = $s_path & $a_fsplit[$i]
        If $f_full_path Then
            $s_hold_ret &= $a_fsplit[$i] & Chr(1)
        Else
            $s_hold_ret &= StringTrimLeft($a_fsplit[$i], $i_path_len) & Chr(1)
        EndIf
    Next

    $s_hold_ret = StringTrimRight($s_hold_ret, 1)
    If $s_hold_ret = "" Then Return SetError(5, 5, 0)

    Return StringSplit($s_hold_ret, Chr(1))
EndFunc

[KaFu]

...and why you don't take the solution proposed by aec in this post !

I'm afraid it's not that easy! Here's a example I tested a while ago while doing research for SMF.

expandcollapse popup

#include "Constants.au3"

; create a file "äßöüßü^.txt" in the script root... notice the difference in the msgbox regarding the output of 
; in the OEM2ANSI version the output for  is '
; in the CHCP1252 version the output for  is 
;
; of course the codepage can be set to anything you need :) .... reference can be found here http://www.microsoft.com/globaldev/reference/cphome.mspx

;---------------------------------------- @ComSpec with _OEM2ANSI ---------------------------------
$line = ""
$foo = Run(@ComSpec & " /c dir", "", @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)
While 1
    $line &= StdoutRead($foo)
    If @error Then ExitLoop
Wend

$output_comspec_oem2ansi = _OEM2ANSI($line)

;---------------------------------------- @ComSpec with chcp 1252 ---------------------------------
$line = ""
$foo = Run(@ComSpec, "", @SW_HIDE, $STDIN_CHILD + $STDOUT_CHILD)
StdinWrite($foo, "chcp 1252" & @CRLF)
StdinWrite($foo, "dir" & @CRLF)
StdinWrite($foo, "exit" & @CRLF)
While 1
    $line &= StdoutRead($foo)
    If @error Then ExitLoop
Wend
$keyword_dir = stringinstr($line,">dir")+6 ; to purge the output of th "chcp 1252" command
$keyword_colon = stringinstr($line,":",0,-1)-5 ; to purge the output of th "exit" command
$line = StringMid($line,$keyword_dir,stringlen($line)-(stringlen($line)-$keyword_colon)-$keyword_dir)

$output_comspec_chcp1252 = $line

;---------------------------------------- Output ---------------------------------

MsgBox(0, "Stdout OEM2ANSI", $output_comspec_oem2ansi)
MsgBox(0, "Stdout chcp 1252", $output_comspec_chcp1252)

Exit

;---------------------------------------- Func ---------------------------------

Func _OEM2ANSI($what)
    $ret = DllCall('user32.dll', 'Int', 'OemToChar', 'str', $what, 'str', '')
    Return $ret[2]
EndFunc   ;==>_OEM2ANSI

Calling @comspec dir always turned out to contain a flaw or the other to me, esp. if your dir and filenames contain unicode characters (Kyrillic, Japanese, Chinese...)...

The only way to go (at least it was clear for me in SMF) was FileFindFirst() & FileFindNext() until I switch to the usage of _WinAPI_FileFindFirstFile() by ascendant to obtain further file-details.

Edit:

This idea "well, optionally maybe you can somehow aquire the standard dos codepage installed, but I droped research in that direction..." is not a good one , imagine someone in China with an English OS version searching for file containing chinese letters...

Edited June 15, 2009 by KaFu

[Tlem]

@SmOke_N

Not working for the $f_full_path.

If I use it for Document and Setting, the path is truncated like this :

ment and Setting\user\...

For the special characters, it's ok for me(But I had already made the modification ).

I think that for an international version of your function, you must take a look about what KaFu said.

Edited June 15, 2009 by Tlem

[SmOke_N]

@SmOke_N

Not working for the $f_full_path.

If I use it for Document and Setting, the path is truncated like this :

ment and Setting\user\...

For the special characters, it's ok for me(But I had already made the modification ).

I think that for an international version of your function, you must take a look about what KaFu said.

I'm not too concerned personally with an international version, I had no intentions of replacing the original _FileListToArray() with this function. I made it specifically for myself to be honest.

You need to provide an actual working example of your issue, because I can't replicate or really understand your issue.

Edit:

And if you had the the international issue fixed with OemToAnsi the way AEC had it, then you were/are calling the dll much more than you need to be.

Edited June 16, 2009 by SmOke_N

[Tlem]

I know that calling the dll isn't really the solution and I know that your function is what you need for yourself, but nevertheless, if you use it and you have a special characters on the name of file or directory, that doesn't work well.

Don't know if American people use characters like '°' or 'éèࣵ...' but if they do, the use of your function will not work correctly because of the use of Dir cmd.

About $f_full_path, this is what I mean :

If I use you function to list files in C:\Documents and Settings\All Users, for example the full path is :

C:\Documents and Settings\All Users\Application Data\My File.txt

and the relative path is :

Application Data\My File.txt

But your function return :

ments and Settings\All Users\Application Data\My File.txt

I talk about that just for help, not to disturb someone.

This forum is used by international people, so he can be indeed to indicate a problem for other countries.

Edited June 16, 2009 by Tlem

[SmOke_N]

I know that calling the dll isn't really the solution and I know that your function is what you need for yourself, but nevertheless, if you use it and you have a special characters on the name of file or directory, that doesn't work well.

Don't know if American people use characters like '°' or 'éèࣵ...' but if they do, the use of your function will not work correctly because of the use of Dir cmd.

About $f_full_path, this is what I mean :

If I use you function to list files in C:\Documents and Settings\All Users, for example the full path is :

C:\Documents and Settings\All Users\Application Data\My File.txt

and the relative path is :

Application Data\My File.txt

But your function return :

ments and Settings\All Users\Application Data\My File.txt

I talk about that just for help, not to disturb someone.

This forum is used by international people, so he can be indeed to indicate a problem for other countries.

the Dir cmd comment with special characters was enough to get me to try that one more time.

However, I still can not replicate your issue with the $f_full_path, I am always getting the relative path.

Could you write an example call using macros that demonstrates your concern so I can see for myself?

Last attempt on special chars:

expandcollapse popup

Func _FileListToArrayEx($s_path, $s_mask = "*.*", $i_flag = 0, $s_exclude = -1, $f_recurse = True, $f_full_path = True)

    If FileExists($s_path) = 0 Then Return SetError(1, 1, 0)

    ; Strip trailing backslash, and add one after to make sure there's only one
    $s_path = StringRegExpReplace($s_path, "[\\/]+\z", "") & "\"

    ; Set all defaults
    If $s_mask = -1 Or $s_mask = Default Then $s_mask = "*.*"
    If $i_flag = -1 Or $i_flag = Default Then $i_flag = 0
    If $s_exclude = -1 Or $s_exclude = Default Then $s_exclude = ""

    ; Look for bad chars
    If StringRegExp($s_mask, "[\\/:><\|]") Or StringRegExp($s_exclude, "[\\/:><\|]") Then
        Return SetError(2, 2, 0)
    EndIf

    ; Strip leading spaces between semi colon delimiter
    $s_mask = StringRegExpReplace($s_mask, "\s*;\s*", ";")
    If $s_exclude Then $s_exclude = StringRegExpReplace($s_exclude, "\s*;\s*", ";")

    ; Confirm mask has something in it
    If StringStripWS($s_mask, 8) = "" Then Return SetError(2, 2, 0)
    If $i_flag < 0 Or $i_flag > 2 Then Return SetError(3, 3, 0)

    ; Validate and create path + mask params
    Local $a_split = StringSplit($s_mask, ";"), $s_hold_split = ""
    For $i = 1 To $a_split[0]
        If StringStripWS($a_split[$i], 8) = "" Then ContinueLoop
        If StringRegExp($a_split[$i], "^\..*?\..*?\z") Then
            $a_split[$i] &= "*" & $a_split[$i]
        EndIf
        $s_hold_split &= '"' & $s_path & $a_split[$i] & '" '
    Next
    $s_hold_split = StringTrimRight($s_hold_split, 1)
    If $s_hold_split = "" Then $s_hold_split = '"' & $s_path & '*.*"'

    Local $i_pid, $s_stdout, $s_hold_out, $s_dir_file_only = "", $s_recurse = "/s "
    If $i_flag = 1 Then $s_dir_file_only = ":-d"
    If $i_flag = 2 Then $s_dir_file_only = ":d"
    If Not $f_recurse Then $s_recurse = ""
    
    Local $i_buffer, $t_input, $t_output, $s_hold_binary
    
    $i_pid = Run(@ComSpec & " /u /c dir /b " & _
                $s_recurse & "/a" & $s_dir_file_only & " " & _
                $s_hold_split, "", @SW_HIDE, 4 + 2)
                    
    ; Collect data from std i/o stream
    While 1
        $s_stdout = StdoutRead($i_pid, False, True)
        If @error Then ExitLoop
        $i_buffer = BinaryLen($s_stdout)
        $t_input = DllStructCreate("byte[" & $i_buffer & "]")
        $t_output = DllStructCreate("char[" & $i_buffer & "]")
        
        DllStructSetData($t_input, 1, $s_stdout)
        DllCall("kernel32.dll", "int", "WideCharToMultiByte", _
                "int", 0, "int", 0, "ptr", DllStructGetPtr($t_input), _
                "int", $i_buffer / 2, "ptr", DllStructGetPtr($t_output), _
                "int", $i_buffer, "int", 0, "int", 0)

        $s_hold_out &= StringRegExpReplace(DllStructGetData($t_output, 1), "(?i)(?:\A|\v)(?:(0+\v+)|(0+))|(\v*0+\z)", "")
        $t_input = 0
        $t_output = 0
    WEnd
    
    ; Replace any extending verticle spaces from the end of the string
    $s_hold_out = StringRegExpReplace($s_hold_out, "\v+\z", "")
    If Not $s_hold_out Then Return SetError(5, 5, 0)
   
    ; Parse data and find matches based on flags
    Local $a_fsplit = StringSplit(StringStripCR($s_hold_out), @LF), $s_hold_ret
    $s_hold_out = ""

    If $s_exclude Then $s_exclude = StringReplace(StringReplace($s_exclude, "*", ".*?"), ";", "|")
    Local $i_path_len = BinaryLen($s_path)
   
    For $i = 1 To $a_fsplit[0]
        If $s_exclude And StringRegExp(StringRegExpReplace( _
            $a_fsplit[$i], "(.*?[\\/]+)*(.*?\z)", "\2"), "(?i)" & $s_exclude) Then ContinueLoop
        If StringRegExp($a_fsplit[$i], "^\w:[\\/]+") = 0 Then $a_fsplit[$i] = $s_path & $a_fsplit[$i]
        If $f_full_path Then
            $s_hold_ret &= $a_fsplit[$i] & Chr(1)
        Else
            $s_hold_ret &= StringTrimLeft($a_fsplit[$i], $i_path_len) & Chr(1)
        EndIf
    Next

    $s_hold_ret = StringTrimRight($s_hold_ret, 1)
    If $s_hold_ret = "" Then Return SetError(6, 6, 0)

    Return StringSplit($s_hold_ret, Chr(1))
EndFunc

Edit:

With this example:

Local $a = _FileListToArrayEx("C:\Documents and Settings\All Users", "*.*", 0, -1, 1, 0)
For $i = 1 To $a[0]
    ConsoleWrite($a[$i] & @CRLF)
Next

I received a return value of (many more than this, this is just a few):

Application Data\VMware
Application Data\Windows Genuine Advantage
Application Data\yahoo!
Application Data\Yahoo! Companion
Application Data\Adobe\Acrobat
Application Data\Adobe\AIR

Edit:

It was leaving null chars sometimes at the beginning of the string.

This option really does slow the thing down... 6000 records in 750 ms is much slower than it used to be.

I have one I wrote in another language that enums the same directory in 100 ms, but it's missing 400 files lol, need to figure out how to fix that... if I can then it would probably be a better command line option than cmd.exe.

Edited June 16, 2009 by SmOke_N

[Tlem]

Doesn't matter for me, it was just to prevent foreign users about special characters and FileList function that used Dir cmd.

For the $f_full_path, I told you just because when I test it, the return was wrong.

Now it seems to work correctly. But in fact, it's not really a problem because the result is the source path + the result of the initial function.

[SmOke_N]

Doesn't matter for me, it was just to prevent foreign users about special characters and FileList function that used Dir cmd.

So you're saying that you can use your special characters with this now?

Now it seems to work correctly. But in fact, it's not really a problem because the result is the source path + the result of the initial function.

I don't understand that at all.

[Tlem]

Yes your new function work well now.

About the $f_full_path don't care about that.