Folder size exclude an ext

[Terenz]

Proprably impossible but i'll try to ask.

There is an API, WMI or any other things that give me a folder size but with a filter, so excluding an extension of a file? Like take this folder ( and his subfolder ) give me the size without any .word file.

I know i can search all the file, exclude the .word, loop all the file with FileGetSize and finally sum the total size but if exist a faster way please tell me.

Thank you

[UEZ]

Afaik, it is not possible to do what you want with a simple command or function.

You have to read the folder recursively and add the size of each file by exluding the extension you want.

[SadBunny]

If your environment is equipped with powershell, you may want to go with a system call to a powershell oneliner like:

(Get-ChildItem c:tmp -recurse | Where-Object {$_.Name -match ".*au3"} | Measure-Object -property length -sum).sum

Put your base directory in the blue part, and any regular expression filter in the argument of the -match switch to filter on any filespec you want. (/edit and if you don't like regex, use any other Get-ChildItem filter, run help Get-ChildItem in powershell for more information, google for specifics.)

It's probably much faster than reading through a directory from within an AutoIt script, but I have not benchmarked it.

Good luck!

Edited March 20, 2015 by SadBunny

[UEZ]

You have an one-liner but you have to understand what's going on behind the scene of PS.

I bet that AutoIt isn't slower than PS in this case.

Edited March 20, 2015 by UEZ

[SadBunny]

You have an one-liner but you have to understand what's going on behind the scene of PS.

I bet that AutoIt isn't slower than PS in this case.

 

I am a novice powersheller, don't know much about its under-the-hood mechanics... All I know is that I tried this on a directory with 90.000 files (my entire steam library, with regex .*) and it had the sum within a second. /edit: Now I must say that this is a set of triple raid-0 ssd's, so that may help. I did it again and timed it - it's actually 0.92 seconds It's good to live in the future.

And whether it's faster or slower - I wrote the oneliner faster and with less effort than writing an autoitscript would have taken me Anyway, it's just a suggestion for TS.

Edited March 20, 2015 by SadBunny

[jguinch]

Maybe you can try with my _DirGetSizeByExtension (in my signature), but it will be slower than PS

Local $iSizeAll = DirGetSize("c:\tmp")
Local $iSizeDoc = _DirGetSizeByExtension("c:\tmp", "doc")
Local $iSizeExludeDoc = $iSizeAll - $iSizeDoc

ConsoleWrite($iSizeAll & " - " & $iSizeDoc & " = " & $iSizeExludeDoc)

[UEZ]

I compared this PS script:

$sw = [Diagnostics.Stopwatch]::StartNew()
(Get-ChildItem "c:\A_DIR\" -recurse | Where-Object {$_.Name -notmatch ".+\.au3"} | Measure-Object -property length -sum).sum
$sw.Stop()
$sw.Elapsed

with this script:

#include <File.au3>

Global $fTimer = TimerInit()
ConsoleWrite("Overall file size: " & FilesGetSize("C:\A_DIR\") &  " bytes / amount of files: "  & @extended & @CRLF)
ConsoleWrite(TimerDiff($fTimer) & " ms" & @CRLF)

Func FilesGetSize($sDir, $sExcludeExt = ".au3", $bRecursiv = True)
    Local $hSearch = FileFindFirstFile($sDir & "\*"), $sFullFileName, $sFAttrib
    If $hSearch = -1 Then Return SetError(1, 0, 0)  ;no files found
    Local Static $iFSize = 0, $iIncludedFiles = 0, $iAllFilesFolders = 0, $iFolders = 0
    While 1
        $sFileName = FileFindNextFile($hSearch)
        If @error Then ExitLoop         ; If there is no more file matching the search.
        $sFullFileName = $sDir & "\" & $sFileName
        $sFAttrib = FileGetAttrib($sFullFileName)
        If  StringInStr($sFAttrib, "D") And $bRecursiv Then
            $iFolders += 1
            FilesGetSize($sFullFileName, $sExcludeExt, $bRecursiv)
        EndIf
        If StringRight($sFileName, StringLen($sExcludeExt)) <> $sExcludeExt Then
            $iFSize += FileGetSize($sFullFileName)
            $iIncludedFiles += 1
        EndIf
        ;$iAllFilesFolders += 1
    WEnd
    FileClose($hSearch)
    Return SetExtended($iIncludedFiles - $iFolders , $iFSize)
EndFunc

My test folder: 50573 files and 5285 folder

 

PS: ~10662,5321 ms

AU3: ~9790.24117279014 ms

 

As I suggested both are more or less in the same range.

Edited March 20, 2015 by UEZ

[SadBunny]

My test folder: 50573 files and 5285 folder

PS: ~10662,5321 ms

AU3: ~9790.24117279014 ms

As I suggested both are more or less in the same range.

 

Sweet Didn't expect that. Tried your script, seems the speeds of the PS way are very constant for me at about .90-1.00 while the AU3 fluctuates between .80 and 1.20. Don't know why that is. But yeah, doesn't seem to make a noticeable difference. Guess it really comes down to preference then  /edit: ah, got it, I was tarballing some very large files on another drive. Yes, .80 vs. .90. Au3 beats PS

I used to use AutoIt for all these kinds of things, but I've taken more and more of a liking to powershell for Windows sysadmin-related automation. It's insanely useful once you kind of surrender to it. Takes a lot of googling to get the hang of it, but once you do, in my opinion at least, it runs circles around AutoIt for sysadmin tasks like these in terms of development efficiency.

Edited March 20, 2015 by SadBunny

[jguinch]

Here is my code, using my suggest in #6 :

expandcollapse popup

$sDir = @SystemDir

Global $fTimer = TimerInit()
ConsoleWrite("Overall file size: " & FilesGetSize($sDir, ".exe") &  " bytes / amount of files: "  & @extended & @CRLF)
ConsoleWrite(TimerDiff($fTimer) & " ms" & @CRLF & @CRLF)

Global $fTimer = TimerInit()
Local $aSizeAll = DirGetSize($sDir, 1)
Local $aSizeDoc = _DirGetSizeByExtension($sDir, "exe", 1)
ConsoleWrite("Overall file size: " & $aSizeAll[0] - $aSizeDoc[0] &  " bytes / amount of files: "  & $aSizeAll[1] - $aSizeDoc[1] & " / amount of folders: " & $aSizeAll[2] & @CRLF)
ConsoleWrite(TimerDiff($fTimer) & " ms" & @CRLF)




Func _DirGetSizeByExtension($sDir, $sExt = "*", $iFlag = 0)
    If NOT FileExists($sDir) Then Return SetError(1, 0, -1)
    If NOT StringInStr(FileGetAttrib($sDir), "D") Then Return SetError(1, 0, -1)

    If $sExt = "*" Then $sExt = "?.*"

    Local $aDirs[1] = [ StringRegExpReplace($sDir, "\\$", "") ]
    Local $iCountDir = 0, $iCountFile = 0, $n = 0
    Local $hSearch, $sFileName
    Local $iSize = 0

    While 1
        $hSearch = FileFindFirstFile( $aDirs[$n]  & "\*.*"  )
        If $hSearch <> -1 Then
            While 1
                $sFileName = FileFindNextFile($hSearch)
                If @error Then ExitLoop

                If @Extended Then
                    $iCountDir += 1
                    If $iCountDir >= UBound($aDirs) Then Redim $aDirs[ UBound($aDirs) * 2]
                    $aDirs[$iCountDir] = $aDirs[$n] & "\" & $sFileName
                Else
                    If StringRegExp($sFileName, "(?i)\." & $sExt & "$") Then
                        $iSize += FileGetSize($aDirs[$n] &  "\" & $sFileName)
                        $iCountFile += 1
                    EndIf
                EndIf
            WEnd
        EndIf
        FileClose($hSearch)

        If $n = $iCountDir Then ExitLoop
        $n += 1
    WEnd

    If NOT $iFlag Then Return $iSize
    Local $aResult[3] = [ $iSize, $iCountFile, $iCountDir ]
    Return $aResult
EndFunc






Func FilesGetSize($sDir, $sExcludeExt = ".au3", $bRecursiv = True)
    Local $hSearch = FileFindFirstFile($sDir & "\*"), $sFullFileName, $sFAttrib
    If $hSearch = -1 Then Return SetError(1, 0, 0)  ;no files found
    Local Static $iFSize = 0, $iCounter = 0
    While 1
        $sFileName = FileFindNextFile($hSearch)
        If @error Then ExitLoop         ; If there is no more file matching the search.
        $sFullFileName = $sDir & "\" & $sFileName
        $sFAttrib = FileGetAttrib($sFullFileName)
        If  StringInStr($sFAttrib, "D") And $bRecursiv Then
            FilesGetSize($sFullFileName, $sExcludeExt, $bRecursiv)
        EndIf
        If StringRight($sFileName, StringLen($sExcludeExt)) <> $sExcludeExt Then
            $iFSize += FileGetSize($sFullFileName)
            $iCounter += 1
        EndIf
    WEnd
    FileClose($hSearch)
    Return SetExtended($iCounter, $iFSize)
EndFunc

PS : 2193,45 ms

UEZ : 720.19 ms

jguinch : 128.77 ms

Both AU3 codes return the same size, but PS returns a diffenrent size...

[UEZ]

I made a mistake in the calulcation for the files without the excluded ones -> I updated the code post #7.

Your code is approx. 4 times faster than my recursive version. 

[Terenz]

My post of yesterday has disappeard?

jguinch,

Plese you can add to your code an optional parameter for make the search not recursive and stop at parent folder? I don't know for DirGetSize if we can use a paramenter like 2+1 or BitOr or only 1 or 2 but in any case i'm intrested only in the total size

Thanks for the help to all

[UEZ]

Terenz,

the DB has crashed and the forum was restored but unfortunately the posts from 6am - 12pm (GMT) were lost. -> '?do=embed' frameborder='0' data-embedContent>>

[Terenz]

Oh, Ok. So maybe other post was missed from this thread, like the answer to my question. Thanks

[jguinch]

@Terenz : please, try this new version, which allows the same filter possibilities than _FileListToArrayRec. Also, like the original  DirGetSize, you can set the $iflag parameter to 2 or 3 to make the function non-recursive.

expandcollapse popup

Local $aSize = _DirGetSizeEx("c:\tmp", "*.xls;*.doc", 1)

ConsoleWrite("Size :" & $aSize[0] & @CRLF)
ConsoleWrite("Files count :" & $aSize[1]  & @CRLF)
ConsoleWrite("Folders count  :" & $aSize[2] & @CRLF)


; #FUNCTION#====================================================================================================================
; Name ..........: _DirGetSizeEx ; Description ...: Returns the size in bytes of a file list by extension.
; Syntax ........: _DirGetSizeEx($sDir[, $sMask = "*"[, $iFlag = 0]])
; Parameters ....: $sDir                - The directory to search in.
;                  $sMask               - [optional] Filter for results. Default is "*" (all).
;                                         Filter for result. Multiple filters must be separated by ";"
;                                         Use "|" to separate 3 possible sets of filters: "Include|Exclude|Exclude_Folders"
;                                             Include = Files/Folders to include (default = "*" [all])
;                                             Exclude = Files/Folders to exclude (default = "" [none])
;                                             Exclude_Folders = only used if $iRecur = 1 AND $iReturn <> 2 to exclude defined folders (default = "" [none])
;                  $iFlag               - [optional] 0 (default) = Returns the size
;                                                    1 = Extended mode is On -> returns an array that contains extended information (see Remarks).
;                                                    2 = Don't get the size of files in subdirectories (recursive mode is Off) ; Return values .: Success = The size in bytes, or a single dimension array
;                  Failure = -1 and sets the @error flag to 1 if the path doesn't exist.
; Author ........: jguinch
; Remarks .......: If you use the extended mode then the array returned from this function is a single dimension array containing the followingelements:
;                    $aArray[0] = Size
;                    $aArray[1] = Files count
;                    $aArray[2] = Dirs Count
;===============================================================================================================================
Func _DirGetSizeEx($sDir, $sMask = "*", $iFlag = 0); OK
    If NOT FileExists($sDir) Then Return SetError(1, 0, -1)
    If NOT StringInStr(FileGetAttrib($sDir), "D") Then Return SetError(1, 0, -1)

    Local $iExtMode = BitAND($iFlag, 1) > 0
    Local $iRecMode = NOT BitAND($iFlag, 2) > 0
    Local $aDirs[1] = [ StringRegExpReplace($sDir, "\\$", "") ]
    Local $iCountDir = 0, $iCountFile = 0, $n = 0, $iSize = 0
    Local $hSearch, $sFileName, $sRegexFilesInclude, $sRegexFilesExclude = "^$", $sRegexFoldersExclude = "^$"

    Local $sRegexMask = StringReplace( StringReplace( StringReplace($sMask, "|", "\|") , "?", "\E(?:.|.?$)\Q"), "*", "\E.*?\Q")
    Local $aFilters = StringSplit($sRegexMask, "\|", 3)

    $sRegexFilesInclude = "(?i)^(?:" & StringRegExpReplace(StringReplace($aFilters[0], ";", "|") , "([^|]+)", "\\Q$1\\E") & ")$"
    If UBound($aFilters) > 1 Then $sRegexFilesExclude = "(?i)^(?:" & StringRegExpReplace(StringReplace($aFilters[1], ";", "|") , "([^|]+)", "\\Q$1\\E") & ")$"
    If UBound($aFilters) > 2 Then $sRegexFoldersExclude = "(?i)^(?:" & StringRegExpReplace(StringReplace($aFilters[2], ";", "|") , "([^|]+)", "\\Q$1\\E") & ")$"

    While 1
        $hSearch = FileFindFirstFile( $aDirs[$n]  & "\*.*"  )
        If $hSearch <> -1 Then

            While 1
                $sFileName = FileFindNextFile($hSearch)
                If @error Then ExitLoop

                If @Extended Then
                    If NOT StringRegExp($sFileName, $sRegexFoldersExclude) Then
                        $iCountDir += 1
                        If $iCountDir >= UBound($aDirs) Then Redim $aDirs[UBound($aDirs) * 2]
                        $aDirs[$iCountDir] = $aDirs[$n] & "\" & $sFileName
                    EndIf
                Else
                    If StringRegExp($sFileName, $sRegexFilesInclude) AND NOT StringRegExp($sFileName, $sRegexFilesExclude) Then
                        $iSize += FileGetSize($aDirs[$n] &  "\" &$sFileName)
                        $iCountFile += 1
                    EndIf
                EndIf
            WEnd

            If NOT $iRecMode Then ExitLoop
        EndIf

        FileClose($hSearch)

        If $n = $iCountDir Then ExitLoop
        $n += 1
    WEnd

    If NOT $iExtMode Then Return $iSize

    Local $aResult[3] = [ $iSize, $iCountFile, $iCountDir ]
    Return $aResult
EndFunc

[Terenz]

Is cool but seems slow of almost double times respect of the script of the post #9

193ms vs 383 ms approx. Same configuration, recursive and same folder. Two separate script lauched many times, the first version was faster in any test

Do you think can you speed up a bit? As i said ( with all due respect to your code ) i need something faster as possible. Thanks

Edited March 23, 2015 by Terenz

[jguinch]

I spend to last 2 hours to try to optimize my code, but no, I cannot make a really faster version, sorry... I cannot suggest you a best way than my first one.

BTW, my >_DirGetSizeEx could be useful for someone else, so I will work on it with enough tests before posting it in the Examples section

Thanks for you return.

[Terenz]

I'm sure it will be useful for many people. I think all that RegExp slow down that function, in my case i'll use your previus version, if no other one has a fastest way. Thanks to you.

Edited March 24, 2015 by Terenz

[jguinch]

The reason is that there are up to 3 RegEx instead of 1, because I added a filter for include files, exclude files and exclude folders.

[jguinch]

I made some changes in my code to have just one regex execution by loop (fusion of two regex into a big one). The result is the same...

If you want to see the result :

expandcollapse popup

; #FUNCTION#====================================================================================================================
; Name ..........: _DirGetSizeEx
; Description ...: Returns the size in bytes of a file list by extension.
; Syntax ........: _DirGetSizeEx($sDir[, $sMask = "*"[, $iFlag = 0]])
; Parameters ....: $sDir                - The directory to search in.
;                  $sMask               - [optional] Filter for results. Default is "*" (all).
;                                         Filter for result. Multiple filters must be separated by ";"
;                                         Use "|" to separate 3 possible sets of filters: "Include|Exclude|Exclude_Folders"
;                                             Include = Files/Folders to include (default = "*" [all])
;                                             Exclude = Files/Folders to exclude (default = "" [none])
;                                             Exclude_Folders = only used if $iRecur = 1 AND $iReturn <> 2 to exclude defined folders (default = "" [none])
;                  $iFlag               - [optional] 0 (default) = Returns the size
;                                                    1 = Extended mode is On -> returns an array that contains extended information (see Remarks).
;                                                    2 = Don't get the size of files in subdirectories (recursive mode is Off) ; Return values .: Success = The size in bytes, or a single dimension array
;                  Failure = -1 and sets the @error flag to 1 if the path doesn't exist.
; Author ........: jguinch
; Remarks .......: If you use the extended mode then the array returned from this function is a single dimension array containing the followingelements:
;                    $aArray[0] = Size
;                    $aArray[1] = Files count
;                    $aArray[2] = Dirs Count
;===============================================================================================================================
Func _DirGetSizeEx($sDir, $sMask = "*", $iFlag = 0); OK
    If NOT FileExists($sDir) Then Return SetError(1, 0, -1)
    If NOT StringInStr(FileGetAttrib($sDir), "D") Then Return SetError(1, 0, -1)

    Local $iExtMode = 0, $iRecMode = 1
    Local $aDirs[1] = [ StringRegExpReplace($sDir, "\\$", "") ]
    Local $iCountDir = 0, $iCountFile = 0, $n = 0, $iSize = 0, $iFolderFilter = 0
    Local $sRegexFilesExclude = "()", $sRegexFoldersExclude = "^$"
    Local $hSearch, $sFileName, $sRegexMask, $iAddFolder, $aRes, $iUBound
    Local $aFilters = StringSplit(  StringReplace( StringReplace( StringReplace($sMask, "|", "\|") , "?", "\E(?:.|.?$)\Q"), "*", "\E.*?\Q"), "\|", 3)
    Local $sRegexFilesInclude = "(?|(\Q" & StringReplace($aFilters[0], ";", "\E)|(\Q") & "\E))$"
    
    If BitAND($iFlag, 1) Then $iExtMode = 1
    If BitAND($iFlag, 2) Then $iRecMode = 0

    If UBound($aFilters) > 1 Then $sRegexFilesExclude = "(?!(?|(\Q" & StringReplace($aFilters[1], ";","\E)|(\Q") & "\E))$)"
    $sRegexMask = "(?i)^" & $sRegexFilesExclude & $sRegexFilesInclude
    ConsoleWrite($sRegexMask & @CRLF)

    If UBound($aFilters) > 2 Then
        $iFolderFilter = 1
        $sRegexFoldersExclude = "(?i)^(?:\Q" & StringReplace($aFilters[2], ";", "\E|\Q") & "\E)$"
    EndIf

    While 1
        $hSearch = FileFindFirstFile( $aDirs[$n]  & "\*.*"  )
        If $hSearch <> -1 Then

            While 1
                $sFileName = FileFindNextFile($hSearch)
                If @error Then ExitLoop

                If @Extended Then
                    $iAddFolder = 1
                    If $iFolderFilter AND StringRegExp($sFileName, $sRegexFoldersExclude) Then $iAddFolder = 0
                    
                    If $iAddFolder Then 
                        $iUBound = UBound($aDirs)
                        $iCountDir += 1
                        If $iCountDir >= $iUBound Then Redim $aDirs[$iUBound * 2]
                        $aDirs[$iCountDir] = $aDirs[$n] & "\" & $sFileName
                    EndIf
                ElseIf StringRegExp($sFileName, $sRegexMask) Then
                    $iSize += FileGetSize($aDirs[$n] &  "\" & $sFileName)
                    $iCountFile += 1
                EndIf
            WEnd

            If NOT $iRecMode Then ExitLoop
        EndIf

        FileClose($hSearch)

        If $n = $iCountDir Then ExitLoop
        $n += 1
    WEnd

    If NOT $iExtMode Then Return $iSize

    Local $aResult[3] = [ $iSize, $iCountFile, $iCountDir ]
    Return $aResult
EndFunc

[Terenz]

Seems not work:

$sDir = @SystemDir

$fTimer = TimerInit()
$aSize = _DirGetSizeEx($sDir, "exe", 1)
ConsoleWrite(Round(($aSize[0]) / 1048576, 2) & " MB / amount of files: " & $aSize[1] & @CRLF)
ConsoleWrite(TimerDiff($fTimer) & " ms" & @CRLF)

$fTimer = TimerInit()
$aSize = _DirGetSizeEx($sDir, "*", 1)
ConsoleWrite(Round(($aSize[0]) / 1048576, 2) & " MB / amount of files: " & $aSize[1] & @CRLF)
ConsoleWrite(TimerDiff($fTimer) & " ms" & @CRLF)

0 MB / amount of files: 0
148.524742669809 ms

906.08 MB / amount of files: 5471
278.603844902076 ms

Don't found any exe file? For all file instead give the correct result