Regular expression challenge for stripping single comments.

guinness · December 23, 2013

My aim is to improve stripping single line comments in a script and even though the regular expression works, when stripping a large file (such as AutoItWrapper) it crashes AutoIt.

So SRE gurus, if you like a challenge this one is for you.

#include <Constants.au3> ; Should be MsgBoxConstants for all those rocking v3.3.10.0.

; Example()

; New line comment.
Func Example() ; This is some comment after a function.
    Local $sText = "; This is a commment as a string and shouldn't be removed." ; This is a comment to explain the; string. Woah! inception.
EndFunc   ;==>Example

; How I currently strip single line comments.
Local $sData = FileRead(@ScriptFullPath)
_StripCommentLines($sData)
MsgBox($MB_SYSTEMMODAL, 'Comment stripping', $sData)


Func _StripCommentLines(ByRef $sData)
    $sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"]).*?\2)*[^;]*)(?:;?[^\r\n]*)', '\1') ; By ProgAndy. http://www.autoitscript.com/forum/topic/108728-stripcomments/#entry765712
EndFunc   ;==>_StripCommentLines

Edited December 23, 2013 by guinness

jchd · December 24, 2013

I don't have time to look closely but I strongly suspect the issue is the heavy backtracking implied by the pattern.

For me, latest AutoIt3Wrapper.au3 doesn't crash with either regexp:

$sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"]).*?\2)*[^;]*)(?:;?[^\r\n]*)', '\1') ; By ProgAndy. http://www.autoitscript.com/forum/topic/108728-stripcomments/#entry765712
    $sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"])[^\r\n]*?\2)*[^;]*)(?:;?[^\r\n]*)', '\1') ; By ProgAndy. http://www.autoitscript.com/forum/topic/108728-stripcomments/#entry765712
    $sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"])\N*?\2)*[^;]*)(?:;?[^\r\n]*)', '\1') ; By ProgAndy. http://www.autoitscript.com/forum/topic/108728-stripcomments/#entry765712

Win7 x64, run in x86. It remains to check that the output is correct.

mikell · December 24, 2013

Even with a regex shortened a bit, this makes no significative difference (on the last release 3.3.10.0)

Local $sData = FileRead("G:\Program Files\AutoIt3\Include\GDIPlus.au3")
_StripCommentLines($sData)
FileWrite("test.txt", $sData)

Func _StripCommentLines(ByRef $sData)
      $sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"]).*?\2)*[^;]*);?.*$', '$1')
;~   $sData = StringRegExpReplace($sData, '(?m)^((?:[^''";]*([''"]).*?\2)*[^;]*)(?:;?[^\r\n]*)', '\1') ; By ProgAndy. 
EndFunc   ;==>_StripCommentLine

guinness · December 24, 2013

jchd,

Thanks. I am not getting any crash now, but will look at your amendments and discuss with wraithdu who seems to be getting the crashes. I don't suck at regular expressions that much as evident in the MVP section.

mikell,

Cheers for your amendments and time as well. I see you changed the last non-capturing group.

mikell · December 24, 2013

Yes, for readability only...

Grouping this part is not necessary and with (?m) the dot can be used as it doesn't match newlines

DXRW4E · December 24, 2013

Hi guinness, in 99.9% of cases is always the fault of Pattern, will have to be so ("K" is really nice and helpful )

#include <Constants.au3> ; Should be MsgBoxConstants for all those rocking v3.3.10.0.

; Example()

; New line comment.
Func Example() ; This is some comment after a function.
    Local $sText = "; This is a commment as a string and shouldn't be removed." ; This is a comment to explain the; string. Woah! inception.
EndFunc   ;==>Example

; How I currently strip single line comments.
Local $sData = FileRead(@ScriptFullPath)
_StripCommentLines($sData)
MsgBox($MB_SYSTEMMODAL, 'Comment stripping', $sData)


Func _StripCommentLines(ByRef $sData)
    ;;$sData = StringRegExpReplace($sData, '(?m)^(?:[^;"' & "']|'[^']*'|" & '"[^"]*")*\K;.*', "") ;; Pattern (?m)^(?:[^;"']|'[^']*'|"[^"]*")*\K;.*
    $sData = StringRegExpReplace($sData, '(?m)^[^;"'']*(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*', "")  ;; Pattern (?m)^[^;"']*(?:[^;"']|'[^']*'|"[^"]*")*\K;.*
EndFunc   ;==>_StripCommentLine



#cs    ;ConsoleWrite($sData & @LF)
 #include <Constants.au3>




Func Example()
    Local $sText = "; This is a commment as a string and shouldn't be removed."
EndFunc   


Local $sData = FileRead(@ScriptFullPath)
_StripCommentLines($sData)
ConsoleWrite($sData & @LF)


Func _StripCommentLines(ByRef $sData)
    $sData = StringRegExpReplace($sData, '(?m)^(?:[^;"' & "']|'[^']*'|" & '"[^"]*")*\K;.*', "")
EndFunc   
#CE

#include <Constants.au3>

Local $sData = FileRead(@DesktopDir & "\test.au3") ;55 MB, 1.122.000 Line
local $fTimerDiff =TimerInit()
_StripCommentLines($sData)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite($fTimerDiff & @LF)


Func _StripCommentLines(ByRef $sData)
    $sData = StringRegExpReplace($sData, '(?m)^[^;"'']*(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*', "")
EndFunc   ;==>_StripCommentLine

#cs
 >"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\DXRW4E\Desktop\New AutoIt v3 Script (2).au3" /UserParams    
+>14:42:18 Starting AutoIt3Wrapper v.2.1.3.3 SciTE v.3.3.7.0 ;  Keyboard:00000409  OS:WIN_8/  CPU:X64 OS:X64    Environment(Language:0409  Keyboard:00000409  OS:WIN_8/  CPU:X64 OS:X64)
>Running AU3Check (3.3.10.0)  from:C:\Program Files (x86)\AutoIt3
+>14:42:18 AU3Check ended.rc:0
>Running:(3.3.10.0):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\DXRW4E\Desktop\New AutoIt v3 Script (2).au3"    
--> Press Ctrl+Alt+F5 to Restart or Ctrl+Break to Stop
1751.5439052816
+>14:42:22 AutoIt3.exe ended.rc:0
>Exit code: 0    Time: 4.351
#CE

Ciao.

Edited December 24, 2013 by DXRW4E

mikell · December 24, 2013

Nice ! and no backreferencing providing max lightness

$sData = StringRegExpReplace($sData, '(?m)^(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*', "")

DXRW4E · December 24, 2013

Hi mikell, how to say is the ABC of RegEx & match string in quotes '(?:[^"]|"[^"]")' servesmust (for more savecompatibility) to matchcheck all the string, not start directly from '"' ect ect

Ciao.

Edited December 24, 2013 by DXRW4E

kylomas · December 24, 2013

If you guys have time could you explain in more detail or break the patterns down for us light-weights? Especially why DWRW4E's pattern is better/different.

Thanks and Merry Christmas!

kylomas

edit:spelling

Edited December 24, 2013 by kylomas

DXRW4E · December 24, 2013

as mentioned in the post above, first use the correct way to readcheck string in quotes (so match correctly ";" or "," ect ect), in this case use of "K" tired much less the regex, as you know Regex lose time when capture Group ect ect, instead with the "K" all is more simple regex does not need to capture that group so normally uses less memory and CPU ect ect, and also more faster, so need to do everything to reduce the options in pattern, because there are the limits of the pointer or do not know in windows hmmmmm, so when there are many MB of data to be check, regex in complicated patterns performingchecking thousands and thousands of options uses much more memory and cpu and is slower ect ect and passes certain limits of windows for this goes to error ect ect, however does not exist basic rules regard to the patterns, all depends on the situation from what you want to do

Ciao.

Edited December 24, 2013 by DXRW4E

guinness · December 25, 2013

Oh wow DXRW4E. I have to have a day for what you just wrote to sink in. Thanks for that. I will probably PM you in the next couple of days if that's OK?

guinness · December 25, 2013

OK, I have isolated the crashing I was talking about.

; Compile AutoIt3Wrapper.au3 and the run the regular expression against the Obfuscated code.
Local $sData = FileRead("C:\Program Files\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper_Obfuscated.au3")
$sData = StringRegExpReplace($sData, '(?m)^[^;"'']*(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*', '')
Exit

This is something I can't work my head around.

DXRW4E · December 25, 2013

I will probably PM you in the next couple of days if that's OK?

OK

OK, I have isolated the crashing I was talking about.

; Compile AutoIt3Wrapper.au3 and the run the regular expression against the Obfuscated code.
Local $sData = FileRead("C:\Program Files\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper_Obfuscated.au3")
$sData = StringRegExpReplace($sData, '(?m)^[^;"'']*(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*', '')
Exit

This is something I can't work my head around.

in my script i personally never use the (?m) , I always use the 'n' or any point of reference, so everything is more faster ect ect and much less stressful for regex

this will have to be Ok

#include <array.au3>
; Compile AutoIt3Wrapper.au3 and the run the regular expression against the Obfuscated code.
Local $sData = FileRead("C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper_Obfuscated.au3")
;;$sData = $aFileData = StringRegExpReplace(@LF & $sData, '\r\n?', @LF)
$sData = StringRegExpReplace(@LF & $sData, '\r(?!\n)', @CRLF)
;$sData = StringRegExpReplace($sData, '(?m)^[^;"'']*(?:[^;"'']|''[^'']*''|"[^"]*")*\K;.*?', '')
$sData = StringTrimLeft(StringRegExpReplace($sData, '\n[^;"''\r\n]*(?:[^;"''\r\n]|''[^''\r\n]*''|"[^"\r\n]*")*\K;[^\r\n]*', ''), 1)
Exit

or ignore #Region comments

#include <Constants.au3> ; Should be MsgBoxConstants for all those rocking v3.3.10.0.

#Region ;aaaaaaa
;bbbbbbbbbbbbbbbb
#EndRegion ;ccccccccccc

; Example()

; New line comment.
Func Example() ; This is some comment after a function.
    Local $sText = "; This is a commment as a string and shouldn't be removed." ; This is a comment to explain the; string. Woah! inception.
EndFunc   ;==>Example

; How I currently strip single line comments.
Local $sData = StringRegExpReplace(FileRead(@ScriptFullPath), '\r(?!\n)', @CRLF)
_StripCommentLines($sData)
MsgBox($MB_SYSTEMMODAL, 'Comment stripping', $sData)


Func _StripCommentLines(ByRef $sData)
    ;; ignore #Region comments
    $sData = StringTrimLeft(StringRegExpReplace(@LF & $sData, '\n\h*(?:#(?:EndRegion|Region)(*SKIP)[^\r\n]*$\f)?[^;"''\r\n]*(?:[^;"''\r\n]|''[^''\r\n]*''|"[^"\r\n]*")*\K;[^\r\n]*', ''), 1)
EndFunc   ;==>_StripCommentLines

Ciao. Edited December 25, 2013 by DXRW4E

guinness · December 25, 2013

Ah, awesome. Thanks for your help.

guinness · December 25, 2013

About your update with the regions, I will keep that in my notes, but I am stripping #EndRegion and #Region in my scripts as well.

DXRW4E · December 25, 2013

everything was to show the usefulness of (*SKIP) is very useful in certain situations, only move offset in the end of the line, so in large files will not lose anything in performance ect ect, really useful

Ciao.

guinness · December 25, 2013

That's something new to me as well as the usage of K. I know I have a lot left to learn with these things. Thanks for your time.

DXRW4E · December 25, 2013

example (I was writing an INF function, and there really is a challenge, a really big mess) translate StrKey in Inf file, so %% = % And %StrKey2% ([strings] section, StrKey2="ect ect ect")

#include <Constants.au3>

$sSectionData = 'aaa%bbb%ccc%ddd%%eee%fff%%ggg'
$sSectionData = StringRegExpReplace($sSectionData, '\r\n?', @LF) ;;@CR serves to use as a delimiter ect ect
$sSectionData = StringRegExpReplace($sSectionData, "%(?:%|[^\n%]+%(*SKIP)$\f)", "%" & @CR) ;;I need "%" & @CR, because during untraslate - StringRegExpReplace($sSectionData, "%\r", "%%") 
;;$aStrKey = StringRegExp($sSectionData, "%([^\r\n%]+)%", 3)
;;ect ect ect
MsgBox($MB_SYSTEMMODAL, 'Comment stripping', $sSectionData)

I wanted to say that options such as (*SKIP) they are really useful

Ciao.

Edited December 25, 2013 by DXRW4E

DXRW4E · December 25, 2013

Hi guinness, another useful thing that will have to get your attention is the (DEFINE) ( how to say byref func or recursive func), example (is not 100% Ok, but so to give an idea)

#include <array.au3>
Global Const $sRefVDSubPattern = '(?(DEFINE)(?<VD>' & '\h*\w+\((?:[^;\(\)"''\r\n]|''[^''\r\n]*''|"[^"\r\n]*"|(?&VD)|\h*\_\h*(?:;[^\r\n]*)*\r?\n\h*)*\)\h*' & '))'

Local $aTest, $sTest = "$Var = StringRegExpReplace($ScriptFullPath & FileRead(@ScriptFullPath), $ScriptFullPath, @CRLF), $Var2 = 'aaaaaaaa', $Var3, $Var4 = FuncName(FuncName(FuncName($Var3, 'aaaa'))), $Var5"


$aTest = StringRegExp($sTest, $sRefVDSubPattern & '(\$\w+)(?>\h*=\h*(?:''[^''\r\n]*''|"[^"\r\n]*")+|\h*=(?&VD)|\h+\_\h*(?:;[^\r\n]*)*\r?\n\h*)*\h*', 3)
;;$aTest = StringRegExp($sTest, $sRefVDSubPattern & '(\$\w+(?>\h*=\h*(?:''[^''\r\n]*''|"[^"\r\n]*")+|\h*=(?&VD)|\h+\_\h*(?:;[^\r\n]*)*\r?\n\h*)*\h*)', 3)

_ArrayDisplay($aTest)

so you can calluse the '(?&VD)' where you want and when you want ect ect, very very useful in complicated situations

Ciao.

Edited December 25, 2013 by DXRW4E

Malkey · December 25, 2013

DXRW4E (An aside, off topic, FYI)

I could not help being distracted by your constant mentioning of electroconvulsive therapy (ECT), formerly known as electroshock. I realise you must be refering to "et cetera, et cetera, et cetera", which was made famous from the movie, "The King and I", and from the book, "Anna and the King of Siam".

An accepted abbreviation of "et cetera" is "etc." or "&c.", where "et" means "and", and "cetera" means "the rest", or "other things". Normally one "et cetera" is all that is needed to convey "and the rest". And, "etc." normally follows a coma, &c.

salut, adios, aloha, etc.

Regular expression challenge for stripping single comments.

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

jchd

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members