[SOLVED] Improving PCRE knowledge, parsing HOSTS file for hostnames.

[jchd]

Argh I didn't test all cases (I was expecting something like this).

Should work better with this one:

Local $aArray = StringRegExp($sData, "(?im)G(?:(?:s*#.*$s*)*|(?:s*)*)*(?:|(?:d{1,3}.){3}d{1,3}s+)([[:alpha:]][w.-/]{2,})s*", 3)

Seem to work reliably with all tests cases submited.

[UEZ]

Looks good now.

Thanks,

UEZ

Edited May 23, 2012 by UEZ

[guinness]

jchd,

The current SRE in post #21 still didn't produce any results. Here's an example of a stripped downed HOSTS file I use.

[BrewManNH]

Isn't there supposed to be a space between the IP addresses and the URL? Does that host file work written like that?

[guinness]

Does that host file work written like that?

Yes. 127.0.0.1 is the address where as the other numbers are part of the hostname, the google was changed from another site as I use my HOSTS file to block malicious websites. Edited May 23, 2012 by guinness

[BrewManNH]

Odd, never seen one that looked like that before. Learned something new today.

[guinness]

I changed my reply above.

[Spiff59]

As the first non-whitespace group of a non-comment line is required to be the forwarding IP, I wouldn't think it necessary to be looking for groups of decimal characters. I think these 3 lines correctly parse a HOSTS file. I am having trouble massaging them all into one statement...

$sData = StringRegExpReplace($sData, '#.*', '')  ; strip comments
    $sData = StringRegExpReplace($sData, '(?m)^s*S*', '') ; strip IP
    Local $aArray = StringRegExp('0 ' & $sData, '[S*]+', 3) ; split words

Edited May 23, 2012 by Spiff59

[guinness]

As the first non-whitespace group of a non-comment line is required to be the forwarding IP, I wouldn't think it necessary to be looking for groups of decimal characters. I think these 3 lines correctly parse a HOSTS file. I am having trouble massaging them all into one statement...

$sData = StringRegExpReplace($sData, '#.*', '')  ; strip comments
    $sData = StringRegExpReplace($sData, '(?m)^s*S*', '') ; strip IP
    Local $aArray = StringRegExp('0 ' & $sData, '[S*]+', 3) ; split words

Produces the same results as my SRE, nice.

Let's hope it can be done with one SRE, I will try to decipher jchd's

I like threads like this, as I already had the answer but have come to understand there are many ways to achieve the same result. Thanks to all who have posted so far.

Edited May 23, 2012 by guinness

[UEZ]

Can you try this?:

#include <array.au3>

$sHosts = _
'# Copyright © 1993-1999 Microsoft Corp.' & @CRLF & _
'#' & @CRLF & _
'# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.' & @CRLF & _
'#' & @CRLF & _
'# This file contains the mappings of IP addresses to host names. Each' & @CRLF & _
'# entry should be kept on an individual line. The IP address should' & @CRLF & _
'# be placed in the first column followed by the corresponding host name.' & @CRLF & _
'# The IP address and the host name should be separated by at least one' & @CRLF & _
'# space.' & @CRLF & _
'#' & @CRLF & _
'# Additionally, comments (such as these) may be inserted on individual' & @CRLF & _
'# lines or following the machine name denoted by a "#" symbol.' & @CRLF & _
'#' & @CRLF & _
'# For example:' & @CRLF & _
'#' & @CRLF & _
'#      102.54.94.97     rhino.acme.com          # source server' & @CRLF & _
'#       38.25.63.10     x.acme.com              # x client host' & @CRLF & _
'127.0.0.1       localhost  #current host' & @CRLF & _
'10.10.10.10 nirvana.somewhere-test.com' & @CRLF & _
'127.0.0.1 08.185.87.03.google.com  08.185.87.04.google.com  08.185.87.05.google.com  08.185.87.06.google.com  08.185.87.07.google.com  08.185.87.08.google.com  08.185.87.09.google.com  08.185.87.1.google.com 08.185.87.10.google.com' & @CRLF & _
 '                          20.20.20.20                                   www.text.com www.test.com www.example.com #dummy' & @CRLF & _
'10.10.10.20 test'

$aFilterHosts = StringRegExp(StringRegExpReplace($sHosts, "(?m:)(#.*)", ""), "s*d+.d+.d+.d{1,3}s*(.*)", 3)
_ArrayDisplay($aFilterHosts)

Btw, what about IPv6 entries?

Br,

UEZ

Edited May 23, 2012 by UEZ

[guinness]

UEZ,

Try my Example and you'll see what is expected, your SRE is stripping comments but it's not separating hostnames into individual elements..

[UEZ]

Ahh, ok! Got it now!

But my Hosts example is not working with your code from post#1.

Br,

UEZ

[guinness]

It is for me, I get an array with 15 hostnames.

Edit: This is correct as I didn't account for space between the forwarding address at the beginning. It was showing 16 when it should have been 15. Fixed, see below and first post too.

expandcollapse popup

#include <Array.au3> ; For _ArrayDisplay.

Example()

Func Example()
    Local $sData = '# Copyright © 1993-1999 Microsoft Corp.' & @CRLF & _
            '#' & @CRLF & _
            '# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.' & @CRLF & _
            '#' & @CRLF & _
            '# This file contains the mappings of IP addresses to host names. Each' & @CRLF & _
            '# entry should be kept on an individual line. The IP address should' & @CRLF & _
            '# be placed in the first column followed by the corresponding host name.' & @CRLF & _
            '# The IP address and the host name should be separated by at least one' & @CRLF & _
            '# space.' & @CRLF & _
            '#' & @CRLF & _
            '# Additionally, comments (such as these) may be inserted on individual' & @CRLF & _
            '# lines or following the machine name denoted by a "#" symbol.' & @CRLF & _
            '#' & @CRLF & _
            '# For example:' & @CRLF & _
            '#' & @CRLF & _
            '#      102.54.94.97     rhino.acme.com          # source server' & @CRLF & _
            '#       38.25.63.10     x.acme.com              # x client host' & @CRLF & _
            '127.0.0.1       localhost  #current host' & @CRLF & _
            '10.10.10.10 nirvana.somewhere-test.com' & @CRLF & _
            '127.0.0.1 08.185.87.03.google.com  08.185.87.04.google.com  08.185.87.05.google.com  08.185.87.06.google.com  08.185.87.07.google.com  08.185.87.08.google.com  08.185.87.09.google.com  08.185.87.1.google.com 08.185.87.10.google.com' & @CRLF & _
            '                          20.20.20.20                                   www.text.com www.test.com www.example.com #dummy' & @CRLF & _
            '10.10.10.20 test'

    ; Strip comments e.g. # This is a single comment. #
    $sData = StringRegExpReplace($sData, '#.*', '')

    ; Strip forwarding IP address e.g. 127.0.0.1
    $sData = StringRegExpReplace($sData, '(?m)^s*(?:d{1,3}.){3}d{1,3}', '')

    ; Parse HOSTS file to create an array with the hostnames in the file.
    Local $aArray = StringRegExp(' HOSTS_Count ' & $sData, '(?:[w.-/]{3,})', 3) ; Past ideas: '((?:d{1,3}.){3}d{1,3})s+(.*?)rn' ... '((?:[w-]*.+)+[w-]*)'
    If @error Then
        MsgBox(4096, '', 'An error occurred when parsing the HOSTS file.')
    Else
        ; Replace the 0th element with the number of items in the array.
        $aArray[0] = UBound($aArray, 1) - 1

        ; Ask whether or not to display the array using _ArrayDisplay, can be slow depending on how large your HOSTS file is.
        If MsgBox(4 + 4096, '', 'Would you like to display the Array? (Could be slow depending on how large you HOSTS file is.)') = 6 Then
            _ArrayDisplay($aArray)
        EndIf
    EndIf
EndFunc   ;==>Example

Edited May 23, 2012 by guinness

[jchd]

I now get it: you want to handle direct hosts with IP too!

That can be cured as well:

#include <Array.au3> ; For _ArrayDisplay.
Example()
Func Example()
    Local $sData = FileRead(@SystemDir & 'driversetcHOSTS')
    ; Parse HOSTS file to create an array with the hostnames in the file.
Local $aArray = StringRegExp($sData, "(?im)G(?:(?:s*#.*$s*)*|(?:s*)*)*(?:|(?:d{1,3}.){3}d{1,3}s+)((?:(?:d{1,3}.){3}d{1,3}.)?[[:alpha:]][w.-/]{2,})s*", 3)
    If @error Then
        MsgBox(4096, '', 'An error ' & @error & ' occurred when parsing the HOSTS file.')
    Else
        _ArrayDisplay($aArray)
    EndIf
EndFunc   ;==>Example

[UEZ]

Here my version stealing a part from jchd's regex:

$aFilterHosts = StringRegExp(StringRegExpReplace($sHosts, "(?m:)(#.*)", ""), "(?m:)((?:(?:d{1,3}.){3}d{1,3}.)?[[:alpha:]][w.-/]{2,})s*", 3)
_ArrayDisplay($aFilterHosts)

Br,

UEZ

Edited May 24, 2012 by UEZ

[jchd]

Hi UEZ,

You're spoiling all the fun, not by borrowing or even stealing anything, but by plain cheating and making a two-line one-liner out of it

Of course I'm well aware that systematically going into depths to achieve a task with only a single RE isn't good practice. Good code mostly demands easy flowing instructions, easy to understand and to maintain.

My purpose was merely to try to answer guinness initial question, albeit whether my proposal is any improvement in practical code is debatable. This is a somehow similar (but vastly simpler) situation to when J. Friedl makes a single RE to check an email address validity (but only at the format level), resulting in either 6598- or 4724-chars REs (in the first edition of his Mastering... book, don't know for subsequent editions). Of course it isn't practical nor good code at any rate.

BTW you have an extraneous invalid space at

... (? :(?: ...

Edited May 24, 2012 by jchd

[UEZ]

Hi UEZ,

You're spoiling all the fun, not by borrowing or even stealing anything, but by plain cheating and making a two-line one-liner out of it

Of course I'm well aware that systematically going into depths to achieve a task with only a single RE isn't good practice. Good code mostly demands easy flowing instructions, easy to understand and to maintain.

My purpose was merely to try to answer guinness initial question, albeit whether my proposal is any improvement in practical code is debatable. This is a somehow similar (but vastly simpler) situation to when J. Friedl makes a single RE to check an email address validity (but only at the format level), resulting in either 6598- or 4724-chars REs (in the first edition of his Mastering... book, don't know for subsequent editions). Of course it isn't practical nor good code at any rate.

BTW you have an extraneous invalid space at

... (? :(?: ...

It is not a 2 liner but rather using more than 1 time StringRegExp() / StringRegExpReplace (concatenation of commands)! Further I'm novice in regex and want to improve it.

Thanks for the regex lession!

Br,

UEZ

Edited May 24, 2012 by UEZ

[jchd]

I was joking of course.

Don't give a dam, we're all noobs in regeps, certainly compared to some crazy stuff you find here and there!

Don't see any lesson here, call it experimentation.

[guinness]

I use the HOSTS file to block malicious sites by using custom HOSTS file found on the internet.

The SREs by jchd and UEZ are not parsing all types of hostnames. If you want I can send you both the HOSTS file via pm due to the nature of the content and you can compare with what mine parses and yours.

[jchd]

Yes, let's see how far we can push that. You seem to have a pretty good idea about how all that may look like, so go ahead.