[SOLVED] Improving PCRE knowledge, parsing HOSTS file for hostnames.

[guinness]

The following code works very well from what I've tested, but I'm wondering if there's a different/better approach to parsing the HOSTS file for the hostnames inside the file. I know there is a HOSTS UDF by JScript, but if there are 5 hostsnames on a single line e.g. 127.0.0.1 www.text.com www.test.com www.example.com it will fail where mine doesn't.

Thanks to anyone who offers improvements.

Working code.

#include <Array.au3> ; For _ArrayDisplay.

Example()

Func Example()
    Local $sData = FileRead(@SystemDir & '\drivers\etc\HOSTS')

    ; Strip comments e.g. # This is a single comment. #
    $sData = StringRegExpReplace($sData, '#.*', '')

    ; Strip forwarding IP address e.g. 127.0.0.1
    $sData = StringRegExpReplace($sData, '(?m)^\s*(?:\d{1,3}\.){3}\d{1,3}', '')

    ; Parse HOSTS file to create an array with the hostnames in the file.
    Local $aArray = StringRegExp(' HOSTS_Count ' & $sData, '(?:[\w\.\-/]{3,})', 3) ; Past ideas: '((?:\d{1,3}\.){3}\d{1,3})\s+(.*?)\r\n' ... '((?:[\w\-]*\.+)+[\w\-]*)'
    If @error Then
        MsgBox(4096, '', 'An error occurred when parsing the HOSTS file.')
    Else
        ; Replace the 0th element with the number of items in the array.
        $aArray[0] = UBound($aArray, 1) - 1

        ; Ask whether or not to display the array using _ArrayDisplay, can be slow depending on how large your HOSTS file is.
        If MsgBox(4 + 4096, '', 'Would you like to display the Array? (Could be slow depending on how large you HOSTS file is.)') = 6 Then
            _ArrayDisplay($aArray)
        EndIf
    EndIf
EndFunc   ;==>Example

Updated: With suggestions by Spiff59.

Edited May 24, 2012 by guinness

[Spiff59]

I notice your SRE to strip comments seems to leave the end-of-line characters in place, so wouldn't inserting "" instead of @CRLF just leave less data to parse later?

On the last SRE, I'd think appending a dummy entry to the front of $sData would save you having to Redim() the array, or shift the first entry to the end (leaving the list out of sync with the actual file), something like?

Local $aArray = StringRegExp("xxx" & $sData, '(?:[w.-/]{3,})', 3)
$aArray[0] = UBound($aArray, 1) - 1

Edited May 20, 2012 by Spiff59

[guinness]

I notice your SRE to strip comments seems to leave the end-of-line characters in place, so wouldn't inserting "" instead of @CRLF just leave less data to parse later?

The reason I did this was another SRE was parsing both the forwarding address and hostname, though it's not needed anymore. Changed.

On the last SRE, I'd think appending a dummy entry to the front of $sData would save you having to Redim() the array, or shift the first entry to the end (leaving the list out of sync with the actual file), something like?

I normally do this, no idea why I didn't do it this time. Thanks Spiff59.

[ripdad]

been playing with this some -- maybe it will help.

#include  ; For _ArrayDisplay.

Example()

Func Example()
    Local $aArray, $sData = '127.0.0.1 DummyEntry' & @CRLF

    ;$sData &= FileRead(@SystemDir & 'driversetcHOSTS.');<- dot

;#cs
    $sData &= '# string one' & @CRLF
    $sData &= '   # string two' & @CRLF
    $sData &= @CRLF & @CRLF & @CRLF
    $sData &= ' 127.0.0.1   localhost' & @CRLF
    $sData &= ' 127.0.0.1' & @TAB & ' www.url1.com # string three' & @CRLF
    $sData &= '  127.0.0.1' & @TAB & '  1www.url2.com # string four' & @CRLF
    $sData &= '   127.0.0.1' & @TAB & '   123.url3.com/page.htm' & @CRLF
    $sData &= ' 127.0.0.1' & @TAB & '    url4.com/folder/text.txt' & @CRLF
    $sData &= '127.0.0.1    www.some-site.com/file.zip' & @CRLF
    $sData &= '# string five' & @CRLF
;#ce

    $aArray = StringRegExp($sData, '(?:d{1,3}.){3}d{1,3}W+(.*?)[^dw-.]', 3)
    $aArray[0] = UBound($aArray, 1) - 1
    _ArrayDisplay($aArray)
EndFunc

Edited May 21, 2012 by ripdad

[guinness]

Thanks ripdad, a different approach with the SRE, exactly what I was looking for to improve my knowledge.

Edit: I think you're missing a forward slash in your SRE if I'm not mistaken & w includes digits so no need for d. Also you can have multiple host names on a single line e.g. 127.0.0.1 localhost localhost2 localhost3, which isn't being picked up with your SRE.

Thanks for participating.

Edited May 21, 2012 by guinness

[ripdad]

If your hosts file is fairly clean, it should work. (I have seen some messy ones in the past)

I left the forward-slash out on purpose. I only wanted the root website address.

You can add it in if you wish.

[guinness]

See above for additional comments, but otherwise a different insight into PCRE.

[ripdad]

-"Thanks for participating"

glad to be of service.

-"multiple host names on a single line"

I put each entry on it's own line, but I threw it out there ... modify how you like.

[jchd]

@ripad,

Also a commented line will silently be processed as if uncommented:

'# string five 127.0.0.1 www.absolutely-forbiden.site'

[ripdad]

jchd,

I didn't see that one coming.

I don't have sites that are commented, so I didn't think of it.

Thanks for pointing it out.

[Spiff59]

Does this work?

Local $aArray = StringRegExp('HOSTS_Count ' & $sData, '[S*]+', 3)

Edited May 22, 2012 by Spiff59

[guinness]

Looks as though that SRE works just like mine does.

Edited May 23, 2012 by guinness

[jchd]

I've found interesting trying to come up with a one-liner regexp, just for the fun. And here it is (AFAIK it works):

#include <Array.au3> ; For _ArrayDisplay.
Example()
Func Example()
    Local $sData = FileRead(@SystemDir & 'driversetcHOSTS')
    ; Parse HOSTS file to create an array with the hostnames in the file.
Local $aArray = StringRegExp($sData, "(?im)G(?:(?:^s*#.*s*)*|(?:^s*)*)*(?:|(?:d{1,3}.){3}d{1,3}s+)([[:alpha:]][w.-/]{2,})s*", 3)
    If @error > 1 Then
        MsgBox(4096, '', 'An error ' & @error & ' occurred when parsing the HOSTS file.')
    Else
        _ArrayDisplay($aArray)
    EndIf
EndFunc   ;==>Example

In fact it isn't that tricky. If someone has a counter-example, please post it.

[Spiff59]

Ouch! That one hurts my eyes to look at and might cause insanity were I to try and decode it!

Kudos for being able to assemble that!. It is dropping the last entry from my HOSTS file though.

[guinness]

It returned @error 1 (I had to change @error > 1 to see it the MsgBox.) The reason being is you can have multiple hostnames on a single line >> http://technet.microsoft.com/en-us/library/cc958812.aspx which is what I do to optimise the HOSTS file.

Otherwise it works jchd if a single hostname is used, thanks for your input.

Edited May 23, 2012 by guinness

[UEZ]

Here is what I got but I cannot get rid of the #... suffix if available.

#include <array.au3>

$sHosts = _
'# Copyright (c) 1993-1999 Microsoft Corp.' & @CRLF & _
'#' & @CRLF & _
'# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.' & @CRLF & _
'#' & @CRLF & _
'# This file contains the mappings of IP addresses to host names. Each' & @CRLF & _
'# entry should be kept on an individual line. The IP address should' & @CRLF & _
'# be placed in the first column followed by the corresponding host name.' & @CRLF & _
'# The IP address and the host name should be separated by at least one' & @CRLF & _
'# space.' & @CRLF & _
'#' & @CRLF & _
'# Additionally, comments (such as these) may be inserted on individual' & @CRLF & _
'# lines or following the machine name denoted by a "#" symbol.' & @CRLF & _
'#' & @CRLF & _
'# For example:' & @CRLF & _
'#' & @CRLF & _
'#      102.54.94.97     rhino.acme.com          # source server' & @CRLF & _
'#       38.25.63.10     x.acme.com              # x client host' & @CRLF & _
'127.0.0.1       localhost  #current host' & @CRLF & _
'10.10.10.10 nirvana.somewhere-test.com' & @CRLF & _
 '                          20.20.20.20                                   www.text.com www.test.com www.example.com #dummy' & @CRLF & _
'10.10.10.20 test'

$aFilterHosts = StringRegExp($sHosts, "(?m:^)s*d+.d+.d+.d+s*(.*)", 3)
_ArrayDisplay($aFilterHosts)

Any help?

Br,

UEZ

Edited May 23, 2012 by UEZ

[jchd]

@guinness, Spiff59

Strange, that was precisely why I had some fun working on it. It works fine here: multiple entries on the same IP work and last site works too. Please can you both post a sample file making it fail (post a file, not a copy'n'paste else the forum is likely to reformat whitespaces, which may make a difference in this case).

@UEZ,

Hi, the problem with the line-by-line approach (^ anchoring) is that you get a single result entry in case of multiple targets for the same IP. That's the whole purpose of my use of G (which would gain to be documented in our help, BTW). Then all the mess around the capture only serve to "eat" unsignificant portions. At least it was the idea... sigh!

Edited May 23, 2012 by jchd

[Spiff59]

@guinness, Spiff59

My actual hosts file is boring, so I snagged the first one off the web and then "doubled-up" one line by adding the testo.com address. Your mega-regexp loses the last entry with this file:

HOSTS.txt

[BrewManNH]

I tested jchd's script with that hosts.txt file and it was able to grab all entries as well as the doubled up entry for testco.com, except for the last entry "127.0.0.1 www.abcsearch.com #[Restricted Zone site]"

[0]|localhost

[1]|test.bleepingcomputer.com

[2]|admin.abcsearch.com

[3]|www.testo.com

[4]|www3.abcsearch.com

I even tested it with the blank lines at the end of the file removed and no CRLF at the end of the last entry and it picked up everything but the last one. The regex is way beyond me so I can't figure out why.

[UEZ]

My test hosts file in post#16 is also not working with jchd's regex.

Br,

UEZ