IPRange Extractor

[Colduction]

Hi again guys!, i had COVID-19 for twice and i couldn't check the forum since 3 or 4 months ago till now! i hope you will get better if you're fighting for beat COVID-19

I have two question, first is about extracting all of the IP Address from an IP Ranges, for e.g: 192.168.1.1-192.255.255.255 (Start and End are variable and will be defined by the user) and for second one, i have a friend that he is Python programmer, he made a IP Parser that it can support large txt files (1TB) and it can parse all of them under 10min and it also supports low-end PCs that have 1 GB RAM!

The list that his program parses are:

#1765497        192.168.1.1     8082
#1765496        192.168.1.1     8084
#1965493        192.168.1.1     8089
#9565495        192.168.1.1     8086
#2565492        192.168.1.1     8081

and it converts very very fast to this:

192.168.1.1:8082
192.168.1.1:8084
192.168.1.1:8089
192.168.1.1:8086
192.168.1.1:8081

I wonder how to do this via AutoIt, if you can help me in this way, i will be happy✌❤

Thanks for your helps.

FIRST_QUESTION_TEST.au3

Edited November 3, 2020 by Colduction

[JLogan3o13]

To both of your questions, what have you tried on your own? I see no code.

More specifically on the first question "extract range of IPs" from what source? Are you saying just give it a start and end, and it spits out every address in between?

Edited November 2, 2020 by JLogan3o13

[Colduction]

Hi @JLogan3o13, i just tried for my second question but it's speed is not better than my friend's script, i've used regex code:

#.+?\t(.+)\t(.+)

 

3 minutes ago, JLogan3o13 said:

Are you saying just give it a start and end, and it spits out every address in between?

Yes, exactly.

[JLogan3o13]

Still not seeing any code, show what you have tried even if it is not working as you would like.

[Colduction]

@JLogan3o13

Test.au3

[pseakins]

For others, here is the contents of Test.au3;

 

Global $g_sRegexPattern_T = "#.+?\t\t(.+)\t\t(.+)"
Global $g_asFR_T = StringSplit(FileRead(@ScriptDir & "\1.txt"), @CRLF, 3)

For $i In $g_asFR_T
    ConsoleWrite(StringRegExpReplace($i, $g_sRegexPattern_T, "$1:$2") & @CRLF)
Next

 

Edited November 2, 2020 by pseakins

[GokAy]

Hey,

For your first question, If you are unfamiliar with the concept, maybe incorporate Network IDs into that with subnet mask?

All IPs in Network 192.168.0.0/16 are 192.168.0.1 - 192.168.255.254 (16 bit subnet mask - 255.255.0.0)

All IPs in Network 192.168.0.0/24 are 192.168.0.1 - 192.168.0.254 (24 bit subnet mask - 255.255.255.0)

Computation would be like this:

4 octets

11111111.11111111.00000000.00000000 = (16 bits) 255.255.0.0

For each octet: (2 ^ 7) * bit + (2 ^ 6) * bit + ..... + (2 ^ 0) * bit 

 

Can also be like (subnet classing, iirc)

 11111111.11111111.11111000.00000000 = (21 bits) 255.255.248.0

This would leave 000.00000000 for the devices to get an IP.

IPs in this network would be: 192.168.248.1 - 192.168.255.254

First IP address of the network (all 0's after the 1 bits of Network ID) is part of the network ID and

Final address of the network (all 1's after the 1 bits of Network ID) is the broadcast IP, no device will get those.

Only mentioned this because your example range ends with 255.  And that is the broadcast IP.

[seadoggie01]

Well... you can start by not splitting on every newline. RegEx will apply to each line:

Global $g_sRegexPattern_T = "#.+?\t\t(.+)\t\t(.+)"
ConsoleWrite(StringRegExpReplace(FileRead(@ScriptDir & "\1.txt"), $g_sRegexPattern_T, "$1:$2") & @CRLF)

[Colduction]

18 hours ago, GokAy said:

because your example range ends with 255.

Hi @GokAy, yes, but it was a test. it can be different also

Anyway, thanks for your guide, you have helpful informations

[Colduction]

18 hours ago, seadoggie01 said:

Well... you can start by not splitting on every newline. RegEx will apply to each line:

Global $g_sRegexPattern_T = "#.+?\t\t(.+)\t\t(.+)"
ConsoleWrite(StringRegExpReplace(FileRead(@ScriptDir & "\1.txt"), $g_sRegexPattern_T, "$1:$2") & @CRLF)

Hi @seadoggie01, it's not very important too, but thanks anyway. i need to solve numbered questions

[seadoggie01]

35 minutes ago, Colduction said:

i need to solve numbered questions

I don't understand what you mean here. Can you explain further? Is there more that you need to do that depends on each individual line? Even if you do, it would seem to be a lot better to apply the RegEx once then loop through lines instead of looping through lines and applying RegEx per line

[pseakins]

Assuming your input file is TAB delimited, this is one way I would do it. The drawback here is that the entire file is read into memory which is not really necessary for this kind of simple conversion. I will probably make a second post where the file is processed line by line. which may or may not be quicker.

#include <File.au3>
#include <FileConstants.au3>
#include <Array.au3>

Dim $aFileData
$hFileOut = FileOpen(@ScriptDir & "\1out.txt", $FO_OVERWRITE)
_FileReadToArray(@ScriptDir & "\1.txt", $aFileData, $FRTA_NOCOUNT, @TAB)

;~  _ArrayDisplay($aFileData)

For $i = 0 To UBound($aFileData) - 1
  FileWriteLine($hFileOut, $aFileData[$i][1] & ":" & $aFileData[$i][2])
Next

FileClose($hFileOut)

 

The thing is, with file handling under Windows, when reading\writing a file, the file is accessed one (or more) sectors (512 bytes) at a time and the the required part of the IO buffer is then swapped to the application. So, two subsequent reads do not necessarily go and hit the disk because this is all handled by the operating system. The OS makes the decisions about what to do with stuff in the buffers. You can get all fancy with your code only to find that the OS is thinking for you. The FileWriteLine() statements in the above code will only be written to the disk when the buffer has sufficient bytes to warrant a disk access.

[pseakins]

This second example avoids loading files into memory. Both my examples dispense with RegEx, which surely would introduce its own overheads given that a regular expression would need to be interpreted by the regex handler. This example can handle any size file.

#include <FileConstants.au3>

$hFilein = FileOpen(@ScriptDir & "\1.txt")
$hFileOut = FileOpen(@ScriptDir & "\1out.txt", $FO_OVERWRITE)
$sLine = FileReadLine($hFilein)
$iEOF = @error
While Not $iEOF
  $sLine = StringSplit($sLine, @TAB)
  FileWriteLine($hFileOut, $sLine[2] & ":" & $sLine[3])
  $sLine = FileReadLine($hFilein)
  $iEOF = @error
WEnd
FileClose($hFilein)
FileClose($hFileOut)

 

Edited November 4, 2020 by pseakins
added FileClose()

[Colduction]

On 11/3/2020 at 9:34 PM, seadoggie01 said:

Is there more that you need to do that depends on each individual line?

Hi @seadoggie01, i want to add progress bar and using Regex without using loop for every lines can't help progress bar, i think using Regex without loop can increase CPU loading

[Colduction]

Hi @pseakins, I hope back to home for testing your codes. Your informations about RAM is valuable, i hope to know more about it.

Thanks sir. 

[seadoggie01]

2 hours ago, Colduction said:

i think using Regex without loop can increase CPU loading

I'd tend to think the opposite. Usually (iirc) RegEx is much faster than attempting to parse the string in AutoIt.

Something else to consider... do you have Excel? You might could beat the python time utilizing Excel... I have no way to test that without a massive file though... hmm...

[Colduction]

On 11/6/2020 at 3:27 AM, seadoggie01 said:

Usually (iirc) RegEx is much faster than attempting to parse the string in AutoIt.

I saw a benchmark on the internet that Original String functions are stronger than Regex

On 11/6/2020 at 3:27 AM, seadoggie01 said:

do you have Excel?

No, i have not, i'd prefer to use just a script (of mine) instead of using other 3rd party programs