How to read or get list of array from sequence of hex?

[HezzelQuartz]

If I have a hex file:
 

FF 43 01 D1 FE 0B 00 F9 F8 5F 02 A9 F6 57 03 A9 F4 4F 04 A9 58 D0 3B D5 F3 03 08 AA 08 17 40 F9 12 12 12 12 F5 03 00 AA E8 07 00 F9 43 FE FF 97 9F 06 00 71 F6 03 00 2A AB 00 00 54

How can I get a list of array start from 43?
 

43
01
D1
FE
...
00
00
54

I tried, something like this

$hFile1 = FileOpen(@ScriptDir & '\example.so', 16)
$bRead1 = FileRead($hFile1)
$aFile1 = StringRegExp($bRead1, '[[:xdigit:]]{2}', $STR_REGEXPARRAYGLOBALMATCH, 3)
_ArrayDisplay($aFile1)
FileClose($hFile1)

But it always start from FF, not 43

Can anyone please give solution or at least any clues for me?

===================================================================================================

I have another question
 

1. Which one is right for my code above [[:xdigit:]]{2} or ([[:xdigit:]]{2})?

2. Is there any better or faster solution to get a list of array other than using StringRegExp?
 

Thank You

Edited 10 hours ago by HezzelQuartz

[pixelsearch]

T

3 hours ago, HezzelQuartz said:

But it always start from FF, not 43

This is because the binary string returned by FileRead starts with "0x" so f you want to start from 43, then you should indicate 5 as offset (last parameter in StringRegExp) to bypass "0xFF"

[HezzelQuartz]

24 minutes ago, pixelsearch said:

T

This is because the binary string returned by FileRead starts with "0x" so f you want to start from 43, then you should indicate 5 as offset (last parameter in StringRegExp) to bypass "0xFF"

Thank You
It worked

Is there any better or faster solution to get a list of array other than using StringRegExp above?

how can I get the size or length of that array list? Should I use Ubound?

Edited 7 hours ago by HezzelQuartz

[pixelsearch]

28 minutes ago, HezzelQuartz said:

Is there any better or faster solution to get a list of array other than using StringRegExp above?

I don't think so. You can't use StringSplit as there are no delimiters. I just tested your script with a binary file of 116KB (237570 bytes length) and the RegExp part was immediate ("There" will be displayed immediately after "Here" in the script below) :

ConsoleWrite("Here" & @crlf)
$aFile1 = StringRegExp($bRead1, '[[:xdigit:]]{2}', $STR_REGEXPARRAYGLOBALMATCH, 5)
ConsoleWrite("There" & @crlf)
ConsoleWrite(Ubound($aFile1) & @crlf) ; 118783 rows

The ArrayDisplay part will take much longer with 118783 rows to display.

3 hours ago, HezzelQuartz said:

Which one is right for my code above [[:xdigit:]]{2} or ([[:xdigit:]]{2})?

I think you don't need a group (...) in your case, this is what I read on a website right now :

What is the purpose of regex capture groups?

Capturing groups allow you to treat a part of your regex pattern as a single unit. This is especially useful when you want to apply quantifiers or modifiers to multiple characters or subpatterns. For example, (abc)+ matches one or more repetitions of "abc" as a whole [...]

 

Edited 6 hours ago by pixelsearch

[HezzelQuartz]

1 hour ago, pixelsearch said:

I don't think so. You can't use StringSplit as there are no delimiters. I just tested your script with a binary file of 116KB (237570 bytes length) and the RegExp part was immediate ("There" will be displayed immediately after "Here" in the script below) :

ConsoleWrite("Here" & @crlf)
$aFile1 = StringRegExp($bRead1, '[[:xdigit:]]{2}', $STR_REGEXPARRAYGLOBALMATCH, 5)
ConsoleWrite("There" & @crlf)
ConsoleWrite(Ubound($aFile1) & @crlf) ; 118783 rows

The ArrayDisplay part will take much longer with 118783 rows to display.

I think you don't need a group (...) in your case, this is what I read on a website right now :

What is the purpose of regex capture groups?

Capturing groups allow you to treat a part of your regex pattern as a single unit. This is especially useful when you want to apply quantifiers or modifiers to multiple characters or subpatterns. For example, (abc)+ matches one or more repetitions of "abc" as a whole [...]

 

It seems that my code doesn't work to 137,496 KB file size....
When I run it, nothing happened for a very long time, and autoit logo suddenly disappeared

Did I do something wrong with the code?

============================================================================================================

When I tried to run my code to 16,000++ KB file size
It worked well

When I tried to run my code to 49,000++ KB file size
Error appear: error allocating memory

Edited 5 hours ago by HezzelQuartz

[pixelsearch]

1 hour ago, HezzelQuartz said:

When I tried to run my code to 16,000++ KB file size
It worked well

This is because the maximum number of elements for an array is 16,777,216 (help file, AutoIt3 Limits/defaults)

When you divide that number of 16,777,216 by 1024 bytes, it gives you the maximal size of your file in KB, i.e. 16,384 KB, that's why your test on 16,000++ KB file size worked, because each byte will become a row in the array and you're still under the limit of 16,777,216 elements

 

 

Edited 3 hours ago by pixelsearch

[HezzelQuartz]

2 hours ago, pixelsearch said:

This is because the maximum number of elements for an array is 16,777,216 (help file, AutoIt3 Limits/defaults)

When you divide that number of 16,777,216 by 1024 bytes, it gives you the maximal size of your file in KB, i.e. 16,384 KB, that's why your test on 16,000++ KB file size worked, because each byte will become a row in the array and you're still under the limit of 16,777,216 elements

 

 

Is there any way to increase the limit?

Actually, I want to compare two hex file by splitting every byte into array, then compare every array

This is my code below
 

$hFile1 = FileOpen(@ScriptDir & '\example1.so', 16)
$bRead1 = FileRead($hFile1)
$hFile2 = FileOpen(@ScriptDir & '\example2.so', 16)
$bRead2 = FileRead($hFile2)

$aFile1 = StringRegExp($bRead1, '[[:xdigit:]]{2}', $STR_REGEXPARRAYGLOBALMATCH, 1)
$aFile2 = StringRegExp($bRead2, '[[:xdigit:]]{2}', $STR_REGEXPARRAYGLOBALMATCH, 1)

For $i = 1 To UBound($aFile1, $UBOUND_ROWS)-1
    If $aFile1[$i] <> $aFile2[$i] Then
        If Mod($i, 4) = 0 Then
            FileWrite(@ScriptDir & '\' & 'Compare Result.txt', $aFile1[$i] & ' ' & $aFile1[$i+1] & ' ' & $aFile1[$i+2] & ' ' & $aFile1[$i+3] & '    ' & $aFile2[$i] & ' ' & $aFile2[$i+1] & ' ' & $aFile2[$i+2] & ' ' & $aFile2[$i+3] & @CRLF)
            $i = $i + 4
        ElseIf Mod($i, 4) = 1 Then
            FileWrite(@ScriptDir & '\' & 'Compare Result.txt', $aFile1[$i-1] & ' ' & $aFile1[$i] & ' ' & $aFile1[$i+1] & ' ' & $aFile1[$i+2] & '    ' & $aFile2[$i-1] & ' ' & $aFile2[$i] & ' ' & $aFile2[$i+1] & ' ' & $aFile2[$i+2] & @CRLF)
            $i = $i + 3
        ElseIf Mod($i, 4) = 2 Then
            FileWrite(@ScriptDir & '\' & 'Compare Result.txt', $aFile1[$i-2] & ' ' & $aFile1[$i-1] & ' ' & $aFile1[$i] & ' ' & $aFile1[$i+1] & '    ' & $aFile2[$i-2] & ' ' & $aFile2[$i-1] & ' ' & $aFile2[$i] & ' ' & $aFile2[$i+1] & @CRLF)
            $i = $i + 2
        ElseIf Mod($i, 4) = 3 Then
            FileWrite(@ScriptDir & '\' & 'Compare Result.txt', $aFile1[$i-3] & ' ' & $aFile1[$i-2] & ' ' & $aFile1[$i-1] & ' ' & $aFile1[$i] & '    ' & $aFile2[$i-3] & ' ' & $aFile2[$i-2] & ' ' & $aFile2[$i-1] & ' ' & $aFile2[$i] & @CRLF)
            $i = $i + 1
        EndIf
    ElseIf $aFile1[$i] = $aFile2[$i] Then
        ContinueLoop
    EndIf
Next
FileClose($hFile1)
FileClose($hFile2)

Or, Is there any way I can split file read every 16,000 KB?
Or can I read from 16,001 KB to 32,000 KB?

Edited 1 hour ago by HezzelQuartz

[Nine]

If I correctly understand, you want to compare 2 files of same length starting at byte 2.  So you could try this :

$hFile1 = FileOpen(@ScriptDir & '\example1.so', 16)
$bRead1 = FileRead($hFile1)
$hFile2 = FileOpen(@ScriptDir & '\example2.so', 16)
$bRead2 = FileRead($hFile2)
FileClose($hFile1)
FileClose($hFile2)

Local $dMid1, $dMid2

For $i = 2 To BinaryLen($bRead1)
  If BinaryMid($bRead1, $i, 1) <> BinaryMid($bRead2, $i, 1) Then
    $iMod = Mod($i, 4)
    $dMid1 = BinaryMid($bRead1, $i - $iMod, 4)
    $dMid2 = BinaryMid($bRead2, $i - $iMod, 4)
    $i = $i + 4 - $iMod
    ConsoleWrite($dMid1 & "/" & $dMid2 & @CRLF)
  EndIf
Next

Works for me with my example files.

ps. I'll let you format the output as you want 

Edited 42 minutes ago by Nine
typo

[HezzelQuartz]

31 minutes ago, Nine said:

If I correctly understand, you want to compare 2 files of same length starting at byte 2.  So you could try this :

$hFile1 = FileOpen(@ScriptDir & '\example1.so', 16)
$bRead1 = FileRead($hFile1)
$hFile2 = FileOpen(@ScriptDir & '\example2.so', 16)
$bRead2 = FileRead($hFile2)
FileClose($hFile1)
FileClose($hFile2)

Local $dMid1, $dMid2

For $i = 2 To BinaryLen($bRead1)
  If BinaryMid($bRead1, $i, 1) <> BinaryMid($bRead2, $i, 1) Then
    $iMod = Mod($i, 4)
    $dMid1 = BinaryMid($bRead1, $i - $iMod, 4)
    $dMid2 = BinaryMid($bRead2, $i - $iMod, 4)
    $i = $i + 4 - $iMod
    ConsoleWrite($dMid1 & "/" & $dMid2 & @CRLF)
  EndIf
Next

Works for me with my example files.

ps. I'll let you format the output as you want 

does it work with very big file?
the size of file I want to compare is 130,000++ KB

Edited 19 minutes ago by HezzelQuartz

[Nine]

Should, what about you try it.  

MAX_BINARYSIZE =

2,147,483,647  :

Maximum bytes of binary data.

Edited 20 minutes ago by Nine

[HezzelQuartz]

1 minute ago, Nine said:

Should, what about you try it.  

MAX_BINARYSIZE =

2,147,483,647  :

Maximum bytes of binary data.

I'm trying to run and learning this code and function now

Thank You for this code

============================================================================================================================
 

Btw, Is there any way I can change the limit of autoit? I mean: "AutoIt3 Limits/defaults"

Sorry for my noob stupid question

[Nine]

Nope.  I mean not those size max limits...

Edited 7 minutes ago by Nine