Generating a bit-field from three ushort values

[lbsl]

Hi lads,

I'm trying to figure out a method to translate one or multiple USHORT values into one large bitfield and i am slowly getting a headache figuring out an effective way to perform this.

In general
The bitfield tells my routine which columns entities are *not* present in a record in the database structure i am reading (The so-called NULL values).
This also means that the byte-block beloging to that field, is not present within that file. This is a very simple form of low-level database compression technique.
I have several different database files, and they contain different amounts of columns with various mixed data-type forms.

The logic of the bit-range is that for each group of columns, one bitfield is stuffed into a USHORT, once the boundaries of the lower logic are reached, the bitfield continues in a next set of a USHORT value.
The USHORT blocks are stored in low to high order order inside the file and have to be swapped to get the correct binary representation.

Below snippet displays the details of the structure, starting with the (in this case) three USHORT bitfields.
The next range of figures is the binary representation of these bitfields *when* shifted (i have done this all manually) and combined in the proper order.
Some of the bitranges are between parentheses: when a column represents a char range, the database structure has two types of bit definitions:odd (01) for numeric fields and even (10) for char fields.
 

bit1 bit2 bit3  bit-range representation of bitfields 3-1    Col-Title column-type
0100 0000 0000  00000000000000000000000000000000000000000001 Column1  (DWORD) 
0800 0000 0000 (00000000000000000000000000000000000000001000)Column2  (CHAR) 
2000 0000 0000 (00000000000000000000000000000000000000100000)Column3  (CHAR)
8000 0000 0000 (00000000000000000000000000000000000010000000)Column4  (CHAR)
0002 0000 0000 (00000000000000000000000000000000001000000000)Column5  (CHAR)
0004 0000 0000  00000000000000000000000000000000010000000000 Column6  (DWORD)
0010 0000 0000  00000000000000000000000000000001000000000000 Column7  (DWORD)
0040 0000 0000  00000000000000000000000000000100000000000000 Column8  (DWORD)
0000 0100 0000  00000000000000000000000000010000000000000000 Column9  (DWORD)
0000 0400 0000  00000000000000000000000001000000000000000000 Columna  (DWORD)
0000 1000 0000  00000000000000000000000100000000000000000000 Columnb  (BOOLEAN)
0000 4000 0000  00000000000000000000010000000000000000000000 Columnc  (BOOLEAN)
0000 0001 0000  00000000000000000001000000000000000000000000 Columnd  (BOOLEAN)
0000 0004 0000  00000000000000000100000000000000000000000000 Columne  (BOOLEAN)
0000 0010 0000  00000000000000010000000000000000000000000000 Columnf  (BOOLEAN)
0000 0040 0000  00000000000001000000000000000000000000000000 Column10 (DWORD)
0000 0000 0100  00000000000100000000000000000000000000000000 Column11 (DWORD)
0000 0000 0800 (00000000100000000000000000000000000000000000)Column12 (CHAR)
0000 0000 1000  00000001000000000000000000000000000000000000 Column13 (DWORD)
0000 0000 8000 (00001000000000000000000000000000000000000000)Column14 (CHAR)
0000 0000 0002  00100000000000000000000000000000000000000000 Column15 (CHAR)

Each database does have a fixed column definition setting, but the order and field-types (char/numeric) within that table can vary per database file.

So figuring out which field is or isn´t present without necessarily knowing it is a char/type or numeric i am using a BitAND() construction with bit pairs of "11" which are in the order of 0x03, 0x0c, 0x30, 0xc0, 0x300, 0xc00, etc.
Regardless of the field type, i would get in that case always a difference if this field is marked as "nul"

My issue
I do know my way around reading 16-bit binaries and getting past the "no-consecutive 32-bit datablocks" issue when using WINAPI_ReadFile() but regarding how to transform multiple USHORT values into one large bitfield using bitshifting methods: I could use some crash course here since depending on the column number i have to construct either a USHORT, ULONG or even a 64/bit hex value...
I know WINAPI_ReadFile() performs the endian correction for the USHORT value, but when i have multiple of these pairs, i need to perform some multiplications (with risk of crashing the program by exceeding  a variable's type value limitation) to add these numbers up and i have the feeling that using bitshifting, works faster and more effective but....
What is the most effective way to combine these USHORT values into one ULONG/SYSTEMLong value?

 

 

[RTFC]

I'd suggest using DllstructCreate. Something like this (pseudocode, completely untested):

$shorts=DllstructCreate("ushort[4]")
DllstructSetData($shorts,1,<your first short>,1)
DllstructSetData($shorts,1,<your second short>,2)
...

$remapped=dllstructcreate("uint64",dllstructgetptr($shorts))    ; 4x16 bit = 1x64 bit
$value=DllStructGetdata($remapped,1)

<call func _BitTest to test individual bits (NB bit IDs are base-0, right to left)>

Func _BitTest($value, $bit)
    Return BitAND(BitShift($value, $bit),1)
EndFunc

 

[lbsl]

I have attempted something like that earlier, but is a real hassle (and slow), i suspect there is no way to go around it as the bitshifter and bitrotater inside AutoIT go nowhere higher than 32-bit rotation (where i need 64-bit rotation).
And slow as in: when i use DllStruct tables, WinAPI_Readfile() picks up in 32-bit blocks. and with blocks of 3 Ushorts there is lacking one ushort and the DLL reads another extra 2 bytes anyway to compensate the lacking two in memory. This means i have the whole hassle of resetting file-pointers the whole time.

 

This is my test-code i am battling with so far to get everything bitshifted:

expandcollapse popup

#include <_Primes.au3>
Dim $ByteOrder[2]
$ByteOrder[0] = 0x03
$ByteOrder[1] = 0x0c
Dim $ByteConverted[2]

Local $ByteOrderPosition = 0
Local $LowByte, $HighByte
;Consolewrite (_IntToAny(0x03*0x10,2,(6*4))&@CRLF)
;Exit
Local $Columns = 24, $Bits = $Columns * 2
;ConsoleWrite(_IntToAny(0xffff,2,20)&@CRLF)
;Exit
Local $Shorts = Ceiling($Bits / 16)
ConsoleWrite("Shorts:" & $Shorts & @CRLF)
Dim $BitBlock[$Shorts]
$BitBlock[0] = 0x000
$BitBlock[1] = 0x000
$BitBlock[2] = 0x020

Local $BinBlock, $raiser
Local $Round = 1
For $x = ($Shorts - 1) To 0 Step -1
    If $Round > 1 Then
        $BinBlock = Int(BitRotate($BinBlock, -16,"D"), 2)
        ConsoleWrite("shifted Binblock:[" & Hex($BinBlock) & "]" & @CRLF)
    EndIf
    ;$BinBlock &= String(_IntToAny($BitBlock[$x],2,20))
    ConsoleWrite("shifted Binblock:[" & Hex($BinBlock) & "] + [" & Hex($BitBlock[$x]) & "]" & @CRLF)
    $BinBlock += $BitBlock[$x]
    $Round += 1
Next
;ConsoleWrite("Binblock:["&_IntToAny($BinBlock,2,48)&"]"&@CRLF)
ConsoleWrite("Binblock:[" & $BinBlock & "]" & @CRLF)
ConsoleWrite("Binblock:[" & Hex($BinBlock) & "]" & @CRLF)
;$BinBlock = _BinToInt($BinBlock)
$BinBlock = _HexToInt($BinBlock)
$BinBlock = _IntToBin($BinBlock)
ConsoleWrite("Binblock hex:[" & $BinBlock & "]" & @CRLF)

For $x = 1 To $Columns

    If $x < 3 Then
        $ByteConverted[$ByteOrderPosition] = $ByteOrder[$ByteOrderPosition]
    Else
        $ByteConverted[$ByteOrderPosition] *= 0x10
    EndIf

    ConsoleWrite("->" & _IntToAny($ByteConverted[$ByteOrderPosition], 2, $Bits) & " | " & _IntToHex($ByteConverted[$ByteOrderPosition]) & @CRLF)
    If BitAND($ByteConverted[$ByteOrderPosition], $BinBlock) Then
        ConsoleWrite(@CRLF & "Column [" & $x & "] not present" & @CRLF)
    EndIf
    $ByteOrderPosition += 1
    If $ByteOrderPosition > 1 Then
        $ByteOrderPosition = 0
    EndIf
Next

I used the Primes UDF from this topic for the binary  convertions:

 

[lbsl]

@Rtfc, thanks for the suggestion so far;
I have reconsidered to simply do it the old fashioned way it was done in the time of lacking 64-bit variables: simply segment the identification part as well:

expandcollapse popup

#include <_Primes.au3>
Dim $ByteOrder[2]
$ByteOrder[0] = 0x03
$ByteOrder[1] = 0x0c
Dim $ByteConverted[2]

Local $ByteOrderPosition = 0
Local $LowByte, $HighByte

;Testing block start
Local $Columns = 21, $Bits = $Columns * 2
Local $Shorts = Ceiling($Bits / 16) ; -> how many USHORTS form the bit-range?

Dim $BitBlock[$Shorts + 1]
;Just for testing, change any value of the bitblocks into one of these values: 1,2,4,8 e.gg 0x0020, 0x0400, 0x8000
$BitBlock[1] = 0x0000
$BitBlock[2] = 0x0000
$BitBlock[3] = 0x0200
;Testing block end


Local $BinBlock, $BitSet, $FormerSet


For $x = 1 To $Columns
    $BitSet = Ceiling($x / 8)

    If $FormerSet <> $BitSet Then
        $FormerSet = $BitSet
        $ByteConverted[0] = 0
        $ByteConverted[1] = 0
    EndIf

    If $ByteConverted[$ByteOrderPosition] == 0 Then
        $ByteConverted[$ByteOrderPosition] = $ByteOrder[$ByteOrderPosition]
    Else
        $ByteConverted[$ByteOrderPosition] *= 0x10
    EndIf

    ConsoleWrite($BitSet & "->" & _IntToAny($ByteConverted[$ByteOrderPosition], 2, $Bits) & " | " &         _IntToHex($ByteConverted[$ByteOrderPosition]) & @CRLF)

    If BitAND($ByteConverted[$ByteOrderPosition], $BitBlock[$BitSet]) Then
        ConsoleWrite(@CRLF & "Column [" & $x & "] not present" & @CRLF)
    EndIf

    $ByteOrderPosition += 1

    If $ByteOrderPosition > 1 Then
        $ByteOrderPosition = 0
    EndIf
Next