array in array

[FMS]

Hello,

The last couple of day's I was searching on this forum for the best way to put array's inside array's.
The best example's i found where a little outdatet (2010) whit a lot of pro's and con's.

Now I've a big script where a lot of computations and big array's are involved, so speed is a big issue.
Also I wanna try the script below but don't know iff speed is a problem this way or maybe there is a better way to do this.

Does somebody know's the best way to put array's inside array's and get the data back from them?
I've made an example of something I was thinking about.
(maybe something totaly wrong but I'm open for sugestions)

I'm doing it this way because I don't think I can access the data inside the array (and doing some calculations to it) some other way if the array is inside another array.
Or is there?

Thanks in advanced.

#include <Array.au3>

Global $aArray[Random(5,10,1)][Random(5,10,1)]
Global $Holder[2][2]

For $x = 0 To UBound($aArray,1) - 1
   For $y = 0 To UBound($aArray,2) - 1
      $aArray[$x][$y] = Round(Random(-1,1),4)
   Next
Next

$Holder[0][0] = $aArray
$aArray = ""
Global $get_array = $Holder[0][0]

_ArrayDisplay($get_array)

 

[iamtheky]

So if you filled a 2x2 array with other 2d arrays, this is how you would call the item in 0,0 of each of those elements. 

#include <Array.au3>

Global $aArray[Random(5,10,1)][Random(5,10,1)]
Global $Holder[2][2]

For $w = 0 to ubound($Holder) - 1

    For $x = 0 To UBound($aArray,1) - 1
       For $y = 0 To UBound($aArray,2) - 1
          $aArray[$x][$y] = Round(Random(-1,1),4)
       Next
    Next

    $Holder[$w][0] = $aArray

    For $x = 0 To UBound($aArray,1) - 1
       For $y = 0 To UBound($aArray,2) - 1
          $aArray[$x][$y] = Round(Random(-1,1),4)
       Next
    Next

    $Holder[$w][1] = $aArray

Next




msgbox(0, '' , ($Holder[0][0])[0][0])
_ArrayDisplay($Holder[0][0])
msgbox(0, '' , ($Holder[0][1])[0][0])
_ArrayDisplay($Holder[0][1])
msgbox(0, '' , ($Holder[1][0])[0][0])
_ArrayDisplay($Holder[1][0])
msgbox(0, '' , ($Holder[1][1])[0][0])
_ArrayDisplay($Holder[1][1])

 

[jchd]

If the context demands storing 2D arrays inside 2D arrays, I see no other better way than your initial post. But maybe your problem can cope as well with a single 4D array; that would make accessing individual elements much faster.

The drawback being that you can't store variable dimension arrays when "mapped" as individual elements of a larger-dimension array. You then need to use the maximum dimensions of all the inner arrays. This isn't an issue if you store arrays inside arrays.

Perhaps there is a better solution if you expose more of the context you're working in.

[RTFC]

13 hours ago, FMS said:

a lot of computations and big array's are involved, so speed is a big issue

If so, I would suggest you do not use arrays at all (and certainly not nested arrays!), but matrices instead.

[jchd]

4 minutes ago, RTFC said:

If so, I would suggest you do not use arrays at all (and certainly not nested arrays!), but matrices instead.

That's a good possibility thanks to your hard work on Eigen, but only if the actual problem semantics is in some numerical domain, something we have little clue about. Tensor calculus may reveal non-trivial.

It all depends on the task to perform.

[RTFC]

@jchd: of course, agreed (and thanks), but the OP suggested a typical case of XYProblem to me, and the twin requirements of "large data sets" (>17M cells? then forget AutoIt arrays altogether) and "lots of computations" just cries out (in my myopic perception of the barely-defined actual problem) for a matrix approach. In any case, a deferred pointer-to-pointer-to-pointer structure (i.e., nested multidim arrays) would seem to me anathema to the aforementioned two stated requirements. Whenever my contexts require 2D nesting (very, very rare), I just tile a large 2D matrix with smaller ones, and use block-based access. Anyroad, I just wanted to flag the possibility of another possible method that might be of use.

[jchd]

You're correct. Let's see how the OP describes his real-world problem and the nature of the computations needed.

You know that unfortunately most users aren't too much math-inclined.

[FMS]

In the past I found the eigen UDF, this because I'm tryng to make an AI (code below (working but not finished)) whish is calculating whit matricies a big plus. @RTFC
unfortunaly When i tried to inplement it , it was a litle confusing for me because I'm not that great in matricies-> math :S
Also din't know how to calculate whit the eigen UDF to do what i want.
That's why i tried to do the same only whit array's.

(does this seems like an XY-problem?
 

 

AI.au3

t10k-images.idx3-ubyte

t10k-labels.idx1-ubyte

settings.set

Edited April 30, 2018 by FMS

[RTFC]

@FMS: Thanks, that's a lot clearer. Cursory inspection of your code tells me you're building a neural net with two hidden layers, in order to associate labels with images. AFAICT, the math involves basic matrix arithmetic and one exponential function (so nothing E4A cannot handle out of the box). Of course, it's entirely up to you whether you'd want to invest the time to refactor your code in matrix terms (I'm of course not going to do this for you), but all those nested loops over 2D arrays you've got now would be gone (each one replaced by a single E4A call), and the computations themselves would be machine-code (dllcall), not AutoIt. But for getting to grips with neural nets, it might be more instructive to stick with pure AutoIt, so you can see what happens at every stage. However, I fear your design won't be fast enough to solve difficult (read: large) problems (myself, I would probably solve classification problems like yours with _Eigen_LDA, or identify the relevant independent sources of variability in the images first through _Eigen_PCA, but that's just me; read the online Tutorials on these functions if you're interested, or study the provided Tutorial scripts; they use different projections into eigenspace; maybe not your cup of tea).

Just thought that if you hadn't been aware of E4A at all, it would have been worth considering (no size limits on inputs, and blazingly fast). Never mind.

One suggestion though. That generic "normelize" function you're using, which Executes whatever expression is fed to it is needlessly slowing down your code no end. Id' write dedicated functions for each usage case, and parse only parameters and/or ByRef arrays to them.

Veel succes ermee, in ieder geval.

Edited April 30, 2018 by RTFC
typos

[FMS]

Bedankt voor je antwoord @RTFC ,

The code I provide in the earlyer post , was me building an AI on pure autoit to get the a grip on AI.
Afther I get the grip I was refactoring it anyway

1st I want to have it working

If I had it working corectly I wanna try the it whit matricies.
This because I don't know how to do and did have the time to get an grip on matricies

The study of the examples in eigen i did before (for the sigmoid )  and was indeed not mine cup of tea.

Quote

(I'm of course not going to do this for you)

Ofcourse I don't want to ask you that , just for 1 thing -> I want to know what is going on under the hood, and that way I don't

But could you give an basic example on whish i can build on it ? (preferbly the calculations i need)

At this point I want to implement matricies calculations (because google says that's the best way) but I realy don't know how.

 

[RTFC]

17 minutes ago, FMS said:

1st I want to have it working

Very sensible; I'd do it that way too.

18 minutes ago, FMS said:

I want to know what is going on under the hood, and that way I don't

If you've downloaded the E4A package, you'll have noticed there's a subdirectory called source, where you'll find the complete C++ code of the (real float part of the) dll. In addition, you've  got the AutoIt wrapper code (Eigen4Autoit.au3),  Eigen's own website + documentation (with lots of simple examples) + forum, plus my own extensive online Help, where every function description includes an example script, just like AutoIt's Help. In your case, I would start by studying the Cwise (cell-wise) functions, as they provide most of what you'll need.

For a basic introduction to matrix math, I recommend the Khan academy's videos as a starting point. That should be enough to get you going.

[FMS]

I'm getting a little off topic on this thread but I'm refactoring the code at this point to use Eigen4Autoit.au3.
And I must say,.... it's easyer then i first thought
good work on the documentation @RTFC, it's very clear

A whole lot of code replaced by (and more to come ) :

_Eigen_Multiply_AB_plusC($M_weights_ih1,$M_Ninputs,$M_bias_h1,$M_outputs_h1)

I'm now at the point of  "activate" the $M_outputs_h1 , whish i did before in the array whit

For $x = 1 To UBound($outputs_h1,1) - 1
    $outputs_h1[$x][1] = sig($outputs_h1[$x][1])
Next

Func sig($x)
   Return 1 / (1 + exp($x * -1))
EndFunc

I'm a little confused to do multiple calculations on an 1 colum matrix in 1 go.
Is this even possible?
The code below uses multiple columns.

$rows=101
$matA=_Eigen_CreateMatrix($rows,3)
_Eigen_SetLinSpaced_Col($matA,0,-10,10)
_Eigen_CwiseUnaryOp_ColCol_InPlace($matA,0,1,"exp")
_Eigen_CwiseScalarOp_ColCol_InPlace($matA,1,2,"+",1)
_Eigen_CwiseBinaryOp_ColCol_InPlace($matA,2,1,"/",$matA)
_matrixDisplay($matA)

 

[RTFC]

1 hour ago, FMS said:

And I must say,.... it's easyer then i first thought
good work on the documentation

Great, glad to hear it.

1 hour ago, FMS said:

'm a little confused to do multiple calculations on an 1 colum matrix in 1 go.
Is this even possible?

Absolutely. A matrix with one dimension of unity is usually called a vector, but for simplicity, everything in E4A works with matrices as basic entity. And the functions are simpler when they act on each cell of the entire matrix, because you don't need to identify which row or col is used. Look at the brackets in your sig function and work from the inside out. Maybe something along these lines (not tested!):

_Eigen_CwiseScalarOp_InPlace($matA,"*",-1)  ; multiply by -1
_Eigen_CwiseUnaryOp_InPlace($matA,"exp")    ; apply exponential function
_Eigen_CwiseScalarOp_InPlace($matA,"+",1)   ; add 1
_Eigen_CwiseUnaryOp_InPlace($matA,"inverse")    ; obtain 1/x

 

[FMS]

nice , why din't I tried it before.
So less code for doing the same thing whit better performance!!

@RTFC aso big tumps up on the " _Eigen_SaveMatrix " and " _Eigen_LoadMatrix " !!!
This makes life so mush easyer

At this point I'm refactoring the dsig function whish I did before like this :

For $k = 1 To UBound($error_o,1) -1
    $dsig_o[$k][1] = dsig($outputs_o[$k][1])
Next


Func dsig($x)
   Return $x * (1 - $x)
EndFunc

I tried to do it like this. (And din't give the expected result )

$M_grad_o = _Eigen_CloneMatrix( $M_error_o , False )
_Eigen_SetOnes ( $M_grad_o )

_Eigen_CwiseScalarOp_InPlace( $M_grad_o ,"-",$M_error_o)
_Eigen_Multiply_AB_InPlace ( $M_grad_o, $M_error_o )

_MatrixDisplay ( $M_grad_o , "$M_grad_o" )

I'm a little stuck in this part , and feel like I'm doing this whole wrong.
Also I think I'm doing this to complex and there is an easyer way , am I wright?

[RTFC]

Classic mistake, easy to make. You're confusing scalar cellwise multiplication with matrix multiplication. Use _Eigen_CwiseBinaryOp_InPlace with operator "*" (or alias "mult") instead. Also see this video, to understand why it failed before.

Edited May 2, 2018 by RTFC

[kylomas]

RTFC & FMS

I watched the video and understand the concept (no math background) but do not understand the practicality of it.   Can you give a real life example of matrix math?

Thanks,

Kylomas

[FMS]

@kylomas For me -> If you look at AI.au3 and specificly at the function "feed" whitin, there are a lot of computations whis are the same across the whole array.
This I'm refactoring now to use E4A.

1 reason for performance and the other for not to use loops.

Maybe not the best explanation I'm also not an math-wizzard but I like the simplicitie.
 

expandcollapse popup

Func feed($runtimes = 10 , $backward = False)

   $feed_forward = True
   rap_rebuild_global($runtimes)

   If $backward Then $feed_backward = True

   If $GUI Then
      If $may_update_GUIdata Then
         GuiCtrlSetData($GUI_inp_correct , 0 )
         GuiCtrlSetData($GUI_inp_wrong , 0 )
      EndIf
   EndIf

   Local $treshold_counter = 0
   For $x = 1 To $runtimes

      If $backward Then $settings[24][1] += 1
      $settings[23][1] += 1
;~ cw("$settings[23][1] = " &  $settings[23][1])
      If $settings[23][1] > UBound($raw_target,1) - 1 Then $settings[23][1] = 1

      If $GUI And $feed_gui_settings Then GuiCtrlSetData($GUI_inp_total_runs , $settings[23][1] )

      $treshold_counter += 1
      If $treshold_counter > $treshold_counter_max Then
         $treshold_counter = 0
         Sleep($treshold_time)
      EndIf

      set_new_input()
      feed_forward()
      rap_report($x)
      If $backward Then feed_backward()

   Next


   $feed_forward = False
   $feed_backward = False
   rap_calculate($runtimes)

   ad($results,"$results" ,True)
EndFunc

 

expandcollapse popup

Func feed($backward = False)
   normelize_input()
   $M_Ninputs = _Eigen_CreateMatrix_FromArray ( $A_Ninputs, True )
   _Eigen_Multiply_AB_plusC($M_weights_ih1,$M_Ninputs,$M_bias_h1,$M_outputs_h1)
   _Eigen_CwiseScalarOp_InPlace($M_outputs_h1,"*",-1)
   _Eigen_CwiseUnaryOp_InPlace($M_outputs_h1,"exp") 
   _Eigen_CwiseScalarOp_InPlace($M_outputs_h1,"+",1) 
   _Eigen_CwiseUnaryOp_InPlace($M_outputs_h1,"inverse")

   _Eigen_Multiply_AB_plusC($M_weights_h1h2,$M_outputs_h1,$M_bias_h2,$M_outputs_h2)
   _Eigen_CwiseScalarOp_InPlace($M_outputs_h2,"*",-1)
   _Eigen_CwiseUnaryOp_InPlace($M_outputs_h2,"exp")
   _Eigen_CwiseScalarOp_InPlace($M_outputs_h2,"+",1) 
   _Eigen_CwiseUnaryOp_InPlace($M_outputs_h2,"inverse") 

   _Eigen_Multiply_AB_plusC($M_weights_h2o,$M_outputs_h2,$M_bias_o,$M_outputs_o)
   _Eigen_CwiseScalarOp_InPlace($M_outputs_o,"*",-1)
   _Eigen_CwiseUnaryOp_InPlace($M_outputs_o,"exp") 
   _Eigen_CwiseScalarOp_InPlace($M_outputs_o,"+",1)
   _Eigen_CwiseUnaryOp_InPlace($M_outputs_o,"inverse")

;~    _Eigen_CreateArray_FromMatrix( $M_outputs_o, $A_outputs_o )
;~    _ArrayDisplay ( $A_outputs_o, "$A_outputs_o" )

   If $backward Then
      set_target()
      $M_target_o = _Eigen_CreateMatrix_FromArray ( $A_target_o, True )
      $M_error_o = _Eigen_CwiseScalarOp ( $M_target_o, "-", $M_outputs_o );==== needs testing

      $M_grad_o = _Eigen_CloneMatrix( $M_error_o , False )
      _Eigen_SetOnes ( $M_grad_o )
      _Eigen_CwiseBinaryOp_InPlace( $M_grad_o ,"-",$M_outputs_o)
      _Eigen_CwiseBinaryOp_InPlace ( $M_grad_o, "*" , $M_error_o )
      _Eigen_CwiseScalarOp_InPlace($M_grad_o,"*",$learning_rate)
;~    _MatrixDisplay ( $M_grad_o , "$M_grad_o" )

;~    $MT_weights_h2o = _Eigen_CloneMatrix_Transposed ( $M_weights_h2o , True )
;~    $M_delta_o = _Eigen_DotProduct ( $M_grad_o , $MT_weights_h2o )
;~    _MatrixDisplay ( $M_delta_o , "$M_delta_o" )

   EndIf
EndFunc

The refactor code isn't finished yet

Edited May 2, 2018 by FMS

[RTFC]

1 hour ago, kylomas said:

Can you give a real life example of matrix math?

For simple answers as to why and where matrix math is useful, see for example here, here, and here. Some examples of applications  of linear algebra (which relies heavily on matrix math) are here.

For some practical examples in the current context, see the online E4A Tutorials on fitting a function through points   ("Regression"),  lossy compression ("PCA"), and classifying flower species ("LDA"). See also the intro webpage "Why Bother?" in the online Help. You don't need to download the E4A package, just reading the Tutorials and studying sample snippets should give you the general idea.

Basically, whenever you're dealing with large numerical data sets, casting them as matrices allows you to perform operations on them (or specific parts thereof, like a row, column, or block) in parallel (that is, affecting all cells in one call, but crucially, you don't need a parallel processor or supercomputer to do this). This means that the size of a particular problem no longer matters. For example, ATLAS particle detection at CERN involves terabytes of data, and uses the Eigen library (see Conclusions section in linked paper) to perform the statistics to extract the significant signature of hunted particles.

In your every day programming, if your code contains lots of (nested?) loops iterating over array cells to do any kind of math, it's probably worth investigating whether a matrix approach would be a more efficient alternative. It also helps to turn problems inside out, and there's some neat tricks in matrix land that can save you tons of time.

[jchd]

To add a little bit to what @RTFC just said, matrices are used in a huge number of domains. One of them you are more or less familiar with involves rotation in 2D or 3D spaces. A rotation in those spaces (as well as in higher dimensions) is usefully performed by a single matrix operation. Rotation in spaces beyond dimension 2 is not as simple as it looks like, because composition of 2 rotations there is no more commutative (the result depends on the order in which both are applied). Again a matrix can represent any rotation. Compared to the mess that Euler angles is, this is the winner path, also since matrix rotations in 3D do not suffer from the gimbal lock issue.

Furthermore, one can define a 4x4 matrix to perform a perspective transform of a 3D scene. Knowing it or not, doing so means using quaternions (which also have a matrix representation).

[FMS]

Here is also a real world problem

I've the (java) code below whis I'm trying to refactor into E4A.
In the end I need to have "weight_ho_deltas" .

// Calculate deltas
 let hidden_T = Matrix.transpose(hidden);
let weight_ho_deltas = Matrix.multiply(gradients, hidden_T);


  static multiply(a, b) {
    // Matrix product
    if (a.cols !== b.rows) {
      console.log('Columns of A must match rows of B.');
      return;
    }

    return new Matrix(a.rows, b.cols)
      .map((e, i, j) => {
        // Dot product of values in col
        let sum = 0;
        for (let k = 0; k < a.cols; k++) {
          sum += a.data[i][k] * b.data[k][j];
        }
        return sum;
      });

  map(func) {
    // Apply a function to every element of matrix
    for (let i = 0; i < this.rows; i++) {
      for (let j = 0; j < this.cols; j++) {
        let val = this.data[i][j];
        this.data[i][j] = func(val, i, j);
      }
    }
    return this;
}

Where i did before in autoit (whish is totaly wrong i think) :

For $k = 1 To UBound($error_o,1) -1
      $delta_o[$k][1] = normelize($grad_o[$k][1] * $outputs_o[$k][1])
   Next

I'm now trying to do whit :

$M_delta_o = _Eigen_CloneMatrix( $M_weights_h2o , False )
_Eigen_Multiply_ABt  ( $M_grad_o , $M_weights_h2o , $M_delta_o )
_MatrixDisplay ( $M_delta_o , "$M_delta_o" )

whish din't work and then tried :

$M_delta_o = _Eigen_DotProduct   ( $M_weights_h2o , $M_grad_o )

Afther that I tried :

$MT_weights_h2o = _Eigen_CloneMatrix_Transposed ( $M_weights_h2o , True )
      _MatrixDisplay ( $MT_weights_h2o , "$MT_weights_h2o" )
      $M_delta_o = _Eigen_CloneMatrix( $M_weights_h2o , False )
      _Eigen_CwiseBinaryOp( $M_grad_o , "*" , $MT_weights_h2o , $M_delta_o )
;~    _Eigen_CwiseBinaryOp_Rowwise ( $M_grad_o, "*", $MT_weights_h2o , $M_delta_o )
      _MatrixDisplay ( $M_delta_o , "$M_delta_o" )
;or whit :
;_Eigen_CwiseBinaryOp_Rowwise ( $M_grad_o, "*", $MT_weights_h2o , $M_delta_o )

I understand what is going on but don't know how to name it in E4A???

Does somebody know's how to calculate it / name in E4A ?

Edited May 2, 2018 by FMS