Eigen4AutoIt - Matrix computing with Eigen

[RTFC]

E4A Version 4.5 is released.

The good news: These new features:

• _Eigen_WriteMatrix_ToTextFile, with five predefined formats (default: csv) and optional user-defined precision.
• bitmask versions of cellwise functions _Eigen_CwiseBinaryMasked, _Eigen_CwiseScalarMasked, and _Eigen_CwiseUnaryMasked, with new test script (#28).
• a fifth cellwise logical operation "Flip," which replaces all non-zero values within scope with zero and all zero values within scope with unity
• a major extension of _Eigen_LDA, now also supporting Bayesian linear and quadratic discriminant analysis
•  Eigen includes for dlls upgraded to stable release version 3.3.5.

Additional details can be found in the Changelog, as usual.

The bad news: Due to compiler limitations (compilation time has mushroomed to take over fifty hours(!) per platform), the dll source has had to be completely re-organised and repartitioned, causing the final output to more or less double in size. I've unsuccessfully tried various alternative compilers and different settings, but those that download this latest version will have to live with this for now.

Furthermore, this will probably be the last release for this year, as my summer break is ending soon. Nevertheless I'll do my best to keep this environment evolving when and where I can...

Edited September 4, 2018 by RTFC
typos

[RTFC]

E4A version 4.6 is released.

This is a minor update, due to Eigen development over the past year seemingly being stagnant.

Various E4A Cwise functions have had Block/Diag variants added where missing, there's a new Show_Matrix function that writes matrix contents to console, and all external (open-source) template libraries have been updated to their latest stable release. The core library now contains 596 functions.

Edited May 29, 2020 by RTFC
typo

[RTFC]

E4A Version 4.7 (the "Lockdown Release") is out. Get it now before stock runs out!

This is a major update, thanks to having to spend way too much time in pandemic lockdown. The library now features over six hundred Eigen functions, over sixteen hundred alias wrappers, and well over two and a half thousand dll functions. This update beefs up integer functions in particular, notably in the context of bitmasks. So for once, no abstruse advanced mathematics.

Some highlights:

• compiler, platform toolkit, and Windows SDK upgraded (long overdue); as a result, the dlls are no longer suffering from elephantiasis
• complete overhaul of CwiseLogicalOp functions, with new operators "nor" and "xcl" ("exclude"), and redundant operator "flip" removed (See the shiny truth tables below);  also a new flag (see _Eigen_SetLogicalBit0only) to control whether these ops affect all (32) bits or just bit 0. I've also created an example script to illustrate how, in combination with bitmask functions (see _Eigen_ConditMask and variants, which now all have _InPlace counterparts as well), these can be leveraged to approach computationally hard, but mathematically simple problems. Output matrix dimensions have also been standardised where possible between all Cwise*Op and ConditMask functions.
• more vector support, including _Eigen_CreateCol/RowVector, _Eigen_IsVector, and _Eigen_CreateVector_FromAcell_Mask, which collects from an input matrix only those cell values that are masked in a separate bitmask matrix. In combination with _Eigen_FindAll (producing offset, row, and column of a targeted value) applied to the bitmask (so target value = 1), all relevant data satisfying a (simple or compound) bitmask condition can now be collected for further processing;
• most MatrixSpecs_Single functions (in section: Matrix Reduction) can now be accessed more easily through their own dedicated function; for example, _Eigen_IsZero, _EigenIsOnes, and _Eigen_GetSum have already proven to be timesavers that also make the code more easily understandable; the same goes for new functions that specifically retrieve the minimum/maximum value or their row/col.
• many matrix creation functions that use an existing matrix as input and result in a changed dimension now return that new size in macro @extended; likewise _Eigen_FindAll and _Eigen_Sort_Unique now return the number of results found in @extended.
• many corrections and clarifications in the online/chm Help pages.

For full details, please consult the download page.

Edited June 12, 2020 by RTFC
link added

[RTFC]

After almost six years, celebrating one thousand downloads of E4A today!

Edited June 1, 2020 by RTFC

[RTFC]

E4A version 4.8 is released, dedicated to @argumentum for digging into the @scriptlinenumber macro. As a result, all user-exposed E4A functions now return the main script's line number when an E4A error is triggered, when running uncompiled scripts. Other features:

• matrix Reduction functions substantially expanded
• new functions for reversing the order of all rows or all cols (content inside each row/col remains untouched)
• sorting all rows or all cols based upon the content of a single col c.q. row

Again, the focus has been on strengthening basic matrix management and specs retrieval, rather than fancy maths. An application example is forthcoming soon published in my BitmaskSudokuSolver thread (v1.1)

Edited July 4, 2020 by RTFC

[RTFC]

E4A version 5,2 is available for download: the "2020 Hindsight" release.

This is a maintenance upgrade, fixing many small issues (and the odd catastrophic bug ), filling some consistency gaps in the function library, streamlining the environment functions, and updating Eigen's source to stable release 3.3.9. Full details can be found in the ChangeLog as per usual. Hope it helps!

EDIT: as every mission has at least one glitch, it's no surprise that the updated MatrixFileConverter.au3 script did not get copied to the bundle subdir before packing it into the setup. So since it's tiny, I'm providing it separately here:

 

EDIT2: as every mission apparently has at least two glitches, I Just discovered a potential issue in _Eigen_StartUp when calling _Eigen_DefineWorkspace repeatedly in x64-mode (luckily this is not a common scenario). See spoiler for a quick patch.

Spoiler

In _Eigen_StartUp: change this:

; in case of repeat calls
    If $releaseAll = True Then
        _Eigen_Cleanup()
        If @AutoItX64 Then _HighMem_ReleaseAll()
    EndIf
    If $EIGEN_DENSE_DLLHANDLE > 0 Then DllClose($EIGEN_DENSE_DLLHANDLE)

    If @AutoItX64 And $EIGEN_ALLOCATE_RAM_GB <> _Max(1, Number(IniRead($E4AIniFilename, "Operational Settings", "EIGEN_ALLOCATE_RAM_GB", $EIGEN_ALLOCATE_RAM_GB))) Then
        _HighMem_CleanUp()
        _HighMem_StartUp($EIGEN_ALLOCATE_RAM_GB, "GB")
    EndIf

into this:

; in case of repeat calls
    Local $newAlloc = ($EIGEN_ALLOCATE_RAM_GB <> _Max(1, Number(IniRead($E4AIniFilename, "Operational Settings", "EIGEN_ALLOCATE_RAM_GB", $EIGEN_ALLOCATE_RAM_GB))))
    If ($releaseAll Or $newAlloc) Then _Eigen_Cleanup()
    If $EIGEN_DENSE_DLLHANDLE > 0 Then DllClose($EIGEN_DENSE_DLLHANDLE)

    ; (re)start
    If (@AutoItX64 And $_HighMem_Active=False) Then _HighMem_StartUp($EIGEN_ALLOCATE_RAM_GB, "GB")

 

EDIT3: the aforementioned patches have now been incorporated into v5.2a (setup re-uploaded). Apologies for any inconvenience caused.

Edited February 2, 2021 by RTFC

[RTFC]

E4A Version 5.3 is finally out.

This is a long-overdue major upgrade, as the library now supports ZLib compression for real and integer matrices stored on file (complex matrices are still always stored as raw binary data). Reloading compressed matrix files is transparent (existing functions are used; the dll automatically redirects to the decompression versions of the function if the compressed-flag is set in the matrix file header (bit 8 in the variable type ID field). Saving matrices individually in compressed form is done through new functions _Eigen_SaveMatrix_Compressed and _Eigen_SaveMatrix_Transposed_Compressed, but alternatively, you can opt to always automatically compress when saving by setting new global flag EIGEN_ALWAYSCOMPRESS to True (stored in Eigen4AutoIt.ini) , through _Eigen_SetAlwaysCompressed. Under the hood, de/compression actually consists of two different algorithms, a fast one for x86 (for small-sized matrices, using a dynamically allocated work buffer, so it may fail if it runs out of memory when trying to de/compress a huge matrix) and a slower one in x64-mode that can handle matrices/files of any size (including > 4 GB). So if you start getting out-of-memory errors when using E4A compression on x86, it's high time to switch to x64-mode if you can. See new test scripts _EigenTest_29a/b_FileCompression.au3 for examples.

You can confirm the integrity of your data before compression vs. after decompression with new File I/O function _Eigen_GetMatrixHash_MD5, which returns the boost library's implementation of MD5 for any E4A matrix in memory. Note that the hash is computed over the matrix contents only; the matrix file header does not reside in memory and is thus excluded here (so computing the matrix file MD5 with a different utility would produce a different hash).

Other new goodies include Transformation functions _Eigen_ClampValue*, to replace all values within a defined range with a single constant, and _Eigen_Rescale*, to replace a given linear range of values with a newly-defined range. Furthermore, in the C(ell)wise sections, there is now support (in CwiseUnaryOp*) for two-way rad/deg conversion and (in CwiseScalarOp*) for new bit-wise functions (for integer matrices only!) shift-left/right ("shl", "shr") and rotate-left/right (the latter also for the LSB/LSW part of values), i.e., "rol", "ror", "rol8", "ror8", "rol16", "ror16". Relevant Cwise test scripts have been updated as well, switching to an integer work environment when demonstrating CwiseScalar operators.

Finally, the Eigen template library itself has been upgraded to latest stable release version 3.4.0.

For the complete list of additions, changes, and fixes, see the History /ChangeLog page in the Help as per usual. I should mention that the online version of the new Help will only be upgraded to v5.3 in the near future, hopefully within the coming week. However, the installer includes a .chm version of the same document in the main Eigen4AutoIt directory.

 

EDIT: and also as per usual, I managed to upload a version without the new rad/deg unary operators enabled, so please download again if you just did. Apologies for the oversight (I briefly moved development to another machine, and then the two versions diverged, and you know the rest...)

Edited May 9, 2022 by RTFC
update to the update

[RTFC]

E4A Version 5.4 is out. The most important new features are:

• File I/O function _Eigen_LoadMatrix_FromTextFile, for fast-loading CSV- and related types of text files into a new E4A matrix. This finally complements existing function _Eigen_SaveMatrix_ToTextFile. It performs limited prior analysis to determine number of rows and columns (or fields), as well as the delimiter used that separates the values; you have to specify yourself how many header lines should be skipped before reading data. Your mileage may vary (should it fail, then the MatrixFileConverter utility may still do the job, albeit far more slowly). Note that complex data expect an even number of data rows, with the first half containing all real parts, immediately followed by the second half containing all imaginary parts.
• Transformation functions_Eigen_Pack/Unpack for de/compressing bit0-only bitmasks to/from full integers (all bits used), which speeds up file I/O and all logical operations by a factor 32.
• Transformation function _Eigen_Rotate, which rotates a matrix by any integer multiple of 45 degrees. When applied to matrix parts (other than _Diag), the cells themselves are rotated (with wrap-around) within the chosen subcontainer (Block, Col, or Row).
• Reduction function _Eigen_GetVectorAngle, to obtain the angle (in radians or degrees) between two vectors (0-180 degrees inclusive, meaning it also distinguishes between parallel and anti-parallel vectors).

Hope it helps.

Edited May 29, 2023 by RTFC

[noellarkin]

Noob question: what python library would this UDF be most similar to/drop-in replacement for?
I used it for cosine similarity a few months back (I was using sentence-transformers py library before) but this one seems a lot more lower level (dealing directly with matrix operations etc).

[RTFC]

Hello again. If you scroll down on Eigen's homepage you'll find a list of Bindings for other languages, including three for Python/numPy: miniEigen (this looks long-dead, actually), Eigency (a bare-bones Cython interface AFAICT), and Eigenpy (probably your best bet; anaconda install here). As I don't use Python myself, I can't really say which is closest to E4A, but from the summary descriptions I gather that none would provide anywhere near the same scope (although Eigenpy offers various matrix decomposition calls, the geometry module, and some vector support). Not sure if that would float your boat, but maybe it's worth a shot.

If you have the time and inclination, you could of course write a Python wrapper library for the E4A dll's yourself (or only the funcs you need), using Eigen4AutoIt.au3 as a template (in fact, much of this could probably be automated, or done by AI). That would preserve this UDF's function naming convention (with predictable matrix part suffices, etc) that I designed to make Eigen more user-friendly. In that case I can make the C++ header files available to you (although the DllCall parameters in the .au3 should contain all info you need).

[noellarkin]

Thanks for the detailed response :) I'm still pretty wet behind the ears when it comes to anything ML related, but I'm really interested in utilizing AutoIt as much as possible, since all my other scripts are in AutoIt. That's primarily why I checked out this UDF.

4 hours ago, RTFC said:

As I don't use Python myself, I can't really say which is closest to E4A

I only mention python because I know about the ML packages that it offers. I'm not partial to using it at all, I'd much rather have my entire codebase in AutoIt :)
I'm curious what languages you usually work with? Most people I've come across working in ML are using Python.

Can you share some applications/use cases of this UDF with autoit that're out in the wild/used in production?

[RTFC]

On 8/27/2023 at 2:33 PM, noellarkin said:

I'm curious what languages you usually work with?

My (computer-related) background is in science (mostly forward and inverse modelling, and nonlinear statistics), not ML. At the moment I mainly use C++ (with boost) and AutoIt, the latter often for fast prototyping with E4A, before rewriting in C++, formerly with CUDA (parallel processing on GPU), lately more with Unreal Engine (for 3D visualisation in real-time). In the past I've often used assembly and various flavours of fortran.

I saw your new matrix thread (great! ) In your first few examples I noticed you fill each cell of your example matrix individually. If the contents don't matter, you could do this more easily/quickly with _Eigen_CreateMatrix_LinSpaced_Colwise(), for example (if starting at zero, cell content would match offset). If the contents do matter, you could fill an AutoIt array first and load that into a matrix (maybe more intuitive for AutoIt users):

Local $arrayR,$arrayA[3][3]=[[1,2,3],[4,5,6],[7,8,9]]
_ArrayDisplay($arrayA,"array A")
$matA=_Eigen_CreateMatrix_FromArray($arrayA)
_MatrixDisplay($matA,"matrix A")

There's unfortunately no way in AutoIt to immediately fill an E4A matrix with an initialising constructor like that (in C++ you can, of course, but then you lose all AutoIt advantages of not having to compile first, error/bounds checks, etc). To be honest, E4A was never designed for hardcoding initialisation data; usually one would load (large, binary, possibly compressed) data files directly into matrices.

There are also many quick AutoIt code examples in the \EigenTest subdirectory of the E4A environment that illustrate specific subsets of functions (Set, Copy, Cwise functions, Swap, Find, Sort, File I/O, bitmasks, decompositions, stats). Feel free to copy from those for your own examples thread, and let me know if you need further info.

Edited September 2, 2023 by RTFC

[RTFC]

E4A Version 5.5 is released.

This is a minor update, mainly providing more support for binary operations on matrices of integer type, notably in file I/O, bitwise & logical operations on a single matrix cell value, CwiseScalarOps with logical operators (+ parsed value), reversing the bit-order in all cells of a matrix (part) with new CwiseUnaryOp operator #37 ("reverseBits"), and Rowwise Pack/Unpack functions for converting integer cells to/from 32 individual bits, 4 bytes, or 2 words. A new test script (#32: BitAndByteOps.au3 in the .\EigenTest subdirectory) illustrates various features. Full details can be found in the History page of the online Help (note that as of this version, the .chm Helpfile is no longer supported/present; just download the latest online Help if you need an offline version). Hope it helps.

Edited November 24, 2023 by RTFC

[noellarkin]

hey, this thread doesn't get much love, but just wanted to say thanks for this, I use the vector operations (cosine similarity, primarily) to compare embeddings and do RAG (retrieval augmented generation) with LLMs