Eigen4AutoIt - Matrix computing with Eigen

[RTFC]

Version 2.4 (finally) released.

Two main additions, one for beginners, one for advanced users:

The MatrixViewer GUI controls an Irrlicht-driven 3D real-time animated bar graph of your matrix of choice, allowing you to follow your own flightpath around it, or auto-circling. It is mainly useful for beginners, to visualise the structure of (relatively small) matrices, and impress casual onlookers with your stratospheric mathematical insights.

For advanced users, Eigen4AutoIt now offers integration with the Pool environment for multi-processing (by yours truly), allowing large calculations to be broken up into parts that can be computed in parallel by different processes. Tutorial, example scripts, and Pool environment are provided.

In addition, the dlls have been upgraded to Eigen version 3.2.4, and have now been compiled against the static runtime, so E4A executables now require no external dependencies other than the dlls themselves.

Edited May 21, 2015 by RTFC

[RTFC]

Version 2.5 released.

The dlls have been upgraded to Eigen version 3.2.7, see Eigen's own Changelog for details.

Otherwise, just some minor additions and internal changes (nothing earth-shattering, I'm afraid), listed in the History/Changelog page of the E4A Help file.

Happy New Year, everyone!

[argumentum]

Eigen4AutoIt: Function Selector: AutoIt3.exe - System Error
The program can't start because MSVCP71.dll is missing from your computer. Try reinstalling the program to fix this problem.

I copy the DLL to the folder where the other DLLs are at but it did not work

Solved: msvcp71.dll and msvcr71.dll were needed in the folder.

Edited December 27, 2015 by argumentum
Solved my issue

[RTFC]

Thanks a lot for letting me know, argumentum. That wasn't supposed to be necessary anymore though, so something went wrong in the dll compilation. This is the first version compiled on a new machine, and I guess my MSVC settings did not get ported properly. I will investigate and update the package, as it's supposed to work without any additional software. Always something to fix....

 

[RTFC]

@argumentum: Right, I've had a long look, and played around with the runtime library settings of Code Generation in MSVC, but I have been unable to identify any msvcr dependency. Here's what dumpbin reveals:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC>dumpbin /dependents d:\Autoit\Eigen\EigenDense.dll
Microsoft (R) COFF/PE Dumper Version 10.00.30319.01
Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file d:\Autoit\Eigen\EigenDense.dll

File Type: DLL

  Image has the following dependencies:

    USER32.dll
    KERNEL32.dll

That's it; no external dependencies whatsoever (I'm not counting the two windows kernel dlls, of course). Same result for the EigenDense_Debug.dll version.

Just as a test, I recompiled EigenDense.dll with /MD (size: 3.82 Mb) instead of the usual setting /MT (size: 8.41 Mb). That much smaller MD version does report msvcr dependencies (so it's not the case that dumpbin fails to detect this):

Microsoft (R) COFF/PE Dumper Version 10.00.30319.01
Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file C:\Users\RTFC\Documents\Visual Studio 2010\Projects\EigenDense\Release\EigenDenseMD.dll

File Type: DLL

  Image has the following dependencies:

    MSVCP100.dll
    MSVCR100.dll
    KERNEL32.dll

As a final test, I loaded the complete, original v2.5 E4A environment on a stick and ran it on a virgin machine without msvc*.dll, and with the latest version of AutoIt installed, and this ran E4A dll calls without a problem, without complaining about missing msvc*.dlls. So I'm stumped as to what is causing the specific problem in your environment, argumentum, I'm sorry. But as far as I can tell, the E4A dlls as I supplied them two days ago should be able to run independently, and their substantial increase in size with respect to MSVC's default /MD version (plus dumpbin's report) suggests to me that the object is statically linked in my /MT version. Until I can find a way to reproduce your issue I won't be able to hunt down the cause, I'm afraid.

Edited December 29, 2015 by RTFC
added link

[argumentum]

... played around with the runtime library settings of Code Generation in MSVC, but I have been unable to identify any msvcr dependency. ...

it only happens in FunctionSelector.* and nowhere else. The error about the DLLs popups twice, then it closes itself without any other errors.

Edit: The error is on IrrlichtWrapper.dll , _IrrStart() returns Seterror(2,0,False) , hope this narrows it down.

Edited December 29, 2015 by argumentum

[RTFC]

Ha! Yes, the irrlicht dll is at the moment just provided-as-is, for convenience sake, so you don't have to download it separately. I did get hold of the source at some point (it being open-source and all), so I suppose I could recompile it as a stand-alone, static resource for E4A, if I spent enough time ironing out various incompatibilities (I spent a stray afternoon on that once, but I hate tinkering with other people's code; I usually just write my own custom tools, which is how E4A came into being too).

Good point though, thanks, I never really thought about it, and never tested my FS module as stand-alone separately, only the core maths components. I guess one could argue that the FS should be able to work independently too. Maybe I'll look into it again, the Xmas break seems like a good time the only feasible opportunity for this in the near future. No guarantees though; I vaguely recall running aground due to some technical issue last time round, one that I was too lazy/lame-brained to fix at the time.

PS was going to sprinkle some emoticons, but that feature no longer seems to work for me at the moment.

Edited December 30, 2015 by RTFC

[RTFC]

An updated version 2.5 is now available, which should work without requiring additional msvcr* dlls.

I managed to recompile static versions of both IrrlichtWrapper and Irrlicht (in the updated bundle), but had to disable DX8 and DX9 support in the latter (tried various ancient SDKs to find a number of missing header files, to no avail; E4A uses OpenGL, so it's not needed here anyway). Ergo, these Irrlicht dll's are not for general Irrlicht usage, so please do not complain if you use them elsewhere and then discover that the DirectX 3D rendering doesn't work, because it's slightly absenterino.

I also spotted (and cleaned up) some stray test messages I'd accidentally left in the MV module.

Apologies to anyone who already downloaded v2.5...

Edited December 30, 2015 by RTFC

[argumentum]

I'm a pain .... , that's how I feel, but, anywayz.
The MatrixViewer.au3 did not display, it flashed something ( 1 mSec ) , so then, I run the MatrixViewer.exe and it told me that

---------------------------
Eigen4AutoIt Fatal Error
---------------------------
Unable to initiate Eigen environment: dll path is undefined or not found:
C:\Users\Jonkers\Documents\Visual Studio 2010\Projects\EigenDense\Release
Please first define the full path where the EigenDense dll files are stored, in environment variable $EIGEN_DLLPATH in Eigen4AutoIt.au3, section Globals. Thank you.
---------------------------
OK   
---------------------------

and the way I did this in my Eigen4AutoIt.au3 to avoid the fixed path was to

Global $EIGEN_DLLPATH="Enabling Eigen4AutoIt to function locally constitutes your acceptance of all terms and conditions of the license agreement (see .chm Help file for details)."

If Not FileExists( $EIGEN_DLLPATH &'\EigenDense.dll' ) Then $EIGEN_DLLPATH = FileGetShortName(@ScriptDir)
If Not FileExists( $EIGEN_DLLPATH &'\EigenDense.dll' ) Then $EIGEN_DLLPATH = FileGetShortName(@ScriptDir&'\..')
If Not FileExists( $EIGEN_DLLPATH &'\EigenDense.dll' ) Then $EIGEN_DLLPATH = FileGetShortName(@ScriptDir&'\..\..')
If Not FileExists( $EIGEN_DLLPATH &'\EigenDense.dll' ) Then $EIGEN_DLLPATH = "Enabling Eigen4AutoIt to function locally constitutes your acceptance of all terms and conditions of the license agreement (see .chm Help file for details)."

also found a fixed path in MVinclude and change it:

Local $exitcode=RunWait("MV.exe " & $paramstring,@ScriptDir)
;Local $exitcode=RunWait("D:\AutoIt\Eigen\MV.exe " & $paramstring)

so, recompiled MV.EXE and MatrixViewer.exe
Everything works.

And, I haven't say thank you, so,
Thank you, this is beautiful.

 

[RTFC]

@argumentum: you're not a pain, argumentum, you helped to improve the E4A environment, so thank you!

Re. MV's fixed path, I was recently changing that module's internals, and I guess I forgot to remove my own hard-coded patches (you may have noticed the remark "; TESTING <<<<<<<<<<<<<<<<<<<" right above it; the original line was immediately below it, commented out, but your own fix does the trick too, of course). I'll try to be more careful next year (but you know how it is with new year's resolutions...).

MV/MatrixViewer's reliance on Eigen4AutoIt.au3 also means that I cannot supply those as precompiled exes in future, hmmmm...

Re. $EIGEN_DLLPATH, this is intentional. I require the user to go into Eigen4AutoIt.au3 once and edit that path manually, so the copyright message has to be consciously encountered. You can store an explicit path there instead, or do something fancier, as you did. But this one is not an oversight.

And it's very nice to hear that you like E4A, much appreciated; I spent a lot of time figuring out its fundamental structure before coding a single line, which is quite different from Eigen's own approach in some respects (no vector definitions, no dinstinction between arrays and matrices, coming up with consistent, predictable function names, alias wrappers, and so forth).

Hope it is helpful in some way (see the Tutorials and test scripts to get a handle on the possibilities).

PS re-patched v2.5 uploaded.

Edited December 30, 2015 by RTFC

[bogQ]

I managed to recompile static versions of both IrrlichtWrapper and Irrlicht (in the updated bundle), but had to disable DX8 and DX9 support in the latter (tried various ancient SDKs to find a number of missing header files, to no avail; E4A uses OpenGL, so it's not needed here anyway).

To recompile for directx9 youl need to instal Directx9 SDK (try google-ing "Microsoft DirectX SDK (June 2010)" around 500-600mb)

I do not know the procedure for Frank Dodd irrlich project to add DX9 support or for your c++ project and its editor that your using, but in "Visual C++ 2008 Express Edition" you only had to install DX9 SDK somewhere before compiling and that is all, no settings needed in project no nothing, and DX9 option worked with defolt irrlicht files.

 

Ouh btw if you need a solo project so that you can update and code your dll for irrlicht, look in my sig for oIrrlicht small starter guide project and start playing with irrlicht dll directly.

 

Edited January 2, 2016 by bogQ

[RTFC]

Hello BogQ, nice to meet you,

Firstly, thanks for your efforts on the Au3 Irrlicht project, great job.

Yeah, I got the SDK for DirectX9 (and using MSVC10), but the Irrlicht source I've got still has many unresolved externals and missing includes for DirectX9 even with that SDK included and with support for DirectX8 explicitly switched off in IrrCompileConfig.h, so after tracing a few other internal flags and getting nowhere fast (the Irrlicht source is pretty big, as you know), and given that I only use OpenGL anyway, I totally chickened out and bravely ran away-away. <a smiley would be here if inserting them had worked for me today; yesterday insertion did function; is the forum software acting up again, or is it my new environment? Weird.>

Perhaps my Irrlicht source would benefit from an update, but I fear because Dodd's wrapper dll is fairly old there may be backward compatibility issues arising, as I read in the Irrlicht Changelog that quite a few functions now work rather differently. It could also be due to the importing of the msvc9 solution to msvc10, perhaps something got mangled or lost? I don't have msvc8, nor do I really wish to go back there.

Is there maybe a particular version of Irrlicht source that is complete and works with the IrrlichtWrapper.dll? I don't really fancy rewriting that entire wrapper library (been there, done that, for Eigen) to make it work for the latest Irrlicht release. Also, the 3D part is mostly eye-candy in this project, and doesn't have to feature the fanciest latest-GPU-ready particle system and physics engine. But I'd be interested to know what the status of the Au3-Irrlicht marriage of convenience is these days.

Edited January 2, 2016 by RTFC

[RTFC]

Bug patch pending next release

in Eigen4AutoIt.au3, Func _Eigen_CreateMatrix_FromArray circa line 6900, please add "Local " to the $matA declaration under Switch $dims, Case 1. So please replace the last line of this section:

Switch $dims
        Case 2
            $rows=UBound($array,1)
            $cols=UBound($array,2)
            Local $matA=_Eigen_CreateMatrix($rows, $cols, $matrixtype)
            If @error Or $matA=-1 Then Return SetError(-1,0,False)

        Case 1
            $rows=UBound($array,1)
            $matA=_Eigen_CreateMatrix($rows, 1, $matrixtype)    ; 2D matrix!

with this:

Switch $dims
        Case 2
            $rows=UBound($array,1)
            $cols=UBound($array,2)
            Local $matA=_Eigen_CreateMatrix($rows, $cols, $matrixtype)
            If @error Or $matA=-1 Then Return SetError(-1,0,False)

        Case 1
            $rows=UBound($array,1)
            Local $matA=_Eigen_CreateMatrix($rows, 1, $matrixtype)  ; 2D matrix!

This bug affects you if you use variable $matA in your own scripts as a global; its ID was inadvertently reassigned globally instead of locally. Apologies for any inconvenience caused. This fix will be implemented in the next regular release.

[RTFC]

E4A version 3.1 released.

This is a major upgrade. Among the most notable changes are:

• The Eigen environment used in the new dll has been upgraded to version 3.3.3, which itself has undergone a significant redesign, allowing for faster processing in essential areas of dense matrix computations.
• E4A's DebugMode is gone, and with it all Debug/Computing mode-related environment functions, and the large secondary dll. Almost all basic mistakes one can make are already tested for in the wrapper library, and the overhead in storage and execution slowdown was just becoming ridiculous. So if you wish to keep relying on Eigen's own assert messages, do not upgrade to version 3.1 or higher; all other users will enjoy a much leaner (21.3 MB), faster work environment.
• The Statistics module now supports Pearson product-moment correlation, both Colwise (standard) and Rowwise (mostly to please researchers in genomics).

For other changes and fixes, see the History/Changelog page on the E4A website or in the Help file.

Edited April 9, 2017 by RTFC
major version number increment

[RTFC]

E4A version 3.2 released.

Another upgrade, this time mostly involving actual UDF capabilities:

• new CwiseUnaryOp functions (rounding-, log-, power-, and trig-related)
• _Eigen_RanSeed() provides a user-defined randomising seed for E4A's Random functions (NB this has nothing to do with AutoIt's SRandom function)
• two cell-specific Swap functions
• new EigenTest_Inversion (#16)
• Transposed matrix File I/O (NB this is a script-breaking change)
• dll now has flag /LARGEADDRESSAWARE set; performance may vary
• several bug fixes in CwiseUnaryOp and CwiseBinaryOp functions

Edited May 2, 2017 by RTFC

[RTFC]

E4A beta version 4.0 released.

Are you sitting comfortably? Because this is going to take a while. This is E4A's biggest overhaul to date. I'll spare you the technical details (see ChangeLog), but users switching to this version (and future upgrades thereof) will notice some big improvements (providing it all works as it's supposed to ), notably:

• a user-friendlier set-up: E4A now relies on a simple INI file (Eigen4AutoIt.ini) for various operational settings, including variable $EIGEN_DLLPATH (full path where the dlls are located). So you never have to patch the source any more (unless you want to); just open the INI file in your favourite text editor to change settings, Save, and done. If absent in the local work directory, it will be generated with default values. Furthermore, several new functions allow you to query basic specs (dims, size, type) of matrices on file (without loading them first) and in memory. And if you've forgotten the exact name of a function, the number of alias wrappers is now close to one thousand. Finally, new flag $EIGEN_Verbose allows for status messages to be sent to console, especially in conjunction with the next item:
• HighMem integration: in x64 environments (machine+OS), AutoIt scripts running in x64-mode (by adding directive #AutoIt3Wrapper_UseX64=Y) will pre-allocate a user-defined amount of virtual memory at initialisation (see new operational variable $EIGEN_ALLOCATE_RAM_GB, also found in the INI file) for storing matrices. That means matrices can now be any size that fits into your virtual RAM (NB this is usually much more than your physical RAM; for example, I have about 128 terabytes (yes, TB) on my dinky old laptop). The usual limit of 2/3 GB per Windows process is thereby lifted. This is made possible by:
• full x64 support: new dynamic link library EigenDense_x64.dll runs in native 64-bit. That means 64-bit registers and memory pointers, and faster processing. Dll usage is automatically selected based upon the contents of macro @AutoItX64 at startup; just add the x64 directive at the top of your script, and you're set.
• Eigen's own version is upgraded to stable release 3.3.4
• Gaussian statistics: probabilities, Z-scores, p-values, one-/two-sided, all you need to determine likelihoods, confidence intervals, and significance, given an underlying normal distribution. New Test script provided.
• Extensive Sort capabilities (see section "Transformation" in the Help): full matrix, row, col, diag, block, rowwise, colwise, and results stored either _InPlace or to new output. Courtesy of the boost libraries, so blazingly fast.The associated new test script compares _ArraySort with converting array to matrix, E4A-Sort, and convert back, which will be faster for sufficiently large arrays.

Naturally, many aspects still await (more) thorough testing, reason why this is a beta release, and all the usual caveats apply. As the license states, do not use E4A in critical infrastructure or high-risk activities. Other than that, feel free to explore these new (and existing) capabilities to the full (I certainly am). Enjoy.

Edited July 24, 2017 by RTFC

[RTFC]

E4A version 4.1 is here. Its most prominent new features are:

• Integration with _HighMem v0.9 (included) means that shared memory parallelisation is now possible (on x64 only!), meaning multiple processes on the same machine can now read from and write to the same matrix (either with or without mutex negotiation, depending on whether race conditions are an issue). At initialisation, some form of IPC is still required to parse matrix specs and relative offset(s) in memory to co-workers (NB not on my to-do list for E4A). This is still very much in early development (as is _HighMem itself), but the latest E4A Test script (#20) already provides proof of concept.
• Linear Discriminant Analysis (LDA), for 2-8 discrete classes and unlimited parameters, with a shiny new Tutorial to boot. This is a tried-and-tested machine learning algorithm for classifying fuzzy multi-dimensional data clouds into distinct groups. Supports training and testing sets, including the option to classify other data sets later on based on criteria derived from the training sets. I use it to sort my sock drawer. Speaking of which...
• Sort_Unique and Freq functions, to construct a sorted Colvector of all unique values found in a matrix, and using that as input, to derive a frequency distribution. All usual matrix part suffices (Col/Diag/Row/Block) are supported.

[RTFC]

E4A version 4.2 is released today (currently 474 public functions), with a major extension of _Eigen_LDA's functionality (see also previous post), now supporting up to 32 classes, extensive diagnostics (if $EIGEN_VERBOSE=True), and optionally using the new open-source Spectra Eigensolver library by Yixuan Qiu. Additionally, new Translate functions perform lookup-replace with a user-supplied table; more specifc cell-targeted ops (Inc/Dec/Add/Sub/Div/Mult); more error checking; conditional mask functions. See the full list in the ChangeLog (in online Help and .chm in the bundle).

Thank you for flying E4A, handling matrices the easy way.

Okay then.

PS: Of course within days of releasing this, I discover a weird edge case affecting Copy_Acol_ToBcol and Copy_Arow_ToBrow, which means that sufficient-size checks can under unlikely circumstances cause a boundary violation issue in Eigen itself (which I cannot fix). So the bundle is patched, and if you're using these two functions a lot, you may benefit from downloading it again. Only for die-hard single Col/Row-copiers; the rest can probably survive till the next regular release. Just thought I'd mention it. Apologies for any inconvenience caused.

PPS: When it rains, it pours. Another week, another bug squashed. For those patch-happy people that cannot wait for the next release, in Func _Eigen_ConditMask_InPlace, please edit the first line to read:

Local $procname="_Eigen_ConditMask_InPlace"

This may seem trivial, but (the tail end of) $procname is actually part of the dll call, and the (formerly) missing "_InPlace" suffix of this variant means the parsed parameters were interpreted incorrectly by the dll, causing a hard crash. Oh well, we live and learn...

PPPS: ...and die a little every time. Still on the subject on Func _Eigen_ConditMask and all its variants, further tests have revealed a deeper issue (in the way Eigen's .select() operator apparently cannot handle fixed values as const parameters), so my advice at this point is to stop using all ConditMask calls for now (they can easily be circumvented by three consecutive Cwise replace calls, which is of course a bit slower), until all kinks have been smoothed out. We are currently working on improving your E4A user experience. Just so you know.

Edited May 11, 2018 by RTFC

[RTFC]

E4A Version 4.3 is released today. New features include:

• _Eigen_Find and _Eigen_FindAll, returning either cell offset(s) or row & col coordinates of a specified value, plus associated funcs _Eigen_ConvertOffset_ToRowCol & _Eigen_ConvertRowCol_ToOffset; to locate cells satisfying other conditions, use _Eigen_ConditMask first, and use its output as input here (finding value unity (1))
• _Eigen_InsertRows/Cols and _Eigen_DeleteRowsCols, to change the dimensions of an existing matrix
• _Eigen_GetMin/MaxValue, _Eigen_GetRow/Col_ofMin/MaxValue wrapper functions
• _Eigen_CreateMatrix_Random_Integer & _Eigen_SetRandom_Integer (also works in integer matrices)

Hope it helps.

Edited June 11, 2018 by RTFC

[RTFC]

E4A version 4.4 has arrived.

The name of the game this time is bitmask support (see _Eigen_ConditMask*, now fully functional), notably:

• a new family of CwiseLogicalOp functions with which to perform BitAND, BitOR, BitXOR, and BitNot, with cell values internally (temporarily) converted to signed integers;
• masked-Set functions _Eigen_SetMaskedConstant*, for filling with a constant value only those cell/row/column contents for which a corresponding bitmask matrix cell contains unity; with new test script;
• masked-copy functions _Eigen_CopyMasked*, for copying only those cell/row/column contents for which a corresponding bitmask matrix cell contains unity; with new test script.

Other new addenda include:

• conditional cellwise functions _Eigen_ConditAll_Block/Colwise/Diag/Rowwise and _Eigen_ConditAny_Block/Colwise/Diag/Rowwise;
• transformation functions _Eigen_Clamp*, to limit values to a specified range; with new test script;
• transformation functions _Eigen_Transpose_Vector/InPlace, for converting between RowVector and ColVector shapes (content unchanged);
• real and integer variants of _Eigen_CwiseScalarOp now also support quotient ("quot," specID 8) and modulo ("mod," specID 9);
• alias wrappers for quickly retrieving the sum and mean value for full matrix and all default parts: _Eigen_GetSumValue/Block/Col/Diag/Row and _Eigen_GetMeanValue/Block/Col/Diag/Row.

More details in the ChangeLog (see Help file/online).