Calculation Mistakes in Large Arrays

[Dana86]

Func C(ByRef $a,$Label);Returns Index Num Col of Seek Label in Array
    Local $lr=UBound($a,1)-1
    Local $lc=UBound($a,2)-1
    Local $iLabel="None"
    For $c=0 To $lc
        $iLabel=$a[0][$c]
        If $iLabel=$Label Then Return $c
    Next
    Debug("[ERROR]Label Not Found! "&$Label)
    Return "[ERROR]"
EndFunc

Func ByCn(ByRef $a,$Row,$Label)
    Return Number($a[$Row][C($a,$Label)],3)
EndFunc

Func ByC(ByRef $a,$Row,$Label)
    Return $a[$Row][C($a,$Label)]
EndFunc

 

 

When I run complex calculations in large arrays derived from .csv files with ByC() (returned without Number()) I get calculation errors. The calculation errors (straight up bogus outputs) seem to be gone after running the returns with Number(). Normally these errors aren't there when running smaller and less complex calculations. I was running 16 parallel processors and calculations still take 30 minutes, when I add the Number() function calculations take up to 24 hours... Is there a faster alternative to Number() or a way to import the arrays as float values & bypass these calculation errors?

 

Thanks!

[RTFC]

Yes, E4A.

[Nine]

Although it is not clear what is your issue (since we do not have a replicable script to test on), there might be a possible solution using scripting dictionary (or maps - beta version).  Provide a full test environment if you want us to propose a precise solution...

[pseakins]

10 hours ago, Nine said:

Although it is not clear what is your issue

I suspect that Dana is trolling us.

[Nine]

1 hour ago, pseakins said:

I suspect that Dana is trolling us.

666 !

[JockoDundee]

9 hours ago, pseakins said:

I suspect that Dana is trolling us.

Fool me once, shame on me…

 

[Dana86]

On 8/10/2021 at 5:03 AM, Nine said:

Although it is not clear what is your issue (since we do not have a replicable script to test on), there might be a possible solution using scripting dictionary (or maps - beta version).  Provide a full test environment if you want us to propose a precise solution...

The data file is too large to upload into this forum. If there any functions that you need I'll provide them. But really I am just getting different calculation results between the use of with and without the use of the function Number(). The miscalculations are reproduceable across a large set of simulations I do, this is just one example. I get completely different results when I replace ByC() with ByCn(). This is a less complex simulation I do for forex, there are more complex simulations for equities that uses parallel processing. 

Edited August 12, 2021 by Dana86

[Dana86]

20 hours ago, Nine said:

666 !

 

22 hours ago, pseakins said:

I suspect that Dana is trolling us.

Not, the type of demeanor that results in a better community of developers???

[636C65616E]

Well, i'm sorry about that, maybe not the answer you're looking for, but dealing with high computation in AutoIt is not really the point of the langague.

So if i was you, i would just code stuff in C ... AutoIt is intended to be a Scripting/Automation language ... I guess some people gonna hate me for that haha

And if you're afraid of C (some people are), you can still get really good perf with any language oriented for calculus (Python, R, Maple, Matlab, ...) or perfs (Ada, Fortran, ... (but if you're afraid of C ... well hahahaha)) but not a scripting language ...

Also, interpreted languages usually s*ck for computation when not explicitly designed for it ...

Edited August 12, 2021 by 636C65616E

[Dana86]

12 hours ago, JockoDundee said:

Fool me once, shame on me…

 

I make my living with Algo-Trading and due to the nature of my work I can not share my code with everyone. 

[Dana86]

1 minute ago, 636C65616E said:

Well, i'm sorry about that, maybe not the answer you're looking for, but dealing with high computation in AutoIt is not really the point of the langague.

So if i was you, i would just code stuff in C ... AutoIt is intended to be a Scripting/Automation language ... I guess some people gonna hate me for that haha

Ya, C and C++ is super fast but its way slower & harder to write. I can write a few thousand lines of code on Au3 in a day vs only a few hundred lines with C or C++.

[636C65616E]

Just now, Dana86 said:

Ya, C and C++ is super fast but its way slower & harder to write. I can write a few thousand lines of code on Au3 in a day vs only a few hundred lines with C or C++.

Then keep praticing, you will end by coding at the same speed ...

[Dana86]

6 minutes ago, 636C65616E said:

Well, i'm sorry about that, maybe not the answer you're looking for, but dealing with high computation in AutoIt is not really the point of the langague.

So if i was you, i would just code stuff in C ... AutoIt is intended to be a Scripting/Automation language ... I guess some people gonna hate me for that haha

And if you're afraid of C (some people are), you can still get really good perf with Python or R ...

Thanks for being honest, I've tried R and Python the process of going from theory to simulation to implementation is much slower. Tho, I changed a few Au3 funcs to reflect python a bit to make my life easier coding between the two languages. 

Edited August 11, 2021 by Dana86

[Dana86]

11 minutes ago, 636C65616E said:

Then keep praticing, you will end by coding at the same speed ...

ByC() was meant to be like the python equivalent of df.loc[] with labels instead of col numbers. Makes experiments more robust, scalable and possible for future data manipulation. 

iProfit=round((Price-df.loc[r,'EntryPrice'])*df.loc[r,'Size'],2)      

[JockoDundee]

2 hours ago, Dana86 said:

I can write a few thousand lines of code on Au3 in a day…

That is fast typing, if nothing else…

3000 lines (few thousand) * 80 chars (avg chars per line) / 5 (avg chars in word) / 8 hours (avg working day) / 60 minutes (avg minutes in an hour) =

100 WPM

Impressive!

[Dana86]

1 hour ago, JockoDundee said:

That is fast typing, if nothing else…

3000 lines (few thousand) * 80 chars (avg chars per line) / 5 (avg chars in word) / 8 hours (avg working day) / 60 minutes (avg minutes in an hour) =

100 WPM

Impressive!

In an experimental data scientist type environment, there is a lot of copy and pasting to get things done faster. 99.9% of experiments fail, so rapid development is very important. Scalability and stability comes later after successful initial experiments. Some days I work for 16 hours on longer experiments. An experiment is like an idea, but a very long drawn out idea that takes many hours to develop and test. Most ideas are are versions of older failed experiments and they get reused.

[pseakins]

8 hours ago, Dana86 said:

I can write a few thousand lines of code on Au3 in a day

An experienced coder will write 60, maybe 80 lines of debugged code in a day. For each of those lines 50 - 100 may have been written, tested, edited and whole sections deleted. Copying and pasting untested experimental code is not exactly "coding" IMO. But, I am retired and earn nothing. You, if you are real and it looks like you may be, may be being paid over $100k so who am I to judge?

[JockoDundee]

12 minutes ago, pseakins said:

Copying and pasting untested experimental code is not exactly "coding" IMO.

I think it only fair to count the lines inserted by any new #include statement, recursively of course

[RTFC]

9 hours ago, Dana86 said:

C++ is super fast

and (at the fear of repeating myself) E4A relies on internal dlls written (by yours truly)  in C++, based upon Eigen and boost. Eigen is used extensively in science, engineering, robotics, and CG. And in x64-mode, data set size is limited only by your virtual memory. Yet to engage with it, you only need a few lines of AutoIt.

[Earthshine]

it's not the number of lines you write, it's what they do. C and C++ are FAR superior to what the OP is doing and not using the appropriate language is a huge mistake and waste of time/energy/money

the argument that you "get more done" is completely false. If you focus on C++ you will become expert in no time and whipping out solutions will be as fast or faster than using an autmation scripting tool.

Basically, in a nutshell--USE THE RIGHT TOOL FOR THE JOB. If speed and accuracy are your main considerations then don't use an interpreted language at all.

 

but if you must use AutoIt then use the UDF that @RTFC created.

Edited August 12, 2021 by Earthshine