Calculation Mistakes in Large Arrays

[JockoDundee]

20 hours ago, Dana86 said:

If there any functions that you need I'll provide them.

Why did you pull your code off, 5 minutes ago?

[Skysnake]

Hey@Dana86 just so you know I make my living off other people's secrets

As to community of developers, this is a volunteer forum and if you can't show us yours, then expect to be left in the dark.

As far as your HUGE CSV files are concerned, how big is big? And how big are those numbers?

I suspect you are trying to mine bitcoin using an AutoIt script. Really,  I  am not sure of that is the right tool.  To validate your first concern, I, too experienced funky calculation errors with AutoIt, and simply assumed I was using the wrong tool, or the tool in the wrong way. So, yes I added Number() to almost anything, and then I dumped all my data (also a lot incoming from CSV) into SQL. I would really like to know how you read those CSV files? Second, SQLite is handy but since it is dynamically typed, it may cause problems. A good solution for big numbers is still SQL but probably one of the larger RDBMSs - MySQL or Postgres. Look at things like bigint if integer, if you are dealing with currency, perhaps unusual data types like money will work for you.

 

Edited August 15, 2021 by Skysnake

[markyrocks]

I will agree c++ is blazing fast.  Without going into too much details about what I've been doing lately (I suspect the outcomes maybe related) I'm making a c# program that downloads text files generally around 700kb each.   I send the file to a c++ dll and it sorts through it and generates a new file based on what I want.   The biggest bottle neck is the download time bc the c++ sorts though a file that size faster than I can download it.  That new file then gets pushed to a class that performs a number of calculations etc and writes reports and saves it, actually 5 different reports based on the number of entries.   It takes like 20 seconds to do 20 files it on a 5 yr old laptop.   I could convert your posted code to c++ in about 5 minutes.  Pm me I'll help you out.  

Edited August 16, 2021 by markyrocks

[JockoDundee]

3 hours ago, markyrocks said:

The biggest bottle neck is the download time bc the c++ sorts though a file that size faster than I can download it.

What’s your download speed?

[markyrocks]

2 hours ago, JockoDundee said:

What’s your download speed?

Lol well dude I know you think you're cute so I'll play along.   When I click da button that starts this series of events in motion the way c# works with webclient object and asynchronous file downloads it assigns a thread to each download.   My laptop is a few years old and it probably doesn't like that very much. But the way that the program is designed said thread calls a on download complete event from that callback my dll functions get called and do all the processing, this callback is also asynchronous.  This is all designed as a background task thats supposed to run at a preset time in the middle of the night so performance isn't a huge concern but I know that I left room for optimization in there.  Well plenty of room but I just wanted it to work and move on to other things.  Multiple functions that loop through a vector when all the looping could be done once.  Even using a vector to store lines is not really efficient bc just pushing back does alot of memory allocations and copying.   Even counting the lines first and allocating once would be a small bump but not too concerned.  C# is such a hog anyways.  But anyways once I have this thing doing its job for reals I'm going to have to keep track of a thread count and wait until a thread is free.  

[pseakins]

12 minutes ago, markyrocks said:

My laptop is a few years old and it probably doesn't like that very much.

FWIW, my docked Dell E6320 laptop is twelve years old and outperforms the original config due to SSD upgrade and early blocking of M$ updates. Still on W7.

[markyrocks]

15 minutes ago, pseakins said:

FWIW, my docked Dell E6320 laptop is twelve years old and outperforms the original config due to SSD upgrade and early blocking of M$ updates. Still on W7.

I put a Samsung Evo in it as soon as I opened it up when new so it is quick in terms of reads and writes but the fact remains its only a dual core with 4 logical threads so anything over 4 threads makes it push to 100% cpu usage.   When I said 20 seconds it was just a guess it could be like 5 I haven't put a timer on it.  It's really not important.   I'll probably bring my desktop back to life when I actually start using this program.   Just kinda hit writers block or whatever.   

[JockoDundee]

7 hours ago, markyrocks said:

Lol well dude I know you think you're cute so I'll play along.

Nah, that’s just you being cute; this is me being cute

 

 

Edited August 17, 2021 by JockoDundee

[markyrocks]

2 hours ago, JockoDundee said:

Nah, that’s just you being cute; this is me being cute

 

you should check out lvalue and rvalue's  and move semantics in c++.  Very interesting stuff.  As far as the previous exchange goes, the size of the file being downloaded is one aspect but that download time totally disregards the hand shake between requester and server, whatever other checks and overheard , ping, current amount of server traffic. 

[JockoDundee]

1 hour ago, markyrocks said:

you should check out lvalue and rvalue's  and move semantics in c++.  

Very interesting stuff.

Do I need to learn c++ first or should I just jump right in with move semantics?

And as long as we’re trading tips, you should check out the Englund Gambit for black -

1.d4   e5 
2.dxe5 Nc6 
3.Nf3  Qe7 
4.Bf4  Qb4+
5.Bd2  Qxb2
6.Bc3  Bb4! and wins

Very interesting as well.

 

[markyrocks]

11 hours ago, JockoDundee said:

Do I need to learn c++ first or should I just jump right in with move semantics?

1

 

There's my cute lil man 😊.   No you don't need to know c++ to understand the concept,  just a basic understanding of oop programming concepts should suffice.   I got faith in ya.   The idea is to steal memory from temporary objects that would just be destroyed otherwise in a way that avoids the construction of a new object then a copy and destroy vs just a move.  I'm still wrapping my mind around it.   I haven't really had much time to play around with it but it definitely seems worthwhile.   You should learn c++ though.   It will open your eyes to a whole new world.

 

Edited August 17, 2021 by markyrocks

[JockoDundee]

3 hours ago, markyrocks said:

You should learn c++ though.

Why? This problem doesn’t exist in c.  *pointers

[markyrocks]

4 hours ago, JockoDundee said:

Why? This problem doesn’t exist in c.  *pointers

c doesn't have constructors so yes the problem doesn't exist.  It also doesn't have anywhere near the amount of oop features.  C is cool, i can dig it.  I actually wrote the following in c earlier it was some homework assignment for some random person.  the goal was to replace every other word after a dot with "abc".  seems to be working.  I wouldn't personally use c for anything besides like driver development or if I was trying to squeeze every ounce of performance out of a program.

https://onlinegdb.com/dSITKxwfO

Edited August 18, 2021 by markyrocks

[JockoDundee]

See what’s weird for me is that you’re telling me to learn C++ cuz it’s great, and start with this move semantics stuff.

But for my first lesson I’m learning how to do an end around the native heap management because it’s messed up?

Why don’t they just use pointers in the constructors?

[markyrocks]

1 hour ago, JockoDundee said:

See what’s weird for me is that you’re telling me to learn C++ cuz it’s great, and start with this move semantics stuff.

But for my first lesson I’m learning how to do an end around the native heap management because it’s messed up?

Why don’t they just use pointers in the constructors?

I'll start off by saying that theres no such thing as native heap managment in c or c++ any heap allocations exist until manually deleted.  Thats not really the point of the video.  Its more about using intermediate containers such as string or vector that allocate memory when theyre created.  So if you create a class that uses a string for certain data members and usually people create a variety of constructors to populate those fields when an instance of an object is created.  The issue becomes that if I create a class constructor that takes a string reference as a parameter (you're supposed to pass most objects by ref for speed, otherwise a copy is created) if I then attempt to instead pass "this is my string" as a parameter you'll get an error bc "this is my string" is a rvalue hence its temporary bc it has no reference, no location to write back to if function tried to modify it.  Now it could be passed if the parameter is marked as const reference.   In that situation tho in said passing of the const char* "this is my string" to a constructor that wants a const string& whats happening is an implicit construction of a string object via string("this is my string") which is temporary bc it will be destroyed as soon as it leaves the scope of the function and as the reference is lost .  The string object performs its own heap allocation and deletes said memory upon destruction.  If you don't "move" that memory into the string variable once the temp is created the initialization of the data member that's going to receive said string will actually construct a new string object and perform a copy on the temp.  in other words in an operation that would require 2 constructions, 2 heap allocations, a copy, and a destruction otherwise gets cut down to 1 construction, 1 allocation. 

That was a mouthful.  Yes you could just pass a pointer but pointers are a touchy subject.  In big projects pointers and easily cause memory leaks and theres a faction of people out there that believe you should never handle raw pointers in c++.  Its one thing if you're the only person working on a project memory management isn't that big of a deal but if you have many people working on a project and pointers start being passed around alot ownership of them can be lost .  

[JockoDundee]

34 minutes ago, markyrocks said:

… in c or c++ any heap allocations exist until manually deleted.

C, agree.  But you are saying that C++ doesn’t have a way to deallocate automatically when an object or variable has no references to it ?

[Earthshine]

if you use the boost libraries that you get automatic garbage collection with C++

https://www.boost.org/doc/libs/1_56_0/libs/smart_ptr/smart_ptr.htm

 

Edited August 18, 2021 by Earthshine

[markyrocks]

On 8/18/2021 at 4:25 AM, JockoDundee said:

C, agree.  But you are saying that C++ doesn’t have a way to deallocate automatically when an object or variable has no references to it ?

Allocations on the heap no.  Variables on the stack are cleaned up once they go out of scope but those are 2 totally different things.  Theres tons of reasons to use one vs the other.  I try to use as many stack allocated variables as possible but one example of why you  need to allocate on the heap is the fact that stack allocated array sizes need to be known at compile time whereas heap allocated arrays do not.  hence if i make an int array int a[]{1,2,3,4,5} thats fine but if you need to dynamically create it via the use of some variable it would need to be on the heap hence int* a= new int [] with a variable in there. (trying to add soemthing in the square brackets is makeing the page do weird things).  in this instance this memory stays allocated regardless of all other factors until delete is called on it.  the way you  would automate heap allocations and deletes is very easily done using a class or struct.

struct A {

    int* array;
    int size;

    A():array(nullptr),size(0) {}//default constructor

    A(int size) :size(size) {  //alocation constructor
        array = new int[size];
    }

    A(int size, int value) :size(size) { //allocation and fill constructor
        array = new int[size];
        for (auto i = array; i < array + size; i++) {
            *i = value;
        }
    }

    ~A() { //destructor
            delete[] array;
        }
};

 

there's essentially and infinite amount of stuff that can be done with something like this.  This is just the foundation on what a container class starts out as.  This can easily be done with a template to make it typeless and operators can be overloaded to expose the underlying array.  Add methods to add elements, remove elements, insert, resize, make a method to iterate through permutations.  before you know it you have a vector and then you can create an iterator class.  Just to clarify the destructor is called when the class or struct goes out of scope but it doesn't automatically clean up the heap allocation. As the above shows you have to actually do it.  Not that big of a deal but for some i guess this is like black magic or something.  

I'd also like to add that the new keyword is essentially just a wrapper on malloc or calloc that automatically throws an exception if it fails.  Delete is just a wrapper for free() or whatever the c memory release function is.  That being said theres not much difference between memory management in c and c++.

Edited August 20, 2021 by markyrocks

[markyrocks]

On 8/18/2021 at 5:29 AM, Earthshine said:

if you use the boost libraries that you get automatic garbage collection with C++

https://www.boost.org/doc/libs/1_56_0/libs/smart_ptr/smart_ptr.htm

 

smart pointers have become a part of the c++ standard Iibrary I believe.  But theres not magic behind these smartpointers its just a class that keeps track of its own memory.  The std::shared_ptr does do reference counting to determine if its its still being used.  I'm not exactly sure how that works but I'm assuming that whenever the copy constructor is called it just increments a ref count and decrements when the destructor is called and the underlying memory isn't deleted in the destructor until the ref count hits zero.  The ref count would need to be static, Now it has me thinking about where it would need to be in the heirarchy for something like that to work right.... hmmmm

[JockoDundee]

43 minutes ago, markyrocks said:

Allocations on the heap no.

So this “managed C++” thing I’ve read about never caught on?  It sounds like the C++ language but using a JVM…. Is that where C# came from?