1d array -> 2d array

[civilcalc]

ok so I have an array...

[0] somedata1 somedata2 somedata3

[1] moredata1 moredata2 moredata3

[2] ......... ......... .........

each seperated by " " (ws)

Can I easily convert this to 2d array?

first dimension varies a lot, second dim is never bigger than 6.

[Melba23]

civilcalc,

This should do it for you:

#include <Array.au3>

Global $aArray_1D[2] = ["somedata1 somedata2 somedata3", "moredata1 moredata2 moredata3"]

Global $aArray_2D[2][3] ; Set these dimensions as required

For $i = 0 To UBound($aArray_1D) - 1
    $aTemp = StringSplit($aArray_1D[$i], " ") ; Split the 1D elements assuming the delimiter is always a space
    For $j = 0 To UBound($aArray_2D, 2) - 1
        $aArray_2D[$i][$j] = $aTemp[$j + 1] ; Get the elements into the 2D array
    Next
Next

_ArrayDisplay($aArray_2D)

M23

[Funtime60]

I know this is old, but is this the fastest possible way?

I came up with something similar that doesn't account for varying number of items in each row. My problem is that it takes well over 6 hours  to run through 230,400 lines at 4MB of an array with each line containing 3 sections. The only thing that I can see slowing it down more than yours is the ProgressSet that I have in the loop.

If anyone reads this I have included my work below which is at the start of a larger project. Sorry for the mess that it likely is, I started this months ago and have be working on it whenever I have a mental break through.

expandcollapse popup

#Region Include
#include <ArrayMultiColSort.au3>    ;Non-standard .au3 added to my Include folder, not applicable to this section
#include <String.au3>
#include <BMP3.au3>                 ;Non-standard .au3 added to my Include folder, not applicable to this section
#EndRegion


#Region Open
Local $testpath = FileOpenDialog("Select A Data File",@ScriptDir & "\","All (*.*)")
Local $testpathpart = StringSplit($testpath, "\")
Local $testfilenamefull = $testpathpart[$testpathpart[0]]
Local $testfilenamepart = StringSplit($testfilenamefull, ".")
Local $testfilename = $testfilenamepart[1]
Local $test = FileOpen($testpath)
Local $testfile = FileReadToArray($test)
#EndRegion


#Region Main Init
Local $aSortData[][] = [[1, 0], [2, 0]]
Local $progress = "N/A"
ProgressOn("Conversion", "Part 1 of 9", "Progress: " & $progress, -1, -1, 16)
#EndRegion


#Region Pack Crct
_ArrayDisplay($testfile, "$testfile")
Local $progress = 0
    ProgressSet($progress, "Progress: " & Round($progress, 2) & "%", "Part 2 of 9")
Local $testsize = UBound($testfile)-1
$testfilerowdata = StringSplit($testfile[0],",")
Local $testfilearray1[1][3] = [[$testfilerowdata[1], $testfilerowdata[2], $testfilerowdata[3]]]
For $testfilerownum = 1 To $testsize Step 1
    $testfilerowdata = StringSplit($testfile[$testfilerownum],",")
    Local $testfilearray2[1][3] = [[$testfilerowdata[1], $testfilerowdata[2], $testfilerowdata[3]]]
    _ArrayConcatenate($testfilearray1,$testfilearray2)
    Local $progress = ((($testfilerownum+1)/($testsize+2))*100)
    ProgressSet($progress, "Progress: " & Round($progress, 2) & "%", "Part 2 of 9")
Next
_ArrayDisplay($testfilearray1, "$testfilearray1")
Local $progress = 100
ProgressSet($progress, "Progress: " & Round($progress, 2) & "%", "Part 2 of 9")
#EndRegion

It should turn this:

15791353,419,294
15791353,420,294
15791353,421,294
15791353,422,294
15791353,423,294
15791353,419,295
15791353,420,295
15791353,421,295
15791353,422,295
15791353,423,295
15791353,419,296
15791353,420,296
15791353,421,296
15791353,422,296
15791353,423,296
15791353,419,297
15791353,420,297
15791353,421,297
15791353,422,297
15791353,423,297
15791353,419,298
15791353,420,298
15791353,421,298
15791353,422,298
15791353,423,298

Into this:  The "|" delimit a column for anyone who hasn't pasted out of _ArrayDisplay before

15791353|419|294
15791353|420|294
15791353|421|294
15791353|422|294
15791353|423|294
15791353|419|295
15791353|420|295
15791353|421|295
15791353|422|295
15791353|423|295
15791353|419|296
15791353|420|296
15791353|421|296
15791353|422|296
15791353|423|296
15791353|419|297
15791353|420|297
15791353|421|297
15791353|422|297
15791353|423|297
15791353|419|298
15791353|420|298
15791353|421|298
15791353|422|298
15791353|423|298

Test File that generated these is below

Thank you for your time. even if you just read this.

TiniTmp.csv

[czardas]

I suggest you use @Melba23's example but I had to test an idea. This automatically resizes the array (on the fly) as required, so you don't need to know the size of the output array in advance. To run the code you need ArrayWorkshop.au3 (found in my signature below). Such code is total overkill for a simple conversion like this (with only two dimensions), but an example of this nature lends itself to testing my own functions and may interest some readers.
 

#include <Array.au3>
#include <StringConstants.au3>
#include <ArrayWorkshop.au3>

; The Array
Local $aTestArray[7] = [ _
"Data Data Data Data", _
"", _
"Data Data Data Data Data",  _
"Data Data Data Data Data Data", _
"Data", _
"Data Data Data Data Data Data Data Data Data Data Data Data", _
"Data Data Data Data Data Data Data Data"]
_ArrayDisplay($aTestArray, "1D Array")

; The Code
Local $aOutputArray = StringSplit($aTestArray[0], " ", $STR_NOCOUNT)
For $i = 1 To UBound($aTestArray) -1
    _ArrayAttach($aOutputArray, StringSplit($aTestArray[$i], " ", $STR_NOCOUNT), 2) ; Add columns
Next

_ArrayTransform($aOutputArray) ; Transform: Rows => Columns

; Display output
_ArrayDisplay($aOutputArray, "2D Array")

 

[Funtime60]

I appreciate the help, however in my inexperienced view it seems you are doing nearly the same thing as me except that your method of connecting the stringsplit pieces togetther causes a 90 rotation of the data which you then correct with another function. I did limited testing, but have found that when converting 128,417 characters on 8160 lines in a .csv file  your code takes ~0.4 seconds longer at ~36.5 seconds which is the best speed difference. This file size is at the smaller end of potential files with mid size files being 1-4 MB which can take multiple hours so your 1% slower, while probably much better written is too much of a cost. Still thank you for your time and effort. I hope this scenario was a good test of your code.

With that said, I'm looking for anything that can beet ~36.1 seconds with this file below. If you think you know how to do it feel free to test it yourself or send it to me.

tmp4.csv

[mikell]

Melba's code applied to your tmp4.csv takes 0.1 second. I don't understand the huge delay you mentioned

 

Edited August 14, 2018 by mikell

[iamtheky]

0.8 seconds with array UDF, and thats about as slow as it gets.

#include<array.au3>
$s = stringtrimright(fileread("Tmp4.csv") , 2)

local $a[0][3]

_ArrayAdd($a , $s , "," , ",")

_ArrayDisplay($a)

 

[Funtime60]

well thanks, not sure why I didn't think to test melba's code or why his is so fast.

[Funtime60]

I have failed to understand how the heck your code works iamtheky, I would be eternally grateful if you could offer any insight.

Never mind, I forgot that you were the author of a UDF that I already had so it looked like you pulled off some black magic. You have saved my code!

Edited August 14, 2018 by Funtime60
I solved my own problem

[jchd]

All the work is done by _ArrayAdd(). Reading the help file will bring light in the darkness.

[Funtime60]

Turns out as I expected, after I fixed this 12 hours became 1.2 minutes. out of 9 sections numbers 2 and 3 were the longest with #2 being hours.

[iamtheky]

1 hour ago, Funtime60 said:

I have failed to understand how the heck your code works iamtheky

i have the same issue, it's all a blur.  Glad you found myriad ways to skin your cat.

[czardas]

14 hours ago, Funtime60 said:

I appreciate the help, however in my inexperienced view it seems you are doing nearly the same thing as me except that your method of connecting the stringsplit pieces togetther causes a 90 rotation of the data which you then correct with another function. I did limited testing, but have found that when converting 128,417 characters on 8160 lines in a .csv file  your code takes ~0.4 seconds longer at ~36.5 seconds which is the best speed difference.

I had expected a time lag and that Melba23's example would run faster. The functions I wrote are intended to accommodate miscellaneous scenarios including auto-determination of the number output dimensions with _ArrayAttach(). Dealing with three, or more, dimensions leads to horrendous syntax and these functions are an attempt to avoid that.

Edited August 15, 2018 by czardas