Problem Writing Array to Excel

wiretwister · August 30, 2015

Hello Autoit Scripters,

Using Autoit v3.3.14.1 in Windows 7, I'm trying to write an array of X rows by 34 columns to an Excel 2000 sheet as follows:

_Excel_RangeWrite ($workbook, $newsheet, $outarray)

It only works up to 160 rows. An array longer than that writes nothing or hangs the system. This needs to work with up to about 32,000 rows. I've displayed $outarray right before the writing function, and it's perfect. I searched around and found an alternative to _Excel_RangeWrite that uses a different approach based on that COM stuff. It gives the same problem! The problem doesn't appear to be affected by how much data is stored in the array, only the number of rows. I must be overlooking something, like maybe a system setting. Any help would be appreciated.

Thanks in advance!

kcvinu · August 30, 2015

Please post your code here

water · August 30, 2015

What is the return value and the value of @error and @extended after _Excel_RangeWrite?

kylomas · August 30, 2015

wiretwister,

This works for me...

#include <array.au3>
#include <Excel.au3>

Local $aArray[32000][34], $st = timerinit()

For $i = 0 To UBound($aArray) - 1
    For $j = 0 To UBound($aArray, 2) - 1
        $aArray[$i][$j] = StringFormat('%05i - %02i', $i, $j)
    Next
Next

ConsoleWrite('Time to populate array = ' & round(timerdiff($st)/1000,3) & ' seconds' & @CRLF)
$st = timerinit()

Local $oExcel = _Excel_Open(false)
ConsoleWrite(@error & ' - ' & @extended & @CRLF)

$oBook = _Excel_BookNew($oExcel, 1)
ConsoleWrite(@error & ' - ' & @extended & @CRLF)

_Excel_RangeWrite($oBook, Default, $aArray)
ConsoleWrite(@error & ' - ' & @extended & @CRLF)

ConsoleWrite('Time to populate spreadsheet = ' & round(timerdiff($st)/1000,3) & ' seconds' & @CRLF)

Win7 x64 Home

Office/XP

Although it takes awhile. See log...

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\Users\ADMIN010\Documents\excel.au3" /UserParams    
+>03:42:13 Starting AutoIt3Wrapper v.15.729.1555.0 SciTE v.3.5.4.0   Keyboard:00000409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64    Environment(Language:0409)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\ADMIN010\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\ADMIN010\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.14.1)  from:C:\Program Files (x86)\AutoIt3  input:C:\Users\ADMIN010\Documents\excel.au3
+>03:42:13 AU3Check ended.rc:0
>Running:(3.3.14.1):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\ADMIN010\Documents\excel.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
Time to populate array = 9.363 seconds
0 - 1
0 - 0
0 - 0
Time to populate spreadsheet = 945.006 seconds
+>03:58:08 AutoIt3.exe ended.rc:0
+>03:58:08 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 955.6

kylomas

water · August 30, 2015

Excel 200 has an undocumented limit of 5641 cells that can be processed using the transpose method.
Try

_Excel_RangeWrite($workbook, $newsheet, $outarray, Default, Default, True)

This forces the UDF to bypass the transpose method (and its limitations) and use function _ArrayTranspose.

water · August 30, 2015

Details about Excel up to version 2000 can be found here: https://support.microsoft.com/en-us/kb/177991

What I have found so far:
Excel 95 - 5461 cells
Excel 97 - 5461 cells
Excel 2000 - 5461 cells
Excel 2002 - 65536 rows (tested by kylomas with an [65537][2] array)
Excel 2003 - 65536 rows (because Excel 2002 and Excel 2010 have the same limit)
Excel 2007 - 65536 rows (because Excel 2002 and Excel 2010 have the same limit)
Excel 2010 - 65536 rows (tested myself with an [65537][1] array)
Excel 2013 - ?

Edited August 31, 2015 by water

kylomas · August 30, 2015

@‌water - Why did my code above work?

water · August 30, 2015

I have no idea. Maybe it is caused by the way the COM error handler has changed.
Could you test your script with AutoIt 3.3.12.0 to check we get the same results?

kylomas · August 30, 2015

@‌water - I ran it initially with the workbook visible. When it finished I navigated to the bottom (row 31999 X 34 columns = appx. 1.08 million cells). Where did the numbers you posted come from? And, why is transposition an issue?

kylomas · August 30, 2015

@‌water - Uninstalled 3.3.14+...installed 3.3.12...running this code...

#include <array.au3>
#include <Excel.au3>

Local $aArray[32000][34], $st = timerinit()

For $i = 0 To UBound($aArray) - 1
    For $j = 0 To UBound($aArray, 2) - 1
        $aArray[$i][$j] = StringFormat('%05i - %02i', $i, $j)
    Next
Next

ConsoleWrite('Time to populate array = ' & round(timerdiff($st)/1000,3) & ' seconds' & @CRLF)
$st = timerinit()

Local $oExcel = _Excel_Open()
ConsoleWrite(@error & ' - ' & @extended & @CRLF)

$oBook = _Excel_BookNew($oExcel, 1)
ConsoleWrite(@error & ' - ' & @extended & @CRLF)

_Excel_RangeWrite($oBook, Default, $aArray)
ConsoleWrite(@error & ' - ' & @extended & @CRLF)

ConsoleWrite('Time to populate spreadsheet = ' & round(timerdiff($st)/1000,3) & ' seconds' & @CRLF)

with this result...

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\Users\ADMIN010\Documents\excel.au3" /UserParams    
+>05:44:04 Starting AutoIt3Wrapper v.15.729.1555.0 SciTE v.3.5.4.0   Keyboard:00000409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64    Environment(Language:0409)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\ADMIN010\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\ADMIN010\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.12.0)  from:C:\Program Files (x86)\AutoIt3  input:C:\Users\ADMIN010\Documents\excel.au3
+>05:44:05 AU3Check ended.rc:0
>Running:(3.3.12.0):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\ADMIN010\Documents\excel.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
Time to populate array = 15.803 seconds
0 - 0
0 - 0
0 - 0
Time to populate spreadsheet = 931.578 seconds
+>05:59:53 AutoIt3.exe ended.rc:0
+>05:59:53 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 949

Note - I changed the open back to "visible" to check that the entire spreadsheet was written. I was able to scroll to the bottom (row 31999) with no problem.

While running the screen does blank out several times and the excel window shows "...not responding..." occasionally.

Hope this helps,

kylomas

edit: my version of excel is 2002 SP3

Edited August 30, 2015 by kylomas
additional info

water · August 30, 2015

Unfortunately transposing is needed because Excel treats the rows of the array as columns and the columns as rows.
Check this code where transposing has been omitted:

#include <Excel.au3>

Local $aArray[3][5] = [[10, 20, 30, 40, 50], [11, 21, 31, 41, 51], [12, 22, 32, 42, 52]]
Local $oExcel = _Excel_Open()
Local $oBook = _Excel_BookNew($oExcel, 1)
$oBook.Activesheet.Range("A1:C5").Value = $aArray

Edited August 30, 2015 by water

water · August 30, 2015

I have learned it the hard way when rewriting the Excel UDF that there is a limit on the transpose method.
Unfortunately you do not find too much reliable information when searching MSDN or the web.

water · August 30, 2015

I searched the web high and low and couldn't find any information about the transpose limit in Excel 2002.
Maybe the limit was already lifted to 65536 rows?
Could you please run the following test with 65536 and 65537 rows in the array?

#include <array.au3>
#include <Excel.au3>

Local $aArray[65536][2] ; <=== Change to 65537 for the second test!
Loca $st = timerinit()

For $i = 0 To UBound($aArray) - 1
    For $j = 0 To UBound($aArray, 2) - 1
        $aArray[$i][$j] = StringFormat('%05i - %02i', $i, $j)
    Next
Next

ConsoleWrite('Time to populate array = ' & round(timerdiff($st)/1000,3) & ' seconds' & @CRLF)
$st = timerinit()

Local $oExcel = _Excel_Open()
ConsoleWrite(@error & ' - ' & @extended & @CRLF)

$oBook = _Excel_BookNew($oExcel, 1)
ConsoleWrite(@error & ' - ' & @extended & @CRLF)

_Excel_RangeWrite($oBook, Default, $aArray)
ConsoleWrite(@error & ' - ' & @extended & @CRLF)

ConsoleWrite('Time to populate spreadsheet = ' & round(timerdiff($st)/1000,3) & ' seconds' & @CRLF)

kylomas · August 30, 2015

Maybe in topic #6 you mean "rows", not "cells".

I will run your test when I get back by my pc...

Edited August 30, 2015 by kylomas

water · August 30, 2015

For Excel 95, Excel 97 and Excel 2000 it is 5461 cells as described in the KB article I mentioned.
In Excel 2010 it is 65536 rows because it doesn't matter how many columns the array has I tested with.

For Excel 2002, 2003 and 2007 it is still undefined in my opinion. For Excel 2007 I somewhere read that it was 65536 cells, but this could be wrong.

As your script works with Excel 2002 and 34000 rows it might me that the limit was already lifted to 65536 rows with this Excel version. The tests with 65536 and 65537 rows with Excel 2002 should show how it works.

kylomas · August 30, 2015

@‌water - Results from the two test runs...

Results from 1ST run...
>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\Users\ADMIN010\Documents\excel test#1.au3" /UserParams    

+>18:31:20 Starting AutoIt3Wrapper v.15.729.1555.0 SciTE v.3.5.4.0   Keyboard:00000409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64    Environment(Language:0409)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\ADMIN010\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\ADMIN010\AppData\Local\AutoIt v3\SciTE 

>Running AU3Check (3.3.14.1)  from:C:\Program Files (x86)\AutoIt3  input:C:\Users\ADMIN010\Documents\excel test#1.au3
+>18:31:21 AU3Check ended.rc:0
>Running:(3.3.14.1):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\ADMIN010\Documents\excel test#1.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
Time to populate array = 1.274 seconds
0 - 1
0 - 0
0 - 0
Time to populate spreadsheet = 11.456 seconds
+>18:31:34 AutoIt3.exe ended.rc:0
+>18:31:34 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 13.73



Results from 2ND run...

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\Users\ADMIN010\Documents\excel test#1.au3" /UserParams    

+>18:32:36 Starting AutoIt3Wrapper v.15.729.1555.0 SciTE v.3.5.4.0   Keyboard:00000409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64    Environment(Language:0409)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\ADMIN010\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\ADMIN010\AppData\Local\AutoIt v3\SciTE 

>Running AU3Check (3.3.14.1)  from:C:\Program Files (x86)\AutoIt3  input:C:\Users\ADMIN010\Documents\excel test#1.au3
+>18:32:36 AU3Check ended.rc:0
>Running:(3.3.14.1):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\ADMIN010\Documents\excel test#1.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
Time to populate array = 1.264 seconds
0 - 1
0 - 0
6 - 169
Time to populate spreadsheet = 0.34 seconds
+>18:32:38 AutoIt3.exe ended.rc:0
+>18:32:38 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 2.647

water · August 31, 2015

That's great. So we now know that the 65536 rows limit was implemented with Excel 2002
But it still doesn't explain, why the OPs script fails

water · August 31, 2015

@wiretwister

How do you populate your array? Do you query a database, Active Directory or some other systems which might return Null or cells with data longer 255 characters?

kylomas · August 31, 2015

That's great. So we now know that the 65536 rows limit was implemented with Excel 2002
But it still doesn't explain, why the OPs script fails

Indeed, now we need to see the code that fails. That was my point, albeit vague, in posting working code in topic #4.

wiretwister · August 31, 2015

Please post your code here

Problem Writing Array to Excel

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Similar Content