Extract data from Website. - (Moved)

[BogdanNicolescu]

Hi to all out there. First of all ....

I wish i haven't had to ask for help ... but here i am.

What i need is something like this:

I have to copy an entry from excel, then go to a site, put what i have in clipboard, search, enter in the first link, copy from some place in page, then return to excel, copy the below next entry, go back to web browser, go back one page and start over again.

I have to do this process aver 10k times ....

 

I do not have any slight idea how to use autoit or have any coding skills or even understandig. So please use baby steps.

is it possible to do this?

[Jos]

Moved to the appropriate forum.

Moderation Team

[Jos]

9 hours ago, BogdanNicolescu said:

is it possible to do this?

I guess it is. So what have you found so far to be the best approach to this challenge doing your research?
We have the excellent Excel Udf to work with the Excel file and several option to work with Websites like the _IE* udfs. 

Jos

 

[BogdanNicolescu]

I'm new to this thing, autoit and coding so i don't know anything about this. I don't know even where to look in the first place ...

And to top it off i need to do this for 10.000+ entries, and if you look closely i have to repeat paste process 20.000+ times, and i have to finish this in the next week or so ...

I was thing something like this:

Copy column A

Hit alt + tab

Enter in search what is copied in clipboard

Hit enter

Click on first result

Copy email address

Alt + tab

Paste in column B

Copy column A

Alt + tab

Hit back button of browser

Enter copied clipboard entry

And repeat what is above. OR!

Copy column A

Hit alt + tab

Enter in search what is copied in clipboard

Hit enter

Click on first result

Copy email address

Hit back button of browser

Alt + tab

Paste in column B

And set after x amount of rows copied to scroll to the next amount.

 

[Jos]

We are here to help you with your AutoIt3 code, not to code it for you! ( at least I am using that approach)
So guess you need to start reading on the 2 topics I gave and start playing and come back when you have questions.
F1 does wonders too in SciTE as that opens the helpfile, which has a lot of information inside.

Jos

[AutoBert]

When source of data is in excel-sheet(s), why not using the excel.au3 to do this job?

[BogdanNicolescu]

On 08.07.2018 at 2:03 PM, AutoBert said:

When source of data is in excel-sheet(s), why not using the excel.au3 to do this job?

because it needs to alt+tab in web browser.

 

Here, let me clarify to you what i want to do:

I have to enter in this site: https://www.siiir.edu.ro/carto/

copy an entry from this excel:  image.png  

then insert here: 

Hit enter and then copy this:

Then go back enter in column j scroll down and repeat.

And i have to do this 10,529 times or how do i like to tell 21,058 copyes and pasties.

 

...............

 

 

Any thoughts?! Should i start now to copy paste or ... does anyone know this program well enough to point me in the right directions?

Oh! And i forgot to tell you, i have like 1 week or so to finish ... so ... if anyone could help me ... 

 

 

[Moonscarlet]

You can use Excel.au3 and IE.au3 to do this and there won't be any need for alt tabbing/ using Send()

Probably, you just need to open/attach to that excel and read the cells (_Excel_Rangeread()) then get the correct IDs/classes (or w/e in that html page) (I guess if they're always in the same place) to extract them and then _Excel_Rangewrite() the results.

Edited July 9, 2018 by Moonscarlet

[BogdanNicolescu]

9 minutes ago, Moonscarlet said:

You can use Excel.au3 and IE.au3 to do this and there won't be any need for alt tabbing/ using Send()

Probably, you just need to open/attach to that excel and read the cells (_Excel_Rangeread()) then get the correct IDs/classes (or w/e in that html page) (I guess if they're always in the same place) to extract them and then _Excel_Rangewrite() the results.

here's the problem: it isn't html page.

At least the rectangle with those rows it isn't, the rest of the page, of course it is, but not the rectangle.

Try for yourself to see what i mean. Me, and another guy have tried to figure out how/what to do. None of us was able to figure out.

The only solution is how i tell you: Copy-paste.

[BogdanNicolescu]

Here is a vid of what i have to do:

https://video.fotp3-2.fna.fbcdn.net/v/t42.9040-2/36746450_220624192102430_8231671746135064576_n.mp4?_nc_cat=0&efg=eyJ2ZW5jb2RlX3RhZyI6InN2ZV9zZCJ9&oh=d39c37f37abfbb47e8f9bf8acca96b7a&oe=5B43ADF4

[wolflake]

If you read column J into an array you don't have to return to the excel sheet again.

Then you can loop through the array and copy each value to the clipboard (ClipPut) and click on the search and paste into the search then (send) an enter key.

If every time the results are in the same place find the mouse position and use that position to click for each record.

When the Contact Edit form comes up try using the same technique of finding the right place for the mouse click and then paste.

Be careful to use sleep pauses to allow for slow browser and server responses.

This isn't the best way but it's pretty simple and you can play with it to get it to work.

Hope this helps.

[BogdanNicolescu]

2 hours ago, wolflake said:

If you read column J into an array you don't have to return to the excel sheet again.

Then you can loop through the array and copy each value to the clipboard (ClipPut) and click on the search and paste into the search then (send) an enter key.

If every time the results are in the same place find the mouse position and use that position to click for each record.

When the Contact Edit form comes up try using the same technique of finding the right place for the mouse click and then paste.

Be careful to use sleep pauses to allow for slow browser and server responses.

This isn't the best way but it's pretty simple and you can play with it to get it to work.

Hope this helps.

I know what you said has some sort of logic, but form me, is not giving me any picture in my head, as i don't know functions and codes. Maybe can you point me into right direction as what literature i need to search/read?

I know the fact that i need to use pause, if only i had to research/edit that ...

[Danp2]

Have you tried contacting the Ministry of Education to see if they will provide this data to you? Honestly, that the only way that I see you obtaining this data without intervention from someone else.

[BogdanNicolescu]

1 minute ago, Danp2 said:

Have you tried contacting the Ministry of Education to see if they will provide this data to you? Honestly, that the only way that I see you obtaining this data without intervention from someone else.

Yeaaahh ... that would be an idea ... but here where i live, all people are busy with living where i live ... so ... even if i request this, the answer would be: NO! or "we can't help you,but you may copy it from our website" which i already attempt to do.

[Danp2]

21 minutes ago, BogdanNicolescu said:

Yeaaahh ... that would be an idea ... but here where i live, all people are busy with living where i live ... so ... even if i request this, the answer would be: NO! or "we can't help you,but you may copy it from our website" which i already attempt to do.

I hear you... but it's your best shot of obtaining the data since you don't currently have the knowledge to do it on your own.

Call them and ask. You might just be surprised by their answer.

[BogdanNicolescu]

1 minute ago, Danp2 said:

I hear you... but it's your best shot of obtaining the data since you don't currently have the knowledge to do it on your own.

Call them and ask. You might just be surprised by their answer.

Or ... i could spend the rest of 4 days and after that the rest of my life in coocoo's nest dressed in a nice white night dress ^^

[BogdanNicolescu]

Here is what i think of, so far. How do i "read" column A?

While 1
Sleep(300)
MouseClick ( "button" [, x, y [, clicks = 1 [, speed = 10]]] ) <= click column A
ClipPut
$SEND_DEFAULT (0) (!{TAB})  to excel ?
OR! should i use:
$SEND_DEFAULT (0) ({ALT}+{TAB})
MouseClick ( "button" [, x, y [, clicks = 1 [, speed = 10]]] ) <= click on search box
Sleep(1000) <= wait
Send($clipboard) <= paste?!
Send ({ENTER}) ?
Sleep(1000)
MouseClick ( "button" [, x, y [, clicks = 1 [, speed = 10]]] ) <= click on first result at location x y
MouseClick ( "button" [, x, y [, clicks = 3 [, speed = 10]]] ) <= clicks on email addres
ClipPut <= copy email address ?!?!?!
MouseClick ( "button" [, x, y [, clicks = 1 [, speed = 10]]] ) <= click on back button
$SEND_DEFAULT (0) (!{TAB})  to excel ?
OR! should i use:
$SEND_DEFAULT (0) ({ALT}+{TAB})
MouseClick ( "button" [, x, y [, clicks = 1 [, speed = 10]]] ) <= position mouse on column j
$sData = ClipGet()
$oExcel.Application.ActiveWindow.ScrollRow = 1
WEnd

 

Edited July 10, 2018 by BogdanNicolescu

[BogdanNicolescu]

Sent an email to those that run the site xD :)) Heh.

Cross fingers to say "NO" ;))

Meanwhile i will try to do the above code tonight when i get home.

[wolflake]

Look at the help file under _Excel_RangeRead to get the A column read into an Array.

Read the example for reading "A1" then use a range instead of a single cell. "A1:A999"

Use a (for next) loop to go through the array of cells you have read.

Use send("^v") to paste.

Use Au3Info.exe in the autoit directory to find the x,y mouse positions for your clicking.  You can drag the finder tool and look at the ControlClick Coords in the tool.

Try running little bits of code to get each part to work then put the parts together.

 

Edited July 10, 2018 by wolflake

[wolflake]

Here is a little code to get your started. for some of your mouse clicks and pasting.  My x,y coordinates are probably not right for your screen so play with them.

MouseClick("",150,425,1,5) ;click on search
sleep(1000)
ClipPut("CASA")
send("^v") ;paste into search field
send("{Enter}") ;trigger search
sleep(1000) ;wait for search to finish 
MouseClick("",150,450,1,5) ;click on row below search to goto edit screen.