Can Autoit do this? - (Moved)

[seadoggie01]

We use MouseClick like this:

; Gets a bunch of pre-defined variables (like $MOUSE_CLICK_LEFT)
#include <AutoItConstants.au3>

; This says, click at (100, 100) on the screen one time with the left mouse button
; N.B. (100, 100) is from the top left corner of your screen
MouseClick($MOUSE_CLICK_LEFT, 100, 100, 1, 10)
    
; This does the same thing because $MOUSE_CLICK_LEFT means "left"
MouseClick("left", 100, 100, 1, 10)

It looks like you're switching functions with variables. Variables start with a dollar sign and represent some other value (like x or y in math). Functions don't have a dollar sign and use parentheses to accept parameters.

The include line at the top of my script is needed if you want to use some variables. You'll see in the help files something like this: Constants are defined in "AutoItConstants.au3".

[Confuzzled]

So just summarising what we have learned so far in this thread:

a) You are trying to screen scrape 'The Washington Post' online news web site pages, and examine and download the comments section on specific articles. We assume you are doing this to save a few mouse clicks, and not for profit or illegal purposes. They have strict user terms and conditions on how you may use their web site, and have a fairly robust paywall.

/rant mode on/ Data theft of intellectual property from news reporting sites is self-defeating, as you well know that the salaried investigative reporter, backed by an experienced editor has slowly but surely been replaced by casual low paid piecework random individuals quoting from social media such as FaceBoob, Instagram and Twitter to generate their advertorial articles that push their own viewpoint. Entirely untrustworthy 'news', full of rubbish. Eventually these well established institutions wither and die from lack of paying readers, and your reliable news feed is no more! EVER and forever. /rant mode off/

Note the following relates specifically to FireFox, my preferred browser:

b) Have you right clicked on your browser page and examined the source code of the web page - what it actually sends to your web browser? You may find references to snippets of code and URLs that may give you pointers in what you might need to access.

c) Have you tried 'Reader View' in FireFox, and something similar in other browsers that strip all the other glamour functionality and just give you the text, the whole text, and nothing but the text? This functionality is built for blind readers that have text to voice capability on their computer. In some countries it is compulsory to have this functionality built in and supported by the web site in question.

d) Have you looked at page style settings on your browser? CSS, Page info, and Developer Tools?

e) Are you using SciTe, the preferred AutoIT development platform, to write your code? If so, does the F1 Help Key work? Are the sections on variables, constants, tutorials, UDFs, and other parts present in the Help File? You may find this a valuable resource to guide you on your journey, to answer some of the fairly basic questions you have posed in previous posts in this thread.

f) Have you tried to write your own html code to specifically access those portions of the Washington Post web site and display them in your browser, ie: embed them into your carefully crafted web page on your own computer system?

I know I am pointing you in an entirely different direction towards researching how web pages are built, transmitted, and displayed, but you will find this knowledge essential when you go back to write your AutoIT code, as The Washington Post tend to update their code from time to time to try and bypass advertising filters and screen scrapers. What you accomplish today will be made obsolete swiftly as they discover your loopholes and patch them with global site changes.

 

Doing a Google search on 'screen scrape washington post' unearthed a few interesting articles, of which https://lingfeiwu1.gitbooks.io/data-mining-in-social-science/content/scraping_articles_from_the_washington_post/ may give you an appreciation of what is involved.

https://github.com/washingtonpost may lead you to Nirvana.

https://www.reddit.com/r/webdev/comments/a8wcl4/web_scraping_i_am_building_an_rss_reader/  will confirm you are not the first to try this.

https://en.wikipedia.org/wiki/List_of_news_media_APIs will give you some idea of third party API's that allow you to access news articles. Some may be pay-per-use, which may be what you are trying to avoid

Best of luck. Unless you are doing this as a learning exercise, it may be far more efficient of your time and effort to just pay the paywall tax and enjoy a rare experience that is rapidly disappearing - news, well presented, as a convenient service for which you pay for what you get.

 

 

Edited October 30, 2020 by Confuzzled

[RetiredGuy]

Hi confused,

What I am trying to do is get a complete copy of my Comments on articles on the Washington Post (WP) for the years 2010 thru half of 2018 which are called the "Old Comments" since the WP implemented a new system then.  I get to these Comments on a special login page on the WP.  Unfortunately the WP isn't at all helpful in this task, the problem is that their communication system drops the connection and sometimes restarts OK, but more often than not, freezes up so I'd have to start over from the top.

If it were not for having to click on the word "More" to get the next page to display and to click on the words "See More" to see the rest of my comment in a long comment, I would be able to copy and paste away and get it all, but I can't because of those words.

I have searched the code the WP has used to run the system for the old comments.  It is composed of HTML and JAVA scripts.  These is no way to identify the tables where the WP has put my Comments and I have looked more than once.  I have tried using "w get" command in my Linux system.  I get a great copy of the front page, but it doesn't have any of my comments nor any links to to my Comments.

So the short answer is I have to get a small program to run in the Console after I login and get my Comments displayed.  I'm not a programmer.  I've using various sized computers since the early 1980's using all sorts of different OS and software packages.

As for SciTe it's yes to all of your questions.  Plus, I reading the book and trying to follow the examples of code in the book and in the Help files, but, not being a programmer, my brain quickly dissolves into confusion or and great big question mark of what does that mean.

I have not tried writing any HTML as I've not done any of that in close to 30 years, plus the WP code is mostly JAVA.

I'm not after news articles, but rather My Comments on news and opinion from those early years as I mentioned above.  I have over 12,000 Comments Of mine that I'm trying to copy and paste down onto my PC so I can do something with them.

I appreciate your taking the time to write me.

Pat

 

 

[Confuzzled]

Just wondering: Does the Internet Archive have a copy of your comments stored with the original articles in a more accessible format?

 

[RetiredGuy]

To answer your question, I have no idea.  These Comments of mine have a link to the article that is stored someplace else on the WP computer.

The good news is, maybe, I did some more with the code and I got it to compile!  I guess now I can take it over and run it and see what happens.

That's some graphic you put in your reply.

Pat

[RetiredGuy]

Well, I tried and now the WP console has shot me down.  My little code is so simple and the compile now looks so clean to me, but what follows are my two attempts to run first the .exe, then after those error messages, I tried to run just my code without the compile and got different error messages. 

Here are the two attempts and the error reports from the console on the WP file of my comments:

===================

 

My changes led to a compile of my code.  Here is the compile report”

"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /ShowGui /prod /in "C:\Users\tpk1\Documents\My12696code.au3"
+>18:06:10 Starting AutoIt3Wrapper (19.1127.1402.0} from:SciTE.exe (4.2.0.0)  Keyboard:00000409  OS:WIN_10/  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\tpk1\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\tpk1\AppData\Local\AutoIt v3\SciTE
-> 1 Change(s) made.
>Running AU3Check (3.3.14.5)  from:C:\Program Files (x86)\AutoIt3  input:C:\Users\tpk1\Documents\My12696code.au3
+>18:06:29 AU3Check ended.rc:0
>Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\aut2exe\aut2exe.exe  /in "C:\Users\tpk1\Documents\My12696code.au3" /out "C:\Users\tpk1\AppData\Local\AutoIt v3\Aut2exe\~AUB0A9.tmp.exe" /nopack /comp 2
+>18:06:30 Aut2exe.exe ended.C:\Users\tpk1\AppData\Local\AutoIt v3\Aut2exe\~AUB0A9.tmp.exe. rc:0
+>18:06:30 Created program:C:\Users\tpk1\Documents\my12696code.exe
+>18:06:31 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 20.86

 

I ran my .exe file on the console of my comments like this:

my12696code.exe

Note that I do Not Have Any Java in that compiled code.

And got this / these error messages:

Uncaught ReferenceError: my12696code is not defined
    <anonymous> debugger eval code:1
debugger eval code:1:1

And the explanation is:

ReferenceError: "x" is not defined

    See Web technology for developersSee JavaScriptSee JavaScript referenceSee JavaScript error referenceReferenceError: "x" is not defined

On this Page

The JavaScript exception "variable is not defined" occurs when there is a non-existent variable referenced somewhere.
Message

ReferenceError: "x" is not defined

Error type

ReferenceError.
What went wrong?

There is a non-existent variable referenced somewhere. This variable needs to be declared, or you need to make sure it is available in your current script or scope.

Note: When loading a library (such as jQuery), make sure it is loaded before you access library variables, such as "$". Put the <script> tag that loads the library before your code that uses it.

============================

Then I tried this: I put my code in looking like this

$looksee = "See More"
$lookmore = "More"
MOUSECLICK("LEFT")
If $looksee > 0 then
mouseclick
EndIf
if $lookmore > 0 then
mouseclick
EndIf

 

And got this Explanation:

Uncaught SyntaxError: unexpected token: identifierdebugger eval code:4:3

SyntaxError: missing ; before statement

    See Web technology for developersSee JavaScriptSee JavaScript referenceSee JavaScript error referenceSyntaxError: missing ; before statement

On this Page

The JavaScript exception "missing ; before statement" occurs when there is a semicolon (;) missing somewhere and can't be added by automatic semicolon insertion (ASI). You need to provide a semicolon, so that JavaScript can parse the source code correctly.
Message

SyntaxError: Expected ';' (Edge)
SyntaxError: missing ; before statement (Firefox)

Error type

SyntaxError.
What went wrong?

There is a semicolon (;) missing somewhere. JavaScript statements must be terminated with semicolons. Some of them are affected by automatic semicolon insertion (ASI), but in this case you need to provide a semicolon, so that JavaScript can parse the source code correctly.

However, oftentimes, this error is only a consequence of another error, like not escaping strings properly, or using var wrongly. You might also have too many parenthesis somewhere. Carefully check the syntax when this error is thrown.

=======================

Can anyone tell me what is wrong?

Pat

 

 

[Confuzzled]

You wrote some Java code too? Post all the code you compiled as well for perusal and comment.

[RetiredGuy]

Nope, no Java, I don't know from "nutting" about Java.  What you see above is what I wrote, FINALLY, that got compiled.  Why the WP computer thinks it's Java is beyond my feeble brain to understand.

Here it is again:

$looksee = "See More"
$lookmore = "More"
MOUSECLICK("LEFT")
If $looksee > 0 then
mouseclick
EndIf
if $lookmore > 0 then
mouseclick
EndIf

That's the code, my code you folks helped me with, that compiled to the .exe file.  I don't know what to say.

[caramen]

Well hmmm... I think WebDriver(Forum UDF) will be the best option here.

You could click on button by string research. And even ClipPut the whole text after reading it with WebDriver as well but in Mozilla or Firefox.

And if you want to learn that, you have to be a bit patient and learn before. It's like starting a game at lvl 5 when you are lvl 0-1./

 

Also in WP you have a PhpMyAdmin database in most of the cases. Which probably contain all your comment in one table. 

Edited November 2, 2020 by caramen

[RetiredGuy]

WebDriver(Forum UDF) will be the best option here. ??

Also in WP you have a PhpMyAdmin database in most of the cases. Which probably contain all your comment in one table.  ??

You just blew my mind.  Could you give a few more details on what and how this works?

I've never heard of any of that.

[caramen]

Read, try examples, train your skill with WebDriver UDF in my signature.

About, the table(s) present or not in your WP database, you have to talk with the site owner. And if you are, google how to access your WP PhpMyAdmin tables to check them. And steps for this totally depend on the site owner chosen provider.

Btw this is not easy tasks. And touching the database without knowledge in PhpMyAdmin can result in a definitive website break. 

Ensure to have backup.

Edited November 2, 2020 by caramen

[RetiredGuy]

"you have to talk with the site owner"

The problem is the site IT dept doesn't want to help and will not even answer questions.  I have written the chief of the dept and never got a reply.

 

[caramen]

So just go to WebDrivers.

I was going to try to explain to you how to ask this to a lvl 0 IT help desk technician. But, just forget it. This is site owner task. When you know what you do.

 

Just go for the WebDrivers option. 

[TheXman]

On 10/31/2020 at 12:54 PM, RetiredGuy said:

$looksee = "See More"
$lookmore = "More"
MOUSECLICK("LEFT")
If $looksee > 0 then
mouseclick
EndIf
if $lookmore > 0 then
mouseclick
EndIf

That's the code, my code you folks helped me with, that compiled to the .exe file.  I don't know what to say.

If that's the entirety of what you have been able to come up with, and if you really need the information you're after, it may be best to hire someone with the skills to get it done or to start manually cutting & pasting the posts yourself.  Because at the level and speed in which you are picking up AutoIt, I seriously doubt that you will be able to write that script anytime soon.  The script that you've posted is SO far off, that it could easily make one think that your goal is/was to get someone here to write the script for you.  But I'm sure that thought never crossed your mind. 

Edited November 2, 2020 by TheXman

[RetiredGuy]

"The script that you've posted is SO far off, that it could easily make one think that your goal is/was to get someone here to write the script for you.  But I'm sure that thought never crossed your mind."

NO, that never crossed my mind.  If I wanted that I would have straight out asked if someone was available for hire to do it for me.

I've been using, learning computers and software packages on unix, dos, supermini computers, mainframes, lan's and PC's.  I'm certainly not afraid to learn and that is what I've been trying to do here.  I've never had the chance to learn programming.  I taught myself html and created an intranet and teaching programs on how to use software packages.

I'm sorry you felt it necessary to insult my intelligence, but you are way off the map with this.

 

 

[TheXman]

3 hours ago, RetiredGuy said:

I'm sorry you felt it necessary to insult my intelligence, but you are way off the map with this.

I wasn't trying to insult your intelligence.  I was making an observation based on the script that you posted and your comment that it is what you were able to come up with based on the help that you've received.  If you're trying to learn AutoIt, you certainly chose a relatively difficult project to start with.   

3 hours ago, RetiredGuy said:

I've been using, learning computers and software packages on unix, dos, supermini computers, mainframes, lan's and PC's.  I'm certainly not afraid to learn and that is what I've been trying to do here.  I've never had the chance to learn programming.  I taught myself html and created an intranet and teaching programs on how to use software packages.

That's great, especially if you're background/career is not in technology.  My career is/was in technology.  I have technically mentored and helped more people than I can count over my career.  Even the vast majority of my posts here are in an effort to help others.  As it says over my avatar and in my About Me, I'm "always willing to help", even if it means letting someone know that they may need to step up, step back, or re-evaluate their efforts.  You shouldn't be so defensive and sensitive.  What I said in my post was not an attack on you or your intelligence.  It was an observation based on your script and what you've said you've gathered from the help that you've received.

Edited November 3, 2020 by TheXman

[CYCho]

1 hour ago, TheXman said:

I'm "always willing to help"

I can attest this statement. As a retired man with no technical background, I was immensely helped by @TheXman and, in fact, many others in this community to learn coding from scratch and build some useful tools of my own. I highly respect those who are willing to help others.

[caramen]

So what is the problem then?

We have someone without development skills. Just try to point him out. And let's see what he will do. 

We all started from scratch here.

If we did it, everyone can.

And what is the point of saying you want we make the script for you ?

It's pointless to say such supposition. I'm always trying to help other too but the most possible without code that they will be able to copy paste into there code. I like to bring the guy to the point I thought before. But there is always a Mr pro that coming to leave a perfect code that the OP will copy paste just after most of my answers. Yeah, this is boring. But it's pretty same as asking for code. A lot of peoples are asking for code, but they do it intelligently so people are not complaining. 

And yeah @TheXman helped me out a lot before. I'm one of them too. 

Edited November 3, 2020 by caramen

[Confuzzled]

RetiredGuy has a need, and the AutoIT is incidental - he wants to download his historical comments from the Washington Post. They deliberately make their site hard to access behind a paywall, and of course their underlying data is not going to be exposed for scraping, easily.

There are a lot of concepts to learn here: Accessing your comments at the website, downloading them, storing them, and presenting them in a format you need. You have to understand HTML, Javascript, CSS, templates, and how the data is stored and accessed as the browser builds the page that it displays to you, and what you have to do to intercept that.

Break your problem down into smaller chunks, and then solve them one-by-one, building on what you have learnt. It is glaringly obvious from the posts above that you have a long way to go, but I sense that you have time, patience, and a purposeful persistence that will win out in the end.

Show that you have mastered the concept of downloading what is displayed to you before you initiate the 'more' command, and then you can build on that, and I am sure those watching will jump in to offer suggestions.

Explain the steps on paper, before you start coding in any language at all. How do you build a table of those articles that have your comments attached and ignore the rest? How you access only your comments amongst all the others? How you download them? What format do you store them in? How do you sort them and present them? Once you can explain that to your mother-in-law or a teenager (that cannot code) so they can understand it in plain English, then you will find that your code will most likely automagically fall into place.

Correspondingly, nobody feels obligated to spoon feed you. Experiment with what you observe, Read about how they want to help you at https://www.washingtonpost.com/view-your-comment-history-on-the-my-profile-tab/d1ec02fd-b3db-44a5-9c45-c66be29323ff_note.html

Research a little on GitHub on what others have done, mostly with other languages other than AutoIT. What did those programmers do to solve the challenges? How did they approach it? The actual code is not that important for your research at the initial stages - the underlying concepts are. Google search is your friend (sorta), as long as you know what questions to ask. Use trigger words like 'screen scrape', 'script', etc and slowly it will fall into place.

Just firing off code that you have cut and pasted from examples will not help you unless you understand the concepts behind what it is there for - why each command is being invoked, how it works, and what the end results are. Unfortunately that is something you have to do for yourself. We can offer suggestions, but you still have to put in the hard yards to reap the rewards.

Once you have mastered the fundamentals, the rest is relatively easy!

Best of luck.

 

PS> I started with punch cards and worked my way forwards. Other than programming a VCR, I still can show those cheeky upstarts a thing or two! Learning is a lifetime challenge, whether formal or not. Never get bored - get even! Persistence vs enthusiasm - experience always wins. Structured learning however where you learn the fundamentals that apply across all languages is something that I would strongly recommend. There are a lot of courses online these days that are free, and find one that is compatible with your learning style. Most universities and colleges offer them. Well worth doing.

Edited November 3, 2020 by Confuzzled