xHarbour and 'SUBMIT'ing Web pages

Hi:

Re: Using xHarbour to 'automate' the reading and submitting of webpages.

Because of David Smith's and Ron Pinkas' help I'm now at the stage of
quickly reading in thousands of webpages from a site. Then parsing the info
and saving it to a database.

The final problem I have is to (programmatically thru xHarbour) enter my
UserName, and Password in the default html page, then 'Press' the Submit
Button (or send a CRLF) on/to the page.

The username, Password, and Login button are in a small frame/window by
themselves. I would normally type my username, then the password, then
'click' on the Submit button -- *but* I would like to automate this
operation.

I have looked thru the innerHTML text, and see both input fields (i.e.,
'txtRemoteUserName' and 'txtRemoteUserPassword') and I also see where the
'submit' button is defined. But I also don't know how to pick out the
frame/window where these inputs are placed

There is some Javascript stuff in there too.

I would like to know what the xHarbour statements are to set username,
and password, then 'click' the button ???

In an earlier post, I have already 'navigated' to the page, read-in the
innerHTML text, parsed and looped -- But this logging in preprocess is the
only thing left to automate it completely.

Thanks for any help offered

--
Mel Smith

in similar post I wrote this

I try to use something like this
PROCEDURE Main
LOCAL oHttp, cHtml, hQuery, aLink, oNode, oDoc, oUrl

oUrl := TUrl():new( "http://www.test.ca" )
oUrl:cUserid := 'user'
oUrl:cPassword := 'password'
oHttp:= TIpClientHttp():new( oUrl)


oHttp:UseBasicAuth()
// Connect to the HTTP server
IF .NOT. oHttppen()
? "Connection error:", oHttp:lastErrorMessage()
QUIT
ENDIF

// download the response
cHtml := oHttp:readAll()
oHttp:close()
? Len(cHtml), "bytes received "

oDoc := THtmlDocument():new( cHtml )

but oDoc return something like this
Object moved - because auth failed

in delphi when I use this component
http://www.badfan.com/delphi/TIE_http_https.html

type
TForm1 = class(TForm)
WebBrowser1: TWebBrowser;
IE1: TIEHTTP;
IdHTTP1: TIdHTTP;

ie1.RequestMethod := 'POST';
ie1.username := 'user';
ie1.password := 'password';
ie1.ExecuteURL("http://www.test.ca");

I can go thru autenhtication

I guess that problem is in oHttppen() because
when we look at httpcln.prg source we see
a hidden metod standardfields that is used in metod get and is not used
in metod open so I guess that metod open does not use oUrl:cUserid and
oUrl:cPassword

hiden metod of TIpClientHttp() uses oUrl:cUserid and oUrl:cPassword

i've tried to add this line
oHttp:Get(oUrl )
befor
IF .NOT. oHttppen()

but I get an error

anybody help, please?
regards

Klas Engwall replaid to me

oHttp:= TIpClientHttp():new( oUrl)

Have you tried turning on trace and analyzing the log file? Like this:

oHttp:= TIpClientHttp():new( oUrl, .T. )


Regards,
Klas

but I did not receive anything in trace.log

Apolinar,

>Klas Engwall replaid to me
>
> oHttp:= TIpClientHttp():new( oUrl)
>
>Have you tried turning on trace and analyzing the log file? Like this:
>
>oHttp:= TIpClientHttp():new( oUrl, .T. )
>
>but I did not receive anything in trace.log

Actually, it is http.log that you want and not trace.log. But why
didn't you say anything? I use the Log() method of the super class to
log ftp sessions, and I saw that the lTrace argument was present in
the http client too so I assumed that it would work. But now I took a
second look at the http client and noticed that one vital thing is
missing: The actual fcreate()ing of the log file. This means that
although the Log() method is called, it is using a file handle that is
NIL.

I haven't tested this, but you could try to solve the problem
temporarily with the following workaround:

oHttp:= TIpClientHttp():new( oUrl, .T. ) // Same as before
oHttp:nHandle := fcreate("http.log")

This will overwrite the previous log when you start a new session, but
that can of course be fixed with a little extra work. At the end of
the session you should probably also close the log file handle:

fclose(oHttp:nHandle)

Plese report your results back to the group.

Regards,
Klas

-------
klas dot engwall at engwall dot com

http://www.engwall.com/clipper/

The LFN Library for Clipper
The LanMan Library for Clipper
The NFPAT1A Timeslice release patch for the Nanforum Toolkit

excuse me klass, I had no time to work more on this and I put it aside
for a while
I tried your advice with
oHttp:nHandle := fcreate("http.log")
and here is what I received in http.log

sorry for xxx.xxx.xx while I am not allowed to say web page
----------------
20080815-13:37:08 :INETCONNECT(xxx.xxx.xx, 80 )
>> A13AA0 <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETSENDALL( A13AA0, 16, GET / HTTP/1.1 )
>> 16 <<

20080815-13:37:08 :INETSENDALL( A13AA0, 18, Host: xxx.xxx.xx )
>> 18 <<

20080815-13:37:08 :INETSENDALL( A13AA0, 36, User-agent: Mozilla/3.0
compatible )
>> 36 <<

20080815-13:37:08 :INETSENDALL( A13AA0, 19, Connection: close )
>> 19 <<

20080815-13:37:08 :INETSENDALL( A13AA0, 2, )
>> 2 <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETRECVLINE( A13AA0, , 500 )
>> HTTP/1.1 302 Object moved <<

20080815-13:37:08 :INETRECVLINE( A13AA0, , 500 )
>> Connection: close <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETRECVLINE( A13AA0, , 500 )
>> Date: Fri, 15 Aug 2008 11:38:02 GMT <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETRECVLINE( A13AA0, , 500 )
>> Server: Microsoft-IIS/6.0 <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETRECVLINE( A13AA0, , 500 )
>> X-Powered-By: ASP.NET <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETRECVLINE( A13AA0, , 500 )
>> Location: xxxxxxxxx/ <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETRECVLINE( A13AA0, , 500 )
>> Content-Length: 131 <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETRECVLINE( A13AA0, , 500 )
>> Content-Type: text/html <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETRECVLINE( A13AA0, , 500 )
>> Set-Cookie: ASPSESSIONIDQSSSAASR=DDODJGBCPFNDLNKOLEDCLIJA; path=/ <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETRECVLINE( A13AA0, , 500 )
>> Cache-control: private <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETRECVLINE( A13AA0, , 500 )
>> <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETERRORCODE( A13AA0 )
>> 0 <<

20080815-13:37:08 :INETRECVALL( A13AA0, , 131 )
>> <head><title>Object moved</title></head>
<body><h1>Object Moved</h1>This object may be found <a
HREF="xxxxxxx/">here</a>.</body>
<<

20080815-13:37:08 :INETCOUNT( A13AA0 )
>> 131 <<

------------------------------------
as I understand username and password are not sent


thanks
apolinar

when I use
oHttp:UseBasicAuth()
before
oHttppen()
I receive this in log
----------


20080815-14:43:42 :INETCONNECT( xxx.xxx.xx, 80 )
>> A13844 <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETSENDALL( A13844, 16, GET / HTTP/1.1 )
>> 16 <<

20080815-14:43:42 :INETSENDALL( A13844, 18, Host: xxx.xxx.xx )
>> 18 <<

20080815-14:43:42 :INETSENDALL( A13844, 36, User-agent: Mozilla/3.0
compatible )
>> 36 <<

20080815-14:43:42 :INETSENDALL( A13844, 19, Connection: close )
>> 19 <<

20080815-14:43:42 :INETSENDALL( A13844, 43, Authorization: Basic
aG90ZWw6SG80MjI2dXXX )
>> 43 <<

20080815-14:43:42 :INETSENDALL( A13844, 2, )
>> 2 <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETRECVLINE( A13844, , 500 )
>> HTTP/1.1 302 Object moved <<

20080815-14:43:42 :INETRECVLINE( A13844, , 500 )
>> Connection: close <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETRECVLINE( A13844, , 500 )
>> Date: Fri, 15 Aug 2008 12:44:37 GMT <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETRECVLINE( A13844, , 500 )
>> Server: Microsoft-IIS/6.0 <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETRECVLINE( A13844, , 500 )
>> X-Powered-By: ASP.NET <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETRECVLINE( A13844, , 500 )
>> Location: xxxxxxx/ <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETRECVLINE( A13844, , 500 )
>> Content-Length: 131 <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETRECVLINE( A13844, , 500 )
>> Content-Type: text/html <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETRECVLINE( A13844, , 500 )
>> Set-Cookie: ASPSESSIONIDQSSSAASR=IHODJGBCPGEIKELCIDDIHBJC; path=/ <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETRECVLINE( A13844, , 500 )
>> Cache-control: private <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETRECVLINE( A13844, , 500 )
>> <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETERRORCODE( A13844 )
>> 0 <<

20080815-14:43:42 :INETRECVALL( A13844, , 131 )
>> <head><title>Object moved</title></head>
<body><h1>Object Moved</h1>This object may be found <a
HREF="xxxxxxx/">here</a>.</body>
<<

20080815-14:43:42 :INETCOUNT( A13844 )
>> 131 <<

---------------
difference is in this
20080815-14:43:42 :INETSENDALL( A13844, 43, Authorization: Basic
aG90ZWw6SG80MjI2dXXX )
>> 43 <<
may be that authorisation is not basicauth and username and password
should be in hFields?
regards
apolinar

Klas & Apolinar:

I believe you folks are talking about code being placed in the page from
the server-side ??

Anyway, I hope you can suggest something in xHarbour for me to automate
the UserName and Password input fields and then click on the Submit button
from the *client's* side where my xHarbour app has to first log in (with my
*own* UserName and Password), then carry on with my automated page download
and parsing.

Of course, I have got to the stage where I can 'activate' the IE object,
then read in a page with innerHTML, then parse the text but I want to place
my name and password in the appropriate fields then 'click' submit
programmatically.

(btw, I didn't really understand how I could use Apolinar's code (with
Klas's fixes) to get this done.

Thank you

-Mel Smith

The diference between yours and mine approach is that you use internet
expolorer object model and I use httpcln xharbour http client object
model. I do not know if this approaches can be mixed

regards

Apolinar

Dear Mel Smith:

On Aug 15, 7:37*am, "Mel Smith" <medsyn...@aol.com> wrote:
...
> * * Anyway, I hope you can suggest something in
> xHarbour for me to automate the UserName and
> Password input fields and then click on the
> Submit button from the *client's* side where my
> xHarbour app has to first log in (with my *own*
> UserName and Password), then carry on with
> my automated page download and parsing.

You are able to set a program-scope variable with the contents of the
entire page.

You are able to set a program-scope variable with various defined
values (the Micro$haft link I provided you).

It works two ways. You can set them, create a web page, whatever.
Since you know the name of the fields, why don't you see if you can
set them. This means all that "blah blah" stuff identifying the
object with which you are communicating goes on the left of the ":=".

This will leave activating the "Submit" control.

David A. Smith

David said:

You are able to set a program-scope variable with the contents of the
entire page.

You are able to set a program-scope variable with various defined
values (the Micro$haft link I provided you).

It works two ways. You can set them, create a web page, whatever.
Since you know the name of the fields, why don't you see if you can
set them. This means all that "blah blah" stuff identifying the
object with which you are communicating goes on the left of the ":=".

This will leave activating the "Submit" control.


David:

I went into the MS IE explorer link you sent me in an earlier post.

The only 'property' that seemed reasonable to use is the: VALUE
Attribute | value Property.

where it implies that I should use: object.value := "melsmith"
and where the 'object' is, I guess, the 'name=txtRemoteUserName'
(i.e., the field for username)

and, as it implies, I should use: object.value := "654321"
and also where (for the password) the object is
'name=txtRemoteUserPassword' (i.e., the field for password)

The only 'method' that seems to be useful is: FORM.submit() ???

I'm trying to understand the syntax of how I can use these in my
xHarbour program ...

Thanks for trying to help but I'm more than a bit baffled (

-Mel Smith

Mel,

> (btw, I didn't really understand how I could use Apolinar's code (with
> Klas's fixes) to get this done.

Sorry for hijacking your thread. Apolinar and I talked a while back
about logging the http session to make it easier to analyze what is
going wrong. My fix was only about how to get the tIP classes to
create the log.

On the subject of sending login credentials and pressing buttons I'm
afraid someone else will have to help you with that.

Regards,
klas

-------
klas dot engwall at engwall dot com

http://www.engwall.com/clipper/

The LFN Library for Clipper
The LanMan Library for Clipper
The NFPAT1A Timeslice release patch for the Nanforum Toolkit