Does anybody know some C or Perl module/function (php is also okay, but
at a lower preference ..) for logging/ reading/ fetching the http
request headers ??

or some other sugesting ways ??

thanks 1024^2 times.

boy
: >

I believe a CGI program gets the HTTP headers as environment
variables, or possibly on stdin.  Any decent document on CGI should
have the details.

--

Nate Eldredge
neldre...@hmc.edu

the general way is to get the redirection url asked from the server and then
sent the necessary info, url, cookies, content (password, username ...etc)
to that
url and then get the url and cookies, and then have them sent back and the
process repeat and repeat till the required page was retrived.

I'm very sure that none of the info(url, cookies, headers, method ... etc)
was missed in the request. And the http response header were also taken
down.

The bot works fine for some web based mail like mail.yahoo.com,
mail.e{*filter*}.com, mail.altavista.com, contact.lycosasia.com ... just some
famous commerical
web based mail.

But the bot just failed for Microsoft hotmail.com

I'ved tried 8 other web based mail and the bot works fine.

I've tried to fit in the necessary http request headers in the bot while
sending a http request, including pragma, content-type, charset, referer,
user-agent ... etc etc.

HTTP/1.0 rfc 1945 was also read and the communication standard was followed
during the communication for the bot.

But just the bot fail ONLY on hotmail.com, can never login, and the of
course can't read the mails.

So the bot is just to simulate the work done from a browser.

So we can read our mails everyday from hotmail.com by browser (Netscap or
IE), and hard to encounter problems.

So the browser works fine, and the bot doesn't work fine, and just fail on
hotmail.com

So what I doubt is that the http request headers sent was different for the
bot and the browsers.

So it would be very great if there's some modules/ ways to log EXACTLY the
full http request headers,  sent from the client through a browser,
or retrived from the server side sent from a client through browser.

Thanks for helping on it and thanks for any other suggestion for it.

thanks 2*1024^2 times.

boy
: >

Well, here is the request:

POST / HTTP/1.0
Connection: Keep-Alive
User-Agent: Mozilla/4.77 [en] (X11; U; Linux 2.2.16 i686)
Host: localhost:8888
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
Content-type: application/x-www-form-urlencoded
Content-length: 214

login=name&domain=hotmail.com&svc=mail&RemoteDAPost=https%3A%2F%2Flogin.msnia.passport.com%2Fppsecure%2Fpost.asp&passwd=passwd&enter=Sign+in&sec=no&curmbox=ACTIVE&js=yes&_lang=EN&beta=&ishotmail=1&id=2&ct=990646153

Note that actual request would be sent to https server.

Hope this helps,

--
Alex Mitrofanov
Brainbench MVP for Unix Programming
http://www.**-**.com/

Why not just look at the network traffic then?  A tool like snoop or
ethereal or tcpdump should let you see all the traffic over a given
network connection.  You can see exactly what the browser is saying to
the server, and what the server is saying back, then compare it to
your program.

Hope this helps.

--

Nate Eldredge
neldre...@hmc.edu

thanks.

@: >

instead of having request sent to https server, the request can also ust be sent to http server.

you can make a html page like this
<form action= http://www.**-**.com/ ; <!--http server-->
<input type=text name=login value=username>
<input type=password name=passwd value=userspasswd>
<input type=submit value='press to submit'>
</form>

and then just fill in the required 2 fundamental info username and password and you can login with browser.

but thanks for your help anyways.

boy
: >

I used netcat to catch it.
Save page into file. Edit form header

<form action=http://localhost:8888 method=post>

Run netcat

nc -lp8888

Load page in browser, fill it and submit.

--
Alex Mitrofanov
Brainbench MVP for Unix Programming
http://www.brainbench.com

I've tried to fit in exactly the same header with the related header info.

While logging into hotmail,

First
POST with the username and password to
http://lc1.law5.hotmail.passport.com/cgi-bin/dologin  (other hidden field content CAN be
ignored), just username, password and the url correct will be okay.

and then the server will give out a response with Set-cookie header, Location header
with a url (response-url), and other headers.

Then the user agent will have to use GET method to get the above url (response-url)
just feedback from response from the server.

And at this GET method process, I failed.

Usually, netscape will post 6 header during the process GET.
1 request line
GET a-path HTTP/1.0

and 6 headers
connection: Keep-Alive
User-Agent: Mozilla/4.7 [en] (Win95; I)
Host: sth
ACCEPT: sth
ACCEPT_CHARSET: sth
ACCEPT_LANGUAGE: sth

and cookie header if some cookie matched.

Thanks for Alex that I use netcat to catch EXACTLY the above http headers.

interesting and strange thing is -- the above header 'HOST' doesn't mentioned in the rfc,
http/1.0 rfc 1945 and http/1.1 rfc 2068 ......
and the perl document from HTTP/Headers also don't talk about the header 'HOST' .............

And when we just use browser bypass the 1st POST action but have had the url responsed (response-url)
noted down and just let the browser GET the url (response-url), we can still
log in in hotmail.

but when I use perl script, or telnet to the host with port 80, host retrived from the above url (response-url),
just any other way to get the url (response-url) not using browser, with
the same header fill in, I failed, failed to login to hotmail ...

Don't know why.

The browser is just to feed in some headers, in the request process with method GET
to the url (response-url), it succeed.

The action using perl script, or to telnet to the host from the url (response-url) 80, with
the 6 headers with corresponding header info fill in, I failed.

So the process between the browser and my telnet action should be the same.

Don't know why hotmail can identify their difference and the login process failed
without using browser.

I've tried to change the file /etc/services and the telnet port is changed to 80, and then
use the telnet action again, just still failed.

Anything I missed ? or some suggestions ?

The similar process was found to be success to other freemails, portals,
I've tried 8 other freemails, and some other portals, tried to login to freemail,
to get pages from portals, and totoally I guess about 30,
And all passed.

But just failed for hotmail ......
and don't have some concrete idea about the failure, and just for hotmail.com.

If anyone interested, then just can randomly get an account from hotmail and then have some scripts or
programms or just telnet manually to the host with port 80 and feed in some headers, and get responses
from the server and retrive required info from the resposne and repeat similar process, to try to login
to hotmail not using browser to have a try on it.

Thanks for helping on it and thanks for any other suggestion for it.

Thanks 3*1024^2

boy
: >

I've tried to fit in exactly the same header with the related header info.

While logging into hotmail,

First
POST with the username and password to
http://lc1.law5.hotmail.passport.com/cgi-bin/dologin  (other hidden field content CAN be
ignored), just username, password and the url correct will be okay.

and then the server will give out a response with Set-cookie header, Location header
with a url (response-url), and other headers.

Then the user agent will have to use GET method to get the above url (response-url)
just feedback from response from the server.

And at this GET method process, I failed.

Usually, netscape will post 6 header during the process GET.
1 request line
GET a-path HTTP/1.0

and 6 headers
connection: Keep-Alive
User-Agent: Mozilla/4.7 [en] (Win95; I)
Host: sth
ACCEPT: sth
ACCEPT_CHARSET: sth
ACCEPT_LANGUAGE: sth

and cookie header if some cookie matched.

Thanks for Alex that I use netcat to catch EXACTLY the above http headers.

interesting and strange thing is -- the above header 'HOST' doesn't mentioned in the rfc,
http/1.0 rfc 1945 and http/1.1 rfc 2068 ......
and the perl document from HTTP/Headers also don't talk about the header 'HOST' .............

And when we just use browser bypass the 1st POST action but have had the url responsed (response-url)
noted down and just let the browser GET the url (response-url), we can still
log in in hotmail.

but when I use perl script, or telnet to the host with port 80, host retrived from the above url (response-url),
just any other way to get the url (response-url) not using browser, with
the same header fill in, I failed, failed to login to hotmail ...

Don't know why.

The browser is just to feed in some headers, in the request process with method GET
to the url (response-url), it succeed.

The action using perl script, or to telnet to the host from the url (response-url) 80, with
the 6 headers with corresponding header info fill in, I failed.

So the process between the browser and my telnet action should be the same.

Don't know why hotmail can identify their difference and the login process failed
without using browser.

I've tried to change the file /etc/services and the telnet port is changed to 80, and then
use the telnet action again, just still failed.

Anything I missed ? or some suggestions ?

The similar process was found to be success to other freemails, portals,
I've tried 8 other freemails, and some other portals, tried to login to freemail,
to get pages from portals, and totoally I guess about 30,
And all passed.

But just failed for hotmail ......
and don't have some concrete idea about the failure, and just for hotmail.com.

If anyone interested, then just can randomly get an account from hotmail and then have some scripts or
programms or just telnet manually to the host with port 80 and feed in some headers, and get responses
from the server and retrive required info from the resposne and repeat similar process, to try to login
to hotmail not using browser to have a try on it.

Thanks for helping on it and thanks for any other suggestion for it.

Thanks 3*1024^2

boy
: >

It's solved.

Thanks all who have helped.

Thanks  3*1024^2 times.

boy
: >

but the browsers Netscape and IE  must have these 2 headers sent, with
values.

So what doesn't specified inRFC but the most commonly used browsers must
have them sent for each communications, and this annonyed people for a long
long time .......

Is that very interesting ??

Anybody know why ??

@: <