The request was aborted: The connection was closed unexpectedly.

Reply

megetron replied on Wed, Aug 25 2010 2:33 AM

rated by 0 users

Yes you right. my mistake. should follow server instructions.

thanks

Reply

megetron replied on Thu, Aug 26 2010 3:30 AM

rated by 0 users

How can I know what is the reason for that? I am using IE7 and I can see  a page, but when trying to crawl the same page, I get the message...

I even replace IP, but stil the same.

Please advice, thank you,

Reply

arachnode.net replied on Thu, Aug 26 2010 10:33 AM

rated by 0 users

Try setting the UserAgent to a browser UserAgent.

Also, many web servers will only allow so many concurrent connections from any one IP.

This is a good post on server behavior: http://arachnode.net/blogs/arachnode_net/archive/2010/04/29/troubleshooting-crawl-result-differences-between-different-crawl-environments.aspx

Reply

arachnode.net replied on Thu, Aug 26 2010 11:22 PM

rated by 0 users

Nice link set:

List of User Agent Strings

Reply

megetron replied on Fri, Aug 27 2010 2:49 AM

rated by 0 users

Hi, after investigating this further:

1, rename user agent several useragents of IE.
2. Changing the webclient.Cs headrs:

             HttpWebRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
            HttpWebRequest.Headers.Add("UA-CPU", "x86");
            HttpWebRequest.Headers.Add("Accept-Language", "en-us");
            HttpWebRequest.Headers.Add("Pragma", "no-cache");

but still the error exists...

So I used fiddler and I find that the page is downloaded partially and stops in the middle. the webserver closes connection.

Now I tried to crawl only one thread in frequence of 10 seconds, and still happens.

Funny thing is that when debug the appllication it's seems like there are 10 threads running and 1 as I defined in the configuration table...

:\