arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Unable to read data from the transport connection: The connection was closed.

rated by 0 users
Not Answered This post has 0 verified answers | 6 Replies | 2 Followers

Top 10 Contributor
229 Posts
megetron posted on Mon, Sep 20 2010 8:38 AM

Hello,

I keep getting this:
Unable to read data from the transport connection: The connection was closed.

I made a research and it seems like it has something to do with a proxy.

I found nice link:

http://forums.asp.net/t/1091361.aspx

a guy named  mayank.prajapati offers a solution. maybe ti can help.

Please let me know if you find this helpfull for the problem?

Thanks,

All Replies

Top 10 Contributor
1,905 Posts

What is the code that fixes your problem?

Do you need to access a proxy server?

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
229 Posts

At start you crawl fine, but after a while you get this:Unable to read data from the transport connection:

After googling a bit:
"Turned out that if the sites failing used cookies – and therefore my request failed. So the correct syntax to fetch pages is instead"
http://aspcode.net/unable-to-read-data-from-the-transport-connection

 

If you would add this to AN it will eorks fine:

 

 

CookieContainer CC = new CookieContainer();
HttpWebRequest.Proxy =
null;
HttpWebRequest.UseDefaultCredentials =
true;
HttpWebRequest.CookieContainer = CC;

Top 10 Contributor
1,905 Posts

I checked my crawl that has been going for a month and I found 2 of the error messages you received.  Wonder why you have so many more?

I made the change and checked it in.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
229 Posts
megetron replied on Tue, Sep 21 2010 11:47 AM

do not make any changes...this is still under investigation.

seems like it still happens, and this code only delays it from happens....

Someone suggested to lower the MTU.

I crawl using a multi-server option. I wi try to crawl differently and let you know.

Top 10 Contributor
229 Posts
megetron replied on Tue, Sep 21 2010 12:31 PM

After changing the MTU(through registry) i still recieve error.

Mike,

I found that this error repeat itself only when you crawls from another machine inside a network, where the SQLServer machine hosted in a different machine.

Foe example: PC1 host the sqlserver, and using dialup connection.

PC2 run the AN executable, and connects to internet through a shared internet connection of PC1.

This scenario will popups this error.

I don't have a solution for this. but the code above doess NOT solve, so check it out please.

At the meantime I must continue with thw crawk, so  I will do the crawl through PC1 from the example above.

If you think you know what the issue, post code here, so I can test this,

Thanks,.

 

 

 

Top 10 Contributor
1,905 Posts

Can you post all of the stack traces for the exceptions?

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (7 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC