arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Throw two exceptions

rated by 0 users
Answered (Verified) This post has 1 verified answer | 3 Replies | 2 Followers

Top 50 Contributor
10 Posts
Henry.Wen posted on Thu, Sep 3 2009 12:35 AM

Hi Mike,i am running the ArachNode.net to get several website images,the WebClient.cs Throw an exception,the view follow:

 

another,from ArachnodeDAO.cs.Maybe my log is full:

 

Does anyone run through such behaviour  or  this is just me?Henry.Wen

Answered (Verified) Verified Answer

Top 10 Contributor
Male
101 Posts
Answered (Verified) Kevin replied on Fri, Sep 4 2009 1:51 PM
Verified by arachnode.net

I will go in sometimes and under

Debugging->Exceptions for CLR, go to System.Net.HttpExceptions

and I'll uncheck both options and these errors (which during debugging I don't care about) should not be shown.

All Replies

Top 10 Contributor
Male
101 Posts
Kevin replied on Thu, Sep 3 2009 8:15 AM

Henry, did you try just tweaking the code real quick to HACK ignoring 404 errors along with the 406 errors already in the code?  Maybe that's a quick way to get around it.  Not sure why the Just My Code option isn't working for you.  I'll try setting it and see if it works for me.

You might post the url you are submitting for crawl (if you haven't done it already) so others can try the crawl and duplicate the issue.

On the error writing the exception to the db, I wonder if there's a data issue related to the language.  Will have to try and duplicate with your particular site.

Kevin

 

Top 10 Contributor
1,905 Posts

Right above the text in your image there's the link to a forum post: http://arachnode.net/forums/p/321/10290.aspx

Thanks!
Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
Male
101 Posts
Answered (Verified) Kevin replied on Fri, Sep 4 2009 1:51 PM
Verified by arachnode.net

I will go in sometimes and under

Debugging->Exceptions for CLR, go to System.Net.HttpExceptions

and I'll uncheck both options and these errors (which during debugging I don't care about) should not be shown.

Page 1 of 1 (4 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC