arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

An unhandled exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll

rated by 0 users
Answered (Verified) This post has 1 verified answer | 5 Replies | 1 Follower

posted on Thu, Dec 24 2009 7:57 AM

Hi,

        I am newbiee here.When I am trying to crawl my website which is very large like a social networking site.The crawling is running fine for few hours later it is throwing the following exception.

An unhandled exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll

Can anyone let me know, what can I do here to get rid of this,can't we crawl this sort of a website.

Waiting for your help.Thanks in advance.

 

 

 

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

Which version are you using?

If you are using Version 1.2 you may encounter this error.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

All Replies

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

Which version are you using?

If you are using Version 1.2 you may encounter this error.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

replied on Thu, Dec 24 2009 7:07 PM

Thanks for your quick reply. You have been helping me a lot in Arachnode.net by giving replies to my posts.

Yes. You are correct. I am using 1.2 only.Can you let me know, what version can I use to get rid of this issue.

FYI : I am just using the open source, not the licensed version

 

 

Top 10 Contributor
1,905 Posts

You would have to upgrade to Version 1.3 or greater, unfortunately.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

replied on Sat, Dec 26 2009 8:16 PM

How can I upgrade to 1,3 version? I have googled but I did not find any download links for 1.3 version.if you don't mind can you give me the downloadable links.

Thanks.

Top 10 Contributor
1,905 Posts

Version 1.3 and above are paid products.  AN is on sale right now, for the holidays, and will be increasing after the 31st.

http://arachnode.net/Content/Commercial.aspx

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (6 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC