arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release
Memory Conditions To Avoid

arachnode.net can be configured to use a set amount of RAM in Megabytes.

If set too high, AN will require virtual memory to store and reference Discoveries and CrawlRequests stored in Cache.cs.  Virtual memory utilization should be avoided.

Also, SQL's cache should be set within the bounds of available physical RAM.

The server setting illustrated allows SQL to use as much RAM as is available.  If set too high, AN and SQL will compete for RAM.  This should be avoided.

If either are set too high, the following condition may result.

Note the 'Cached', 'Available' and 'Free' in Physical Memory (MB).  Low numbers should be avoided.

In summary, ensure that both AN and SQL are restricted to available physical RAM.


Posted Fri, Aug 27 2010 5:01 PM by arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC