arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release
New arachnode.net Version / Platform (AN.Next)

AN.Next is aimed at the non-SQL Server inclined/non-Mid- to Senior level Developer market and will use the DistributedCache for state, storage and indexing and is configured much like memcached is.

AN.next will be able to use SQL Server for storage but isn't required to do so, and will allow distribution and fault tolerance much like Cassandra, but without requiring implementing transactional replication.

Plugins are accomplished through multicast delegates, thereby making crawl pipeline modification even easier.

Class overview.  (Subject to change)

 


Posted Sun, Dec 5 2010 4:35 PM by arachnode.net
Filed under:
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC