arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Browse Site by Tags

Showing related tags and posts across the entire site.
  • Disallowed Directories

    Hi Mike, Following our conversation, I think it might be a good idea to add the feature to limit the crawl to a specific Directory. For example - If I have a crawl request with "http://edition.cnn.com/US/" as URI, I will want only results from this directory down (i.e http://edition.cnn.com...
    Posted to Feature Requests by Anat on Sun, Dec 13 2009
  • Batch Submit Crawl Requests Via EngineAction and CSV File(s)

    I used to create manual t-sql scripts that would "batch up" domains I wanted to crawl, and submit them via the crawl insert stored procedure. However, this is manual, requires t-sql, and manually inserting is not a good idea versus doing it through the engine. So, how about we add an EngineAction...
    Posted to Feature Requests by Kevin on Tue, Sep 15 2009
Page 1 of 1 (20 items)
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC