arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Browse Forum Posts by Tags

Showing related tags and posts for the Bug Reports forum. See all tags in the site
  • Re: Feedback on latest build and playing

    I think I'll put some thought into how to auto balance domains across priorities in the priority queue. I did some testing on 4 or 5 other priority queues I found on the NET and the arachnode.net and the one I wrote has a good deal of distance between it and the next fastest one. So, there is some...
    Posted to Bug Reports (Forum) by arachnode.net on Tue, Feb 17 2009
  • Re: Feedback on latest build and playing

    Now that the version in source control crawls by priority, I need to determine an algorithm to filter by priority. e.g. When the Engine is populated with CrawlRequests, the Engine will crawl all CrawlRequests by priority, but this means from batch to batch to batch the Engine is still crawling by order...
    Posted to Bug Reports (Forum) by arachnode.net on Sun, Feb 15 2009
Page 1 of 1 (2 items)
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC