arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Browse Site by Tags

Showing related tags and posts across the entire site.
  • Operations: selecting and managing a collection of crawl requests

    Hi Mike and all, I'm evaluating AN for a client and my goal is to crawl a collection of URIs on a regular schedule. Essentially I will run AN as a service and schedule crawls for various sites by specifying those site URIs to AN. My understanding is that the table CrawlRequests is working memory...
    Posted to General Questions by jamesy on Wed, Jul 6 2011
  • Help Needed

    Hi, I would greatly appreciate any help, I am new to AN, i managed to get it up and running, It was actually not bad, just 4 easy steps and it's up and running, now here is what i am trying to achieve, 1) I have a list of web sites approximately 200 sites (these are job sites, job aggregators, companies...
    Posted to General Questions by vishal on Tue, Oct 6 2009
  • Re: Crawl pages created or modified 30 day ago

    Let's go with what I communicated over IM. There are several ways to achieve what (I think and hope) I understand your needs to be. The switches and modifications from the post were to support a batch-style analysis - but we actually need to implement a continuous crawling mechanism, which is what...
    Posted to General Questions by arachnode.net on Tue, Aug 11 2009
Page 1 of 1 (20 items)
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC