arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE
Does arachnode.net scale? | VS2008/2010/2012 & SQL2008/2012 | Download the latest release

Browse Site by Tags

Showing related tags and posts across the entire site.
  • Re: What new feature should I finish first?

    Hey there! Find Anonymizer.cs. Enable the CrawlAction in cfg.CrawlActions. The plugin performs a string replacement on the AbsoluteUri submitted for crawling, so that the HttpWebRequest is fulfilled through an intermediary. crawlRequest.Discovery.Uri = new Uri (_anonymizerAbsoluteUri + crawlRequest.Discovery...
    Posted to Feature Requests by arachnode.net on Sun, Jun 20 2010
  • Re: Anonymouse Crawling

    I added an Anonymizer plugin to the branch so you can see how this would be implemented. Don't forget to check out the DB too... (Branch is a branch, but quite viable...) This code is checked into the trunk now. Mike
    Posted to Feature Requests by arachnode.net on Mon, Jun 1 2009
Page 1 of 1 (20 items)
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE

copyright 2004-2013, arachnode.net LLC