arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Browse Forum Posts by Tags

Showing related tags and posts for the arachnode.net group. See all tags in the site
  • Re: Crawling several sites with 1.2 version

    1.) You will need to write separate rules for each site, but one plugin will work. Else, how would the plugin know what information you want to pull? You can use UserDefinedFunctions.ExtractDomain or UserDefinedFunctions.ExtractHost to perform the filtering/switching. 2.) The easiest would be to Create...
    Posted to General Questions (Forum) by arachnode.net on Sun, Jul 26 2009
Page 1 of 1 (1 items)
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC