arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Browse Forum Posts by Tags

Showing related tags and posts for the General Questions forum. See all tags in the site
  • Re: Setting arachnode up for rss-collection

    Just for others that read this post... when troubleshooting AN, start by looking in the database tables 'Exceptions' and 'DisallowedAbsoluteUris'. For your question: Take a look at cfg.AllowedDataTypes. This table controls what you are allowed to crawl. First step it to make sure that...
    Posted to General Questions (Forum) by arachnode.net on Thu, Aug 20 2009
Page 1 of 1 (1 items)
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC