arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2005/2008/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop
Mongo/Raven/MySQL/Hadoop Does arachnode.net scale? | VS2008/2010/2012 & SQL2008/2012 | Download the latest release

Browse Forum Posts by Tags

Showing related tags and posts for the Forums application. See all tags in the site
  • Possible Robots.txt Bug - Release 2.5

    Am I doing something wrong or is this a bug? What would you recommend I do? My install is from the release-2.5 tag. I realized that robots.txt wasn't working in my installation, so I walked through the code. When it's trying to read the robots.txt file, I found that on line 375 of SiteCrawler...
    Posted to Bug Reports (Forum) by bscott on Wed, May 11 2011
  • Parameter is not valid - may be 304 response issue for images and maybe more

    In reviewing Parameter is not valid errors in Exceptions table, it appears that this may be due to 304 responses. In testing, I get this error on images returned. Using firebug to see what exactly comes back when these particular absoluteuri images are requested, I see that the first time I get a 200...
    Posted to Bug Reports (Forum) by Kevin on Fri, Sep 4 2009
Page 1 of 1 (2 items)
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE

copyright 2004-2014, arachnode.net LLC