arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop
Search the Live Index Does arachnode.net scale? | Download the latest release

Browse Forum Posts by Tags

Showing related tags and posts for the Forums application. See all tags in the site
  • "value cannot be null" Exception Message

    When I have SaveDiscoveredWebPagesToDisk and SaveDiscoveredFilesToDisk set to false, I get the Exception listed below in the Exceptions table regularly (I believe for every CrawlRequest). It appears that it tries to add a file to the Lucene.net index even though it isn't saving the files to the hard...
    Posted to Bug Reports (Forum) by bscott on Wed, May 25 2011
  • Re: Possible Robots.txt Bug - Release 2.5

    Ok, I updated to the latest version of Arachnode from the trunk. The previously described issue is resolved in this version, as you said. The robots.txt is still not being handled properly, though. I walked through the code in RobotsDotTextManager.cs, and discovered what the problem is. It appears that...
    Posted to Bug Reports (Forum) by bscott on Fri, May 13 2011
  • Possible Robots.txt Bug - Release 2.5

    Am I doing something wrong or is this a bug? What would you recommend I do? My install is from the release-2.5 tag. I realized that robots.txt wasn't working in my installation, so I walked through the code. When it's trying to read the robots.txt file, I found that on line 375 of SiteCrawler...
    Posted to Bug Reports (Forum) by bscott on Wed, May 11 2011
  • Throw two exceptions

    Hi Mike,i am running the ArachNode.net to get several website images,the WebClient.cs Throw an exception,the view follow: another,from ArachnodeDAO.cs.Maybe my log is full: Does anyone run through such behaviour or this is just me?Henry.Wen
    Posted to Bug Reports (Forum) by Henry.Wen on Thu, Sep 3 2009
  • Re: v1.0 first comments

    Hey! Thanks for checking this out for me... I had a pretty serious situation occur over the weekend so I didn't have the time to test a fresh install. Here's what I did to fix the bugs you found. 1.) Updated the release. \source and \database are at the 'root' of the .zip folder. 2.)...
    Posted to Bug Reports (Forum) by arachnode.net on Mon, Jan 5 2009
Page 1 of 1 (5 items)
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC