arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Browse Site by Tags

Showing related tags and posts across the entire site.
  • 404 is returned because trailing slash is not used

    When I crawl this site: http://www.jenkinskling.com the following response is returned: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <title>R a z o r B a l l</title> <meta http-equiv="Content-Type" content="text...
    Posted to General Questions by canuckbbp on Thu, Nov 10 2011
  • Re: Exceptions query, do my results look typical

    Looks normal to me. You can always spot check a few. Things to keep in mind. 1.) Sites may only allow you to retreive X Discoveries in any given period. 2.) This value varies widely from site to site, athough you may not hit their thresholds due to crawling speeds and attempted politeness. (Round-robin...
    Posted to General Questions by arachnode.net on Fri, Apr 9 2010
  • Re: Problem downloading from php sites

    It doesn't look like your page ' http://www.napoli2nord.it/aziende.php#bandi?bandi.php ' is experiencing any errors that aren't caused by your configuration. In the DisallowedAbsoluteUris table, ' http://www.napoli2nord.it/aziende.php#bandi?bandi.php ' isn't listed. Unless...
    Posted to General Questions by arachnode.net on Sat, Mar 27 2010
  • Throw two exceptions

    Hi Mike,i am running the ArachNode.net to get several website images,the WebClient.cs Throw an exception,the view follow: another,from ArachnodeDAO.cs.Maybe my log is full: Does anyone run through such behaviour or this is just me?Henry.Wen
    Posted to Bug Reports by Henry.Wen on Thu, Sep 3 2009
  • Re: The remote server returned an error: (404) Not Found

    404 errors don't come from robots.txt files. Check out 'UserAgent' in the 'Configuration' database table. You are welcome.
    Posted to Bug Reports by arachnode.net on Fri, Aug 7 2009
Page 1 of 2 (40 items) 1 2 Next >
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC