arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Browse Site by Tags

Showing related tags and posts across the entire site.
  • Re: HTTP POST Authentication

    1.) Look into writing a CrawlAction with a CrawlActionType of 'PreRequest'. You can make whatever modifications to the WebClient you desire. 2.) The CredentialCache functionality is very, very new and was requested by another user - and he was happy with the results. So, I'm open to suggestions...
    Posted to Feature Requests by arachnode.net on Sun, May 24 2009
  • Re: Credential Cache

    Right. The code is in the trunk and not part of a release... yet... http://arachnodenet.svn.sourceforge.net/viewvc/arachnodenet/trunk/SiteCrawler/Crawler.cs?revision=179&view=markup
    Posted to General Questions by arachnode.net on Thu, May 21 2009
  • Re: Deploy error

    I haven't explicitly tried downloading from an HTTPS site where a username/password combo was required.
    Posted to Bug Reports by arachnode.net on Fri, May 1 2009
  • Re: Deploy error

    Actually, I just added an overload to Crawler.cs so you can pass in a CredentialCache. Try that? Thanks! Mike
    Posted to Bug Reports by arachnode.net on Fri, Apr 17 2009
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC