arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Browse Site by Tags

Showing related tags and posts across the entire site.
  • Search Keyword in Single Input URL

    Hi All, I have downloaded the demo application from arachnode.net website, I want to search keyword in a single URL that the user will input. Can anyone suggest me the change required in demo application so that i can give harcoded input for URL in the code and get results for the keyword from URL. Any...
    Posted to General Questions by Syed Rizwan ul haq on Mon, Jul 13 2015
  • Re: Is it possible to create many instances of the crawler running at the same time.

    I am making a change to the core which will allow you to adjust the CrawlActions, CrawlRules and EngineActions before Crawling, per Crawl instance, just like you can with ApplicationSettings and WebSettings. http://arachnode.net/blogs/arachnode_net/archive/2010/05/06/controlling-configuration-from-code...
    Posted to Feature Requests by arachnode.net on Fri, May 7 2010
  • Re: Getting started: crawling multiple sites

    Looks like you have been registered for quite some time, mopiola. Which version of AN are you using? I wouldn't add CR's to the CR table, but rather add them as shown in Program.cs. This table is used by the Cache/Engine. Read this post about restricting a crawl: http://arachnode.net/forums/t...
    Posted to General Questions by arachnode.net on Wed, Apr 28 2010
  • Re: crawling specific web sites for tag words

    Hey - I'm back from my mini-vacation to the Washington coast. The site was disallowed as, by default, arachnode.net follows robots.txt rules. If you want to turn off the robots.txt behavior, check the 'CrawlRules' table, find the robots.txt rule and turn it off. No worries on being new to...
    Posted to General Questions by arachnode.net on Thu, Aug 6 2009
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC