arachnode.net
An open source .NET web crawler written in C# using SQL 2005/2008
All Tags » crawling a single site » lucene.net (RSS)

Browse Forum Posts by Tags

Showing related tags and posts for the Forums application. See all tags in the site

  • Re: How to restrict crawl to single domain?

    After all of the CrawlRequests are crawled, if you have CreateCrawlRequestsFromDatabaseHyperLinks set to true or CreateCrawlRequestsFromDatabaseHyperLinks set to true then arachnode.net will create CrawlRequests from those AbsoluteUris. There is an order in which CrawlRequests are processed. a.) CrawlRequests...
    Posted to General Questions (Forum) by arachnode.net on 11 Feb 2009
Page 1 of 1 (1 items)
An open source .NET web crawler written in C# using SQL 2005/2008

copyright 2004-2010, arachnode.net LLC

Powered by Community Server (Non-Commercial Edition), by Telligent Systems