-
OK, how about this: If you want to crawl 500 domains you would configure arachnode.net to restrict Crawls to those 500 domain only like the posts above describe how to do. Then, make sure your settings in Application.config are set as shown. The Crawl process works like this if you have the settings...
-
After all of the CrawlRequests are crawled, if you have CreateCrawlRequestsFromDatabaseHyperLinks set to true or CreateCrawlRequestsFromDatabaseHyperLinks set to true then arachnode.net will create CrawlRequests from those AbsoluteUris. There is an order in which CrawlRequests are processed. a.) CrawlRequests...