-
I am making a change to the core which will allow you to adjust the CrawlActions, CrawlRules and EngineActions before Crawling, per Crawl instance, just like you can with ApplicationSettings and WebSettings. http://arachnode.net/blogs/arachnode_net/archive/2010/05/06/controlling-configuration-from-code...
-
Looks like you have been registered for quite some time, mopiola. Which version of AN are you using? I wouldn't add CR's to the CR table, but rather add them as shown in Program.cs. This table is used by the Cache/Engine. Read this post about restricting a crawl: http://arachnode.net/forums/t...
-
Hey - I'm back from my mini-vacation to the Washington coast. The site was disallowed as, by default, arachnode.net follows robots.txt rules. If you want to turn off the robots.txt behavior, check the 'CrawlRules' table, find the robots.txt rule and turn it off. No worries on being new to...