Re: Getting started: crawling multiple sites

arachnode.net — Wed, 28 Apr 2010 19:42:11 GMT

Looks like you have been registered for quite some time, mopiola.

Which version of AN are you using?

I wouldn't add CR's to the CR table, but rather add them as shown in Program.cs. This table is used by the Cache/Engine.

Read this post about restricting a crawl: http://arachnode.net/forums/t/739.aspx

To restrict to a specific directory you would need to write a plugin. There is a file, AbsoluteUri.cs, which you may or may not have... (it's not in a demo as the SiteCrawler project is compiled/obfuscated/encrypted)... which shows how you would accomplish filtering. There is also a new property of Discoveries and CrawlRequests (IsStorable) which allows you to crawl through a site and only store what you want, rather than being blocked by IsDisallowed. Very handy, but only in the newest release version. If you purchase a license I will write the plugin for you.

Let me know how you would like to proceed. I would love to help you!

- Mike

Search results matching tags 'IsStorable' and 'IsDisallowed'

Re: Getting started: crawling multiple sites