-
Hello again...I am the same user as this question: http://arachnode.net/forums/t/1445.aspx I just bought a license and am still wondering this same question. Here is the question again: I would like to restrict part of a domain from crawling. I am trying to crawl a site and a "site-map like page"...
-
I would like to restrict part of a domain from crawling. I am trying to crawl a site and a "site-map like page" has many links that go to many different parts of the site. (E.g: the footer and header of the page contains many different types of links that go to various pages that are not of...
-
Megetron, I have a situation where I build lists of particular web sites I want to crawl. I do it via a sql script that will submit the crawl requests, but it may give you some ideas. In the sql I just do something like this: declare @id bigint declare @url varchar(255) declare @depth int declare @restrictcrawl...
-
They correspond to the enum: namespace Arachnode.SiteCrawler.Value.Enums Flags ] { [ public enum UriClassificationType : byte { None = 0, Domain = 1, Extension = 2, FileExtension = 4, Host = 8, Scheme = 16 } } RestrictCrawlTo means that the Crawl won't crawl WebPages that aren't the same UriClassificationType...