Showing related tags and posts across the entire site.
-
Check out the CrawlRule AbsoluteUri.cs. Find all references to it and read the code. This rule shows how arachnode.net can parse and filter absoluteuris before and after crawling, and how to completely ignore those that don't confirm to your rules. AN does use regular expressions, but a better way...
-
Address has been merged with AbsoluteUri. The AbsoluteUri CrawlRule references the following tables: If you wanted to Crawl msn.com, delete all rows in the tables above, insert msn.com into DisallowedDomains, and set IsDisallowed=true for msn.com and set the following setting in CrawlRules.config: negateIsDisallowedForAbsoluteUri...