Showing related tags and posts across the entire site.
I do believe that the WebClient already takes this in account, FWIW - need to check for sure.
The WebPages table stores InitiallyDiscovered, LastDiscovered and LastModified. If no CrawlRequests exists in the CrawlRequests table and all HyperLinks exist in the WebPages table, then the crawler will move on to crawling the WebPage(s) that were LastDiscovered the longest ago. arachnode.net doesn't...