Showing related tags and posts across the entire site.
-
I think I'll put some thought into how to auto balance domains across priorities in the priority queue. I did some testing on 4 or 5 other priority queues I found on the NET and the arachnode.net and the one I wrote has a good deal of distance between it and the next fastest one. So, there is some...
-
Now that the version in source control crawls by priority, I need to determine an algorithm to filter by priority. e.g. When the Engine is populated with CrawlRequests, the Engine will crawl all CrawlRequests by priority, but this means from batch to batch to batch the Engine is still crawling by order...