arachnode.net

thrombibulator

http://www.codinghorror.com/blog/archives/001301.html

"must be installed on a physical computer"

must be installed on a physical computer

Memory Consumption Greatly Improved!

Many thanks to Megetron for helping pinpoint some of the memory issues that crept up between Version 1.1 and Version 1.2

It is now possible to crawl at an infinite depth (int.Max) without consuming all of the RAM on your machine.

10 threads crawling at 128 MB of RAM maximum, from 'http://msn.com' at a Depth of int.Max is properly respecting the maximum memory settings and has done so for the last hour.  Before the memory fixes, it was expected that AN would consume up to and beyond the maximum memory settings, often doubling what was requested.

Additionally, the new memory fixes allow AN to crawl faster as well, with each thread processing 2.5 times per seconds what each thread could process before.

Thanks to all! 

...and focuses on improvements to Encoding, respecting the LastModified HttpResponse header as well as some subtle improvements to memory management when crawling at large Depths.

Thanks!