arachnode.net v2.0
An open source .NET web crawler written in C# using SQL 2005/2008

arachnode.net

My Bio

Hi!  I'm Mike Anderson, the creator of arachnode.net.  I live in Seattle, Washington.

I'm a fan of NClassifier, SharpNLP and OpenTextSummarizer.

I enjoy running, lifting weights and playing the guitar, and, of course, coding arachnode.net.

Follow me on twitter: http://twitter.com/arachnode_net

Join the arachnode.net group on Facebook: http://www.facebook.com/groups.php?ref=sb#/group.php?gid=166721755872

I hope you enjoy the experience of arachnode.net.

dotnetdotcom.org - contact me please...

http://search.arachnode.net/Search.aspx?query=computer&discoveryType=WebPage&pageNumber=1&pageSize=10&shouldDocumentsBeClustered=1

Announcements

  • 12-01-2009
  • thrombibulator

    thrombibulator

  • 10-01-2009
  • Great marketing link!
  • 09-29-2009
  • Just a random quoted string not found in Google.com

    "must be installed on a physical computer"

    must be installed on a physical computer

  • 09-04-2009
  • Memory Consumption Greatly Improved!

    Memory Consumption Greatly Improved!

    Many thanks to Megetron for helping pinpoint some of the memory issues that crept up between Version 1.1 and Version 1.2

    It is now possible to crawl at an infinite depth (int.Max) without consuming all of the RAM on your machine.

    10 threads crawling at 128 MB of RAM maximum, from 'http://msn.com' at a Depth of int.Max is properly respecting the maximum memory settings and has done so for the last hour.  Before the memory fixes, it was expected that AN would consume up to and beyond the maximum memory settings, often doubling what was requested.

    Additionally, the new memory fixes allow AN to crawl faster as well, with each thread processing 2.5 times per seconds what each thread could process before.

    Thanks to all!  Big Smile

  • 08-20-2009
  • Version 1.3 is coming...

    ...and focuses on improvements to Encoding, respecting the LastModified HttpResponse header as well as some subtle improvements to memory management when crawling at large Depths.

    Thanks!

My Comments

Milo wrote Re:
on 07-07-2009 9:28 AM

You damn right she is :D

An open source .NET web crawler written in C# using SQL 2005/2008

copyright 2009, arachnode.net LLC

Powered by Community Server (Non-Commercial Edition), by Telligent Systems