arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Browse Site by Tags

Showing related tags and posts across the entire site.
  • Exceptions query, do my results look typical

    When I run the following query SELECT LEFT(Message, 39) AS Expr1, COUNT(ID) AS Expr2 FROM Exceptions AS Exceptions_1 GROUP BY LEFT(Message, 39) HAVING (LEFT(Message, 39) LIKE 'The remote name could not be resolved:%') UNION SELECT Message AS Expr1, COUNT(ID) AS Expr2 FROM Exceptions GROUP BY...
    Posted to General Questions by DataMan on Fri, Apr 9 2010
  • Crawl specific domains, take screenshot, run javascript, ID ads, write to DB ... any ideas?

    We want to capture info about AD SPOTS; - on specific list of approx 10,000 domains - capture a screenshot png / jpg - run javascript on each page (browser specific?) - read js and identify ad spot SIZES - identify PLACE on page that the adspot is located - relate this location back to the screenshot...
    Posted to General Questions by jpntol on Sat, Feb 14 2009
  • Re: Walking thru console program.cs part 1

    Kevin - I will answer all of your posts. Just got back from band practice - need to get to bed so I'm fresh for work tomorrow. I really appreciate all of the questions and posts! OK - back... 1.) The static objects are artifacts used by helper code that was moved to the Utilities project. They shouldn't...
    Posted to General Questions by arachnode.net on Tue, Jan 27 2009
Page 1 of 1 (20 items)
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC