arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Browse Site by Tags

Showing related tags and posts across the entire site.
  • Re: What is the best method to parse html tags!

    Milan Solanki: > What would be the best method of parsing html I do this daily. I use biterscripting for parsing our own web pages, and extracting all kinds of info from it in all kinds of formats. You can start with the sample script posted at http://www.biterscripting.com/SS_WebPageToText.html as...
    Posted to General Questions by JenniC on Wed, Nov 4 2009
  • Re: Plugin help

    Templater.cs is ALPHA code. 1.) HtmlAgilityPack is a memory hog, and should only be used when you absolutely need it. It is used in the templater code because I need XPATH support. 2.) ExtractText does a much, much better job of stripping out tags than the HtmlAgilityPack does, and it's faster as...
    Posted to General Questions by arachnode.net on Tue, Aug 11 2009
  • Re: Plugin help

    arachnode.net already contains support for the HtmlAgilityPack - however, the HtmlAgilityPack is a HUGE memory hog and has an extremely negative impact on crawling rate. If you can avoid it, don't use it. If you have to use it, change the configuration setting for 'ExtractWebPageMetaData'...
    Posted to General Questions by arachnode.net on Fri, Aug 7 2009
Page 1 of 1 (20 items)
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC