arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Renderer, agilityPack question

rated by 0 users
Answered (Verified) This post has 1 verified answer | 1 Reply | 2 Followers

Top 25 Contributor
23 Posts
JCrawl posted on Wed, Jun 10 2015 3:35 PM

Hello 

2 simple questions,

1) what is the Renderer project used for, when would you need to enable this option etc. 

2) I have read on the forum that the HTML agility pack is a memory hog and does not cleanup until memory pressure is experienced, Is there a recommended alternative that has the same functionality as the htmlagilitypack.

Let me know 

 

Thank you 

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

1.) The Renderers are used for Rendering dynamic content: http://arachnode.net/blogs/arachnode_net/archive/2015/04/01/ajax-dynamic-content.aspx

2.) I remember writing this, and I think this post is 4+ years old.  I haven't had any issues with the AgPack.  It's probably fine now.

Thanks!
Mike 

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (2 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC