arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Can I run the crawler without a SQL DB?

rated by 0 users
Answered (Verified) This post has 1 verified answer | 2 Replies | 2 Followers

Top 500 Contributor
1 Posts
davidfg posted on Sat, Aug 27 2016 6:23 PM

Hi there,

I just discovered this application; it sounds great but I'm confused about the SQL requirements.

In the installation instructions, it looks like arachnode.net requires SQL, but AN .NET does not; is that the case? What is the difference between the two?

Just to give some context, my needs are basically {html/text + results of js execution} for a personal project; so I'm looking for something very light-weight in terms of installation/deployment ideally.

I've used htmlagility pack in the past, but it doesn't return the data resulting for js scripts execution (as far as I know).

Thanks a lot!

PS - Maybe an entry in the FAQ about SQL requirements will help others with similar doubts in the future?

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

1.) You can overrride the MS-SQL requirements by using: https://workingcopy.arachnode.net/Console/Override/WorkingCopy_ArachnodeDAOFileSystem.txt

2.) AN.Next was deprecated in favor of using the overrides for AN.

3.) You would use the Renderers to display dynamic data / JavaScript/AJAX.

Thanks!
Mike 

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

All Replies

Top 10 Contributor
1,905 Posts

Please forgve my delay in responding - life happened.

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

1.) You can overrride the MS-SQL requirements by using: https://workingcopy.arachnode.net/Console/Override/WorkingCopy_ArachnodeDAOFileSystem.txt

2.) AN.Next was deprecated in favor of using the overrides for AN.

3.) You would use the Renderers to display dynamic data / JavaScript/AJAX.

Thanks!
Mike 

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (3 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC