arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release
arachnode.net Help

In summary: If you need help, I will help you directly.  Skype: arachnodedotnet (chat) and TeamViewer.com (shared desktop)

Recently, I have received a few questions around, "Hey!  arachnode.net seems like a great product but I'm having trouble getting started.  There are quite a few configuration options and I am not sure where to begin.  Will you help me?"

The short answer is:  "Thank you.  I am very proud of the work and appreciate your compliments.  Yes, there are quite a few configuration options and AN does take a bit of reading and time to set up.  Yes, I will help you."

Further: "The number of configuration options comes from the number of crawling scenarios as requested by you, the user and customer.  Each option represents a real world requirement and/or challenge.  I have done my best to provide defaults which represent the most commonly requested options.  After over 1,000 customers there is rarely a crawling scenario which I have not encountered and should you require my assistance, or even an evaluation of your crawling configuration I will gladly review your setup and offer my input."

This post comes on the edge of updating the demo to include AN.Next which does not require MS-SQL Server to be installed and configured.  Therefore, you may simply open the Visual Studio Solution, press F5 and you are off and crawling.

Setup is rather straightforward for this: http://arachnode.net/Content/InstallationInstructions.aspx (Full installation instructions and summary / AN.Next Section)

For arachnode.net there are still a few setup steps that need to be executed in order to crawl.  For this there are a few answers:

1.) Take a moment to work through the full installation steps if the summary seems too daunting.  The 'step-by-step' is a great way to learn about AN, and although it is extremely detailed it will answer most questions you will have.

2.) Run through the steps needed to create a Plugin.  http://arachnode.net/Content/CreatingPlugins.aspx  (Full installation instructions and summary)

3.) Read Console\Program.cs and Console.Next\Program.cs.  My comments in these two files will answer most of what you may ask.

If you still require assistance, I will help you directly.  Skype: arachnodedotnet (chat) and TeamViewer.com (shared desktop)

Sincerely,
Mike Anderson, arachnode.net, llc

 


Posted Wed, Feb 13 2013 10:19 AM by arachnode.net
Filed under:
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC