arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2005/2008/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop
Mongo/Raven/MySQL/Hadoop Does arachnode.net scale? | VS2008/2010/2012 & SQL2008/2012 | Download the latest release

Another getting started Question

rated by 0 users
Answered (Verified) This post has 1 verified answer | 4 Replies | 3 Followers

Top 150 Contributor
2 Posts
squip posted on Fri, Aug 28 2009 2:22 AM

Hi there,

First of all - please excuse me for asking dumb questions but for some reason I cannot get this working like the rest of the community.

I am deploying arachnode for a college project demonstrating the usage of web crawlers alongside databases. I have installed it completely and the program will compile however - I am not sure how to configure it.

Am I required to configure using the SQL tables only or do I edit the configuration project > ApplicationSettings.cs?

 

Ok time for another question: When the application compiles it will run a web application. When I try to execute this application I get a directory view of  [arachnode.net root]\web - should this be displaying the demonstration page (just like on this website?)

 

I thank you for your help - I hope I am not wasting too much of your time :)

Regards,
Patrick
(New Zealand)

(also is the descriptions missing on the documentation page?)

Answered (Verified) Verified Answer

Top 10 Contributor
1,751 Posts

You'll need to edit cfg.Configuration, cfg.CrawlActions and/or cfg.CrawlRules.

If you see the directory view then you haven't selected a startup page (Search.aspx) - StartupPage is a solution option, which I don't check in because your breakpoints are not my breakpoints (and other options) and vice versa... :)

Which documentation page are you referring to?

MAIN INSTALLATION DOCUMENTATION LINK: http://arachnode.net/Content/InstallationInstructions.aspx

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

All Replies

Top 10 Contributor
1,751 Posts

You'll need to edit cfg.Configuration, cfg.CrawlActions and/or cfg.CrawlRules.

If you see the directory view then you haven't selected a startup page (Search.aspx) - StartupPage is a solution option, which I don't check in because your breakpoints are not my breakpoints (and other options) and vice versa... :)

Which documentation page are you referring to?

MAIN INSTALLATION DOCUMENTATION LINK: http://arachnode.net/Content/InstallationInstructions.aspx

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 150 Contributor
2 Posts
squip replied on Fri, Aug 28 2009 5:04 PM

Hay Mike,

Thanks for the swift reply - I had a look in those table and until I can get this going I am happy enough with the default settings. But  I am still having a hard time getting this going (I feel silly)

SO I un commented your code within the test crawl block in Program.CS but I still cannot get this to run - it looks exactly like it does in the screenshot. I was under the impression it would insert a row into crawlrequests (since I cant really manually do this)

Another silly Question - How do I go about checking in the startup page -  IF I can get this going It would be great to have an example search engine running for my project.

As for the documentation, http://arachnode.net:8081/ the descriptions are missing (or is this meant to be like this?)

Regards,

Patrick

Top 10 Contributor
1,751 Posts

I bet your SQL connection is hanging... check the first call to GetVersion in Crawler.cs...

        /// <summary>
        /// Checks for exceptions.
        /// </summary>
        private void CheckForExceptions()
        {
            ArachnodeDataSet.VersionDataTable versionDataTable = _arachnodeDAO.GetVersion();

Here's how to set the Start Page:

Documentation, yes - they are supposed to be blank - supposed to remind me to add them.  Big Smile  AN really needs a small book as every so often I'll forget how something works - yes, there is a lot arachnode.net can do... Smile

You can insert rows into the table directly - I HIGHLY recommend using the API tho - ArachnodeDAO.InsertCrawlRequest(...);

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
1,751 Posts

Also, I'm fixing some "less-than-minor" things that user:megetron found - so, please get the latest from the trunk and check back often.  :)

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (5 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE

copyright 2004-2014, arachnode.net LLC