arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE
Does arachnode.net scale? | VS2008/2010/2012 & SQL2008/2012 | Download the latest release

Threading doesn't complete on time?

rated by 0 users
Not Answered This post has 0 verified answers | 5 Replies | 2 Followers

Top 25 Contributor
16 Posts
sebastian_h posted on Wed, Jan 26 2011 12:35 PM

Hi Mike,

I've used your console app as an example for another console app I'm writing, and copied out the general structure (in regards to where you instantiate stopwatch, crawler engine, inject crawlrequests, etc.). AFAIK I've kept the structure pretty much similar.

The error I'm encountering at the moment is during some crawls, the green "Elapsed. Press any key to terminate arachnode.net" message appears and I assume the crawl is complete. However a few seconds later, I get output appearing on the screen as if the database is just being written to.

Further examination of the dbo.Hyperlinks table shows that it is empty when the message appears, however gets populated when the further output appears. Essentially, the termination message is getting displayed before the program has finished executing.

Do you have any idea what may be causing this? FYI I've upgraded the solution to VS2010, and in my console application I'm using the Entity Framework to access some arachnode tables.

 

Many thanks.

All Replies

Top 10 Contributor
1,714 Posts

For how long after do you see activity in the HL table?

Are you using the .zip code or the latest from SVN?

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 25 Contributor
16 Posts

It seems to be outputting more discovery results - it'll run for another minute or so.

It never gave any problems when I was running it in VS2008. I got this copy from SVN in December.

 

Top 10 Contributor
1,714 Posts

Try updating your source from SVN.

No piece of software is without bugs and I do fix and check in.  Big Smile

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 25 Contributor
16 Posts

Thought I'd upload a screenshot to help you out

 

Top 10 Contributor
1,714 Posts

Try updating your source from SVN.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (6 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE

copyright 2004-2013, arachnode.net LLC