arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

How Close A Crawl Application ?

rated by 0 users
Answered (Verified) This post has 1 verified answer | 6 Replies | 2 Followers

Top 50 Contributor
8 Posts
TileCheng posted on Wed, Feb 18 2009 1:02 AM

I Write a Crawl Test  Console Application.

Now,How I Can Know  The Crawl Application End? And  When ?

Do Have a Common Method ?

Thank s.....

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by TileCheng

You know what?  There isn't a way to know this, explicitly, without examining the database table.  It was my imagination that arachnode.net would continuously crawl.

If you examine the CrawlRequests table and the Discoveries table and they are empty, your Crawl is complete.  Depending on settings in Application.config, arachnode.net may still be crawling, but will be crawling past your originally specified crawl, if 'createCrawlRequestsFromDatabaseHyperLinks' is true and/or 'createCrawlRequestsFromDatabaseWebPages' is true.

I'm moving this thread to the feature requests forum.  I'll add an event that will signal when the crawl is complete.

-Thanks!

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

All Replies

Top 10 Contributor
1,905 Posts
Verified by TileCheng

You know what?  There isn't a way to know this, explicitly, without examining the database table.  It was my imagination that arachnode.net would continuously crawl.

If you examine the CrawlRequests table and the Discoveries table and they are empty, your Crawl is complete.  Depending on settings in Application.config, arachnode.net may still be crawling, but will be crawling past your originally specified crawl, if 'createCrawlRequestsFromDatabaseHyperLinks' is true and/or 'createCrawlRequestsFromDatabaseWebPages' is true.

I'm moving this thread to the feature requests forum.  I'll add an event that will signal when the crawl is complete.

-Thanks!

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 50 Contributor
8 Posts

OK。 NO thanks

I Hope you quickly add a event  to close a crawl application........

I hope....

Top 10 Contributor
Male
101 Posts
Kevin replied on Thu, Feb 19 2009 11:43 AM

Cool idea - thx!

 

Top 50 Contributor
8 Posts

Hello Mike,

Have you added event which notifies when crawl is complete? i badly beed that :(

Thanks,
Jd

Top 10 Contributor
1,905 Posts

I haven't yet.  It's fairly simple to do.  Send me a message to remind me?

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
1,905 Posts

This feature is part of the 1.1 Release.  Coming soon...

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (7 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC