arachnode.net
An open source .NET web crawler written in C# using SQL 2005/2008
IT Professionals & Windows Deployment Professionals: SmartDeploy Enterprise is the first hardware-independent imaging toolset that uses boot time driver-injection, simplifying deployment and easing distribution by reducing total image count. [LINK]

crawl specific sites

rated by 0 users
Not Answered This post has 0 verified answers | 1 Reply | 2 Followers

Top 75 Contributor
3 Posts
Roel posted on 16 Mar 2009 2:17 AM

Hi,

First thanks for your work, it works good!!

I setup my arachnode like http://arachnode.net/forums/t/103.aspx. How do I call the crawl within the console?

Now its configured like this:

crawler.Crawl(

 

new CrawlRequest(new Discovery(http://www.nu.nl), 1, false));

But the nu.nl URI is in the database now.

Thanks!!

Roel

All Replies

Top 10 Contributor
1,244 Posts

I am not sure I understand your question.  Can you ask it another way?

For best service when you require assistance:  Big Smile

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

An open source .NET web crawler written in C# using SQL 2005/2008.

Twitter: http://twitter.com/arachnode_net

arachnode.net provides custom crawling and contracting resources.  Please ask.

C# crawler, C# web crawler, C# site crawler

Page 1 of 1 (2 items) | RSS
An open source .NET web crawler written in C# using SQL 2005/2008

copyright 2004-2010, arachnode.net LLC

Powered by Community Server (Non-Commercial Edition), by Telligent Systems