arachnode.net
An open source .NET web crawler written in C# using SQL 2005/2008
Forums » arachnode.net » General Questions » !!!How to restrict crawl to single domain? (Again)
IT Professionals & Windows Deployment Professionals: SmartDeploy Enterprise is the first hardware-independent imaging toolset that uses boot time driver-injection, simplifying deployment and easing distribution by reducing total image count. [LINK]

!!!How to restrict crawl to single domain? (Again)

rated by 0 users
Not Answered This post has 0 verified answers | 1 Reply | 2 Followers

Top 50 Contributor
8 Posts
TileCheng posted on 17 Feb 2009 10:36 PM

I Crawl a List Page: Such As:  http://www.zhaojijiaju.com/news.asp

And It has Many sub pages : Such as: http://www.zhaojijiaju.com/view_news.asp?id=75

                                                                     http://www.zhaojijiaju.com/view_news.asp?id=25

                                                                    http://www.zhaojijiaju.com/view_news.asp?id=78

 

How I Can  Do  a CrawlRequest   In  The DataTable  "dbo.CrawlRequests"??

quickly,Thanks!!!!

 

All Replies

Top 10 Contributor
1,244 Posts
Reply
arachnode.net replied on 26 May 2009 8:28 PM

I am not sure I understand the question.

I think you want to submit a depth of more than 1 to crawl deeper into a website's structure.

For best service when you require assistance:  Big Smile

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

An open source .NET web crawler written in C# using SQL 2005/2008.

Twitter: http://twitter.com/arachnode_net

arachnode.net provides custom crawling and contracting resources.  Please ask.

C# crawler, C# web crawler, C# site crawler

Page 1 of 1 (2 items) | RSS
An open source .NET web crawler written in C# using SQL 2005/2008

copyright 2004-2010, arachnode.net LLC

Powered by Community Server (Non-Commercial Edition), by Telligent Systems