arachnode.net
An open source .NET web crawler written in C# using SQL 2005/2008
Forums » arachnode.net » General Questions » Crawl URL Question
IT Professionals & Windows Deployment Professionals: SmartDeploy Enterprise is the first hardware-independent imaging toolset that uses boot time driver-injection, simplifying deployment and easing distribution by reducing total image count. [LINK]

Crawl URL Question

rated by 0 users
Not Answered This post has 0 verified answers | 1 Reply | 2 Followers

Top 50 Contributor
8 Posts
TileCheng posted on 17 Feb 2009 10:10 PM

I Crawl  it  :  http://www.zhaojijiaju.com/news.asp

and Crawl  it's  SubPage  Infomation   http://www.zhaojijiaju.com/view_news.asp?id=75,

I will  Insert  What Data  To  DataTable  "CrawlRequests"??

Now I Do  :Absoluteuri: http://www.zhaojijiaju.com/news.asp,   depth:4,   restricttourihost:true;   priority:1

 

How  You Do??? thanks

All Replies

Top 10 Contributor
1,244 Posts
Reply
arachnode.net replied on 26 May 2009 8:29 PM

This is correct.

For best service when you require assistance:  Big Smile

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

An open source .NET web crawler written in C# using SQL 2005/2008.

Twitter: http://twitter.com/arachnode_net

arachnode.net provides custom crawling and contracting resources.  Please ask.

C# crawler, C# web crawler, C# site crawler

Page 1 of 1 (2 items) | RSS
An open source .NET web crawler written in C# using SQL 2005/2008

copyright 2004-2010, arachnode.net LLC

Powered by Community Server (Non-Commercial Edition), by Telligent Systems