arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE
Sign in
|
Join
|
Help
Home
Features
Compare
Download
License
Live Demonstration
Support
Contact
API Documentation
Blogs
Creating Plugins
Forums
Frequently Asked Questions
Installation Instructions
Working Copy
All Tags
»
frequency
»
politeness
(
RSS
)
Does arachnode.net scale?
| VS2008/2010/2012 & SQL2008/2012 |
Download the latest release
Browse Forum Posts by Tags
Showing related tags and posts for the
General Questions
forum. See
all tags in the site
Sorry, but there are no more tags available to filter with.
Frequency - Politeness
Took me a while to figure out where you keep MaximumNumberOfWebRequestsPerHostPerDay=500 ThreadSleepTimeInMillisecondsBetweenWebRequests=1000 They are simply parameters to the Rules.Frequency stored in the cfg.CrawlRules so now I know it all works. In the Program.cs you have the loop that turns off all...
Posted to
General Questions
(Forum)
by
polfilm
on Thu, Jul 22 2010
Page 1 of 1 (1 items)
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE
copyright 2004-2013, arachnode.net LLC