Hi Mike,
Just a thought, why not making plugins fro the program.cs pages to keep some order ?
sometimes to add the crwalrequest is a very complex mission, and there are huge algorythms. and you want to keep each domain in a different page.
I can find it helpfull, I don't know what the priority for such, but defiently it will increase the flexibility.
Please let me know.
Thanks,
Megetron:
Not sure what you are asking - what is a specific use case?
Thanks!
Mike
For best service when you require assistance:
An open source .NET web crawler written in C# using SQL 2005/2008.
Twitter: http://twitter.com/arachnode_net
arachnode.net provides custom crawling and contracting resources. Please ask.
C# crawler, C# web crawler, C# site crawler
Exactly like you did with the Plugin system. that every plugin can be enable\disable through the settings DB.
There are scenarios where you want to trigger a new crawl request to a certain website, and I think it will be helpfull if you can trigger a certain crawl request to a certain website,
Instead of writing : New CrawlRequest(htatp://domain.com,...) in Program.CS page, There will be a new plugin page that will be names "domain_program.cs" and there you will enter: New CrawlRequest(htatp://domain.com,...)
So, if there is Domain1.com, and Domain2.com, then, I will have 2 pages Domain1_program.cs and domain2_program.cs.
in each of this pages I will have the New CrawlRequest(...) line, and it can be enable and disable by the settings table, inside database.
I hope I explained myself better.
Please let me know what you think.
You can turn Plugins on and off by making the Dictionaries that store the CrawlRules/CrawlActions 'public' inside of RuleManager.cs.
Would be a cool feature tho... will keep it in mind.