Hi.
I think this is a great product, and I'm excited to start crawling with it.
Is it possible with the current version to make an anonymouse crawl?
If true, how can I do so? If not, can it be added to your future features list?
Thanks, Sagie
I added an Anonymizer plugin to the branch so you can see how this would be implemented. Don't forget to check out the DB too...
(Branch is a branch, but quite viable...) This code is checked into the trunk now.
Mike
An open source .NET web crawler written in C# using SQL 2008.
Join the arachnode.net group on Facebook: http://www.facebook.com/groups.php?ref=sb#/group.php?gid=166721755872
Twitter: http://twitter.com/arachnode_net
arachnode.net is provides custom crawling and contracting resources. Please ask.
http://bit.ly/TOFX4
This functionality isn't coded, but it is possible, and rather easily.
You would want to create a PreCrawlRequest CrawlAction and change the AbsoluteUri to your anonymizer.
Take a look at ManageLuceneDotNetActions.cs. This is a PostCrawlRequest CrawlAction.
If you can't figure out how to get a plug-in going, find the other posts on the site that talk about plug-ins and keep pinging me. It would be fairly easy for me to implement. :)
Thanks,I've figured out how to add an action, rule, etc., and I will be happy to contribute it when I'll finish