So I've been trying to figure out how to get AN to only return pages that have certain words on them. I would think that there would be a table or a text file somewhere that you fill and voila, only pages with those terms on it would be returned.
How would I accomplish that? I've been trying to figure out CrawlRules and trying to figure out how to write a plugin but can't seem to get anywhere.
Thanks
You mean filter them - only allow certain pages into the system that contain specific words?
Look at Source.cs in the SiteCrawler project.
-Mike
An open source .NET web crawler written in C# using SQL 2005/2008.
Join the arachnode.net group on Facebook: http://www.facebook.com/groups.php?ref=sb#/group.php?gid=166721755872
Twitter: http://twitter.com/arachnode_net
arachnode.net provides custom crawling and contracting resources. Please ask.
http://bit.ly/TOFX4
C# crawler, C# web crawler, C# site crawler