Showing related tags and posts across the entire site.
-
I am making a change to the core which will allow you to adjust the CrawlActions, CrawlRules and EngineActions before Crawling, per Crawl instance, just like you can with ApplicationSettings and WebSettings. http://arachnode.net/blogs/arachnode_net/archive/2010/05/06/controlling-configuration-from-code...
-
Good morning, I need to pass some data into AN for processing by a plugin. I could obviously write that data into the DB and pick it up in the CrawlAction later, but I'm already retrieving that data before calling into AN as a matter of necessity and I'd rather not be more redundant than I have...
-
Hi there! You are right. This can be accomplished with a custom crawl action and isn't difficult... There isn't an existing CrawlAction that does what you want, but it would not be difficult to implement. Glad to hear you are hooked on arachnode.net. I am too! Which Version are you using, by...
-
Let's go with what I communicated over IM. There are several ways to achieve what (I think and hope) I understand your needs to be. The switches and modifications from the post were to support a batch-style analysis - but we actually need to implement a continuous crawling mechanism, which is what...
-
Templater is a piece of code that can look at a webpage and extract the 'meat' of the page - it can look at a blog site and tell you which xpath will select the main post, the titles, or looking at a forum site, which posts are the forum posts. It basically solves a tough problem in web scraping...