arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE
All Tags » plugins (RSS)
Does arachnode.net scale? | VS2008/2010/2012 & SQL2008/2012 | Download the latest release

Browse Site by Tags

Showing related tags and posts across the entire site.

  • New Documentation Links

    http://arachnode.net/Content/InstallationInstructions.aspx http://arachnode.net/Content/CreatingPlugins.aspx
    Posted to arachnode.net : blog by arachnode.net on Fri, Jul 9 2010
    Filed under: installation, plugins, documentation

  • passing state into AN

    Good morning, I need to pass some data into AN for processing by a plugin. I could obviously write that data into the DB and pick it up in the CrawlAction later, but I'm already retrieving that data before calling into AN as a matter of necessity and I'd rather not be more redundant than I have...
    Posted to General Questions by offbored on Wed, Feb 10 2010

  • Re: Plugin help

    Templater is a piece of code that can look at a webpage and extract the 'meat' of the page - it can look at a blog site and tell you which xpath will select the main post, the titles, or looking at a forum site, which posts are the forum posts. It basically solves a tough problem in web scraping...
    Posted to General Questions by arachnode.net on Sun, Aug 2 2009

  • Re: Crawling several sites with 1.2 version

    There isn't an explicit tutorial - but these are the steps... 1.) Find one of the existing plugins. 'Anonymizer.cs' is the simplest and shortest. 2.) Create a new class using the name of your choice. 3.) Examine the 'CrawlActions' database table and follow the present pattern. That's...
    Posted to General Questions by arachnode.net on Sun, Jul 26 2009

  • Re: Crawling several sites with 1.2 version

    1.) You will need to write separate rules for each site, but one plugin will work. Else, how would the plugin know what information you want to pull? You can use UserDefinedFunctions.ExtractDomain or UserDefinedFunctions.ExtractHost to perform the filtering/switching. 2.) The easiest would be to Create...
    Posted to General Questions by arachnode.net on Sun, Jul 26 2009

  • Re: Anonymouse Crawling

    I added an Anonymizer plugin to the branch so you can see how this would be implemented. Don't forget to check out the DB too... (Branch is a branch, but quite viable...) This code is checked into the trunk now. Mike
    Posted to Feature Requests by arachnode.net on Mon, Jun 1 2009
Page 1 of 2 (40 items) 1 2 Next >
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE

copyright 2004-2013, arachnode.net LLC