arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop
Search the Live Index Does arachnode.net scale? | Download the latest release

Indexing and data mining

rated by 0 users
This post has 1 Reply | 2 Followers

Top 150 Contributor
Posts 3
john12890 Posted: Thu, Apr 18 2013 4:26 AM

Hi all,

Just going through the Demo version of AN, I would like to know how this tool indexing the data? Is the tool do the Data Mining of the crawled data? I would like to discuss in detail in this regard.

My issue is that I've to crawl specific data from a authenticated web page and do the analytics on that. Will this tool can help me out?

Thanks.

Top 10 Contributor
Posts 1,905

Indexing is performed either through SQL Full Text indexing and/or Lucene.NET.

There are a large number of functions and sections of helper code for use in data mining.

Here: http://arachnode.net/forums/p/1437/13003.aspx#13003

Here: Look at 'UserDefinedFunctions' -> http://arachnode.net/search/SearchResults.aspx?q=UserDefinedFunctions

Yes, AN can crawl authenticated web sites.

Thanks.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (2 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC