arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop
Search the Live Index Does arachnode.net scale? | Download the latest release

Updating indexes

rated by 0 users
Answered (Verified) This post has 1 verified answer | 1 Reply | 2 Followers

Top 25 Contributor
23 Posts
victor posted on Wed, Jan 11 2017 2:26 PM

How does arachnode keep the index up to date?

When I run crawler with a list of websites second time after initial crawl, what does actually happen? Does AN update the index for websites, which were already crawled before?

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by victor

Which index?

The lucene.net index?  If so, yes.  The most recent result will be found in the lucene.net index.

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (2 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC