Re: Full-Text Indexing / Storage Question

arachnode.net — Sat, 17 Jul 2010 18:28:41 GMT

Hello there! Not many questions about how SQL Full Text Indexing (FTI) is implemented, so I am glad you asked. And, for anyone that doesn't know, FTI in SQL 2008 is vastly improved over 2005 and worth a second look.

The code in [Console]\Program.cs, and the database table cfg.Configuration instructs the crawler to NOT store the WebPage Source in SQL, rather to store in on disk. (The configuration shown below is slightly different than the default, per my own project requirements.)

ApplicationSettings.InsertWebPageSource = false;

ApplicationSettings.SaveDiscoveredWebPagesToDisk is the SQL to DISK complement.

If I set ApplicationSettings.InsertWebPageSource = true; and crawl for a bit, and then examine the WebPages table I get this:

Where the Source was '0x' before making the configuration change, now varbinary Source is present.

Next, examine the Full Text Index catalogs.

Examine the properties of the arachnode.net_ftc_WebPages catalog.

By default, the AbsoluteUri and ResponseHeaders columns are indexed. By adding data to the Source column, the Source column now becomes searchable.

The default configuration is to index to disk, which is accomplished by [Plugins]\ManageLuceneDotNetIndexes.cs.

Helper code in [Console]\Program.cs may have set your on-disk indexing directories:

Also, the LUKE toolbox is included in the source.

Worth reading: http://arachnode.net/blogs/arachnode_net/archive/2010/04/08/adding-custom-fields-to-the-lucene-net-indexes.aspx

This details how plugins (ManageLuceneDotNetIndexes.cs) are used to extend functionality: http://arachnode.net/Content/CreatingPlugins.aspx

Tags: http://arachnode.net/tags/lucene.net/default.aspx

Tags: http://arachnode.net/tags/plugins/default.aspx

Does this answer your question?

Sincerely,
Mike

Search results matching tags 'LUKE' and 'lucene.net'

Re: Full-Text Indexing / Storage Question