arachnode.net

An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Sign in | Join | Help

Search the Live Index

Does arachnode.net scale? | Download the latest release

April 2010 - arachnode.net : blog

arachnode.net : blog

Home
Contact

Syndication

Recent Posts

Tags

Web Crawlers

Archives

Sort by: Most Recent | Most Viewed | Most Commented

Troubleshooting differing crawl environments and differing crawl results...

1.) Ensure that your crawling code and configurations are identical . After crawling, use SQL Data Compare to compare your results. http://www.red-gate.com/products/SQL_Compare/index.htm - SQL Compare: Used to ensure your SQL database code is identical...

Published Thu, Apr 29 2010 9:05 AM by arachnode.net

Filed under: ISP Throttling, crawl results, troubleshooting, ISP
Adding custom fields to the lucene.net indexes...

1.) Create a new plugin which extends ManageLuceneDotNetIndexes.cs 2.) Update cfg.CrawlActions to reference the new plugin. 3.) Copy the code from CreateDocument(...) into the CreateDocument(...) function in the new class. Adding the CreateDocument(....

Published Thu, Apr 8 2010 4:49 PM by arachnode.net

Filed under: extending ManageLuceneDotNetIndexex.cs, custom plugins

An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC