How do I keep AN.Next alive and ready for new CrawlRequests after a Crawl has completed...

arachnode.net posted on Fri, Mar 23 2012 9:49 AM

rated by 0 users

Let's say you'd like to keep AN.Next alive and waiting for new CrawlRequests, but don't want to instantiate a new Crawler each time your calling application has something for AN.Next to do...

Look in CoreConfiguration.xml.

  <KeepCrawlerAlive value="true" />

Look in Crawler.cs.

        /// Keeps the Crawler threads active even when there are no CrawlRequests to crawl.

        /// </summary>

        public bool KeepCrawlerAlive { get; set; }

if (!areAnyThreadsCrawling)

                    {

                        if (CrawlRequestsToCrawl.Count == 0 && !KeepCrawlerAlive)

                        {

                            break;

                        }

                    }

Also, if you elect to use the Renderers, like in AN 2.6, by enabling the following Plugin...

            //_crawler.CrawlRequestPlugins += crawlRequestPlugins.DownloadDataAndDecodeHtml;

            _crawler.CrawlRequestPlugins += crawlRequestPlugins.DownloadDataRenderAndDecodeHtml;

Look in Program.cs for the following code:

/**/

            //necessary for the Rendering functionality if you have enabled the Plugin 'DownloadDataRenderAndDecodeHtml'

            //if (_crawler != null)

            //{

            //    //may be null if all configuration settings are not initialized in the database

            //    while (!_hasCrawlCompleted)

            //    {

            //        Application.DoEvents();

            //    }

            //}

            /**/

You'll need to uncomment this to prevent a COM context switching exception that may occur on longer running crawls.