List of Questions

Reply

arachnode.net replied on 15 May 2009 9:35 PM

rated by 0 users

Hey J -

It's late for me - all good questions... let me get back to you over the weekend!

But, good news - Version 1.2 is close... I'm hoping that within a week we'll have it ready!!!

Mike

Reply

arachnode.net replied on 19 May 2009 8:18 AM

rated by 0 users

1.) Check out the stored procedure and remove the line shown below: (I have removed this from the SP that will be part of Version 1.2)

dbo

.arachnode_usp_arachnode.net_RESET_DATABASE - EXEC dbo.arachnode_omsp_CrawlRequests_INSERT @Datetime, 'http://arachnode.net/', 4, 0, 1, null

2.) That would work.  An even easier way would be to submit CrawlRequests with a high depth, say '100' and restrict crawling to CrawlRequests and WebPages.

3.) I think this actually may be a bug that I fixed for the upcoming Version 1.2 release.  :(  (Sorry...)  If you stop a crawl it doesn't check when saving the requests back to the DB.

4.) Check out the 'Integration' project in the solution.  This is where the terms get split.  The switch you found in the DB is for extracting Text and XML from the WebPages (inserted into WebPages_MetaData) and for enabling creating of an HtmlAgilityPack HtmlDocument.  (The HtmlDocument functionality is very memory intensive.)

5.) Yes - disallowQueryStrings.

<?

xml version="1.0" encoding="utf-8" ?>

<

crawlRules>

<

rule assemblyName="Arachnode.SiteCrawler" typeName="Arachnode.SiteCrawler.Rules.AbsoluteUri" isEnabled="true" order="1" ruleType="3" outputIsDisallowedReason="true" disallowNamedAnchors="true" disallowQueryStrings="false" ...