Does arachnode.net scale? | VS2008/2010/2012 & SQL2008/2012 | Download the latest release

Search

Re: How to avoid creating discoveries ?

1. Commented all unnecessary code in program.cs - What does this mean? What did you comment? 2. Created the below plugin and changed plugin name in program.cs as well as in database crawlactions table - OK, good. 3. Since I dont need any discoveries, I just commented the below section in Engine.cs - You don't need to comment out anything in AN

Posted to Plugins (Forum) by arachnode.net on Thu, May 23 2013
Re: How to run schedule crawler (program.cs) automatically

https://www.google.com/webhp?sourceid=chrome-instant&rlz=1C1CHFX_enUS531US531&ion=1&ie=UTF-8#rlz=1C1CHFX_enUS531US531&sclient=psy-ab&q=task%20scheduler&oq=&gs_l=&pbx=1&fp=6c7f8a5fed4db490&ion=1&bav=on.2,or.r_cp.r_qf.&bvm=bv.46865395,bs.1,d.cGE&biw=1262&bih=862

Posted to General Questions (Forum) by arachnode.net on Thu, May 23 2013
Re: How to avoid creating discoveries ?

Look in Console\Program.cs - this is where the settings are set. Set IsStorable = true to store the HyperLink (crawlRequest.IsStorable and/or the discovery.IsStorable) - in your case, use the 'Discovery' method, and not the 'CrawlRequest' method.

Posted to Plugins (Forum) by arachnode.net on Wed, May 22 2013
Re: How to avoid creating discoveries ?

Set ApplicationSettings.InsertHyperLinks = true; and use IsStorable in the plugin and let the CrawlRequestManager.cs insert the HyperLinks.

Posted to Plugins (Forum) by arachnode.net on Tue, May 21 2013
Performance Tools

http://www.passmark.com/download/pt_download.htm https://www.google.com/webhp?sourceid=chrome-instant&rlz=1C1CHFX_enUS531US531&ion=1&ie=UTF-8#rlz=1C1CHFX_enUS531US531&sclient=psy-ab&q=byte+to+megabits&oq=byte+to+megabits&gs_l=hp.3...2050808.2057905.0.2058079.19.18.0.0.0.1.330.2701.0j14j2j1.17.0...0.0...1c.1.14.psy-ab.yUCazaa7RCQ&pbx

Posted to arachnode.net : blog (Weblog) by arachnode.net on Thu, May 16 2013
Re: Getting IndexReader is closed error :(

Looks like you have made heavy changes to both files. I'm not sure I can help you with this since so much code has changed. Meaning: I wouldn't be able to tell since it isn't AN code and I can't repro here. I do know that the default service was AOK with multiple requests - so something about the modifications you have made is breaking

Posted to General Questions (Forum) by arachnode.net on Wed, May 1 2013
Re: Indexing and data mining

Indexing is performed either through SQL Full Text indexing and/or Lucene.NET. There are a large number of functions and sections of helper code for use in data mining. Here: http://arachnode.net/forums/p/1437/13003.aspx#13003 Here: Look at 'UserDefinedFunctions' -> http://arachnode.net/search/SearchResults.aspx?q=UserDefinedFunctions Yes

Posted to Anonymous Forum (Forum) by arachnode.net on Fri, Apr 19 2013
One Billion WebPages+

Yes, that is over 1 Billion!

Posted to arachnode.net : blog (Weblog) by arachnode.net on Fri, Apr 12 2013
Re: Getting IndexReader is closed error :(

1.) Have you changed any code? If so, what? 2.) What is frequently, multiple times mean, speficially? Concurrently? In serial fashion?

Posted to General Questions (Forum) by arachnode.net on Thu, Apr 11 2013
Re: Crawling of webpage where data comes with scrollbar

As you solved it: Check Fiddler for the requested AbsoluteUri - in this case additional AbsoluteUris are generated with QueryStrings, which makes it VERY easy.

Posted to General Questions (Forum) by arachnode.net on Thu, Apr 11 2013

Page 1 of 133 (1324 items) 1 2 3 4 5 Next > ... Last » | More Search Options