arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE
Forums » arachnode.net » General Questions » google
Does arachnode.net scale? | VS2008/2010/2012 & SQL2008/2012 | Download the latest release

google

rated by 0 users
Answered (Verified) This post has 1 verified answer | 5 Replies | 2 Followers

Top 10 Contributor
30 Posts
pp.ps posted on Wed, Mar 10 2010 8:06 AM

Hello,
How to crawl google search result http://www.google.com/search?hl=pl&q="pp.ps"+synergia++b2b+2d&btnG=Szukaj&lr=&aq=f&oq=

When depth is set to 1 only google addresses are found, if depth is set to 2 crawler crawl to deep. 

Answered (Verified) Verified Answer

Top 10 Contributor
1,714 Posts
Reply
Answered (Verified) arachnode.net replied on Wed, Mar 10 2010 12:02 PM
Verified by arachnode.net

Read up on 'RestrictDiscoveriesTo': http://arachnode.net/search/SearchResults.aspx?q=restrictdiscoveriesto

And, take a look at the FAQ: http://arachnode.net/Content/FrequentlyAskedQuestions.aspx (Usage #4)

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

All Replies

Top 10 Contributor
1,714 Posts
Reply
Answered (Verified) arachnode.net replied on Wed, Mar 10 2010 12:02 PM
Verified by arachnode.net

Read up on 'RestrictDiscoveriesTo': http://arachnode.net/search/SearchResults.aspx?q=restrictdiscoveriesto

And, take a look at the FAQ: http://arachnode.net/Content/FrequentlyAskedQuestions.aspx (Usage #4)

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
1,714 Posts
Reply
arachnode.net replied on Wed, Mar 10 2010 1:34 PM

What is 'to deep'?

The first step in understand what AN is doing when you don't understand what AN is doing is to examine the Exceptions and DisallowedAbsoluteUris table.

Search results on what 'Depth' means: http://arachnode.net/search/SearchResults.aspx?q=depth

It is helpful to me to see what is in your WebPages and other tables.  The more information the better.

 

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
1,714 Posts
Reply
arachnode.net replied on Wed, Mar 10 2010 1:58 PM

Please post what is currently in your table and what you expect.  I am having a tough time trying to figure ou what you want.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (6 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2005/2008/CE

copyright 2004-2013, arachnode.net LLC