arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Arachnode support for crawling frames

rated by 0 users
Answered (Verified) This post has 1 verified answer | 2 Replies | 3 Followers

Top 100 Contributor
4 Posts
rakmagik posted on Mon, Jul 26 2010 5:52 PM

Hi,

I could not find much information on crawling frames using Arachnode. Could you please let me know if Arachnode supports frames and if so, how do I configure it to crawl frames?

I did a crawl on a site that had frames but did not get any urls that were referenced withing the frames.

 

Thanks

 

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

What is the site you are trying to crawl?

Sincerely,
Mike 

EDIT: This is fixed.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

All Replies

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

What is the site you are trying to crawl?

Sincerely,
Mike 

EDIT: This is fixed.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
229 Posts

frames are usuallu ignored by crawlers. you can write your own plugin to extract the frame html/page and create a crwalrequest

Page 1 of 1 (3 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC