arachnode.net v2.0
An open source .NET web crawler written in C# using SQL 2005/2008

crawl per country / langiage

rated by 0 users
Not Answered This post has 0 verified answers | 3 Replies | 2 Followers

Top 10 Contributor
155 Posts
megetron posted on 05-22-2009 10:11 AM

Hello,

Is there an option crawling sites from a specic country? or better, by a specific language?

 

All Replies

Top 10 Contributor
Male
927 Posts

1.) You could filter by extension...

2.) Is there a language tag that you know of that is returned in the HTML headers?  If so, you could write a plug-in to achieve this functionality.

An open source .NET web crawler written in C# using SQL 2008.

Join the arachnode.net group on Facebook: http://www.facebook.com/groups.php?ref=sb#/group.php?gid=166721755872

Twitter: http://twitter.com/arachnode_net

arachnode.net is provides custom crawling and contracting resources.  Please ask.

http://bit.ly/TOFX4

Top 10 Contributor
155 Posts

I am not sure thier is one. but UNCODE characters used in most websites, and I guess that if most of the characters are lets say japanese characters, so it is a japanese website. the crawler can make a language check for a page, and estimate the precenteges by reading the text of the website.

is it possible?

Top 10 Contributor
Male
927 Posts

Yes, this is possible, and actually might be rather easy for you to implement.

Look at Source.cs - this is a CrawlRule that can filter content based on the content of the page.  :)

An open source .NET web crawler written in C# using SQL 2008.

Join the arachnode.net group on Facebook: http://www.facebook.com/groups.php?ref=sb#/group.php?gid=166721755872

Twitter: http://twitter.com/arachnode_net

arachnode.net is provides custom crawling and contracting resources.  Please ask.

http://bit.ly/TOFX4

Page 1 of 1 (4 items) | RSS
An open source .NET web crawler written in C# using SQL 2005/2008

copyright 2009, arachnode.net LLC

Powered by Community Server (Non-Commercial Edition), by Telligent Systems