arachnode.net v2.0
An open source .NET web crawler written in C# using SQL 2005/2008

Browse Forum Posts by Tags

Showing related tags and posts for the Forums application. See all tags in the site
Sorry, but there are no more tags available to filter with.
  • How to restrict crawl to single domain?

    Hello Guys, I am stuck if anyone have idea of it !!! well i want to crawl only the links which belong to my domain e.g. http://www.mydomain.com , i want to crawl all the pages under http://www.mydomain.com like http://www.mydomain.com/1.aspx , http://www.mydomain.com/2.aspx etc... and one of the page...
    Posted to General Questions (Forum) by jaydeep on 02-05-2009
  • DisallowedFileExtensions are case sensitive

    Perhaps this is no longer a case in versions past 0.9 but i just discovered that I can't exclude .jpg extension with a single ".jpg" entry because ".JPG" is already there. First, i have not found a way to disable extracting files into a database so they all end up in Files table...
    Posted to General Questions (Forum) by polfilm on 01-04-2009
Page 1 of 1 (2 items)
An open source .NET web crawler written in C# using SQL 2005/2008

copyright 2009, arachnode.net LLC

Powered by Community Server (Non-Commercial Edition), by Telligent Systems