arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Crawling using a keywords

rated by 0 users
Answered (Verified) This post has 1 verified answer | 1 Reply | 2 Followers

Top 150 Contributor
3 Posts
Jerry posted on Fri, Aug 21 2015 7:20 PM

There are key words in my hand. 10 pieces. only to find pages with keywords on a site that I want to save. Not all the pages of the site. Is it possible?

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by Jerry

Sure, just filter and use the 'IsDiallowed' functionality.

https://arachnode.net/Content/CreatingPlugins.aspx

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (2 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC