arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Proxies

rated by 0 users
Answered (Verified) This post has 1 verified answer | 1 Reply | 2 Followers

Top 25 Contributor
23 Posts
JCrawl posted on Wed, Jul 1 2015 3:46 PM

Been busy and have not had time to get back to this.

 

Where is the best place / price to get proxies. How many do you suggest.

 

Thank you 

 

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts

Any place will work - as long as they have anonymous proxies.

There isn't any good way to tell for sure that you'll have the best experience with any one site as you may purchase proxies and the very next day the site you're trying to crawl adds this provider's range(s) to their flagged list.

Look for sites which have an online tester, and advertise to work with X site(s) somesite.com.

You can usually find ANONYMOUS proxies for ~$1.

Two per thread is what I like to use.

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (2 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC