arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release
How to set web proxy in arachnode code.

If you are running the AN (arachnode crawler) in a ntetowrk secured by some proxy server , then you may need to specify the proxy server settings in the AN code. 

You can do this as follows:

Go to SiteCrawler project in the AN code solution , then to Components folder.

You will find a class with name webClient.cs.

In this file search for the code 'HttpWebRequest.Proxy = null'

set the webproxy here , eg:

HttpWebRequest.Proxy = new WebProxy("your proxy url");

The web proxy is set now.

 

Happy Crawling.


Posted Thu, Mar 21 2013 10:43 AM by Abhishek Gahlout

Add a Comment

(required)  
(optional)
(required)  
Remember Me?
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC