arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Credential Cache

rated by 0 users
Answered (Verified) This post has 1 verified answer | 7 Replies | 3 Followers

Top 50 Contributor
10 Posts
Mr. Wobbet posted on Thu, May 21 2009 9:27 AM

According to http://arachnode.net/forums/p/194/10029.aspx#10029 there is code somewhere that allows me to provide credentials to login protected sites using the CredentialCache. But when I scan the latest code downloaded from SourceForge there are no references to it anywhere.

What am I missing?

rjsjr

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

Big Smile  It does work...

Check this out: http://msdn.microsoft.com/en-us/library/system.net.credentialcache.aspx

Each HttpWebRequest has its Credentials set before each request.

 

if

 

 

(Crawler.CredentialCache != null

)

{

HttpWebRequest.Credentials =

 

Crawler

.CredentialCache;

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

All Replies

Top 50 Contributor
10 Posts

I'm missing the trunk from SVN is what I'm missing...

Top 10 Contributor
1,905 Posts

Right.  The code is in the trunk and not part of a release... yet...

http://arachnodenet.svn.sourceforge.net/viewvc/arachnodenet/trunk/SiteCrawler/Crawler.cs?revision=179&view=markup

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 50 Contributor
Female
9 Posts
Anat replied on Sun, Jun 20 2010 7:22 AM

Hi Mike,

Regarding to the issue of using the Credential Cache to crawl into sites that requires user/password authentication, I know this is already available in the release but the last thing I found is that you said you didn't try to use it yet...Confused

So I was wondering if you did do so and if you have any code examples for using it?

 

Thanks!

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

Big Smile  It does work...

Check this out: http://msdn.microsoft.com/en-us/library/system.net.credentialcache.aspx

Each HttpWebRequest has its Credentials set before each request.

 

if

 

 

(Crawler.CredentialCache != null

)

{

HttpWebRequest.Credentials =

 

Crawler

.CredentialCache;

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 50 Contributor
Female
9 Posts
Anat replied on Sun, Jun 20 2010 2:13 PM

Smile

Thanks Mike!

I did try it but it's still not working for some reason. my code (inside GetWebResponse) looks like the following - 

 CredentialCache cc = new CredentialCache();

 cc.Add(new Uri(absoluteUri), "Digest", new NetworkCredential("myUser", "myPassword","MyUriDomain"));

HttpWebRequest.Credentials = cc;

Am I doing anything wrong there?

Also, do you know a site that asks for credentials that this would work on for sure? Thanks again.

Top 10 Contributor
1,905 Posts

Looks correct.

When I tested it, I followed the advice from MSDN, and created a new site and varied the credential requirements from IIS.

Try these links:

  1. CredentialCache.Add Method (String, Int32, String ...

    Subject = "Test Email using Credentials"; NetworkCredential myCreds = new NetworkCredential("username", "password", "domain"); CredentialCache ...
    msdn.microsoft.com/en-us/library/59x2s2s6.aspx - Cached - Similar
  2. How to pass CredentialCache.DefaultCredential from Client to ...

    (I will also do perf. test on real network after) Thanks Tag: ASMX Web Services and XML Serialization How to pass CredentialCache. ...
    www.windows-tech.info/13/7105471f65820110.php - Cached
  3. Trouble with Proxy.Credentials = CredentialCache ...

    3 posts - 2 authors - Last post: Mar 26, 2007 Credentials = CredentialCache.DefaultCredentials ... NET Development Server the web application is testing in. My username is set up on my ...
    forums.asp.net/t/1089887.aspx - Cached - Similar Sending email‎ - 5 posts - Dec 28, 2007
    how to pass client credentials(integrated windows)‎ - 4 posts - Apr 17, 2006

    More results from forums.asp.net »

  4. HOW TO: Pass Current Credentials to an ASP.NET Web Service

    The DefaultCredentials property of the CredentialCache class contains the system ... To test the HelloWorld WebMethod, click the HelloWorld link. ...
    support.microsoft.com/kb/813834 - Cached - Similar
  5. Impersonation and CredentialCache.DefaultCredentials gives HTTP ...

    Apr 7, 2010 ... Now I am trying to impersonate myself in some initial testing. .... Net.CredentialCache.DefaultCredentials encrypted? ...
    stackoverflow.com/.../impersonation-and-credentialcache-defaultcredentials-gives-http-401-unauthorized - Cached - Similar
  6. DO NOT REPLY [Bug 48958] mod_ldap, ldap credential cache ...

    Apr 1, 2010 ... command caused the error: if test `pwd` = "/usr/local/src/httpd-2.0.63"; .... DO NOT REPLY [Bug 48958] mod_ldap, ldap credential cache ...
    osdir.com/ml/bugs-httpd/2010-04/msg00002.html - Cached
  7. SSL Web Service Proxy Authentication - .NET Framework answers

    Private Sub Test() > > > > > > Dim oCD As New cdprocess.CDProcess > > > Dim cache As CredentialCache = New CredentialCache ...
    bytes.com › topic.netanswers - Cached - Similar
  8. Credentialcache.defaultnetworkcredentials.dump() - O'Reilly Forums

    2 posts - 2 authors - Last post: Feb 25, 2009 CredentialCache.DefaultNetworkCredentials returns a ... The Art of Application Performance Testing, |-- Twitter API: Up and Running ...
    forums.oreilly.com/.../Credentialcache-defaultnetworkcredentials-dump-/ - Cached
  9. Credential cache not working - Application Forum at ObjectMix.com

    11 posts - 5 authors - Last post: Nov 21, 2006 This is a discussion on Credential cache not working within the Apache forums in Application .... (This is my preferred testing vehicle...) ...
    objectmix.com/apache/685073-credential-cache-not-working.html - Cached
  10. kerberos, sspi cache vs mit credential cache

    I need to install kerberos Server for security testing . .... copied to mit credential cache (leash import). Is this required? ...
    www.mofeel.net/552-comp-protocols-kerberos/3411.aspx - Cached

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 50 Contributor
Female
9 Posts
Anat replied on Thu, Jul 1 2010 12:41 AM

Thanks Mike!

I'll follow the links and try it again.

Page 1 of 1 (8 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC