I figured out how to get arachnode.net to craw my site. The problem now is it keeps adding /images to the uri like this:
dt:07/21/2009 9:33:58 AM|ot:ProcessCrawlRequest|tn:1|crd:0|AbsoluteUri:http://skircr.com/images/images/images/images/new_sidemenu/section_about.gif
It found the original which is at http://skircr.com/images/new_sidemenu/section_about.gif and I'm not sure why it continues to add more and more /images to the uri.
Anybody have a clue as to what its doing?
Stephen
I'll get a crawl going and see what I can see with your site.
So, I did and I'm not seeing the repeating AbsoluteUris. Quick question: Which version are you using? 1.1? 1.2?
Also, you can turn off breaking on WebExceptions in the Debug > Exceptions menu. This is a VS/solution option and not an AN one.
For best service when you require assistance:
Skype: arachnodedotnet
You have a page on your site that is invalid, but serves up valid HTML and relative links, thereby creating a spider trap.
I'll crawl your site, if you don't mind and see if you've found a bug. Let me know, OK?
Mike
Go right ahead and thanks for the help. I did comment out the
throw new WebException(webException.Message, webException);
in WebClient.ca line 170 as it kept trowing 404 and 406 errors and I would have to keep hitting F5 to continue execution.
I am now using the CVS version. Just downloaded this morning and will be trying it out today.
I'll turn it off and see what happens.
Cool. Let me know - I didn't see any repeating AbsoluteUris in the DB using 1.2.
Everything seems to crawl just fine now. Thanks for you help.
You are very welcome.
I have following the instructions, but I still get webException, what shall I do?
How can i turn off this exception?
thanks for any help!
I find that while "robots.txt" does not exist, a "WebException" will be produced, for examplewww.taobao.com" what shall I do?
This is the correct behavior for robots.txt.
Did you follow the instuctions for the solution listed here: http://arachnode.net/forums/p/321/10290.aspx ?
(Debug > Exceptions...)
-Mike
Oh, I forgotted it, Thank you very much!
You are very welcome!
Mike you are a genius!! thanks so much for all your help!!
