Since I have set the crawler not to save WebPages in database, I get lots of error...
maybe this is got something to do with the fact the the crawler speed up, and no delays between request which cause some errors.
I will post them one by one, Created ID AbsoluteUri1 AbsoluteUri2 HelpLink Message Source StackTrace2010-05-01 19:05:33.967 171263 http://www.lyricsoncall.com/lyrics/widespread-panic/jaded-tourist-lyrics.html http://www.lyricsoncall.com/lyrics/widespread-panic/jaded-tourist-lyrics.html NULL Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: startIndex mscorlib at System.Globalization.CompareInfo.IndexOf(String source, String value, Int32 startIndex, Int32 count, CompareOptions options) at System.Globalization.CompareInfo.IndexOf(String source, String value, Int32 startIndex) at System.String.IndexOf(String value, Int32 startIndex) at Arachnode.Plugins.CrawlActions.MasterPagesMusicSela.PerformAction(CrawlRequest crawlRequest, ArachnodeDAO arachnodeDAO) at Arachnode.SiteCrawler.Managers.ActionManager.PerformCrawlActions(CrawlRequest crawlRequest, CrawlActionType crawlActionType, ArachnodeDAO arachnodeDAO)
You need to store a reference in the WebPages table so that the foreign key references will resolve.
Since you have already downloaded the stream, you have the data to process. You have to insert the WebPage row, but you don't have to insert the Source. And, if you know what you are doing (which you do), you don't have to save the source to disk. But you do need the record in the DB, so the HyperLink can be sourced.
Do this make things clear?
Also, it appears that you are using an old version.
-Mike
For best service when you require assistance:
Skype: arachnodedotnet
You have to save the WebPages if you save any other data.
Also, the error is coming from your Plugin, so you may have caused additional errors.
Yes I know that, I hoped you changed this logic.
why should I change the complete page if all i want to save is one link inside it?
The performance is really going dpwn with that, cant we save it to cache instead of database?
thanks.