crawling continues

InvestisDev posted on Fri, Mar 2 2012 5:37 AM

rated by 0 users

Hello,

At this time, crawling is taking too much time to crawl a site that had hyper links amount 145 and web pages around 92.

It is taking lots of time and so i need to exclude images to be excluded right now.

One more thing, while crawling a site, after some time the record count in the table "Hyperlinks" and "Webpages" are not increasing but crawler is still continues with same URL crawling multiple times. (i think it goes into a loop )

Let me know if you need any further detail for the same.

Thanks,