absoluteuris batch processing

megetron posted on Sun, Aug 23 2009 10:26 AM

rated by 0 users

Hello,

If I amy suggest a new feature that might be usefull for others.

The feature is called a batch processing of specific  absoluteuris that specify according to regular expression or so.

for example lets say I wish to add to crawlrequests only absolute uris looks like "watch.php?catid={0}&pid={1}

Below is how I implemented this. what do you think of adding a support for such? a more flexible way to populate the CR table.

                string[] sPID = { "23&pid=1", "23&pid=2", "23&pid=3", "23&pid=4",
                                  "22&pid=1", "22&pid=2", "22&pid=3", "22&pid=4",
                                  "21&pid=1", "21&pid=2", "21&pid=3", "21&pid=4","21&pid=5",
                                  "27&pid=1",
                                  "15&pid=1", "15&pid=2", "15&pid=3", "15&pid=4","15&pid=5","15&pid=6",
                                  "24&pid=1","24&pid=2","24&pid=3",
                                  "28&pid=1", "28&pid=2", "28&pid=3", "28&pid=4","28&pid=5",
                                  "20&pid=1", "20&pid=2", "20&pid=3", "20&pid=4","20&pid=5", "20&pid=6", "20&pid=7", "20&pid=8", "20&pid=9","20&pid=10",
                                  "34&pid=1","34&pid=2",
                                  "29&pid=1","29&pid=2",
                                  "30&pid=1","30&pid=2","30&pid=3"
                                };

                foreach (string s in sPID)
                {
                    _crawler.Crawl(new CrawlRequest(new Discovery("mydomain.com/watch.php?catgid="+s), int.MaxValue, UriClassificationType.Domain, UriClassificationType.Domain, 1));
                }