I have one site for which i am crawling it with the DepthLevel=6.
while checking the "webpages" table, the "Absoluteuri" field is having URLs that includes the pagination number links too.
like if on a page, there is a pagination to show a list of items. And on pagination numbers, we have the url like "http://www.xyz.com/aboutus.aspx?page=1&lang=en" and "http://www.xyz.com/aboutus.aspx?page=2&lang=en". I dont want to include such URLs. for this i can use "Disallowed by query string" to stop including such urls. But in many other case I need the url that has query string parameter too. like "http://www.xyz.com/aboutus.aspx?lang=en". so in such case i can not use "Disallowed by query string" because its just an example of "lang" field even there are many which i want to allow.
Please suggest me a way to achieve this.