arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Search Keyword in Single Input URL

rated by 0 users
Answered (Verified) This post has 1 verified answer | 3 Replies | 2 Followers

Top 100 Contributor
5 Posts
Syed Rizwan ul haq posted on Mon, Jul 13 2015 8:57 AM

Hi All,

            I have downloaded  the demo application from arachnode.net website, I want to search keyword in a single URL that the user will input. Can anyone suggest me the change required in demo application so that i can give harcoded input for URL in the code and get results for the keyword from URL.

 

Any help will be appreciated.

 

 

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

Like this: https://search.arachnode.net/Search.aspx?query=absoluteuri:www.nbc.com/today%20share&discoveryType=WebPage&pageNumber=1&pageSize=10&shouldDocumentsBeClustered=1

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

All Replies

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

Like this: https://search.arachnode.net/Search.aspx?query=absoluteuri:www.nbc.com/today%20share&discoveryType=WebPage&pageNumber=1&pageSize=10&shouldDocumentsBeClustered=1

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 100 Contributor
5 Posts

As per your suggestion, I hard coded value for Query

 

 protected void uxBtnSearch_Click(object sender, EventArgs e)

        {

            string t1 = "absoluteuri:www.nbc.com/today share";

 

            if (uxTbQuery.Text != string.Empty)

            {

                if (Request.QueryString["shouldDocumentsBeClustered"] == "0")

                {

                    Response.Redirect(Request.Url.LocalPath + "?query=" + t1 + "&discoveryType=" + uxRblDiscoveryType.SelectedValue + "&pageNumber=1&pageSize=" + WebSettings.PageSize + "&shouldDocumentsBeClustered=0", true);

                }

                else

                {

                    Response.Redirect(Request.Url.LocalPath + "?query=" + t1 + "&discoveryType=" + uxRblDiscoveryType.SelectedValue + "&pageNumber=1&pageSize=" + WebSettings.PageSize + "&shouldDocumentsBeClustered=1", true);

                }

            }

        }

 

Final Url look like this

http://localhost:56830/Search.aspx?query=absoluteuri:www.nbc.com/today%20share&discoveryType=WebPage&pageNumber=1&pageSize=10&shouldDocumentsBeClustered=1

 

Still getting no search results

 

Top 10 Contributor
1,905 Posts

The point of "absoluteuri:www.nbc.com/today share" was to illustrate you can instruct Lucene.NET to search a specific AbsoluteUri.

What you did instead was force a query, so that no matter what you search for you will likely never get results, unless you crawled nbc.com/today and wanted to search for 'share'.

Use the Lucene.NET syntax - there is no need to modify the code.

Or, if you really want to modify the code, don't add 'share' and use the actual AbsoluteUri you want to use - I am guessing you didn't want to search nbc.com?

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (4 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC