arachnode.net

How to avoid creating discoveries ?

Dinesh — Tue, 21 May 2013 13:48:16 GMT

Hi All,

I wrote a small Plugin (crawlaction) which will extract the specific content from website and save the content into database. Here is piece of code from my plugin. I am taking the crawl requests (websites) from CrawlRequests.txt and this .txt file has only one website name such as acquia.com. So, I want to execute the below PerformAction method only once but this method executing more than 100 times. I think the acquia.com has 100+ discoveries, thats why the PerformAction method executing 100 + times. How to avoid creating discoveries or how to avoid multiple times execution ?

public override void PerformAction(CrawlRequest crawlRequest, ArachnodeDAO arachnodeDAO)

{

_crawlRequest = crawlRequest.Parent.Uri.AbsoluteUri.ToString();

// Get the URL specified

var webGet = new HtmlWeb();

var document = webGet.Load(_crawlRequest);

//hyperlink weblinks

var companyLnks = document.DocumentNode.SelectNodes("//a");

if (companyLnks != null)

{

foreach (var lnk in companyLnks)

{

if (lnk.Attributes["href"] != null)

{

if (lnk.Attributes["href"].Value.Contains("https://www.facebook.com/"))

{

_fbLink = lnk.Attributes["href"].Value.ToString();

}

_activityDateTime = DateTime.Now.ToString();

arachnodeDAO.InsertWebLinks(_crawlRequest, _fbLink, _activityDateTime);

How to run schedule crawler (program.cs) automatically

Dinesh — Thu, 23 May 2013 09:07:03 GMT

I would like run my crawler everyday at 9:00 PM automatically. How do I accomplish this task ?

Getting IndexReader is closed error :(

InvestisDev — Thu, 11 Apr 2013 20:57:22 GMT

Hello Mike,

We are using the older service.asmx in which we have the Hits object and on making a loop off it getting field values.

If we will hit this service.asmx multiple times frequently to hit this service at the same time (almost), it throws error that IndexReader is closed. The frequency of this error increased. Please suggest some way to fix this.

Thanks,

Indexing and data mining

john12890 — Thu, 18 Apr 2013 11:26:17 GMT

Hi all,

Just going through the Demo version of AN, I would like to know how this tool indexing the data? Is the tool do the Data Mining of the crawled data? I would like to discuss in detail in this regard.

My issue is that I've to crawl specific data from a authenticated web page and do the analytics on that. Will this tool can help me out?

Thanks.

Is there any way to run crawler as web service?

Abhishek Gahlout — Tue, 16 Apr 2013 05:51:27 GMT

Hi , I need to know that is there any mechanism to run crawler as web service ? If yes then how?

Actually I want to run crawler for some website links in some particular cases on my website.

So can I run crawler as webservice?

Crawling of webpage where data comes with scrollbar

Abhishek Gahlout — Thu, 11 Apr 2013 09:23:02 GMT

Hi , I need to know , how should I crawl a webpage where the information is loading when page scrolls. eg: please visit the link

http://www.wikido.com/us/ny/ny

Though there are thousand of records for this page but at a time only 60 records are shown. further records are shown when we scroll the vertical scroll bar.Crawler only fetches data foar 60 records at a time as when page first loads it only shows 60 records. So to crawl and get complete information , how should I crawl these kind of webpages.

Crawling of webpages where data comes after page load

Manoj — Wed, 10 Apr 2013 09:15:13 GMT

Hi , Please tell me that how can I crawl the webpages where the information comes after the page loading is completed. Eg: http://www.spotcrime.com/#new%20york

When I crawl this webpage there is no relevant information in the view source which gets stored in the DB , though that information is available on the webpage. The data comes after the page gets loaded by some javascript function.

What steps should I follow to crawl such webpages.?

Crawl specific domains, take screenshot, run javascript, ID ads, write to DB ... any ideas?

jpntol — Sat, 14 Feb 2009 13:53:24 GMT

We want to capture info about AD SPOTS;

- on specific list of approx 10,000 domains

- capture a screenshot png / jpg

- run javascript on each page (browser specific?)

- read js and identify ad spot SIZES

- identify PLACE on page that the adspot is located

- relate this location back to the screenshot (draw it on the screenshot)

- write all to DB

- repeat above for next level down (max 1)

- repeat all weekly, compare and highlight differences

Are we looking at right toolkit with arachnode? Anyone out there wants to have a crack at coding this?

Can AN crawl the authenticated website content ?

john12890 — Fri, 05 Apr 2013 08:18:52 GMT

Hello

I am new to use the AN crawler, We would like to crawl the content from authenticated web page. Is this possible ? and also would like crawl no of likes, no of followers from Facebook and Twitter every day . Is this possible ??

Thanks

John Adams

Requested registry access is not allowed while installing

john12890 — Fri, 05 Apr 2013 14:50:56 GMT

We downloaded the DEMO_3.0 - VS2010 - SQL2008.zip file and followed the installation steps provided by arachnode, but suddenly we got an error while installing like "Requested registry access is not allowed" and "securityexception was unhandled by user code". Can anyone help me on this ?

Regards

John Adams

Crawling of webpage with HTML form

Abhishek Gahlout — Tue, 02 Apr 2013 05:15:14 GMT

I have a question which is as follows:

If I have a webpage with a html form. In that html form I have a dropdown and a submit button with other controls. Now after clicking the submit button , depending on the combobox selected value , some result page is shown with some information. Now If i crawl that website , will I get the result in DB for each possible pages which would have been formed by all the possible values of the dropdown and submitting them by cHi Mike, I have a question which is as follows: If I have a webpage with a html form. In that html form I have a dropdown and a submit button with other controls. Now after clicking the submit button , depending on the combobox selected value , some result page is shown with some information. Now If i crawl that website , will I get the result in DB for each possible pages which would have been formed by all the possible values of the dropdown and submitting them by clicking submit button.licking submit button.

eg: go to the following link : http://nces.ed.gov/globallocator/

In this webpage I can get the result including school details zipcode wise or state-city wise if I successfully submit the html form . So is our Crawler able to fetch the resultant data (by automatically filling the form and submit that form ). Please help in this regard.

start another program while crawling

samarhtc — Sat, 15 Dec 2012 15:28:40 GMT

Hi I have a question regarding the crawling. I have made a console program that I want to start running right after the crawling has started. I have noticed that the crawling is first started in the section mentioned below

"Console, Program.cs file"

catch (Exception exception) { System.Console.WriteLine(exception.StackTrace); }

//necessary for the Rendering functionality. if (_crawler != null) { //may be null if all configuration settings are not initialized in the database... while (!_hasCrawlCompleted && _crawler.AreRenderersEnabled) { Application.DoEvents(); } }

System.Console.ReadLine();

//-----------------------------// The crawling begins now, that is after System.Console.ReadLine(); //-----------------------

//This is hit when the crawler is finished. if (_crawler != null && _crawler.Engine != null) { _crawler.Engine.Stop(); }

What I'm wondering about is how can I start my program right after the crawling has begun. Shall I just call it after System.Console.ReadLine, this does not work, because at that time the crawler has finished crawling. Please note that for personal reasons I do not want to write a plugin. I just want to make a call to my (console) program right after the crawling has begun, and thereby have the two (crawler and my program) be running at the same time.

//----------------------------------------------------------------------------------------------

"Function.dll" has to be signed.

samarhtc — Sat, 15 Dec 2012 15:22:29 GMT

Hi I'm having a problem with the database. I'm not able to work with the arachnode.net database, since SQL Server is keep saying that "Function.dll" has to be signed. I'm not sure what this means, however I have gotten a signed certificate (myCA.cer and mySPC.pfx). My question is now, what do I do? Shall I sign the function.dll or function project with this certificate, if so then how is this done exactly.

thanks

Error : Type 'AbsoluteUri' already exists or you do not have permission to create it.

jaydeep — Tue, 03 Feb 2009 12:37:23 GMT

Hello,

I am new to arachnode.net and trying to make it work for me.

After all set up and instalation while running i get deployement erroe which says Type 'AbsoluteUri' already exists or you do not have permission to create it.

Can you tell me the resolution?

Thanks in advance.
JD

Site is not crawling completely

InvestisDev — Tue, 11 Dec 2012 05:53:11 GMT

Hello Mike,

i was crawling "http://www.bodycote.com" site. I had set the depth level to 6 while crawling this site.

if you will search the keyword "china" it will show some results but that result is not including "http://www.bodycote.com/en/contact-directory/asia/china.aspx" whose depth level is < 6.

i had even checked the DB too. the pages on the level of china and below are not in the list of hyperlinks or webpages tables.

Even i am downloading the pages so if checking the "DownloadWebPages" folder, there is only one page thats of "world.aspx".

Please suggest some way to solve this.

Thanks,

Consistency and fault tolerance

Shakir — Tue, 11 Dec 2012 13:22:30 GMT

Hello dear

How do you implement Replication consistency and Fault tolerance in this lovely and massive crawler ?

Thank you in advance

AN.Next configuration

koolworld — Mon, 10 Dec 2012 19:19:36 GMT

Hi Mike,

We're looking to use AN.Next over the main Arachnode due to it being simply to learn and use. Can i just confirm the following:

1. Can it use SQL rather than CE

2. Process files such as MS Office and PDF files

3. Use lucene to create the index and add to the lucene index fields to suit our situation (i know we created our own Lucene index class which inherited from ManageLuceneDotNetIndexesso we could have our own index fields - does AN.Next use these same plugins?)

4. Are the configuration settings taken from the database? There seems to be a configuration file passed to the crawler currently.

5. Can you confirm the minimum requirements for the database objects so that we can strip out any tables which aren required?

Sorry about all the questions!

Thanks
Dan

IndexSearcher not releasing file

InvestisDev — Wed, 21 Nov 2012 06:28:44 GMT

Hello Mike,

i had created one separate application which will have a list of URLS to crawl. Sequentially it will crawl all the sites and Copies index files into separate folders from were service.asmx will access the index files.

Now for example there is one folder "www.abc.com" having all the index files and some one is accessing the index file through the service.asmx. At this time if i will remove all the index files to copy the new one, it says _0.fdt file is in use.

Can you suggest a way to release the resource and remove the files to override the new index files.

Note: can i use Global.IndexSearcher.Close() to release the resource. [Please explain how can i use it]

Thanks,

deploy to production server

samarhtc — Fri, 30 Nov 2012 15:47:57 GMT

Hi I have arachnode.net running on a development server. I would like to deploy this solution to another server (production server). There will not be any development on the production server. When I right-click the solution and choose deploy nothing happens, so it seems. I was expecting that a dialog box would popup and ask me which folder I want to deploy to, I would then choose a folder and afterwards copy paste the files from development server to production server.

Is there something I have misunderstood?

dbo.convertsource function does not exists anymore, how to convert source column in webpages tabel

samarhtc — Tue, 20 Nov 2012 14:52:48 GMT

I would like to convert the Source column inside webpages table into text. In a similar post the answer is to use the function dbo.convertsource, but this function does not exits anymore in the arachnode.net database or rather its nowhere to find. I have used this to find it :

SELECT [TEXT] FROM SYS.SYSCOMMENTS WHERE [TEXT] LIKE '%ConvertSource%'

I have a "the newest version" that is a paid version of arachnode.net from sommer 2012

So what can I do now?

disable encoding of source column within webpages table

samarhtc — Wed, 21 Nov 2012 12:21:37 GMT

I would like to disable the encoding of source column within the webpages table.

What I want instead is to have the crawled data be inserted into the Source column as text (nvarchar).

Where do I do this? (which solutions, which method?) Another thing is that I have a paid version of arachnode.net from this summer, however I am not able to debug into the SiteCrawler project. I am able to set the Console as start up project and start debugging from here, but as soon as I hit the Crawl method (Arachnode.SiteCrawler.Crawler.Crawl(Arachnode.SiteCrawler.Value.CrawlRequest)) I cannot continue into this with the debugger. Why not, this is a paid version.

P.S. I'm running Visual Studio 2010 Ultimate, SQL Server 2008 R2, Windows 2008, IIS 7 and I have already tried deleting all break points and starting a new.

Thanks.

debugging arachnode.net solution

samarhtc — Thu, 18 Oct 2012 13:11:07 GMT

Our company have bought a version of Arachnode.net I am currently trying to modify its behaviour. However I cannot seem to debug the solution that is my break points are never hit. I have set some break points in different projects, I then run (F5) the Console project and crawl a site, the crawling is done but the break points are never hit. Or rather the break point are hit from the DataAccess project, the ArachnodeDAO.cs file, but I cannot get it to hit anything within the SiteCrawler project

I have also tried to set a break point right at the beginning of the crawler that is inside program.cs inside the console project and then F10/F11 all the way, but still no result its like the program flow never goes into other projects.

The visual studio is set on debug mode.

Am I missing something here.

Making the crawler work faster

samarhtc — Wed, 21 Nov 2012 16:12:56 GMT

I'm in a situation where I have to crawl some very large websites. The resulting db is apx 120 gb. This is actually ok, but the time it takes to crawl one website is way to long. I'm crawling with 100 thread, but still it takes days.

So I was wondering how to minimize the crawl time. What I need from the crawling is the webpages that is raw data only meaning html code only, no images, or anything else.

I was thinking if I somehow disable the encoding of the Source column within the webpages table, maybe this would speed up the process, since now AN does not have to encode/decode the webpages it simply insert the html inside the source column (or a new column next to it source_2!) in nvarchar format.

what do you think, any suggestion would be appreciated.

crawler not crawling

bmchunu — Tue, 17 Jul 2012 22:56:04 GMT

(Please visit the site to view this media)

(Please visit the site to view this media)The crawler is not crawling

Settings below and logs attached. Not sure what else I can send through. I had the demo version working fine, not sure what's wrong here!

Key Value

AssignCrawlRequestPrioritiesForFiles true

AssignCrawlRequestPrioritiesForHyperLinks true

AssignCrawlRequestPrioritiesForImages true

AssignCrawlRequestPrioritiesForWebPages true

AssignEmailAddressDiscoveries true

AssignFileAndImageDiscoveries true

AssignHyperLinkDiscoveries true

ClassifyAbsoluteUris true

ConsoleOutputLogsDirectory C:\Users\user\Downloads\arc\arc1\Console\bin\Debug\ConsoleOutputLogs

CrawlRequestTimeoutInMinutes 1

CreateCrawlRequestsFromDatabaseCrawlRequests true

CreateCrawlRequestsFromDatabaseFiles true

CreateCrawlRequestsFromDatabaseHyperLinks true

CreateCrawlRequestsFromDatabaseImages true

CreateCrawlRequestsFromDatabaseWebPages true

DesiredMaximumMemoryUsageInMegabytes 1024

DownloadedFilesDirectory C:\Users\user\Downloads\arc\arc1\Console\bin\Debug\DownloadedFiles

DownloadedImagesDirectory C:\Users\user\Downloads\arc\arc1\Console\bin\Debug\DownloadedImages

DownloadedWebPagesDirectory C:\Users\user\Downloads\arc\arc1\Console\bin\Debug\DownloadedWebPages

EnableConsoleOutput true

ExtractFileMetaData false

ExtractImageMetaData true

ExtractWebPageMetaData false

HttpWebRequestRetries 5

InsertDisallowedAbsoluteUriDiscoveries true

InsertDisallowedAbsoluteUris true

InsertEmailAddressDiscoveries true

InsertEmailAddresses true

InsertExceptions true

InsertFileDiscoveries true

InsertFileMetaData false

InsertFiles true

InsertFileSource true

InsertHyperLinkDiscoveries true

InsertHyperLinks true

InsertImageDiscoveries true

InsertImageMetaData true

InsertImages true

InsertImageSource false

InsertWebPageMetaData false

InsertWebPages true

InsertWebPageSource true

MaximumNumberOfCrawlRequestsToCreatePerBatch 1000

MaximumNumberOfCrawlThreads 10

MaximumNumberOfHostsAndPrioritiesToSelect 10000

OutputConsoleToLogs true

OutputStatistics false

SaveDiscoveredFilesToDisk true

SaveDiscoveredImagesToDisk true

SaveDiscoveredWebPagesToDisk true

SqlCommandTimeoutInMinutes 60

UserAgent http://arachnode.net 1.4

VerboseOutput false

CacheTimeoutInMinutes 15

CreateCrawlRequestsForMissingFilesAndImages false

DownloadedFilesVirtualDirectory \DownloadedFiles

DownloadedImagesVirtualDirectory \DownloadedImages

LuceneDotNetIndexDirectory C:\Users\user\Downloads\arc\arc1\Console\bin\Debug\LuceneDotNetIndex

MaximumNumberOfDocumentsToReturnPerSearch 200

MaximumPageTitleLength 64

PageSize 10

About the Calculation of "relevancyscore" field

InvestisDev — Fri, 27 Apr 2012 05:22:45 GMT

Hello Mike,

I want to know the logic of the field "relevancyscore" (score) value, how the scoring of the records can be calculated.

Please explain it with an example like if i have 10 results returned how the value of "relevancyscore" will be calculated.

Thanks,