arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop
Search the Live Index Does arachnode.net scale? | Download the latest release

crawler not initiating

rated by 0 users
Answered (Verified) This post has 1 verified answer | 2 Replies | 3 Followers

Top 100 Contributor
3 Posts
shoaib posted on Sun, Aug 16 2009 10:21 AM

i built the project and it gave the following error

Error    2    CREATE ASSEMBLY for assembly 'Arachnode.Functions' failed because assembly 'Arachnode.Functions' is not authorized for PERMISSION_SET = UNSAFE.  The assembly is authorized when either of the following is true: the database owner (DBO) has UNSAFE ASSEMBLY permission and the database has the TRUSTWORTHY database property on; or the assembly is signed with a certificate or an asymmetric key that has a corresponding login with UNSAFE ASSEMBLY permission.    Functions

however it started running inspite of this exception, but on running it gave the following exception but it keeps on running

System.TypeInitializationException was unhandled
  Message="The type initializer for 'Arachnode.Console.Program' threw an exception."
  Source="Arachnode.Console"
  TypeName="Arachnode.Console.Program"
  StackTrace:
       at Arachnode.Console.Program.Main()
       at System.AppDomain.nExecuteAssembly(Assembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()

 

Note: i have made the console project as 'start up' project and also executed the alter command 'ALTER DATABASE [arachnode.net] SET TRUSTWORTHY ON'

i am using the 2005 version, not switched to 2008 yet, you guys launched the 2008 in midway

i wonder why these errors are comming

 

also how to setup crawling restrictions and where to specify crawling url, and how to know if the crawler is running, where to check the indeces and where to query the indexed data, i'd be thankful

 

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

Is SQL set up for remote connections?

Once you get it crawling I can answer the other questions... (tired... Smile)

Also, thanks for answering, Megetron!

[EDIT: You will need to run the RESET procedure in the DB the first time you run AN]

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

All Replies

Top 10 Contributor
229 Posts

Hello and welcme to AN.

Make sure you have run stored procedure called: arachnode_usp_arachnode.net_RESET_DATABASE and make sure you are using SQL2005 database file when restore.

check that: http://arachnode.net/forums/p/303/10444.aspx#10444

to start crawling set CrawlRequest table. to create a crawl restriction you can always create a new rule (for refference check the absoluteuri.cs) to know crawl is running is simple, when pressing F5 on cnsole mode you will get the black command window that will prompt you with the current actions. you can check indexs on webpages table and disallowedabsoluteuri table is a very important table there you find all of the uris that did not pass your rule/restriction,

to query the data create your own plugin if needed. for refference check Anonymizer.cs which is the most simple one.

Have nice crawling.

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

Is SQL set up for remote connections?

Once you get it crawling I can answer the other questions... (tired... Smile)

Also, thanks for answering, Megetron!

[EDIT: You will need to run the RESET procedure in the DB the first time you run AN]

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (3 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC