arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

AN & Remote SQL Servers

rated by 0 users
This post has 4 Replies | 2 Followers

Top 25 Contributor
Posts 27
egecko Dog [&] Posted: Tue, Jun 2 2015 10:25 PM

Hey there, just wanted to report a bug I found.  When we moved AN to a dedicated box, we left the SQL Server it uses in the cloud.  In theory all I should have needed to do was to update AN's connection string and spin it up on the new box.  However, when I did this, it was having trouble connecting to the proper database and instead kept trying to loop back to a locally installed instance of SQL Server.  I ended up tracing this back to the way the ArachnodeDAO class initializes itself.  When this class first instantiates itself, it initializes all the data adapters then fires its constructor which is passed the proper connection string to use, however when the constructor executes its calls to GetVersion() and ExecuteSql2(), the adapters' connection strings remain set at the default connection string.  To get around this problem, I added code to update all of the adapaters' connection strings:

            _allowedDataTypesTableAdapter.Connection.ConnectionString = connectionString;

            _allowedExtensionsTableAdapter.Connection.ConnectionString = connectionString;

            _allowedSchemesTableAdapter.Connection.ConnectionString = connectionString;

            _businessInformationTableAdapter.Connection.ConnectionString = connectionString;

            _configurationTableAdapter.Connection.ConnectionString = connectionString;

            _contentTypesTableAdapter.Connection.ConnectionString = connectionString;

            _crawlActionsTableAdapter.Connection.ConnectionString = connectionString;

            _crawlRequestsTableAdapter.Connection.ConnectionString = connectionString;

            _crawlRulesTableAdapter.Connection.ConnectionString = connectionString;

            _disallowedAbsoluteUrisTableAdapter.Connection.ConnectionString = connectionString;

            _disallowedAbsoluteUrisDiscoveriesTableAdapter.Connection.ConnectionString = connectionString;

            _disallowedDomainsTableAdapter.Connection.ConnectionString = connectionString;

            _disallowedExtensionsTableAdapter.Connection.ConnectionString = connectionString;

            _disallowedFileExtensionsTableAdapter.Connection.ConnectionString = connectionString;

            _disallowedHostsTableAdapter.Connection.ConnectionString = connectionString;

            _disallowedSchemesTableAdapter.Connection.ConnectionString = connectionString;

            _disallowedWordsTableAdapter.Connection.ConnectionString = connectionString;

            _discoveriesTableAdapter.Connection.ConnectionString = connectionString;

            _documentsTableAdapter.Connection.ConnectionString = connectionString;

            _domainsTableAdapter.Connection.ConnectionString = connectionString;

            _emailAddressesTableAdapter.Connection.ConnectionString = connectionString;

            _emailAddressesDiscoveriesTableAdapter.Connection.ConnectionString = connectionString;

            _engineActionsTableAdapter.Connection.ConnectionString = connectionString;

            _extensionsTableAdapter.Connection.ConnectionString = connectionString;

            _filesTableAdapter.Connection.ConnectionString = connectionString;

            _filesDiscoveriesTableAdapter.Connection.ConnectionString = connectionString;

            _filesMetaDataTableAdapter.Connection.ConnectionString = connectionString;

            _hostsTableAdapter.Connection.ConnectionString = connectionString;

            _hyperLinksTableAdapter.Connection.ConnectionString = connectionString;

            _hyperLinksDiscoveriesTableAdapter.Connection.ConnectionString = connectionString;

            _imagesTableAdapter.Connection.ConnectionString = connectionString;

            _imagesDiscoveriesTableAdapter.Connection.ConnectionString = connectionString;

            _imagesMetaDataTableAdapter.Connection.ConnectionString = connectionString;

            _prioritiesTableAdapter.Connection.ConnectionString = connectionString;

            _schemesTableAdapter.Connection.ConnectionString = connectionString;

            _versionTableAdapter.Connection.ConnectionString = connectionString;

            _webPagesTableAdapter.Connection.ConnectionString = connectionString;

 

            for(int i=0; i < _queriesTableAdapter.SqlCommands.Count(); i++)

                _queriesTableAdapter.SqlCommandsIdea.Connection.ConnectionString = connectionString; 

Notice the last for loop is needed because the connection string is actually buried in the SqlCommands[] array instead of off the adapter like the others.

Also, a quick note we changed from using a Trusted Windows login to a SQL Server login with credentials instead.  When this was done, the user account had dbo and role membership to all of the schemas and datareader and datawriter roles, but was still having trouble.  I was able to trace this back to a lack of permissions on the user account to view the server state.  If you are provisioning a user account, you need to make sure that user can read the server state, the TSQL to do this is:

GRANT VIEW SERVER STATE TO [<insert user login name here>]

 

eGecko

Top 10 Contributor
Posts 1,905

Thank you.

When I upgraded the /trunk to VS2012, the upgrade process did some "interesting" things with the connection strings.

I will check this out and fix 'er up.

Aloha,
Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
Posts 1,905

Hmm...

I used Console and updated ConnectionStrings.config to 'arachnode.net2' after changing the DB name to 'arachnode.net2'.

Where are you seeing the bug?  Stack trace?

Thank you!
Mike 

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 25 Contributor
Posts 27
egecko replied on Wed, Jun 3 2015 8:14 PM

I've been working in Arachnode.Service so I do not know if Console has the same issue or not.  Technically it does not throw an exception so getting a stack trace isn't really going to be possible, but because the connection strings are not properly set it when it does, it ends up trying to hit the wrong server to perform the version check..  I only noticed it because I was watching the ports it was trying to open via the resource monitor.  The problem was in the ArachnodeDAO.cs file when the constructor invoked its call to CheckVersion() immediately upon starting up; this should be approximately line 167. Note, the actual connection string parameter passed into the constructor contains the proper value, it's just the data adapters that are ending up using the default arachnode connection string which points to a local instance on port 1433 (the port was actually the dead give away in resource monitor since we run non-standard port configurations and AN was slamming 1433 instead of what I had updated the connection string value to).  The constructor does perform some additional logic after the CheckVersion() call so it may be just a matter of how things are currently sequenced and invoked -- I didn't really spend a lot of time analyzing the code but rather just identified the problem and came up with a quick fix for it (I'll leave the proper fix to the proper author and clean up anything later while merging any changes that get published. =D)

 

eGecko

Top 10 Contributor
Posts 1,905

Perhaps did you not rebuild - I don't believe VS will copy *.config files unless code has changed?

I fired up the Service and all worked for me.  (updated ConnectionStrings.config to 'arachnode.net2' and changed the DB name to 'arachnode.net2'.

I changed ConnectionStrings.config back to 'arachnode.net' and I get this:

Perhaps you are using VS2013, and there was an 'upgrade' which took place?

I am adding in the Proxy code from the Console to the Service as well.

 

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (5 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC