arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Exception: Cannot insert duplicate key row in object 'dbo.Extensions' with unique index 'IX_Extensions'. Cannot insert the value NULL into column 'ExtensionID', table 'arachnode.net.dbo.Extensions_Discoveries'; column does not allow nulls. INSERT fails.

rated by 0 users
Answered (Verified) This post has 1 verified answer | 1 Reply | 2 Followers

Top 10 Contributor
229 Posts
megetron posted on Thu, Sep 3 2009 3:39 PM

Another exception:

Created ID AbsoluteUri1 AbsoluteUri2 HelpLink Message Source StackTrace
2009-09-04 01:16:12.200 5763 NULL NULL NULL Cannot insert duplicate key row in object 'dbo.Extensions' with unique index 'IX_Extensions'.  Cannot insert the value NULL into column 'ExtensionID', table 'arachnode.net.dbo.Extensions_Discoveries'; column does not allow nulls. INSERT fails.  The statement has been terminated.  The statement has been terminated. .Net SqlClient Data Provider    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)     at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)     at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)     at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)     at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()     at System.Data.SqlClient.SqlDataReader.get_MetaData()     at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)     at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)     at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)     at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)     at System.Data.SqlClient.SqlCommand.ExecuteScalar()     at Arachnode.DataSource.ArachnodeDataSetTableAdapters.QueriesTableAdapter.InsertDisallowedAbsoluteUri(Nullable`1 ContentTypeID, Nullable`1 DiscoveryTypeID, String WebPageAbsoluteUri, String DisallowedAbsoluteUriAbsoluteUri, String Reason, Nullable`1 ClassifyAbsoluteUri, Nullable`1& DisallowedAbsoluteUriID) in E:\DEVELOPMENT\yeshira12\DataSource\ArachnodeDataSet.Designer.cs:line 13307     at Arachnode.DataAccess.ArachnodeDAO.InsertDisallowedAbsoluteUri(Int32 contentTypeID, Int32 discoveryTypeID, String webPageAbsoluteUri, String disallowedAbsoluteUriAbsoluteUri, String reason) in E:\DEVELOPMENT\yeshira12\DataAccess\ArachnodeDAO.cs:line 848

 

 

absoluturi1 and absoluturi2are NULL but it happens with regular absoluturis

AbsoluteUri1 AbsoluteUri2
http://sratim.co.il/movies/actors/view.aspx?id=2339 http://sratim.co.il/movies/view.aspx?id=46210

 

and sometimes even the 1st uri is NULL and the second one is fill with a real absolute uri

do you see what goes wrong?

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by megetron

This occurs because we don't lock this table when inserting rows.

We do check using EXISTS but don't lock using a TRANSACTION for performance reasons.  If this error is present, the record is safely recorded.

You can ignore this error.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (2 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC