Hi, short intro on our setup:
2x SOFS server in cluster + 3 MD1220 JBODS hosting SMB3 shares for library and workload for Hyper-V/SQL
3x Hyper-V hosts in cluster for Tenants
3x Hyper-V hosts in cluster for Fabric
2x SQL 2012 in guest cluster with shared VHDX
2x VMM 2012R2 in guest cluster
2x FileServer functioning as library (also guest cluster)
1x WAP for Tenant Self Service
1x WAP for Administration
1x SPF + 1X SMA + 2x SCO + 2x SCOM + 1x DPM + 1xSCCM
We have found some issues between SQL and the storage part, I didn't find the trigger creating the issue but see below log for details
----------------------------------------------------
------------------- Error Report -------------------
----------------------------------------------------
Error report created 7/10/2014 1:33:06 PM
CLR is not terminating
----------------------------------------------------
--------------- Bucketing Parameters ---------------
----------------------------------------------------
EventType=VMM20
P1(appName)=vmmservice.exe
P2(appVersion)=3.2.7634.0
P3(assemblyName)=Utils.dll
P4(assemblyVer)=3.2.7634.0
P5(methodName)=Microsoft.VirtualManager.DB.SqlRetryCommand.ExecuteNonQuery
P6(exceptionType)=Microsoft.VirtualManager.DB.CarmineSqlException
P7(callstackHash)=9d33
SCVMM Version=3.2.7634.0
SCVMM flavor=C-buddy-RTL-AMD64
Default Assembly Version=3.2.7634.0
Executable Name=vmmservice.exe
Executable Version=3.2.7510.0
Base Exception Target Site=140722485938904
Base Exception Assembly name=System.Data.dll
Base Exception Method Name=System.Data.SqlClient.SqlConnection.OnError
Exception Message=Unable to connect to the VMM database because of a general database failure.
Ensure that the SQL Server is running and configured correctly, then try the operation again.
EIP=0x00007ffc950c5bf8
Build bit-size=64
----------------------------------------------------
------------ exceptionObject.ToString() ------------
----------------------------------------------------
Microsoft.VirtualManager.DB.CarmineSqlException: Unable to connect to the VMM database because of a general database failure.
Ensure that the SQL Server is running and configured correctly, then try the operation again. ---> System.Data.SqlClient.SqlException: The DELETE statement conflicted with the REFERENCE constraint "FK_tbl_ST_StoragePhysicalDiskToFileServerNodeMapping_tbl_ST_StoragePhysicalDisk".
The conflict occurred
The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Microsoft.VirtualManager.DB.SqlRetryCommand.ExecuteNonQuery()
--- End of inner exception stack trace ---
at Microsoft.VirtualManager.DB.SqlRetryCommand.ExecuteNonQuery()
at Microsoft.VirtualManager.DB.ImageLibrary.StoragePhysicalDiskDBAccess.Remove(StoragePhysicalDiskData stData, ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.StrgLibObjectContainerBase.RemoveDeletedChildren(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.StrgLibObjectContainerBase.AddOrUpdateObjectInDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.LibObjectBase.UpdateDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.StrgLibObjectContainerBase.UpdateChildObjects(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.LibObjectBase.UpdateDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.Deployment.StrgRefresherBase.RefreshProvider(StrgProvider prov, Guid refresherOrTaskID, Boolean isLocked, Boolean forceDiscovery)
at Microsoft.VirtualManager.Engine.Deployment.StrgRefresherBase.RefreshData(StrgProviderReference providerRef, Nullable`1& auditTaskId)
at Microsoft.VirtualManager.Engine.RefreshDriver`1.RefreshThreadFunction(Object obj)
*** Carmine error was: DatabaseFailure (2605); SQLServer; 547
[s#15b499] ExecuteNonQuery()
[s#15b496] LibObjectBase.UpdateDB(this: (StrgArray#10570), catalog: (ImgLibCatalog#18b4b))
[s#15b484] LibObjectBase.UpdateDB(this: (StrgProvider#1056f), catalog: (ImgLibCatalog#18b4b))
Full call stack from when the exception was thrown: at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at Microsoft.VirtualManager.DB.CarmineSqlException..ctor(SqlException sqlException)
at Microsoft.VirtualManager.DB.SqlRetryCommand.ExecuteNonQuery()
at Microsoft.VirtualManager.DB.ImageLibrary.StoragePhysicalDiskDBAccess.Remove(StoragePhysicalDiskData stData, ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.StrgLibObjectContainerBase.RemoveDeletedChildren(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.StrgLibObjectContainerBase.AddOrUpdateObjectInDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.LibObjectBase.UpdateDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.StrgLibObjectContainerBase.UpdateChildObjects(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.LibObjectBase.UpdateDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.Deployment.StrgRefresherBase.RefreshProvider(StrgProvider prov, Guid refresherOrTaskID, Boolean isLocked, Boolean forceDiscovery)
at Microsoft.VirtualManager.Engine.Deployment.StrgRefresherBase.RefreshData(StrgProviderReference providerRef, Nullable`1& auditTaskId)
at Microsoft.VirtualManager.Engine.RefreshDriver`1.RefreshThreadFunction(Object obj)
at Microsoft.VirtualManager.Utils.Multithreading.ThrottledThreadPool.<>c__DisplayClass3.<ScheduleNextThread>b__2(Object dummy)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
--- SqlException details -----------------
System.Data.SqlClient.SqlException (0x80131904): The DELETE statement conflicted with the REFERENCE constraint "FK_tbl_ST_StoragePhysicalDiskToFileServerNodeMapping_tbl_ST_StoragePhysicalDisk". The conflict occurred
The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Microsoft.VirtualManager.DB.SqlRetryCommand.ExecuteNonQuery()
ClientConnectionId:60625fa5-b80b-49e1-bdd4-cefe54a942e4
Error = 547
Index #0
Source: .Net SqlClient Data Provider
Number: 547
State: 0
Class: 16
Server: SEGOTN99016-002\SCVMMDB,1435
Message: The DELETE statement conflicted with the REFERENCE constraint "FK_tbl_ST_StoragePhysicalDiskToFileServerNodeMapping_tbl_ST_StoragePhysicalDisk". The conflict occurred
Procedure: prc_ST_RemoveStoragePhysicalDisk
Line: 12
Index #1
Source: .Net SqlClient Data Provider
Number: 3621
State: 0
Class: 0
Server: SEGOTN99016-002\SCVMMDB,1435
Message: The statement has been terminated.
Procedure: prc_ST_RemoveStoragePhysicalDisk
Line: 12
--- Caller StackTrace -----------------
FileName:; Method:ExecuteNonQuery(); lineNo:0; ilOffset:178.
FileName:; Method:Remove(); lineNo:0; ilOffset:43.
FileName:; Method:RemoveDeletedChildren(); lineNo:0; ilOffset:76.
FileName:; Method:AddOrUpdateObjectInDB(); lineNo:0; ilOffset:14.
FileName:; Method:UpdateDB(); lineNo:0; ilOffset:206.
FileName:; Method:UpdateChildObjects(); lineNo:0; ilOffset:42.
FileName:; Method:UpdateDB(); lineNo:0; ilOffset:206.
FileName:; Method:RefreshProvider(); lineNo:0; ilOffset:806.
FileName:; Method:RefreshData(); lineNo:0; ilOffset:163.
FileName:; Method:RefreshThreadFunction(); lineNo:0; ilOffset:82.
FileName:; Method:<ScheduleNextThread>b__2(); lineNo:0; ilOffset:27.
FileName:; Method:RunInternal(); lineNo:0; ilOffset:112.
FileName:; Method:Run(); lineNo:0; ilOffset:4.
FileName:; Method:System.Threading.IThreadPoolWorkItem.ExecuteWorkItem(); lineNo:0; ilOffset:0.
FileName:; Method:Dispatch(); lineNo:0; ilOffset:162.
-------- exception.InnerException.ToString() -------
System.Data.SqlClient.SqlException (0x80131904): The DELETE statement conflicted with the REFERENCE constraint "FK_tbl_ST_StoragePhysicalDiskToFileServerNodeMapping_tbl_ST_StoragePhysicalDisk". The conflict occurred
The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Microsoft.VirtualManager.DB.SqlRetryCommand.ExecuteNonQuery()
ClientConnectionId:60625fa5-b80b-49e1-bdd4-cefe54a942e4
----------------------------------------------------
--------------- exception.StackTrace ---------------
----------------------------------------------------
at Microsoft.VirtualManager.DB.SqlRetryCommand.ExecuteNonQuery()
at Microsoft.VirtualManager.DB.ImageLibrary.StoragePhysicalDiskDBAccess.Remove(StoragePhysicalDiskData stData, ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.StrgLibObjectContainerBase.RemoveDeletedChildren(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.StrgLibObjectContainerBase.AddOrUpdateObjectInDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.LibObjectBase.UpdateDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.StrgLibObjectContainerBase.UpdateChildObjects(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.ImageLibrary.LibObjectBase.UpdateDB(ImgLibCatalog catalog)
at Microsoft.VirtualManager.Engine.Deployment.StrgRefresherBase.RefreshProvider(StrgProvider prov, Guid refresherOrTaskID, Boolean isLocked, Boolean forceDiscovery)
at Microsoft.VirtualManager.Engine.Deployment.StrgRefresherBase.RefreshData(StrgProviderReference providerRef, Nullable`1& auditTaskId)
at Microsoft.VirtualManager.Engine.RefreshDriver`1.RefreshThreadFunction(Object obj)
----------------------------------------------------
------------- StackTrace from handler --------------
----------------------------------------------------
This is the call stack from where the exception was caught, not where it was thrown.
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonReport.WriteReportTextFile(TextWriter reportFile)
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonReport.Send()
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonExceptionReport.Send()
at Microsoft.VirtualManager.Utils.Diagnostics.WatsonCenter.ReportException(Exception e, WERReportOptions options, String& localReportPath)
at Microsoft.VirtualManager.Engine.TaskRepository.TaskRepository.HandleCriticalRefresherFailure(Object sender, CriticalRefreshFailureEventArgs eventData)
at Microsoft.VirtualManager.Engine.RefreshDriver`1.RefreshThreadFunction(Object obj)
at Microsoft.VirtualManager.Utils.Multithreading.ThrottledThreadPool.<>c__DisplayClass3.<ScheduleNextThread>b__2(Object dummy)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
Any ideas so far?
MCITP SA - MCP Active Directory - MCP Infrastructure - MCP Server Virtualization www.it101.eu