Sitecore OMS Error: Overwhelming Change Notification

Today, our websites generated a stack trace error with an “Out of Memory Exception” message for the whole world to see.

Immediately looking at the IIS logs, we discovered 500 internal server errors over a 3-minute time span. Using the time stamp generated by IIS, I then compared against the error logs created by Sitecore (typically in [website path]\data\logs).

Looking at the Sitecore logs, sandwiched between two Analytics errors, I see the following:

4304 14:29:31 ERROR Failed to insert Analytics data
Exception: System.Data.SqlClient.SqlException
...

4276 14:30:18 INFO  **************************************************
4276 14:30:18 WARN  Sitecore shutting down
4276 14:30:18 WARN  Shutdown message: Overwhelming Change Notification in C:\inetpub\wwwroot
Overwhelming Change Notification in C:\inetpub\wwwroot
Change in GLOBAL.ASAX
HostingEnvironment initiated shutdown
CONFIG change

4304 14:30:26 ERROR Failed to insert Analytics data
Exception: System.Data.SqlClient.SqlException
...

Having no idea what “Overwhelming change notification” meant of what it was referring to, our team contacted Sitecore directly. Turns out that this error is a result of a bug in OMS 1.0.1 (running under Sitecore 6.1). An unhandled exception appeared in the Sitecore OMS module due to improper handling of the AnalyticsLogger class. Their support team were able to reproduce the issue and generate the following error message related to the one triggered above:

ERROR Unhandled exception detected. The ASP.NET worker process will be terminated.
Exception: System.IndexOutOfRangeException
Message: Index was outside the bounds of the array.
Source: mscorlib
   at System.Collections.Generic.List`1.Add(T item)
. . .

Currently, the only work around for this is to disable error logging for Sitecore OMS by following the steps below:

  1. Open the \Website\App_Config\Include\Sitecore.Analytics.config file.
  2. In Sitecore.Analytics.config file, comment the following pipeline’s processor:

    <!– <processor type=”Sitecore.Analytics.

    Pipelines.HttpRequest.StartDiagnostics, Sitecore.Analytics” patch:after=”processor[@type=’Sitecore.Pipelines.HttpRequest.StartMeasurements, Sitecore.Kernel’]” /> –>

Tough choice here…. Keep your OMS error logging or take your chances with random OMS exceptions?

Advertisements

About Dave
Certified Sitecore Developer.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: