Sitecore Analytics Data Loss: Max Size of Insert Queue Reached

For high traffic websites using Sitecore Analytics, you may run into instances where you find the following warning in your Sitecore logs:

Analystics: Max size of insert queue reached. Dropped 2680.

While this may seem like just a warning, the truth is, you’re losing data.

The warning message tells you exactly how many database writes were attempted to be queued, but dropped in the current session.

Further Analysis

Sitecore Analytics runs as a multi-threaded background process collecting thread requests to write data to the Analytics database. When the number of requests queued is greater than the threshold set in your Analytics.Config file, warnings are written for every 25 failures discovered and those additional requests are discarded with no way to recover the data.

Your only solution is to increase your MaxQueueSize attribute in the Analytics.Config file. Just make sure your servers can handle the load.

Using reflector, the method used to queue requests to be written to the database shows the vulnerability. Keep in mind this method is used in

  • Creating campaign events
  • Updating visitor identity
  • All HTTP and Media request handling

The public static method Enqueue, exposed in the DatabaseSubmitter class of Sitecore.Analytics.Data places tracking in queue to be updated in the Analytics database.

private static readonly int maxSize = Settings.GetIntSetting("Analytics.MaxQueueSize", 500);

...
public static void Enqueue(IDatabaseSubmittable databaseSubmittable)
{
    Assert.ArgumentNotNull(databaseSubmittable, "databaseSubmittable");
         lock (syncQueue)
    {
        if (queue.Count > maxSize)
        {
            AnalyticsManager.SetStatusFailed(AnalyticsManager.STATUS_QUEUE_FULL);
            
            failed++;
            if ((failed % 0x19) == 1)
            {
                Log.Warn( string.Format("Analystics: Max size of insert queue reached. Dropped {0}.", failed), typeof(DatabaseSubmitter));
            }
        }
        else
        {
            queue.Add(databaseSubmittable);
        }
    }
}

Lets hope future versions of Sitecore Analytics will provide better ways of dealing with excessive load than dropping data without having a way to recover it.

Advertisements

About Dave
Certified Sitecore Developer.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: