Sitecore Analytics Data Loss: Max Size of Insert Queue Reached

For high traffic websites using Sitecore Analytics, you may run into instances where you find the following warning in your Sitecore logs:

Analystics: Max size of insert queue reached. Dropped 2680.

While this may seem like just a warning, the truth is, you’re losing data.

The warning message tells you exactly how many database writes were attempted to be queued, but dropped in the current session.

Further Analysis

Sitecore Analytics runs as a multi-threaded background process collecting thread requests to write data to the Analytics database. When the number of requests queued is greater than the threshold set in your Analytics.Config file, warnings are written for every 25 failures discovered and those additional requests are discarded with no way to recover the data.

Your only solution is to increase your MaxQueueSize attribute in the Analytics.Config file. Just make sure your servers can handle the load.

Using reflector, the method used to queue requests to be written to the database shows the vulnerability. Keep in mind this method is used in

  • Creating campaign events
  • Updating visitor identity
  • All HTTP and Media request handling

The public static method Enqueue, exposed in the DatabaseSubmitter class of Sitecore.Analytics.Data places tracking in queue to be updated in the Analytics database.

private static readonly int maxSize = Settings.GetIntSetting("Analytics.MaxQueueSize", 500);

...
public static void Enqueue(IDatabaseSubmittable databaseSubmittable)
{
    Assert.ArgumentNotNull(databaseSubmittable, "databaseSubmittable");
         lock (syncQueue)
    {
        if (queue.Count > maxSize)
        {
            AnalyticsManager.SetStatusFailed(AnalyticsManager.STATUS_QUEUE_FULL);
            
            failed++;
            if ((failed % 0x19) == 1)
            {
                Log.Warn( string.Format("Analystics: Max size of insert queue reached. Dropped {0}.", failed), typeof(DatabaseSubmitter));
            }
        }
        else
        {
            queue.Add(databaseSubmittable);
        }
    }
}

Lets hope future versions of Sitecore Analytics will provide better ways of dealing with excessive load than dropping data without having a way to recover it.

Advertisements

Tracking Selenium Commands Using C# and SQL Server 2008

When running a suite of Selenium tests, I wanted an easy way of looking at the pages visited as well as commands executed during each unit test. To solve this problem, I designed a solution built on top of the core Selenium classes that records all unit test data to a SQL Server 2008 database.

In this post, details of the following are covered:

  • Back End Design – Database schema overview.
  • Core logic – The class that drives tracking of the test.
  • Example Unit Test – Example of tracking a simple unit test.

Not all code required to communicate with the database is listed in this post. You can either download the source from the project on SourceForge by going to https://sourceforge.net/projects/seleniumtrack/ or browse the source in the project’s SVN repository.

The Back-End Design

The SQL Server database consists of 5 tables:

  • Test Suite – All test suites executed.
  • UnitTest – All unit tests for each test suite.
  • CommandLog – Commands executed for each unit test.
  • PageLog – Page or URL accessed by a command.
  • Cookies – Cookies values, if any, available when viewing a page.

tracking database schema

Tracking Database Schema


 

Core Logic

There’s one main class the directs the traffic, TrackDefaultSelenium. Inheriting from DefaultSelenium, the commands start, stop, open and click are recorded.

It’s here where test suites and unit test records are saved in the database. For each command targeted at the TrackDefaultSelenium class, records are saved which can be easily tied back to their respective unit tests.


    public class TrackDefaultSelenium : DefaultSelenium, ISelenium

    {

        private string _screenshotPath = @"c:\build\SeleniumTests\Screenshots\";

        private string _currentUrl;

        private int _testSuiteId;

        private int _unitTestId;

        private bool _captureScreenShots = false;

 

        private string _testSuiteName;

        private string _unitTestName;

        public TestStatus UnitTestStatus { get; set; }

 

        public enum TestStatus

        {

            Running,

            Completed,

            Failed

        }

Constructors – Kicks off the creation of a test suite.


        public TrackDefaultSelenium(ICommandProcessor processor, string testSuiteName, string unitTestName, bool CaptureScreenshots)

            : base(processor)

        {

            this._currentUrl = "";

            this._testSuiteName = testSuiteName;

            this._unitTestName = unitTestName;

            this._captureScreenShots = CaptureScreenshots;

            this._testSuiteId = Tracking.CreateTestSuite(this._unitSuiteName);

        }

 

        public TrackDefaultSelenium(string serverHost, int serverPort, string browserString, string browserURL)

            : base(serverHost, serverPort, browserString, browserURL)

        {

            this._currentUrl = "";

            this._testSuiteId = Tracking.CreateTestSuite(this._unitSuiteName);

        }

Command OverridesStart creates the unit test, whereas, click and open record page hits.


        void ISelenium.Start()

        {

            this.UnitTestStatus = TestStatus.Running;

            this.commandProcessor.Start();

            this._unitTestId = Tracking.CreateUnitTest(this._testSuiteId, this._unitTestName, GetLocation());           

        }

 

        void ISelenium.Click(string locator)

        {

            this.commandProcessor.DoCommand("click", new string[] { locator });

            int fileId = RecordPageHit("click", GetLocation());

 

            if (this._captureScreenShots)

                CaptureScreenShot(fileId);

        }

 

        void ISelenium.Open(string url)

        {

            this.commandProcessor.DoCommand("open", new string[] { url });

            RecordPageHit("open", GetLocation());

        }

 

        void ISelenium.Stop()

        {

            Tracking.UpdateUnitTestStatus(this._unitTestId, UnitTestStatus.ToString());

            this.commandProcessor.Stop();

        }

Tracking and Screenshot Capture Methods


        private void CaptureScreenShot(int fileId)

        {

            if (!System.IO.Directory.Exists(this._screenshotPath + this._testSuiteId))

                System.IO.Directory.CreateDirectory(this._screenshotPath + this._testSuiteId);

 

            if (!System.IO.Directory.Exists(this._screenshotPath + this._testSuiteId + @"\" + this._unitTestId))

                System.IO.Directory.CreateDirectory(this._screenshotPath + this._testSuiteId + @"\" + this._unitTestId);

 

            CaptureEntirePageScreenshot(this._screenshotPath + this._testSuiteId + @"\" + this._unitTestId + @"\" + fileId + @".jpg", "");

        }

 

        private int RecordPageHit(string command, string Url)

        {

            // record the command

            int CommandId = Tracking.CreateCommandLogEntry(this._unitTestId, command, "");

            int PageId = Tracking.CreatePageLogEntry(CommandId, Url);

 

            // attempt to obtain all cookie values without causing test to fail

            string ASPNetSessionId = "";

            try { ASPNetSessionId = GetCookieByName("ASP.NET_SessionId"); } catch { }

 

            // record all cookie values

            if (ASPNetSessionId != "")

                Tracking.CreateCookieEntry(PageId, "ASP.NET_SessionId", ASPNetSessionId);

 

            // save the URL

            this._currentUrl = Url;

 

            return CommandId;

        }

    }

}

Example Unit Test

In this example, we hit a w3school’s page, check the page title and record every command directed at the Selenium server.


using System;

using System.Text;

using System.Text.RegularExpressions;

using System.Threading;

using NUnit.Framework;

using Selenium;

using Web.Diagnostics.Selenium;

 

namespace Web.Diagnostics.UnitTests.TripPages

{

    [TestFixture]

    public class HeaderMastheadLinks : TestSuiteBase

    {

        private ISelenium selenium;

        private StringBuilder verificationErrors;

 

        [SetUp]

        public void SetupTest()

        {

            this.BaseUrl = "http://www.w3schools.com/";

 

            //target the Selenium RC Server

            selenium = new TrackDefaultSelenium(new HttpCommandProcessor("localhost", 4444,

                "*chrome", this.BaseUrl), "Example Test Suite", "Example Unit Test", true);

 

            // test with an anonymous (not logged in, not recognized) user

            selenium.DeleteAllVisibleCookies();

 

            selenium.Start();

            verificationErrors = new StringBuilder();

        }

 

        [TearDown]

        public void TeardownTest()

        {

            try

            {

                selenium.Stop();

            }

            catch (Exception)

            {

                // Ignore errors if unable to close the browser

            }

            Assert.AreEqual("", verificationErrors.ToString());

        }

 

        [Test]

        public void Header()

        {

            try

            {

                selenium.Open("http://www.w3schools.com/");

                selenium.Click("link=Learn HTML5");

                selenium.WaitForPageToLoad("30000");

                Assert.AreEqual("HTML5 Tutorial", selenium.GetTitle());

 

                (selenium as TrackDefaultSelenium).UnitTestStatus =

                    TrackDefaultSelenium.TestStatus.Completed;

            }

            catch (Exception e)

            {

                (selenium as TrackDefaultSelenium).UnitTestStatus =

                    TrackDefaultSelenium.TestStatus.Failed;

                selenium.Stop();

                throw (e);

            }

        }

    }

}

Notice how the script matches that of one generated by the Firefox plugin. The only difference here is how we’re instantiating the Selenium object. Instead of using the DefaultSelenium object, we’re using our custom Selenium tracking object TrackDefaultSelenium.

Looking at the data captured from the test above, we get the following:

UnitTestResults

Partial Sitecore Cache Clearing by Item

Sometimes you just want to clear part of the Sitecore cache and not the entire thing. Taking a page out of the Staging module philosophy, you can easily clear cache by specific items.

First, make sure you understand the basics of Sitecore cache and how requests are handled.

The example method below, clears an enumerated list of items by GUID given a database context.


public static void ClearDataItemCache(Database database, IEnumerable ids)

{

    Cache prefetchCache = GetPrefetchCache(database);

    foreach (ID id in ids)

    {

        if (!ID.IsNullOrEmpty(id))

        {

            database.Caches.ItemCache.RemoveItem(id);

            database.Caches.DataCache.RemoveItemInformation(id);

            database.Caches.StandardValuesCache.RemoveKeysContaining(id.ToString());

            if (prefetchCache != null)

            {

                prefetchCache.Remove(id);

            }

        }

    }

}

Abandoning the ASP.NET User Session in Sitecore with OMS

Be careful when clearing out a user’s ASP.NET session, if you’re using Sitecore OMS. If programmatically clearing one’s session using code similar to one or all of the following:


Context.Session.Abandon();

HttpContext.Current.Request.Cookies.Remove("ASP.NET_SessionId");

Response.Cookies.Add(new HttpCookie("ASP.NET_SessionId", string.Empty));

You may notice that in your Sitecore logs, you’ll have entries that match:

    Violation of PRIMARY KEY constraint ‘PK_Session_1’. Cannot insert duplicate key in object ‘dbo.Session’.

The primary key violation is an indicator that you’ve cleared out the users ASP.NET session, but neglected to also clear the Analytics Session as well. Since both sessions are so tightly bound in the Analytics data (just take a look at the table schema for the table Sessions), breaking this relationship will cause loss of data.

All recording of page events via OMS produce errors for the given session after hitting the piece of code to abandon the ASP.NET session. You lose all Analytics tracking due to this error, which means any pages a user visits after abandoning the ASP.NET session will be lost.

The Fix

In addition to clearing the ASP.NET session, also clear the Analytics session:


HttpContext.Current.Request.Cookies.Remove("SC_ANALYTICS_SESSION_COOKIE");

Response.Cookies.Add(new HttpCookie("SC_ANALYTICS_SESSION_COOKIE", string.Empty));

Thanks to Sitecore Support for providing the analysis and recommended approach.

Running Remote Desktop Console from Command Line

I always forget the syntax for using Remote Desktop from the command line.

mstsc.exe /v:MyPCName /console /f

Troubleshooting SMTP Virtual Server Settings in Windows Server 2008

One of our development servers today needed to be configured to send emails from our web site to users and staff within the organization. We knew that the application was coded properly, as the email piece was functioning on other servers using the same code base.

Within Windows Server 2008, you must first add the IIS 6.0 Manager Components. We had this piece installed and configured using the IP of the web server and the default port of 25. Everything looked fine, but for some reason email wasn’t sending.

What I really needed was a way to troubleshoot the problem. I couldn’t step through the code, so I needed a way to determine what about the SMTP virtual server message delivery settings were incorrect. I then found a helpful KBA on troubleshooting via telnet.

By using telnet and following the steps in the KBA referenced above, I was able to see at which step during the email process the failure occurred. Making an EHLO command, the failure showed a domain different than what I was expecting. Instead of “oattravel.com”, I saw the name of the server. Going back into the SMTP virtual server properties, I quickly realized where the problem was.

The SMTP Advanced Delivery Options required domains for both:

  • Masquerade domain
  • Fully-qualified domain name

After changing the settings above, then retracing the steps to telnet back into the server and send an email, I then got a successful “Queued message for delivery” status.

Telnet test SMTP settings blur

One last note… make sure your SMTP Service is set to start up automatically. Ours wasn’t. For every IISRESET, the SMTP virtual server would stop and remain stopped until manually started again.

SMTP Service Automatic Startup

Set the SMTP Service to automatic startup

Why Can’t I Copy Assemblies from the GAC?

Today, I needed to obtain a copy of an assembly referenced in a web.config of one of our applications. The assembly referenced resembled the following:

<assemblies>

  <add assembly="Oracle.DataAccess, Version=10.1.0.200, Culture=neutral, PublicKeyToken=89B483F429C47342"/>

</assemblies>

I tried making a copy of an assembly from the GAC to my file system. Simple task, right? That’s what I thought until I right-clicked on a file only to find a few options, none of which had “copy”.

cannot copy file from explorer

After googling around, I discovered that I had a few options. I opted to copy from the command line:

copy assembly from the command line

By going into the GAC assembly directory from the command line and drilling deeper into the directory tree, you will eventually find the DLL you’re looking for.