I am currently working on a Sitecore project that uses rules to apply personalisation based on a GeoIP lookup using the MaxMind Lite Database, utilising the Rule Engine like this is one of the use cases that makes Sitecore so powerful, unfortunately, I was running into strange behaviour that was causing me to pull my hair out.

The origin of the issue was; "an incorrect DataSource applied for traffic that was originating in Australia". At first, it seemed to be related to a design choice that Sitecore had made when looking up GeoIp data, something that has been around since the days of DMS. See this KB for more information. So we applied this patch and felt that was it, however, the incorrect DataSource issue remained. To try and work out what was going on exactly I created a custom condition, to allow for a debugging breakpoint. After setting up the required items and deploying the assemblies, I loaded the website and observed no break on the first request. However, subsequent requests within the session the breakpoint would be reached, and I started to see a pattern to appear. "Even though the WhoIs information was correctly set the personalisation would never happen on that first request."

Looking back up through the Sitecore pipelines (with a little help from DotPeek), I concluded that the issue was the Tracker had not been initialised, and this was causing the mvc.customizeRendering pipeline to halt, in the Sitecore.Mvc.Analytics.Pipelines.Response.CustomizeRendering.Personalize processor.

public override void Process(CustomizeRenderingArgs args)
{
  Assert.ArgumentNotNull((object) args, nameof (args));
  if (args.IsCustomized || !Tracker.IsActive)
    return;
  this.Evaluate(args);
}

I confirm this by setting up DotPeek to be a symbol server and setting a breakpoint in Sitecores code directly, see this on how to allow it, and observed that the IsActive was false. I then turned to the Log files just to see a bunch of errors relating to establishing a connection to xConnect.  I followed this blog article from Kam Figy and double checked everything, nothing seemed to work. I supected that it boiled boil down to an issue with the Client Certificate and its thumbprint set in the ConnectionStrings.config.

I want to expand a bit on what Kam had put together before moving on to how this issue was solved. His blog article assumes some knowledge of tools that I did not have before this week.

The certlm  application, this is a Microsoft Management Console Snap-in. I have looked at this in the past but never took the time to actually understand what it is exactly. I fired up with a run command to certmgr.msc, the strange thing was that I did not see the certs I was interested in, and it turned out that it was defaulting (for me) to the Current User location. This location was incorrect and what I wanted was to look at the Local Computer location. To get this, I needed to run the command mmc and add the Snap-in my self

As a side note, you can also use PowerShell to view the install certs and their ThumbPrints
Get-ChildItem -path cert:LocalMachine
this will list all the Cert groups on the LocalMachine


Get-ChildItem -path cert:LocalMachine/My
this will list all the certs in the My group

After verifying that the thumbprints were the same I doubled checked IIS to see what cert was actually in use on the binding, this was also correct. At this point, I had far less hair than when I started down this rabbit hole, what was going on? The only option I had was going back to DotPeek. I strongly believe no matter how good documentation is available or how great a blog article is, nothing compares to being able to work things out from the source code.

Back neck deep in the source code and log files I decided to take a look at the Sitecore.Xdb.Common.Web.CertificateWebRequestHandlerModifier.Process(HttpClientHandler handler) method as this call was all through the exceptions in the log files. The line that jumped out ate me was

X509Certificate clientCertificate = CertificateWebRequestHandlerModifier.FindClientCertificate(this.StoreName, this.StoreLocation, this.FindType, this.FindValue, this.AllowInvalidClientCertificates);

When this returned null an exception was thrown with a message of "The certificate was not found." this was it, I placed a breakpoint at this line and attached to the w3c process.

  • The breakpoint was hit
  • I moved my mouse to see what the thumbprint was
  • Wait, What, how is that not the same?

Somehow an old thumbprint was still being used, was from a Cert that was updated a few months back. I did a find all for this string and found it in the Web.Config, for whatever reason connection string element was directly in the Web.Config and not referencing out to the ConnectionStrings.config file as I had assumed. I updated the Thumbprints there, redeployed and the errors (that had accelerated my baldness) had disappeared.

So the moral of the story if it seems like you are missing something simple you probably are, and look at the code. The code won't lie to you, it might be convoluted and hard to understand, but it tells the truth. Oh don't assume anything, aussieviking always told me this, I should remember that more often.

The log entries I was seeing:

FATAL [Experience Analytics]: Failed to synchronize segments. Message: The certificate was not found.. Details:    at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.GetAll(NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.GetAll(NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsManager.GetSegmentsToSynchronize()
   at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsManager.SynchronizeAllSegments()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsProcessor.<Process>d__4.MoveNext()
ERROR Exception when executing agent aggregation/aggregator
Exception: Sitecore.XConnect.XdbCollectionUnavailableException
Message: An error occurred while sending the request.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock(Func`1 taskFactory)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.Initialize(XmlNode configNode)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert, IFactoryHelper helper)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert)
   at Sitecore.Configuration.DefaultFactory.CreateObject(String configPath, String[] parameters, Boolean assert)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient(String clientConfigPath)
   at Sitecore.Analytics.Processing.AsyncPoolScheduler`2..ctor(IAsyncProcessingPool`1 pool, IXdbContextFactory xdbContextFactory, ExpandOptions options, Int16 maxBatchSize)
   at Sitecore.Analytics.Aggregation.InteractionAggregationAgent.<ExecuteCoreAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Analytics.Core.Agent.<ExecuteAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Analytics.Core.AsyncBackgroundService.<ExecuteAgentAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Analytics.Core.AsyncBackgroundService.<RunAsync>d__26.MoveNext()

Nested Exception

Exception: System.InvalidOperationException
Message: The certificate was not found.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.CertificateWebRequestHandlerModifier.Process(HttpClientHandler handler)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateRequestHandler()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateClient()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteAsync>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteGetAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.<Refresh>d__4.MoveNext()
ERROR Exception when executing agent aggregation/pathAnalyzerLiveAgent
Exception: Sitecore.XConnect.XdbCollectionUnavailableException
Message: An error occurred while sending the request.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock(Func`1 taskFactory)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.Initialize(XmlNode configNode)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert, IFactoryHelper helper)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert)
   at Sitecore.Configuration.DefaultFactory.CreateObject(String configPath, String[] parameters, Boolean assert)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient(String clientConfigPath)
   at Sitecore.PathAnalyzer.Processing.Agents.TreeAggregatorAgent.Execute()
   at Sitecore.Analytics.Core.BackgroundService.Run()

Nested Exception

Exception: System.InvalidOperationException
Message: The certificate was not found.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.CertificateWebRequestHandlerModifier.Process(HttpClientHandler handler)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateRequestHandler()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateClient()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteAsync>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteGetAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.<Refresh>d__4.MoveNext()
ERROR Cannot start analytics Tracker
Exception: System.InvalidOperationException
Message: The certificate was not found.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.Analytics.DataAccess.Dictionaries.DataStorage.ReferenceDataClientDictionary.EnsureDefinitionType(String definitionTypeName)
   at Sitecore.Analytics.DataAccess.Dictionaries.DataStorage.ReferenceDataClientDictionary.LoadAs[T](Object key)
   at Sitecore.Analytics.DataAccess.Dictionaries.AverageCounterExtensions.MeasureMilliseconds[T](AverageCounter counter, Func`1 func)
   at Sitecore.Analytics.DataAccess.Dictionaries.ReferenceDataDictionary`2.Get(TKey key, LookupStrategy strategy)
   at Sitecore.Analytics.DataAccess.Dictionaries.UserAgentsDictionary.Register(String userAgentName)
   at Sitecore.Analytics.Tracking.CurrentVisitContext.set_UserAgent(String value)
   at Sitecore.Analytics.Pipelines.CreateVisits.InitializeWithRequestData.Process(CreateVisitArgs args)
   at (Object , Object )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Analytics.Pipelines.CreateVisits.CreateVisitPipeline.Run(CreateVisitArgs args)
   at Sitecore.Analytics.Tracking.StandardSession.CreateInteraction(HttpContextBase httpContext)
   at Sitecore.Analytics.Pipelines.InitializeTracker.CreateVisit.Process(InitializeTrackerArgs args)
   at (Object , Object )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Analytics.Pipelines.InitializeTracker.InitializeTrackerPipeline.Run(InitializeTrackerArgs args)
   at (Object , Object )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Analytics.Pipelines.StartTracking.StartTrackingPipeline.Run(StartTrackingArgs args)
   at Sitecore.Analytics.DefaultTracker.StartTracking()

Exception: Sitecore.XConnect.XdbCollectionUnavailableException
Message: An error occurred while sending the request.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock(Func`1 taskFactory)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.Initialize(XmlNode configNode)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert, IFactoryHelper helper)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert)
   at Sitecore.Configuration.DefaultFactory.CreateObject(String configPath, String[] parameters, Boolean assert)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient(String clientConfigPath)
   at Sitecore.PathAnalyzer.Processing.Agents.TreeAggregatorAgent.Execute()
   at Sitecore.Analytics.Core.BackgroundService.Run()

Nested Exception

Exception: System.InvalidOperationException
Message: The certificate was not found.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.CertificateWebRequestHandlerModifier.Process(HttpClientHandler handler)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateRequestHandler()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateClient()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteAsync>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteGetAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.<Refresh>d__4.MoveNext()
ERROR Exception when executing agent aggregation/aggregator
Exception: Sitecore.XConnect.XdbCollectionUnavailableException
Message: An error occurred while sending the request.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock(Func`1 taskFactory)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.Initialize(XmlNode configNode)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert, IFactoryHelper helper)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert)
   at Sitecore.Configuration.DefaultFactory.CreateObject(String configPath, String[] parameters, Boolean assert)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient(String clientConfigPath)
   at Sitecore.Analytics.Processing.AsyncPoolScheduler`2..ctor(IAsyncProcessingPool`1 pool, IXdbContextFactory xdbContextFactory, ExpandOptions options, Int16 maxBatchSize)
   at Sitecore.Analytics.Aggregation.InteractionAggregationAgent.<ExecuteCoreAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Analytics.Core.Agent.<ExecuteAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Analytics.Core.AsyncBackgroundService.<ExecuteAgentAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Analytics.Core.AsyncBackgroundService.<RunAsync>d__26.MoveNext()

Nested Exception

Exception: System.InvalidOperationException
Message: The certificate was not found.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.CertificateWebRequestHandlerModifier.Process(HttpClientHandler handler)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateRequestHandler()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateClient()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteAsync>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteGetAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.<Refresh>d__4.MoveNext()
ERROR Exception when executing agent aggregation/pathAnalyzerHistoryAgent
Exception: Sitecore.XConnect.XdbCollectionUnavailableException
Message: An error occurred while sending the request.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock(Func`1 taskFactory)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.Initialize(XmlNode configNode)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert, IFactoryHelper helper)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert)
   at Sitecore.Configuration.DefaultFactory.CreateObject(String configPath, String[] parameters, Boolean assert)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient(String clientConfigPath)
   at Sitecore.PathAnalyzer.Processing.Agents.TreeAggregatorAgent.Execute()
   at Sitecore.Analytics.Core.BackgroundService.Run()

Nested Exception

Exception: System.InvalidOperationException
Message: The certificate was not found.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.CertificateWebRequestHandlerModifier.Process(HttpClientHandler handler)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateRequestHandler()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateClient()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteAsync>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteGetAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.<Refresh>d__4.MoveNext()