Neil Killen

So I’ve been involved in several projects where I’ve had to setup geo-distributed Sitecore environments and/or scale the Sitecore Content Management and Content Delivery roles along with the plethora of roles XP Scaled provides. I’ll share some of those experiences here and some hurdles and how to overcome them.

Geo-Distributed Content Delivery

You can configure one or more content delivery servers for improved scalability and better performance. If you expect to have high numbers of visitors or want to configure servers in different geographic locations, ideally nearest to your customers.

In order to reduce latency you will want your web database in the same region as your CD. If you have multiple clusters in different regions you will need to consider how you plan to keep these web dbs up-to-date. Normally you would implement multiple publish targets, however Sitecore also supports geo-replication of the web database. See my post on Publish Targets vs Azure Geo replication for more details.

What other services and or roles do you need to support Content Delivery and where should these be located? More on this later…

Geo-Distributed Content Management

You cannot talk Geo-distributed Content Management without considering the master database. The Content management role and the Master database are intrinsically linked and there is a lot of interaction between these two entities. When looking at scaling the master database, Sitecore makes the following recommendations:

• A dedicated server for the Master database or a server shared with the Core database.
• The Content Management server(s) and the Master database should be physically close or in the same sub-region in Azure if possible.
• The Content Management server(s) and the Master database should be physically close to the largest group of authors.
• Sitecore does not support replication of the Master database. You’ll run into all sorts of issues if you try this concurrency, indexing etc – biggest thing though you’re in an unsupported state.

While it is possible to have a Content Management role in a separate region to the master database I strongly oppose this design. As we discovered, while Sitecore will function, the latency introduce between CM and master database causes significant performance issues. Trust me, I have tried this on Sitecore 9.3 and we noticed significant latency while trying to install OOTB Sitecore packages.

As you can see the latency introduced attempting to create Sitecore items was somewhere in the magnitude of 50x slower compared to having the CM and Master database co-located. We expected there would be some latency, but I don’t think anyone expected it would be this severe. DON’T DO IT!

Scenario’s where you might need to scale CM.

1. Number of Content Authors – with a high number of concurrent Authors causing poor performance. Adding an additional CM will allow you to distribute load between multiple CM instances.
   • CM must live in the same region as the Master DB.
2. Reduce load/processing on CM instance – if you have a lot of additional processing
   • Use the Indexing Sub Role – add an additional CM but its main purpose is Indexing.
   • Any heavy Processing Tasks that require compute can be configured to run only on the additional CM, Agents, DEF etc.
   • Consider utilizing the Dedicated Publishing Service https://dev.sitecore.net/Downloads/Sitecore_Publishing_Service.aspx
   • All CMs will live in the same region as Master DB.
3. Multiple Sites with different Authors could potentially be split into multiple separate Instances:
   • Each CM could be located in different regions.
   • Each instance would have its own separate Master DB.
   • CM could be located nearest the Authors for each Site.

What if my Content Authors are distributed geographically?

This is a common scenario and isn’t easily solvable. Following Sitecore recommendations Content Management must be near to the content authors and the Content Management role must reside is the same region or as close to the master database. So what are your options?

As mentioned previously if the Sitecore implementation has multiple sites with content authors for the various sites located in different regions you could split into multiple separate instances. Each site would have a single content management role each their own separate master database.

Alternatively you could introduce a neutral region. While this breaks the recommendation of having the content management instance close to the content authors it does not favor one region over another. The neutral region would contain the master database and all other shared Roles and resources. Here is an example of a XP Scaled Sitecore 9.3 geo-distributed environment with a neutral region that is running very successfully in Production for a client. 

While a solution like this has the potential to add latency between the Content Authors and Content management who are located in Region A and B. This latency is balanced and is less noticeable than the latency you would introduce by having the CM instance in each region to the Master database. Also there are solutions you can introduce to improve the performance of the CM like edge caching, if it becomes problematic.

Here are some question I had running through my head when designing this infrastructure along with my colleagues and was able to get clarification from documentation and Sitecore Support:

Q. Where should the xConnect Collection Service reside in each Region where the Content Delivery Roles reside or in the neutral region only?
“Deploy the xConnect Collection service as close as possible to the xDB Collection database. It is more important to reduce latency between the service and the database than between the service and clients such as Content Delivery.” (see scaling and configuring the xconnect collection search role).

Q. Cortex Processing Engine is it best to put it in the Neutral region?
“Multiple instances of the Processing Engine can process tasks in parallel. All instances must connect to one centrally located Processing Tasks database and one centrally located Processing Storage database.” (see scaling and configuring the sitecore cortex processing engine role).

Q. The Reference Service needs to reside in each region should the Reference Data Database also reside in each region and can it be Geo-replicated?
“Deploy the Reference Data service as close as possible to the Content Delivery (CD) environment because the CD environment generates the most load on the application.” (see scaling and configuring the reference data service role).

Support: “If you have separate Reference data for each service, you will not get the whole analytics data together. We do not recommend separate instance and Replication, it may work, but it was not officially tested.”

Other Considerations

• Indexing – with multiple CMs you need to make one CM instance your dedicated Indexing role.
• Dedicated Publishing Service – will reduce load on your CM instance and significantly improve the speed of publishing.
• Images – if images are stored on the filesystem they need to be kept in sync somehow. Maybe consider Blob storage supported OOTB 9.3.
• Optimization – follow Sitecore recommended best practices for tuning: indexes, caches, etc
• Solr/Azure search – if you’ve have multiple CD clusters supporting the different geographic locations to reduce latency between your CD and Azure Search instance you should consider an Azure search instance in each region.
• Redis – hmm this is a whole topic by itself maybe in a future post.

Conclusion

• Is it possible to setup a geo distributed Sitecore implementation? Yes
• Are there things you need to be aware of when considering a geo-distributed Sitecore infrastructure? Yes
• Is it straightforward? No Not really and no two Sitecore implementations are ever the same.
• But depending on the circumstances you DO have different options available.
• Engage Sitecore in the design process.
• Share your experiences with the community!
• Test, Test & Test 

Useful links

• Content Delivery Role
• Content Management Role
• Scaling and configuring Master Database
• Support for SQL Scaling Features
• Configure multiple Content Management instances
• Scaling and configuring Content Management
• Sitecore configurations and topology for Azure
• Sitecore Deployment Options

Hope this was useful!