Sitecore Publishing Service in a geo-distributed environment

If you’ve never used the publishing service before it’s an awesome opt-in for Sitecore to provide blazingly fast publishing by performing bulk publish operations. If you are needing to improve performance over your existing Sitecore publishing or reduce load on your Content Management Role then you should consider looking into setting up Publish service available since Sitecore 8.2. In this post I’ll share some discoveries while setting up Publish Service v4.2 on Sitecore 9.3 on Azure PaaS. It was pretty straightforward and painless! It just works really well!

image

Geo-Distributed Environment

My scenario is fairly complex I was setting up a distributed environment with multiple regions containing Content Delivery Roles and a neutral region with all the shared resouces to support these regions with two Content Management roles in the neutral region. We also implemented Azure Geo replication of the web database to keep content delivery regions in sync including the shared web database.

Q. How is this going to impact Publish Service, is there anything I need to do to ensure it operates successfully?

“It shouldn’t really as the Publish service when is starts creates a manifest of what needs to be published and moves that data to the destination Target. With Geo-Replication you only have a single target – the Web database and once that gets updated SQL replication kicks in and updates the web dB in each region.

Setting it Up

I’m not going to go into Specifics here as there are some really good posts that go into more details.

    1. Create a new Azure Web App.
    2. Once the app is created, download the publishing service zip file from dev.sitecore.net , upload it to Azure, extract and configure the service.
    3. Select App Service Editor under development tools.
    4. Delete the hostingstart.html file and upload the publishing service zip file.
    5. Extract the files, go to the Azure console and type the following: unzip -o *.zip
    6. If you list the directory contents, you should see the contents of the zip file extracted.
    7. Configure the connection strings by run the following command: Sitecore.Framework.Publishing.Host.exe configuration setconnectionstring core “<your core db connection string>” – Repeat for master and web
    8. Upgrade the DB Schema by running the following command: Sitecore.Framework.Publishing.Host.exe schema upgrade -f
    9. Test the service: try to hit this url http://<service host or domain anme>/api/publishing/maintenance/status

The service should return status:0 in json response.

Once the service is installed and running follow the installation guide to install the Sitecore Publishing Service module on Content Management Roles and configure the service url to point to the Publish service instance. This setting is located in App_Config\Modules\PublishService\Sitecore.Publishing.Service.config.

For each Content Delivery Role you need to deploy some config and dlls Sitecore do. It provide an install package for this. You can create your own WDP and it should contain the following:

  • App_Config/Modules/PublishingService/Sitecore.Publishing.Service.Delivery.config
  • bin/Sitecore.Framework.Conditions.dll
  • bin/Sitecore.Publishing.Service.Abstraction.dll
  • Sitecore.Publishing.Service.Delivery.dll
  • Sitecore.Publishing.Service.dll

Disabling the Publish Service

When troubleshooting I found myself wanting to disable the Service so I could verify an operation worked OOTB in Sitecore 9.3 and it was something that had been introduced by the publish Service. Thankfully this is straightforward and details can be found in KB154093.

Related Media Items

We were experiencing an issue with related media items when you published an item the related media items were not being published. On further investigation and with the help of Sitecore support we discovered this was caused by the configure links database. Publish service is configured by default to use the core database {publishing service webroot}/config/sitecore/publishing/sc.publishing.xml.

The installation guide details information for a scaled scenario and mentions you should use the web database instead if you have a scaled environment. You can verify on the where the link database resides by checking ShowConfig.aspx on the CM and searching for link database.

To resolve the issue

  1. Go to the file {publishing service webroot}/config/sitecore/publishing/sc.publishing.xml
  2. Change the Links connectionstring to use “Web” instead of “Core”
  3. Restart the publishing service (this is required for new config changes to take effect) and check the status http://<service host or domain anme>/api/publishing/maintenance/status 
  4. Hopefully you should get a status:0 if not then you’ll probably get some warning message like: The specified CGI application encountered an error and the server terminated the process.  
  5. No need to panic this is normal! You’ll need to test the Publish Service from the console.
  6. From Azure Portal fire up Kudu and attempt to start the service in Development mode as this allows us to see verbose messages.
  7. To run in development mode we run the following command from the installation folder with the environment flag: Sitecore.Framework.Publishing.Host.exe –environment development   
  8. Follow any output message/instructions that appear to resolve the problem.rebuildschemamessage
  9. Once you have resovled the Publish service and it start successfully Rebuild the link database.
  10. Check publishing items and related media.

Remember in the same way you don’t update Sitecore configs directly the same is true for the publish Service you can override config setting in a custom config file. Making your upgrade path easier in the future. To override default configuration settings create your configuration file in the wwwroot/config/global directory and your filename should be prefixed with sc. for example sc.publishing.custom.xml.

Connection Timeout

In production we ran into an issue whereby the Publish Service would stop running this would manifest itself to content authors with the following error when attempting to publish:

Also the Publishing dashboard displays no records. Checking the publish service status and it would appear to be ok api/publishing/maintenance/status {0}. So those are the symptoms. Restarting the publish app-service instance would recover the ability to publish – until it occurred again. This is more of a workaround.

Performing root cause analysis we discovered the following errors in the publish Service logs:

2020-07-12 04:08:51.175 +01:00 [Information] Executing Cleanup Task : “PublishOperationCleanupTask-60832bb9509e4a37855285a8346a6a53”

2020-07-12 04:09:21.697 +01:00 [Error] There was an error adding 1 publisher operations. – Error : “Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.”

The clean up task is responsible for removing old data from the Publishing_ManifestStep and Publishing_PublisherOperation tables. These tables are created and used by the service in the master database during the install. On inspection of these two tables we could see they contained an excessive amount of older data that had not been cleared out.

To resolve the issue we had to increase the value of CommandTimeout the setting is located in sc.publishing.sqlazure.xml file. The default value is 2mins once the old records have been cleared out this value could potentially be set back to the default value.

Useful Links

Hope this was useful!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s