Handy JMeter Script to crawl your site

JMeter is my tool of choice for load testing. This isn’t necessarily a load test but rather a useful script to crawl site urls using a sitemap. By hitting every accessible url allows Sitecore to populate all the various caches and provides you with a sense of the potential sizes of the various Sitecore caches and helps you identify any tuning that may be required.

Quick overview of the script

This post assume you are already familiar with JMeter. If not and you would like to get started may I suggest checking out the following posts:

User Defined Varibles – this contains several defined variables please review and configure accordingly for your own site:

  • ConcurrentUsers – number of concurrent users/threads (default 1).
  • RampUp – time taken to ramp up Concurrent Users (default 1).
  • NumberOfExecutions – number of times to repeat test for each users/thread (default 1).
  • Domain – specify the domain of your site.
  • SitemapUrl – set this to your sitemap url this will be appended onto the end of the domain variable.
  • SitemapUrlRegEx – this is regular expression to extract urls from the sitemap. If your sitemap follows the XML schema for the Sitemap protocol you should not need to change this.
  • DateTime – datetime stamp format included in the name of the exported results file.
  • ResultsPath – path to store summary report results file.

Traverse Sitemap Thread Group – the main and only thread group which configures the following parameters:

  • Set the number of threads
  • Set the ramp-up period
  • Set the number of times to execute the test

Get Sitemap HTTP Request – using the SitemapUrl makes a request to get your sitemap.

Get Site Urls Xpath Extractor – this post processor uses the SitemapUrlRegEx to extract your site URL’s from the sitemap and populates the SiteUrls array.

Traverse Site Urls Controller – this for each logic controller takes the SiteUrls array as an input variable and populates the SiteUrl output variable.

${SiteUrl} HTTPRequest – makes a request to the SiteUrl variable.

The script is available on GitHub and was created using JMeter v5.3.

Happy Site Crawling!

Leave a comment