Leveraging Sitecore Data Exchange Framework to access your Data

Neil Killen

Eric Stafford (aka Sitecore Superman) and myself had the opportunity to attend the Integration Bootcamp at Symposium.  The day was divided into 2 sections: xConnect in the morning with Todd Mitchell (xConnect Product Manager) and Niels Kuhnel (Sitecore Architect) in the afternoon Adam Conn (Director of Product Management) did a deep dive into the Data Exchange Framework. In this post will I take you through some of my takeaways on Data Exchange Framework which I shared recently with the Atlanta Sitecore User Group.

What do we call it?

First, we need to clarify one thing and that’s how Data Exchange Framework is abbreviated. There appears to be some confusion in the community and that’s because Sitecore has a fondness for the letter x.

Adam Conn was insistent that if we left the boot camp knowing one thing – it should be: “the Data Exchange Framework is DEF, not DXF!! And that referring to is as DXF is like Sitecore with a capital ‘C'”.

Enough said!!

What is it?

  • A Framework Designed for transferring Data into and out of Sitecore.
  • It’s an ETL – Extract, Transform and Load the 3 steps of data integration.
    • Extract – you’ve got a source system you are extracting data from.
    • Transform – we change the data to be compatible with the target system.
    • Load – we load the data it into the target system.

DEF_ETL

  • It standardizes how we do data integrations with Sitecore.
  • It promotes re-usability by providing patterns and components that help you with your integration leaving you to focus on the other tasks associated with data integration.
  • It is highly configurable and extendable allowing you to integrate with almost any 3rd party system.
  • It enables Synchronization of Systems and processes.
  • It reduces the amount of effort required to integrate 3rd party systems.

Why Do we need it?

Every Sitecore implementation I’ve worked on has had an element Data Integration with 3rd party application.

What happens to our integration when a new version of Sitecore is released?

  • Is it going to work or will it require further development which inevitably would require full regression testing?

Who is going to able to support our custom integrations?

  • Are you always going to be the go-to person when there is an issue?
  • Or will it be built in such a way that anyone can easily pick it up and understand it and be able to resolve any issues?

What happens when requirements for the integration change?

  • Will the customer have to come back to the developer to make those changes?

There has never been a standardized way of integrating data with Sitecore and these are some of the questions we need to consider when building custom integrations.  The Data Exchange Framework helps answers these questions by providing a standard way of integrating with Sitecore. So when a new version of Sitecore is released you know your integration will be compatible or at least have an upgrade path. Following the standard set patterns and practices for implementing DEF makes supporting your integrations easier and transferable. Due to DEF implementation of being highly configurable when requirements change those changes could be, in most scenarios applied through modifications to the existing DEF configuration and not require developers to get involved.

DEF – Release Train

  • June 2016 – Version 1.0 released – For Sitecore 8.1 Update 1
    • Fully configurable synchronization processes and data mapping
  • August 2016 – Version 1.1 released
    • Improved logging and troubleshooting features
    • Improved performance and reliability for synchronization processes
    • Bug fixes
  • December 2016 – Version 1.2 released – For Sitecore 8.1 update 1 or later
    • Improved logging and troubleshooting features
    • Improved performance and control over synchronization processes
    • Bug fixes
    • Support for Sitecore on Azure
  • February 2017 – Version 1.3 released
    • Improved control over work queue processing
    • Improved feedback when stopping pipeline batch that is running
    • Added support for multiple languages when working with Sitecore items
    • Bug fixes
  • April 2017 – Version 1.4 released
    • Usability enhancements
    • Work queue enhancements
    • Value mapping enhancements
    • SDK for provider developers
    • Bug fixes
  • August 2017 – Version 1.4.1 released
    • Usability enhancements
    • Work queue enhancements
    • Value mapping enhancements
    • SDK for provider developers
    • Bug fixes
  • October 2017 – Version 2.0 released – For Sitecore 9
    • Performance and API improvements.
    • Additional Providers

What Providers are available OOTB?

With DEF 2.0 Sitecore has updated the existing providers so they are compatible and released some new providers:

These providers offer a great starting point to understand how the providers work and help you build your own providers if necessary.

What do you need to get going?

In order to get up and running with DEF, you will need to install it.  This is done through Sitecore packages.

  1. Download and install the framework
    • this can 5-10 minutes to install as there are a lot of Sitecore templates and items to be installed.
    • once installed if navigate in Content Editor to Sitecore > System you will notice a new node ‘Data Exchange’.  This is where you can create new Tenants and configure your data integrations.
    • a node is also been added to Sitecore > Settings > Data Exchange – this is where the various configuration options for the framework itself are located.
  2. Download and Install the providers
    • these are installed separately.
    • they are made available as separate packages and that makes it easier to make updates to individual providers.
    • you can install all the OOTB providers or only the ones you need.
  3. Get Busy with configuration
    • now that its installed you need to configure DEF so that it knows about your data integrations.
    • there is a lot to be configured and when you first dive into the various settings it can seem a bit overwhelming.

Framework Components

It is really helpful if you have a basic understanding of the various framework components that you’ll need to configure. The configuration can seem a little overwhelming initially, but once you have an understanding of these components then the need for all the configuration makes sense.

defconfigcomponents_st

  • Tenant – is used to organize and isolate settings so that multiple configurations can coexist.
  • Endpoints – Allow you to define the Source and Destination endpoints.
  • Data Objects – These are the objects you are going to read data from and to.  For example, if your target system is Sitecore then you need to create a Sitecore template item to put the data in.
  • Value Accessors and Accessor Sets – values accessors are like the property getters and setters in a C# class. They associate the component used to read a value with the component used to write a value.  A value Accessor Set is used to model an object that can serve as a source object or a target object.
  • Mappings – allow you to map source values with target values and apply rules and tasks for transforming those values if necessary.
  • Pipelines – are a collection of pipeline steps that run in a specific order and each step performs a specific task. Each pipeline has a pipeline processors associated with it. A pipeline processor implements the logic to run a pipeline. Pipelines are similar to Sitecore pipelines with processors and context.
  • Pipeline Context –  is used to pass information between the different pipeline step processors that are run when a pipeline is run. When a pipeline is started, a pipeline context is created. When a pipeline step processor is run, this context is passed to the processor. The same pipeline context is passed to each processor. This provides a way for data to be passed from one processor to another.
  • Pipeline Batch – a pipeline batch is like a unit of execution for a pipeline in the DEF. these can be executed manually, through a Sitecore scheduled task or remotely using a console app or Windows service using the Data Exchange Framework Remote SDK.

These components are covered in depth on the Data Exchange Framework Component reference documentation.

Demo

During the user group, I did a quick demo utilizing the SQL Provider – to read from the customer table in Microsoft’s example Northwind database and create items in Sitecore.

def_sqlcustomerdemo3

I’ll walk through the necessary configuration required to set up this integration and some other cool features  in a follow-up post.

Building Custom Providers

As well as the OOTB Providers Sitecore you can build your own custom providers.  Sitecore has provided some good resources to help you get started. However, all these resources are currently for DEF 1.4.1 and below as they have not yet released the documentation for DEF 2.0.  But they are still a valuable resource to help you get started understanding the principles and patterns required to create your own provider. So go ahead and check out the Developers Guide for 1.4.1 and the API Guide.

Sitecore also provide the Data Exchange Framework SDK to help you create your custom providers. Once you’ve developed your custom provider you then need to create all the templates and configuration items required for DEF to support and use your provider.  The SDK provides tools to helps you with this and does most of the mundane work for you to make it much easier.

There are some good examples of custom providers online I’d encourage you to check these ones out:

Before you dive into and start creating your own providers I’d urge you to look at the existing OOTB providers as you can extend these existing providers and perhaps reuse a lot of what Sitecore have already implemented.

Well, I hope this has provided some helpful pointers to get you up and running with DEF. I know I’m really excited what Sitecore is doing with it and can’t wait to start implementing it in future integration projects. Look out for my followup posts where I’ll walk you through utilizing the SQL provider, configuring the verification database and run through the setup of xDB to xConnect migration toolkit.