I discovered an issue with the Resolve Sitecore Pipeline step in DEF 1.4.1 on Sitecore 8.2 where my Pipeline Batch process would occasionally return and attempt to update the wrong item. In this post, I will explore how that is possible and provide a simple solution to prevent it from occurring.
I ran into an issue with Sitecore’s Data Exchange Framework v1.4.1 where my pipeline batches would intermittently create thousands of duplicate Sitecore items. This caused a bit of management overhead having to clean out the duplicates. Following some investigation, I decided to add some defensive coding by introducing a Custom Resolve Sitecore Item Processor to replace Sitecore’s OOTB pipeline step and prevent duplicates from being created.
So in this short post, I’ll cover a few common mistakes that you might make when starting out configuring your DEF project, I know I certainly did and a fellow colleague hit the same issues on a recent project he just started. Hopefully, it will prevent you from falling into the same trap and banging your head against the wall.
Symptoms: You see the following error is logged when you run your pipeline batch and you have a Resolve Sitecore Item Pipeline Step. You will probably see something like this in your Sitecore logs:
[Date/Time] ERROR An item with the same key has already been added.
…
[Date/Time] ERROR Pipeline step processing will abort because a critical error occurred during processing.
Possible Cause: You see this occur during the Resolve Sitecore Item pipeline step processor. It usually means you have duplicate field name on the item you are trying to create in Sitecore. Check the template of the Sitecore item you are attempting to create including any inherited templates and resolve any duplicate fields names.
Symptoms: You have a pipeline batch with pipeline steps that read data from a source, iterates the items and create an item in Sitecore. The Item is created with the correct identifier, however, the fields on the Sitecore item are not populated.
Possible Cause: Most likely this is an issue with Data Location field has been set incorrectly on a pipeline step. Go through each step and check the location value.
Symptoms: You have a Resolve Sitecore Item Pipeline step followed by an Apply Mapping Pipeline step. Your new Item gets created however, all the fields are null. You see something similar to the following in the logs (Debug mode):
[Date/Time] DEBUG Value converted for search. (pipeline: Handle Single Row Pipeline, pipeline step: Resolve Sitecore Item Pipeline Step, field: UniqueId, original value: XXX-XXXX-XXXXX, converted value: XXX-XXXX-XXXXX)
[Date/Time] DEBUG Starting search for item. (pipeline: Handle Single Row Pipeline, pipeline step: Resolve Sitecore Item Pipeline Step, field: UniqueId, value: XXX-XXXX-XXXXX)
[Date/Time] DEBUG Item was not resolved. Will create it. (pipeline: Handle Single Row Pipeline, pipeline step: Resolve Sitecore Item Pipeline Step, identifier: XXX-XXXX-XXXXX)
2/27/2019 9:52:34 PM DEBUG New item was created. (pipeline: Handle Single Row Pipeline, pipeline step: Resolve Sitecore Item Pipeline Step, identifier: XXX-XXXX-XXXXX)
[Date/Time] ERROR Pipeline step processing will abort because the pipeline context has no target assigned. (pipeline: Handle Single Row Pipeline, pipeline step: Apply Mapping Pipeline Step, plugin: Sitecore.DataExchange.Plugins.SynchronizationSettings, property: Target)
[Date/Time] DEBUG Pipeline processor finished processing pipeline steps. (pipeline: Handle Single Row Pipeline, completed pipeline steps: 3)
Possible Cause: Check the Resolve Sitecore Item Pipeline Step and look the Object Resolution section ensure the Resolved Object Location is set: Pipeline Context Target and not Pipeline Context Source. If that is set correctly then you might want to check you the Converter and Processor Types correctly:
If you are using a custom Processor & Converter ensure you have set the SupportIds attribute on your Converter class so that it matches your custom template.
Hope this helps!
With the General Data Protection Regulation (GDPR) about to go into effect on May 25th, you should be aware of what Personally Identifiable Information (PII) could be stored in Sitecore 9 xConnect.
These fields are easily identifiable if you have installed Sitecore Data Exchange Framework 2.0+ and the xConnect Provider (provides the ability to read and write data to the collection and reference data services of Sitecore xConnect).
When transferring data between systems while some data will map directly you inevitably will come across data objects that need be transformed from the source format into a format that is compatible with the target object. Sitecore Data Exchange framework provides a mechanism and flexibility to reuse to Transform source data objects into the required format for the target data object.
A Value Reader allows you to read a value from a Source object. While it can read the value as is, they can also be used to transform the value. With each version of DEF Sitecore has provided a number Value Readers OOTB.
With DEF version 1.4.1 you have access to the following values readers:
With Sitecore 9.0.1 having just been released I was pretty excited to have a look at some of the improvements covered in the release notes. Not too mention a newer version of Sitecore Data Exchange Framework 2.0.1 and the OOTB providers.
Having a Sitecore 9 instance already installed and all its prerequisites means installing Sitecore 9.0.1 quick and easy to get up and running.
Update-Module SitecoreInstallFramework
Package = "$PSScriptRoot\Sitecore 9.0.1 rev. 171219 (OnPrem)_xp0xconnect.scwdp.zip" Package = "$PSScriptRoot\Sitecore 9.0.1 rev. 171219 (OnPrem)_single.scwdp.zip"
Don’t forget to run the applicable Post Installation Steps covered in Chapter 6.
Now, where did I put that DEF 2.0.1 package download!!
If you are familiar with Data Exchange Framework you will know that apart from being able to run a number of pipelines, a Pipeline Batch can also provide summary information about the last time the batch was run:
Prior DEF 2.0 the log messages were written to the Sitecore log file (by default) and written to the Messages field. You could also control the maximum number of messages that were written and displayed in the messages field.
With Sitecore 9.0 and the introduction xConnect you might be wondering what you are going to do with all that analytics data sitting in MongoDB, when you upgrade your Sitecore 8.x instance to Sitecore 9. Well, Sitecore have created a tool called the xDB migration tool. The tool uses the Sitecore Data Exchange Framework 2.0 and it provides the ability to read from MongoDB (source system) and write to the xConnect server (target system). If you are not familiar with the Data Exchange Framework I have covered it in an earlier post.
The xDB migration Tool provides a pre-configured mapping of data from the MongoDB database to the xConnect server in the form of a DEF Tenant. The Tool includes two data migration processes.
I recently presented at Atlanta Sitecore Usergroup and gave a demo using Data Exchange Framework (DEF) 2.0 and the SQL Provider – to read customer records from a SQL table and create items in Sitecore. You can read more about the DEF 2.0 here. In this follow-up post, I’ll walk through the DEF configuration required for the demo.
Eric Stafford (aka Sitecore Superman) and myself had the opportunity to attend the Integration Bootcamp at Symposium. The day was divided into 2 sections: xConnect in the morning with Todd Mitchell (xConnect Product Manager) and Niels Kuhnel (Sitecore Architect) in the afternoon Adam Conn (Director of Product Management) did a deep dive into the Data Exchange Framework. In this post will I take you through some of my takeaways on Data Exchange Framework which I shared recently with the Atlanta Sitecore User Group.
First, we need to clarify one thing and that’s how Data Exchange Framework is abbreviated. There appears to be some confusion in the community and that’s because Sitecore has a fondness for the letter x.
Adam Conn was insistent that if we left the boot camp knowing one thing – it should be: “the Data Exchange Framework is DEF, not DXF!! And that referring to is as DXF is like Sitecore with a capital ‘C'”.
Enough said!!
Neil Killen 