Let's face it, sometimes our implementation isn't perfect. Even with the most meticulously designed data layer, a completely up to date Web SDK implementation utilizing the latest data object logic and server-side forwarding and Datastreams, you're bound to occasionally see the dreaded Failed Record in your Adobe Experience Platform dataset. So what options do we have to find and fix these problems?

Discovering Records That Failed Experience Platform Ingestion Into A Dataset

So this article is specific to customers that are ingesting data into an Experience Platform Dataset. This means it'll be helpful for customers ingesting data for use by Customer Journey Analytics, Real-Time CDP, or Journey Optimizer. In my example, I'm ingesting data into a Dataset via the Web SDK. And evvvvery once in a while I see a Failed batch in the Dataset, like this:

If you click the links in the left, you'll often times get a little more information about why the record didn't pass the automated, built-in validation when data is ingested into Experience Platform. These errors are often due to formatting, type mismatches, etc. Here's some helpful documentation on types of Experience Platform Validation.

I've been trying to figure out this specific error lately, but it's difficult to debug without additional information. Here's a screenshot of the validation error:

Getting Additional Context On When Your Ingestion Validation Errors Are Happening

OK so sometimes the error is easy to debug and resolve. But this one in particular has been bugging me. I need additional context to understand when this is happening in the data collection process. I was told that I can easily access the full XDM object that failed by using the Data Access API for Adobe Experience Platform but API work is my least favorite kind of work. So I went digging into the UI. And guess what! I found something amazing:

Imagine what I can do with this information. I can see exactly what URL the user was on when this XDM record failed, and all of their details that are included in XDM: location, events, ECID, and more! Of course, I threw that ECID into Customer Journey Analytics, found the exact journey the user was on, and resolved the issue with a little bit of cleanup in my Launch rule.

Oh wait, you probably want to know how I got there? OK fine, I'll share my secret.

Click on Sources, then Dataflows:

Then, apply a Filter to Adobe Data Collection (or if you're not using the Web SDK / Mobile SDK, whatever Source you used instead)

Then, you'll want to scroll through the Dataflows in the listing to find the one associated with the Dataset that you care about and click it. You'll now see a similar list of Batches ingested into your Dataset with a list of any that have Records that Failed due to Validation:

Simply click any "Dataflow" that has a Failed Record included in it, and you'll see a page that looks very similar to what we saw earlier, but with one additional (amazing) button: Preview Error Diagnostics

AND THIS, my friends, is how I found the complete XDM object that failed without having to understand what a Bearer Access Token is!

Use this information wisely - for those of you using the Web SDK to ingest data for Customer Journey Analytics, Real-Time CDP, or Journey Optimizer, I'm confident you'll find it to be incredibly useful!

Go forth and debug those failed dataset records, you Rockstar you!