Azure Data Factory Publishing Error

I am using Azure Data Factory v2. It is bound to an Azure DevOps GIT repository. I made some changes in my branch, including deleting some obsolete items. Then I merged my changes back to the ‘master’ branch.

Now, I’m trying to publish my changes from the ‘master’ branch to the Azure Data Factory. When I do, it says it’s deployed 33 of 33 changes, and then it fails, saying:

“Publishing Error

The document cannot be deleted since it is referenced by <some Pipeline>.”

I searched high and low looking for some evidence that the offending Pipeline existed anywhere in the GIT repo. It did not.

Then, I discovered that you can change from the Azure DevOps GIT version of the Data Factory to the actual Data Factory version by selecting the latter from the dropdown in the top left corner of the Data Factory editor:

This saved the day. I was able to locate and delete the offending Pipeline(s) directly from the actual Data Factory. Then, when I switched back to Azure DevOps GIT mode, it allowed me to publish the changes from the ‘master’ branch. Woohoo!

6 thoughts on “Azure Data Factory Publishing Error

  1. We had the same issue and we found that in de Data factory environment there where still pipelines we created before switching to CI/CD and these where causing the issue. After deleting these pipelines is was indeed solved.

  2. I have the same problem but your solution does not work in my case. I want to delete an Azure Databricks Linked Service. It says it is referenced by pipeline X. I can see this pipeline when I switch to the Data Factory mode (= not Git linked). I delete the unnecessary pipeline, but I can not save the changes (no Save all button visible) and I can not Publish either (Publish button visible but it says “you have git enabled, publish from Data Factory mode is not possible”). So my only option in this mode is to Discard all changes. After which I am back to square one.

  3. Happy to say that with a few additional steps this resolved my issue, much like you was running into these errors. Sadly my first attempt to fix things was a painful process of deleting and recreating things but to no avail.

    What I then found was switching to Live mode was that all these services/previous versions of things that came up in the error were present.

    As mentioned above you cannot actually delete as suggested when attached to Git or DevOps as its blocked. However what worked for me was to disconnect Git make the changes and publish them in the Live environment and then reenable Git.

    I think the mistake is clicking “Import files to repository” when migrating your ADF instance, it gives you a little warning saying its a bad idea to do this on a populated branch, and I think this is the symptom. Perhaps with a few things containing the same name it creates a lot of invalid references.

    Alternatively disconnect from Git, make sure your Live environment has the necessary elements (check this thoroughly first time around I found out my Live environment had 1/10 of the existing development effort). Once Live looks correct and everything is as expected with no extra files, Create a new git branch and “import files” to that branch.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.