I am using Azure Data Factory v2. It is bound to an Azure DevOps GIT repository. I made some changes in my branch, including deleting some obsolete items. Then I merged my changes back to the ‘master’ branch.
Now, I’m trying to publish my changes from the ‘master’ branch to the Azure Data Factory. When I do, it says it’s deployed 33 of 33 changes, and then it fails, saying:
The document cannot be deleted since it is referenced by <some Pipeline>.”
I searched high and low looking for some evidence that the offending Pipeline existed anywhere in the GIT repo. It did not.
Then, I discovered that you can change from the Azure DevOps GIT version of the Data Factory to the actual Data Factory version by selecting the latter from the dropdown in the top left corner of the Data Factory editor:
This saved the day. I was able to locate and delete the offending Pipeline(s) directly from the actual Data Factory. Then, when I switched back to Azure DevOps GIT mode, it allowed me to publish the changes from the ‘master’ branch. Woohoo!
6 thoughts on “Azure Data Factory Publishing Error”
We had the same issue and we found that in de Data factory environment there where still pipelines we created before switching to CI/CD and these where causing the issue. After deleting these pipelines is was indeed solved.
I have the same problem but your solution does not work in my case. I want to delete an Azure Databricks Linked Service. It says it is referenced by pipeline X. I can see this pipeline when I switch to the Data Factory mode (= not Git linked). I delete the unnecessary pipeline, but I can not save the changes (no Save all button visible) and I can not Publish either (Publish button visible but it says “you have git enabled, publish from Data Factory mode is not possible”). So my only option in this mode is to Discard all changes. After which I am back to square one.
I got a reply from Microsoft support regarding the issue I had:
“You have 2 options :
– You can use the powershell commandlet to delete the pipeline from the factory (https://docs.microsoft.com/en-us/powershell/module/azurerm.datafactories/Remove-AzureRmDataFactoryPipeline?view=azurermps-6.13.0)
– Or you can create a new temporary pipeline in GIT, with the same name as in the Data Factory mode. This pipeline shouldn’t have reference on the linked service. Publish the pipeline, this will overwrite the existing definition of the pipeline in the factory. Now you will be allowed to delete the linked service. Finally you can delete the temporary pipeline you created.”
I used the second option, which worked great.
That’s awesome. Thanks!
I had the same problem, Git-linking prevented the publish of deleting the “bad” pipeline. After deleting the pipeline with PowerShell (I used Az Module, https://docs.microsoft.com/en-us/powershell/module/az.datafactory/remove-azdatafactoryv2pipeline) the publish worked smoothly. Thank you guys for sharing this, it helped a lot!
Happy to say that with a few additional steps this resolved my issue, much like you was running into these errors. Sadly my first attempt to fix things was a painful process of deleting and recreating things but to no avail.
What I then found was switching to Live mode was that all these services/previous versions of things that came up in the error were present.
As mentioned above you cannot actually delete as suggested when attached to Git or DevOps as its blocked. However what worked for me was to disconnect Git make the changes and publish them in the Live environment and then reenable Git.
I think the mistake is clicking “Import files to repository” when migrating your ADF instance, it gives you a little warning saying its a bad idea to do this on a populated branch, and I think this is the symptom. Perhaps with a few things containing the same name it creates a lot of invalid references.
Alternatively disconnect from Git, make sure your Live environment has the necessary elements (check this thoroughly first time around I found out my Live environment had 1/10 of the existing development effort). Once Live looks correct and everything is as expected with no extra files, Create a new git branch and “import files” to that branch.