If you work with Azure Data Factory, you probably noticed that the deployment process for this tool is unique. It is different from an API or website deployment as it generates ARM Templates and certainly at some point in time questioned “Why is so difficult to deploy it?”. Well, it doesn’t have to be…

Azure Data Factory is a great orchestration tool for the Big Data process. It has many integrations and capabilities that make the Data Engineer life very easy.

Although when we talk about deployments there are some tricks, for example, the publish button inside the workspace, that is necessary to generate the ARM Templates to be deployed. Also, there is no deployment into the development environment, as you are already working there as an editor tool. This process’s a kind different comparing to other “traditional developments”, and this manual step sends shivers down DevOps engineers spines:

Deployment from Azure Data Factory Workspace — Image from Microsoft Docs

It often creates discussions and “workarounds” to deal with this manual step. In some solutions, people move the generate ARM Templates from the adf_publish manually to the feature branch and create a new commit. In others, there is an integration using more than one repo, but again, start the deployment should be – again – a manual step.

On the older approach, the best solution from my point of view is to deploy using Azure Powershell, but it creates additional complexity as you need to control manually all elements that should be deployed, like linked services and pipelines.

Fortunately, there is a new way to generate the ARM templates to be deployed, and you don’t need the workarounds mentioned above.

In this strategy, you basically validate and “build” your Data Factory JSON files into ARM Templates ready to be used. To make it work, you need to create a build pipeline and consume the ADFUtilities NPM package. This is much cleaner and the manual steps disappear :).

Deployment with new Azure Data Factory flow — Image from Microsoft Docs

For more information, check the official documentation here.

Continue reading: https://towardsdatascience.com/azure-data-factory-ci-cd-made-simple-building-and-deploying-your-arm-templates-with-azure-devops-30c30595afa5?source=rss—-7f60cf5620c9—4

Source: towardsdatascience.com