Forum Discussion
Solved
How to save Azure Data Factory work (objects)?
Hi,
I'm new to Azure Data Factory (ADF). I need to learn it in order to ingest external third-party data into our domain. I shall be using ADF Pipelines to retrieve the data and then load it into an Azure SQL Server database.
I currently develop Power BI reports and write SQL scripts to feed the Power BI reporting. These reports and scripts are saved in a backed-up drive - so if anything disappears, I can always use the back-ups to install the work.
The target SQL database scripts, the tables the ADF Pipelines will load to, will be backed-up following the same method.
How do I save the ADF Pipelines work and any other ADF objects that I may create (I don't know what exactly will be created as I'm yet to develop anything in ADF)?
I've read about this CI/CD process but I don't think it's applicable to me. We are not using multiple environments (i.e. Dev, Test, UAT, Prod). I am using a Production environment only. Each data source that needs to be imported will have it's own Pipeline, so breaking a Pipeline should not affect other Pipelines and that's why I feel a single environment is suffice. I am the only Developer working within the ADF and so I have no need to be able to collaborate with peers and promote joint coding ventures.
Does the ADF back-up it's Data Factories by default? If they do, can I trust that should our instance of ADF be deleted then I can retrieve the backed-up version to reinstall or roll-back?
Is the a process/software which saves the ADF objects so I can reinstall them if I need to (by the way, I'm not sure how to reinstall them so I'll have to learn that)?
Thanks.
Hello AzureNewbie1,
I do not know the internals of how Azure performs backups and disaster recovery for ADF in the event of an issue with Azure. However, if you want to take manual backups of your ADF environment to put your mind at ease, the easiest way I can think of is to export it as an ARM template (JSON file). See the image below on what I am referring to. Click on export and save this file in a location that you are comfortable with. To import, refer to the following document (if you only have one environment, be careful not to unintentionally overwrite it):
The export process could be done manually or you should be able to automate it by using Azure DevOps Pipelines to schedule a PowerShell or Azure CLI script.
Kind Regards,
Zack
2 Replies
Hello AzureNewbie1,
I do not know the internals of how Azure performs backups and disaster recovery for ADF in the event of an issue with Azure. However, if you want to take manual backups of your ADF environment to put your mind at ease, the easiest way I can think of is to export it as an ARM template (JSON file). See the image below on what I am referring to. Click on export and save this file in a location that you are comfortable with. To import, refer to the following document (if you only have one environment, be careful not to unintentionally overwrite it):
The export process could be done manually or you should be able to automate it by using Azure DevOps Pipelines to schedule a PowerShell or Azure CLI script.
Kind Regards,
Zack- zack-sponaugle - thank you.
