Large-scale Data Analytics with Azure Synapse - Workspaces with CLI
One of the challenges of large scale data analysis is being able to get the value from data with least effort. Doing that often involves multiple stages: provisioning infrastructure, accessing or moving data, transforming or filtering data, analyzing and learning from data, automating the data pipelines, connecting with other services that provide input or consume the output data, and more. There are quite a few tools available to solve these questions, but it's usually difficult to have them all in one place and easily connected. This article will cover what Azure Synapse is and how to start using it with Azure CLI.
ADF\Synapse Analytics - Replace Columns names using Rule based mapping in Mapping data flows
In real time, the column names from source might not be uniform, some columns will have a space in it, some other columns will not. For example, Sales Channel Item Type, Region Country Unit Price It is a good practice to replace all the spaces in a column name before doing any transformation for easy handling. This also helps with auto mapping, when the sink column names do not come with spaces! Select transformation in data flow makes it simpler to automatically detect spaces in column names and then remove them for the rest of the dataflow. Consider the below source, with the given column names. Here as we can see, few columns have spaces, few columns like Region and Country do not have spaces in it. Using the below configuration in select transformation, we can get rid of the spaces in the column names with a simple expression. In the Input columns, Click on Add mapping button, and choose Rule-based Mapping. Then give the below expression: on Source1's column: true() on Name as column: replace($$,' ','') What it does? It will return true() for all the columns which have ' ' (space) in it and replace it with '' (no space). Upon data preview, we get to see the below result, As we are seeing here, all the columns with spaces are coming without spaces in between. If not for the Rule based mapping, one would have to manually remove space from all the columns. It would be a nightmare if the number of columns are more! Thanks to rule-based mapping!
