execute command activity stage data stage versions

Then, use Add Dynamic Content in the Data Flow activity properties. Use pipeline activities like Lookup or Get Metadata in order to find the size of the source dataset data. The Core Count and Compute Type properties can be set dynamically to adjust to the size of your incoming source data at runtime. Set logging level of your data flow activity executionĭynamically size data flow compute at runtime Only if the data flow reads or writes to Azure Synapse Analytics If you're using an Azure Synapse Analytics source or sink, the folder path in blob storage account used for PolyBase staging Only if the data flow reads or writes to an Azure Synapse Analytics Also learn the needed configurations for Azure Blob and Azure Data Lake Storage Gen2 respectively. If your Azure Storage is configured with VNet service endpoint, you must use managed identity authentication with 'allow trusted Microsoft service' enabled on storage account, refer to Impact of using VNet Service Endpoints with Azure storage. If you're using an Azure Synapse Analytics source or sink, specify the storage account used for PolyBase staging. Can only be specified if the auto-resolve Azure Integration runtime is used

The type of compute used in the spark cluster. The number of cores used in the spark cluster. If not specified, the auto-resolve Azure integration runtime will be used. The compute environment the data flow runs on. The reference to the Data Flow being executed 'referenceName': 'MyDataFlowIntegrationRuntime', 'referenceName': 'MyStagingLinkedService', Select other options as required to complete your configuration. Select an existing data flow or create a new one using the New button. Checkpoint key option after publishing a new data flow activity with change data capture enabled data flow resource is shown as below. All existing data flow activity will use the old pattern key for backward compatibility. Data flow activities use a guid value as checkpoint key instead of “pipelinename + activityname” so that it can always keep tracking customer’s change data capture state even there’s any renaming actions. Select the new Data Flow activity on the canvas if it is not already selected, and its Settings tab, to edit its details.Ĭheckpoint key is used to set the checkpoint when data flow is used for changed data capture. Search for Data Flow in the pipeline Activities pane, and drag a Data Flow activity to the pipeline canvas.

To use a Data Flow activity in a pipeline, complete the following steps:

If you're new to data flows, see Mapping Data Flow overview Create a Data Flow activity with UI Use the Data Flow activity to transform and move data via mapping data flows.

Execute command activity stage data stage versions