Data Factory

Cloud based Data integration Service.

Can be think as SSIS on cloud.

Key Components:

Linked Service

Dataset

Pipeline

Activity

Key Features:

  1. Better automation power like In build scheduling mechanism (Json based).
  2. Fault Tolerant Pipeline execution, we can Re tries for Transient failure.
  3. Monitoring and Alerts.
  4. Staging feature for intermediate processing.

Monitoring: To Do add screenshots

 

Copy Activity

In build copy activity- Blob to SQL.

Need Data manger – To copy from legacy systems.

 

Custom .Net Activity

Custom .Net activity – Need to add new Azure Batch account for compute.

 

Data Factory-Visual Studio Extension

To Write Custom .Net activity code we have to install the Data Factory VS Extension. Installation path from Visual Studio VS -> Tools-> Extension and Updates.

Afterward we can use the Data Factory Templates.

DF

From VS we can also publish the Data Factory project into cloud with all dependencies (package/dlls).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s