Skip to main content

Azure Data Load (ETL) Process using Azure Functions Step by Step Example

Data Load (ETL) Process using Azure Functions


Azure Functions 

are serverless and are a great solution for processing data, integrating systems, working with the internet-of-things (IoT), and building simple APIs and microservices. Consider Functions for tasks like image or order processing, file maintenance, or for any tasks that you want to run on a schedule.

Here we are talking about ETL process implementation using Azure Functions, even though Azure Data Factory is out there but if you are a c# developer you will love it. You can leverage all the benefits of the App Service Plan and/or Consumption Plan(Pay As you Go) along with Event-Driven Process and Programming Model

Azure Durable Function

Durable Function is an extension of Azure Function that lets you write stateful functions in a serverless compute environment. It allows you to define stateful workflows by writing orchestrator functions and stateful entities by writing entity/Activity functions using the Azure Functions programming model. All other things like state management, checkpoints, and restarts for you, will be taken care of by azure durable function engine and allowing you to focus on your business logic.

The primary requirement is the reader should be familiar  with Azure function and durable functions

Business Requirement

 We have CSV file dropped into the azure blob storage/container and that file should be process and data saved into the Azure SQL Server.

Design and Architecture

  • Azure Function: It's a blob trigger function and starter for the data load process.
  • Azure Durable Functions - Orchestrator: It's an orchestrator function that will manage the workflow/data flow activities function and all the executions 
  • Azure Durable Functions - Activity: An azure function that will actually process the CSV data and will insert into the azure SQL database
  • Azure SQL Server: will be in used to keep processed data
  • SendGrid: will be used to send emails and acknowledgment on process completion
  • Application Insights: can be used to logging the exception event etc..
work flow

Development Environment Setup:

  1.  Visit MSDN for the step by step example  here is a link
  2.  Required NuGet package Microsoft.Azure.WebJobs.Extensions.DurableTask

Code and Example : 

Here is a list of code screenshots
Start Function: its a blob trigger azure function that will execute automatically once any CSV file will be dropped into the container "samples-workitems"

Start Function


Orchestrator Function : 

it will manage the life cycle of data workflow 
Orchestrator Function


Activity Functions
perform actual data manipulation and communication with the database.

Activity Functions


Solutions and NuGet pkg:
Solutions and NuGet pkg

Bonus Points: 

  1. Use Cunsputions Plans only if you are sure that you function execution time will no exceed 10minutes limit
  2. Use App Service Plan if need to configure vnet andother securtity and if your function will need more than 10 minutes to complete the task just you need to configure function time out in host.json
  3. Visit for more Application Patterns 


Labels:

Comments

Post a Comment

Popular posts from this blog

How to Make a Custom URL Shortener Using C# and .Net Core 3.1

C# and .Net Core 3.1:  Make a Custom URL Shortener Since a Random URL needs to be random and the intent is to generate short URLs that do not span more than 7 - 15 characters, the real thing is to make these short URLs random in real life too and not just a string that is used in the URLs Here is a simple clean approach to develop custom solutions Prerequisite:  Following are used in the demo.  VS CODE/VISUAL STUDIO 2019 or any Create one .Net Core Console Applications Install-Package Microsoft.AspNetCore -Version 2.2.0 Add a class file named ShortLink.cs and put this code: here we are creating two extension methods. public   static   class   ShortLink {      public   static   string   GetUrlChunk ( this   long   key ) =>            WebEncoders . Base64UrlEncode ( BitConverter . GetBytes ( key ));      public   static   long   GetKeyFromUrl ( this   string   urlChunk ) =>            BitConverter . ToInt64 ( WebEncoders . Base64UrlDecode ( urlChunk )); } Here is the Calling Sampl
Post a Comment
Read more

azure function error unknown argument --port

How to run Azure Function app on a different port in Visual Studio or  azure function error unknown argument --port How to Fix it? Update Project Properties -> Debug to following put the following command  "host start --port 7071 --pause-on-error" Finally, it works 
Post a Comment
Read more

Microsoft Graph API upload large file to SharePoint

How to Upload a file to SharePoint using Microsoft Graph API - C#/.Net? To make it easier to upload large files, a number of entities in Microsoft Graph support plus some extra file uploads. Instead of attempting to upload the entire file in a single request, the file is divided into smaller pieces and a single request is used to upload a single slice. To make this process easier, the Microsoft Graph SDKs include a large file upload task that handles the uploading of the slices. Azur AD Setup:  you need to complete the following steps to configure the azure ad. Step - 1: Register an application with the Microsoft identity platform Sign in to the  Azure portal . If you have access to multiple tenants, use the  Directories + subscriptions  filter in the top menu to switch to the tenant in which you want to register the application. Search for and select  Azure Active Directory . Under  Manage , select  App registrations  >  New registration . Enter a Display  Name  for your applicatio
Post a Comment
Read more