Azure Data Factory- Azure functions as pipeline activity- Part1

In my previous post I talked about how easy it is to use data factory to copy your data from on-premise to Azure cloud with full security, scalability and easy options and configurations provided by ADF to play with your option.

While working on ADF I found integration of Azure functions with ADF is again a very cool thing which give so much flexibility to user when playing around your data.

While working on one of my projects, I came across a situation where I had to copy .pgp files from on-perm and transform data and then store to Azure SQL. When I started working with ADF I realised ADF doesn’t support .pgp files. Though I was able to copy .pgp data from on-perm to Azure blob using ADF copy pipeline with binary format but got stuck with data transformation part. And here come the saviour “Azure function” which really worked well for decrypting my files and transforming my data. Then I stored the decrypted and transformed data into .txt in blob and with ADF copy activity easily copied them to SQL.

So this is how my pipeline looked like.

Well here I am not focusing on copy activity configuration and if you want to see more about this copy activity then my previous post will help :

Azure Data Factory- Copy data from on-perm SFTP server to Azure blob.

So now let’s talk about the configuration of azure function here. Below are some point which we have to consider while using azure function in our ADF pipeline.

1. Your function app should be HTTP-triggered.
2. The return type of the Azure function has to be a valid JObject.
3. Azure Functions times out after 230 seconds regardless of the functionTimeout setting you've configured in the settings.
4. For long running functions follow an async pattern or use Durable Functions.

Now coming to sample code for azure function. I have done this in c# but yes you can write your functions in your preferred language just keeping above points in mind.

So in this sample code my function app is decrypting my .pgp file which i copied in my pipeline copy activity.

Now coming to configuration part of azure function in ADF pipeline.

First you have a create a linked service to connect to your function app where you need to pass the function url .

Now basic configuration to call your function in your pipeline.

Here based on your Method type you can select GET, POST, DELETE,PUT etc. In my case my function is using POST calls.

Now here comes the tricky part to pass the body to your function.

If it is static value you have to send to call your azure function then it is pretty much easy to pass in you body.

But in my case I need to pass the file names dynamically which was getting created with my previous activity.

So this was the value I was using in my case.

image1

image2

Which will be different when you create your pipeline but point here is this is how you will create dynamic body to your function app.

And now you are all ready to use your azure function in your ADF pipeline.

This is really awesome option Azure has provided in ADF which saved me while dealing with .pgp encrypted files.

Hope this is will help someone. :). Happy Learning !!!!