Home > Azure, Computers and Internet > Azure Logic App That Writes Service Bus Messages to A Data Lake

Azure Logic App That Writes Service Bus Messages to A Data Lake


The title says it ALL!  Um, no it doesn’t.  The title gets right to the point, but by no means is that ALL.  This article will talk about everything you need to setup all of this as well.  Don’t do what I did and JUMP right into the Logic App, because you may miss something along the way and you will have to delete steps and redo things.  But that’s what I did. I jumped right into it and wanted things to just WORK dammit!  My main hiccup was that I wanted to use the default security so I DIDN’T have to set up an Application Registration through Azure Active Directory (AAD).  I don’t advise that.  PLUS, my browsers, all three: Chrome, Edge, and IE, were blocking the popup and the connection would fail.  I explicitly turned OFF popup blocking in all three browsers and yet Azure was giving me an error that the popup was being blocked.  I spent WAY too long trying to figure out how to ALLOW the popups and head down the “easy” path, when I should have been looking at connecting to resources using Service Principles (AAD Application Registrations).

I currently do all my Azure testing in an Enterprise account where I am allotted $150 a month, and most months I only use about $50 of that (mostly MooVee Picker).  This means that I try to keep things lean and mean and run the “Azure AD Free” version of my Default Directory.

I’m also assuming you are SOMEWHAT familiar with navigating through the Azure Portal (https://portal.azure.com) You may not be an expert, but you at least know how to navigate.

A Table of Contents (of sorts – no page numbers OR hotlinks)

  1. Create Azure Active Directory Application Registration
  2. Create Service Bus Namespace
    1. Create Topic
    2. Create Subscription
  3. Create Data Lake
    1. Add access for AAD App Reg
    2. Add new folder “log01”
  4. Create Logic App
    1. Trigger from Subscription
    2. Complete the message
    3. Test
    4. Write Message to Data Lake
    5. Test

Create Azure Active Directory Application Registration

Where is Azure Active Directory?  It’s not in the “All Resources” view.  If you’re like me, which you might not be, you have AAD at the TOP of your favorite’s menu on the left.  I like it there because that “resource” isn’t in the “All Resources” view.

Make AAD your favorite! (fig. 1)

Select “All Services”, search for “Active”, and Azure Active Directory should show up in the filtered list.  Your entry is probably not marked as a favorite.  Go ahead and mark it and it will be at the BOTTOM of your favorites list in your menu.  Drag it to the top (if you want to be like me).  Otherwise have fun searching for this resource all the time.  I forget if you have to create one for the first time, but I don’t think you do.  I believe this AAD resource is part of your subscription.

(fig. 2)

Once you select your AAD favorite icon, you will be able to select the “App registrations” section under the “Manage” section and create a “New registration”.

(fig. 3)

Enter a name for your Application Registration.  I chose the highly original name “DemoLakeAppReg” (fig. 3).  I added the “AppReg” suffix so if this name shows up in a listing that I don’t recognize, I know where it came from.  As the screen says “(this can be changed later)”.  In this run through I will omit the Redirect URI, unless it’s absolutely needed for this demo and then I’ll just edit this text and you will never know (I’m re-reading this and I didn’t need it).  Once you select your options, click on the “Register” button at the bottom of the screen (not pictured to save space).

(fig. 4)

When you register your application, you will be taken to its overview.  This will display the client and tenant keys that you will need later.  Please note the “Certificates & secrets” section that we will visit a bit later.

Create Service Bus Namespace

Add a service bus (namespace) by clicking on the plus icon, search for “service bus”, and select that new resource.  That will take you to the “Create namespace” form. (fig. 5) Enter a unique name for your service bus namespace.  I started with “demo-bus” and that was already taken, so I started to enter “ncm” after “demo” and was successful after I typed the “n” so I went with it.  Demon Bus it is!  I chose the Premium tier, because I wanted to see what the cost differences are compared against another service bus that I have that uses the lowest tier.  Choose the best plan that suits you since it won’t affect this example.

(fig. 5)

When creating this namespace it was the first time I had to choose a resource group, and I created a new resource so I could filter out all of the other resource groups that I have going on in my subscription. (fig. 5)

(fig. 6)

Create a Topic and just call it “demolake” (fig. 6); choosing all the defaults is fine.  Select your new topic in the overview and create a new subscription and call it “demolakesub” and stick with the defaults. (fig. 7)

(fig. 7)

Create Data Lake

When you select the “All Resources” icon and then filter down by [your resource group name here] (why do I hear GLaDOS’s voice here?) or DemoLake if you’re like me. (fig. 8)  Select the “Add” option, search for “data lake”, and I chose Data Lake Storage Gen1.

(fig. 8)

The name “demolake” was not taken so I jumped on that one, sorry about that.  Maybe try “demonlake”?  It’s available at the moment at the time I’m writing this.  I chose my default subscription, my existing DemoLake resource group, and changed my location to “Central US”. (fig. 9)

(fig. 9)

I know you want to jump right in and create your Logic App, but DON’T!  You should set up your data lake access before you forget.  You don’t HAVE to do it now.  You will successfully create our Logic App, but you will see errors when you finally test it because the Logic App won’t have rights to your new data lake.

(fig. 10)

Your FIRST instinct might be to head to the “Access Control (IAM)” section, but that won’t get you access to the data in the lake (or write to it).  The IAM section will allow you to set up standard Azure access to “manage” this resource.

You must set up access in the “Data explorer”. (fig. 10) Once you’ve launched the Data Explorer, select the root “demolake” and then “New folder” to create a folder called “log01”. (fig. 11) This will also prove out our access to a “child” of the root data lake (setup in the next step).  Since file system structure is rather important in the lake, keeping the data organized makes it easier to find and transform from its raw state into something usable.

(fig. 11)

Make sure that the root folder “demolake” is selected and select the “Access” option which will add another form to fill out to the right in the standard Azure Portal fashion. (fig. 12)

NOTE: If you do this in the REVERSE order (allow access to root and include children, THEN create the folder) that folder will NOT inherit the access.  My initial understanding of the option was that anything you create from then on would inherit from root, but it doesn’t.  I got access errors the first time doing it the “reverse” way and I had to fix it.

(fig. 12)

Search for the Application Registration which we created in the first step and make sure you “Select” it as well.  There might be quite a few in your list, so searching for it might be quicker. (fig. 13)

(fig. 13)

Once the group has been selected, you will be prompted with permissions.  Go ahead and add ALL of the permissions and have them propagate to all of the children as well.  This IS a demo after all.  You can experiment with other permissions and creating permissions on individual folders later.  Once you select “Ok” you will see your new “Assigned permissions”. (fig. 14)

(fig. 14)

Create Logic App

Search for “Logic App” in the new resource screen found by clicking the “+” in the left menu.  Select the resource found and the form below will be displayed. (fig. 15) Fill out the form and create the App.  Note that I used my existing resource group “DemoLake” just to be organized.  I kept “Log Analytics” off just so I didn’t have to create space for them.

(fig. 15)

I now have my four resources (don’t forget the AAD Application Registration) and I need to flesh out the Logic App, because right now it does absolutely nothing.

(fig. 16)

Select the DemoLakeApp. When the overview comes up you may see the “Edit” button disabled, because Azure is attempting to be helpful and showing you a bunch of templates from which to start.  Scroll down and choose the “blank” template, because we can search for the one we want when adding the first step.

Make note of the Disable button (fig. 17), because that might come into play if you encounter errors or if you don’t “complete” your messages where they stay on the bus and are continually processed costing YOU money.

(fig. 17)

Search for “service” in your trigger step once you’ve chosen a “blank” template.  “Service Bus” should be the first trigger category that comes up.  It may not be the FIRST one, but it’ll be up there.

(fig. 18)

Select “Service Bus” once you found it (fig. 18) and you will be presented with the service bus triggers that will execute your logic app.  Select the following option that is pictured “When a message is received in a topic subscription (peek-lock)”. (fig. 19)

(fig. 19)

When you select the service bus trigger you will need to configure it.  The Logic App Designer will prompt you to create a connection to the subscription.  Enter a name, which will be a friendly name if you create multiple connections to this namespace and select the “Service Bus Namespace”.  I chose my “demon-bus”. (fig. 20)  When this connection is created, Azure will create an “API Connection” resource and automatically add it to my DemoLake resource group. It is viewable in the “All resources” view in the portal (just in case you feel like deleting it and creating a different one).

(fig. 20)

Now that the connection is created (or selected if you’re doing this a second time), choose the Topic and Subscription to monitor and pick a frequency.  The Logic App will POLL using this frequency, but if there are no messages waiting in the Subscription then this trigger will be “skipped”.  You will see all the successes and failures on the Logic App Overview screen and there is also a summary of each evaluation, including all of the skipped triggers.

(fig. 21)

You COULD jump right to creating the Data Lake step, but it’s a good idea to test the Service Bus Connection.  You will need to add this step anyway, because successfully processed messages should be “completed” (removed from the Subscription queue). The designer even reminded you when you created that first trigger step since it was in “peek-lock” mode.

Click on the “New step” button and select the Topic and Subscription names from the dropdown lists.  When you click in the “Lock Token” field the “Dynamic content” form will display and select the Lock Token.  That dynamic content will be placed in the Lock Token field. (fig. 22)

(fig. 22)

Click the “Save” button and you will be able to run your first test. (fig. 23)  Send a message to the “demolake” topic.  Easier said than done, I know.  Do you have a tool to do this?  I built one using both the .Net Framework and a different one using .Net Core.  If you have made it THIS far, and do not have a way to test your topic, then let me know.  I’ll make the code available on GitHub.  I use Service Bus Explorer written by Paolo Salvatori (of Microsoft) to monitor my Service Bus Namespaces.

Since you are familiar with creating Logic Apps, you COULD create a Logic App that sends a message to your topic to test it.

(fig. 23)

You can see on the Logic App overview page that I had a successful run. (fig. 24)  You can also see that the Logic App triggered 16 times.  When I set this up initially, I’ll admit that I went STRAIGHT to the data lake step which certainly makes things harder to diagnose when there are issues.  It’s better to take smaller steps, not necessarily BABY steps, where you can incrementally verify this process.

(fig. 24)

To get the message to the data lake you need a STEP to do this.  I was GOING to say this is the “last” step, but it’s actually the SECOND step and we are just implementing it LAST.  Semantics, I know.  Launch the Logic App Designer, hover over the arrow pointing to the last step, and click that plus “+” button.  The Designer will prompt you to add a step.  Search for “lake” and select the “Append file (concurrent)” action. (fig. 25)

(fig. 25)

The Designer will prompt you to create a connection to the data lake.  Initially I wanted to just go the EASY route, but the popups NEVER came up for me.  Select the “Connect with Service Principal” link/button. (fig. 26)

(fig. 26)

All four fields are needed.  For the “Connection Name” call it “demolake” so it matches your account name on the next prompt. (fig. 27)

Did you write those other IDs down or paste them into a text file?  NO?  Don’t worry, just launch another tab from the “Microsoft Azure” branding in the upper left and navigate to your favorite Azure Active Directory.  Manage the “App registrations” which is in a picture WAY back at the beginning of this blog.  Since you are a multi-tab master, if you need this picture for reference (fig. 4) then launch this blog in another tab and then scroll down a bit.

Select the DemoLakeAppReg where the Application (client) and Directory (tenant) IDs are listed in the overview.  If you hover over those IDs, the copy buttons will appear.  Copy and paste the IDs to their appropriate spots in the Azure Data Lake step. (fig. 27)

(fig. 27)

Select the “Certificates & secrets” on your DemoLakeAppReg and create a new secret. (fig. 28)  Name it whatever you want and click “Add”.  Once it shows you the secret, make sure you COPY it just like you did the other IDs and paste it into the “Client Secret” field. (fig. 27)

(fig. 28)

With all of the fields filled in, go ahead and “Create” the connection.

For the “Account Name” field this is ABSOLUTELY the name of your data lake.  DON’T do what I did and just put some display name, but since I added “AccountName” as a suffix (BatLakeAccountName) the first time I was able to find my issue pretty quickly.  This field MUST be “demolake” (or [insert data lake name here]).

Enter the “File path” log01/logicapp01 which will append to a file in the log01 folder. (fig. 29)  Add whatever values you like in the “Content” field.  You can add dynamic content like you did when choosing the Lock Token above when setting up the previous Logic App step. (fig. 22)  Since there is no dynamic content for the CURRENT step, selecting “See more” will show the fields from the previous step(s). (fig. 29)

(fig. 29)

Make sure you save this and by default it is enabled, so if you’ve got messages stacked up in that Topic then they will be processed..  Run your message publish test and you will see the file in the log01 folder of your demolake.  YAY! (fig. 30)

(fig. 30)

In summary…  Well you can scroll up to the table of contents and get the gist of what’s covered.  Oh wait a second.  In summary, we created four resources and Azure created two API Connections to support the Logic App PLUS all of the topics covered in the table of contents.

(fig. 31)

You made it!! I hope that this has helped you as I had to do MUCH trial and error, as well as look at Microsoft documentation and a few other blog posts, some of which were VERY helpful and some were NOT. Good or bad, feedback is always appreciated.

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: