Skip to main content

Use Azure Data Lake Storage for Ingest Instance

This guide covers how to create and configure Azure Data Lake Storage for an Ingest instance in Jet Analytics Data Integration.

  • If you are building a new Jet Analytics Data Integration environment from scratch, consider using one of the Reference Architectures for deploying Jet Analytics Data Integration in Azure.
  • If you deployed one of the Azure Marketplace templates with Azure Data Lake, the necessary Data Lake resources already exist. Skip step 1 and proceed to step 2.
  • If you have an existing Jet Analytics Data Integration environment in Azure without a Data Lake storage account, start at step 1.

Step 1: Create an Azure Storage Account

Note: If you already have a Data Lake Storage account, skip this step.

  1. In the Azure portal, go to Create a new Resource > Storage account — blob, file, table, queue > Create storage account.
  2. Assign a Subscription name, Storage account name, Location, and other required properties.
  3. Set Account kind to StorageV2 (general-purpose v2).
  4. On the Advanced tab, set Hierarchical namespace to Enabled.
  5. It is not necessary to create a container now — this is done from within Jet Analytics Data Integration in step 5.

For more details, refer to the Microsoft Azure documentation on creating Azure Data Lake Storage Gen2.

Step 2: Create an App Registration

To access Data Lake resources from Jet Analytics Data Integration, configure an App Registration in the Azure portal.

Note: You need your Azure Administrator to register the application in the Azure Portal and note the Application ID, Client ID, and Client Secret while registering the application. These details are required when configuring the Ingest instance storage.

Note:

The following access control steps describe the minimum permissions required in most cases. In production environments, you may adjust these permissions to align with your business and compliance requirements. See the Microsoft Azure documentation for details.

  1. In the Azure Portal, go to Azure Active Directory > App Registrations and click New Registration.
  2. Enter a name and select Accounts in this organizational directory only. The Redirect URI is the URL where your application is hosted. Click Register.
  3. On the new App Registration, go to Certificates & secrets and create a New Client Secret. The secret is encrypted after saving — record it somewhere safe. It will appear after you click Add.F

Step 3: Enable App Registration access to Data Lake

  1. Navigate to the resource group where your Data Lake resources are located and open the Data Lake storage account resource.
  2. Select Access Control (IAM) and click Add a role assignment.
  3. Add the App Registration you created to the Owner role for the resource.
Note:

After adding or removing role assignments, wait at least 5 minutes before executing a task within the Ingest instance. Changes can take up to 30 minutes to take effect. See Troubleshoot Azure RBAC for details.

Step 4: Configure Ingest instance storage

In the Jet Analytics Portal, add a new Ingest instance or edit an existing one.

  1. Storage Type: Select Azure Data Lake Gen2 storage.
  2. Storage Account: Enter the name of your Azure Storage Account. Only the resource name is required, not the full URL.
  3. Container name: Enter a name for the Azure Storage Container. The container will be created in Jet Analytics Data Integration in step 5.
  4. Tenant ID: Also known as the Directory ID, found under the properties of Azure Active Directory.
  5. Application ID: The Application ID of the App Registration created in step 2. Found in the Azure portal under Azure Active Directory > App Registrations.
  6. Application Key: The client secret created in step 2 for the App Registration.
  7. Timeouts: Timeouts for communicating with the Azure Storage account. The defaults are suitable for most situations but may need to be increased for slow connections or exceptionally long data transfers.
  8. Transfer to Prepare instance — Limit memory use: This setting applies only if you are not using Azure Data Factory for data transfer. For more information contact Product Support.

Step 5: Create the storage for the Ingest instance (container)

Before executing transfer tasks in an Ingest instance, the data storage must be created.

  1. In Jet Analytics Data Integration, right-click on the instance and select Edit Instance.

  2. Click Create Storage… to create the storage container for the Ingest instance.

Troubleshooting

Hostname error

Error:Service request failed: Invalid URI: The hostname could not be parsed. (System.UriFormatException)

Ensure that the details entered in the Portal for the Ingest instance do not contain spaces.

Was this article helpful?

We're sorry to hear that.