Skip to main content

Fabric Lakehouse Ingest Instance Storage

Use Microsoft Fabric Lakehouse as Ingest instance storage to ingest data sources into Microsoft Fabric OneLake Delta Parquet format.

Fabric Lakehouse Ingest instance storage is available as part of the standard, premium, or enterprise package.

For more information on standard, premium, or enterprise package, contact Product Support.

A public preview of this feature is currently available. The following functionality is not yet supported in the public preview:

  • Jet Analytics SAP Table data source
  • Ingest Storage Management tasks
  • Ingest Security Roles
  • Transfer to Prepare instance with Snowflake storage
  • ADF Transfer to Prepare instance

Prerequisites

  1. Ask your Azure Administrator to register an app in the Azure Portal and note the Tenant ID, Application ID, and Client Secret. You will need these details in the Add an Ingest Instance with Fabric Lakehouse Storage section.
  2. In the Fabric/Power BI Admin Portal, enable Allow service principals to use Power BI APIs as described in Fabric Lakehouse Inges2t Storage — App Registration Issues, to grant the App Registration access to the Fabric workspace.
  3. In the Fabric portal, create a new workspace or navigate to an existing one. Click Manage access and grant the App Registration account Member access to the workspace. Click Workspace settings, then under Data Engineering/Science click Spark settings, select Runtime version 1.2, and click Save.
  4. Go to the Admin Portal.

  5. Under Tenant, scroll down to OneLake settings and enable Users can access data stored in OneLake with apps external to Fabric.

  6. Under Tenant, scroll down to Admin API settings and ensure that Service principals can access read-only admin APIs is enabled and a security group is allowed. The security group should contain the relevant service principals.

Add an Ingest instance with Fabric Lakehouse storage

Note:

You can connect to an existing Lakehouse created directly in the Fabric portal, or you can create the Lakehouse from within Jet Analytics Data Integration.

  1. Add a new Ingest instance and select the storage type Microsoft Fabric Storage.

  2. Enter the Workspace name for the existing Fabric workspace.
  3. Provide a name for the Lakehouse.
  4. Enter the Tenant ID for the tenant associated with Fabric.
  5. Enter the Application ID for the App Registration.
  6. Enter the Application Key (the client secret value) for the App Registration.

Create the Fabric Lakehouse Ingest instance storage

  1. Open the Jet Analytics Data Integration application and open the Ingest instance.
  2. If the Lakehouse has not yet been created, right-click the instance in the Solution Explorer and select Edit Instance, then click Create Storage.

Objects created in Fabric

A notebook is created and run each time a transfer task executes for a data source in a Fabric Ingest instance. These notebooks can be monitored in the Fabric portal under Monitor.

The notebook creates a Delta Parquet table in the Lakehouse (named LoadTables_<data source name>_<uuid>) and a temporary Parquet file in the Lakehouse file folder. The data from the temporary file is transferred to the Delta Parquet table, after which the temporary file is deleted. JSON files are also created in the Lakehouse file folder to capture data source metadata.

Was this article helpful?

We're sorry to hear that.