# OpenMetaData

### Overview

OpenMetaData is an end-to-end metadata management platform that enables unlocking the value of data assets in common use cases of data discovery and governance, as well as emerging use cases related to data quality, observability, and people collaboration.

OpenMetaData on Nuvolos supports the ingestion of files stored on Azure File Shares, which allows you to track operations performed on files stored in Azure File Shares.

### Setting up OpenMetaData

Add a new **OpenMetaData** application to your Nuvolos working instance. OpenMetaData runs inside a **VS Code** application and includes a pre-installed **Airflow** environment for executing ingestion workflows created in OpenMetaData.

#### Starting your application

Start the OpenMetaData application after adding it to your Nuvolos instance. On the first launch, initialization can take a few minutes because both the **OpenMetaData** and **Airflow** databases are set up in the background. When startup completes, the application opens in a **VS Code** interface. You can use VS Code to access Airflow, create or refine DAGs, and install additional packages from the built-in terminal if needed..

#### Opening OpenMetaData

To open OpenMetaData, open the **VS Code** [Command Palette](https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette) and run:

```
OpenMetaData: Show OpenMetadata
```

OpenMetaData opens in a new VS Code tab. Select **Sign in with Auth0** to log in. On first sign-in, a new OpenMetaData user is created automatically.

If you are a **Nuvolos space administrator** when the application is first started, your OpenMetaData user is created with administrator privileges. Otherwise, a non-privileged user is created. OpenMetaData checks this administrator mapping only on first start, so if you receive Nuvolos admin privileges later, an existing OpenMetaData administrator must grant you the corresponding role inside OpenMetaData.

### Adding an Azure File Share storage:

To add Azure File Share storage in OpenMetaData:

1. Open Settings > Storages > Add New Service.
2. Select AZFS from the available storage services.
3. Enter a name for the storage service.
4. Provide the Azure File Share name and a [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-get-info?tabs=portal#get-a-connection-string-for-the-storage-account) with read access to that file share.
5. Select Test Connection to confirm that OpenMetaData can access the Azure File Share with the supplied credentials.

### Adding an ingestion pipeline for Azure File Share

OpenMetaData can create an ingestion pipeline that maps folders and files in Azure File Share into OpenMetaData containers and objects. This pipeline is implemented as an Airflow DAG managed by OpenMetaData.

To create the pipeline:

1. Open the AZFS storage service you created.
2. On the *Ingestions* tab, select *Add Metadata Ingestion*.
3. Optionally enter a pipeline name.
4. Set *Storage Metadata Config* Service to AZFS.
5. Provide the Azure File Share connection string and the file share name.
6. Configure the ingestion schedule.
7. Select Add & Deploy to create and deploy the pipeline in Airflow.

### Running the ingestion pipeline

To execute the ingestion pipeline manually, select Run. To inspect execution details, open the Logs link to view the corresponding Airflow logs.

#### Viewing the Airflow DAG

To open the Airflow interface in VS Code, open the **Command Palette** and run:

```
Airflow: Show Airflow
```

Airflow opens in a new VS Code tab, where you can inspect the DAG created for the ingestion workflow.

#### Checking the newly ingested metadata:

After the ingestion pipeline completes, open *Explore > Containers* to review the newly ingested metadata for the Azure File Share.