Datahub - how to add custom data source

To add a custom data source to DataHub, a metadata management and data discovery platform, you'll typically need to create a custom plugin or connector that interfaces with your specific data source. This is particularly useful if you're working with a data system that isn't natively supported by DataHub.

Here's a general approach to adding a custom data source to DataHub:

1. Understand the DataHub Architecture

Before proceeding, it's important to understand DataHub's core architecture:

• DataHub Server: Centralized backend that stores and manages metadata.
• DataHub UI: The frontend interface for users to explore and search for metadata.
• DataHub Ingestion Framework: A framework that allows ingestion of metadata from various data sources (relational databases, NoSQL stores, etc.).

To add a custom data source, you'll primarily interact with the DataHub Ingestion Framework, which is responsible for extracting metadata from data sources and loading it into DataHub.

2. Create a Custom Connector (Ingestion Source)

The custom data source you want to add to DataHub will need to be integrated through an ingestion source. You’ll need to implement the ingestion logic for the specific data source. Here's a general process for adding a custom connector:

Step 1: Set Up the Ingestion Framework

Make sure you have the DataHub Ingestion Framework installed and configured. The ingestion framework is available as a Python library.

You can install it via pip:

pip install datahub

Step 2: Create a Custom Plugin

You'll need to create a new plugin or connector by subclassing the appropriate base classes. The ingestion framework supports connectors for different data sources, so you'll be creating a new connector class that implements the required methods.

Example of creating a custom connector:

Here’s a skeleton example of how a custom ingestion connector might look like:

from datahub.ingestion.source.metadata import MetadataSource
from datahub.metadata.schema_classes import DatasetSnapshotClass, DatasetPropertiesClass
from datahub.ingestion.source.state import SourceState

class MyCustomDataSource(MetadataSource):
    def __init__(self, config: dict):
        super().__init__(config)
        self.config = config

    def get_snapshot(self, dataset_name: str):
        """
        This method fetches the metadata for a given dataset from your custom data source.
        It should return an instance of DatasetSnapshotClass.
        """
        # Implement logic to connect to your data source and retrieve metadata
        # Example: Fetch dataset properties and schema
        properties = DatasetPropertiesClass(
            description="Description of the dataset",
            tags=["tag1", "tag2"],
            # Add more properties as needed
        )
        
        snapshot = DatasetSnapshotClass(
            urn=f"urn:my-custom-data-source:{dataset_name}",
            aspects=[properties],
        )
        
        return snapshot

    def get_all_datasets(self):
        """
        This method returns a list of all datasets available in the custom data source.
        """
        # Fetch all datasets (e.g., from an API or direct database query)
        datasets = ["dataset1", "dataset2"]  # Example datasets

        return datasets

    def close(self):
        """
        Optionally close any open connections or cleanup tasks.
        """
        pass

In the above example:

• get_snapshot: This method should return the metadata (schema, properties, etc.) for a specific dataset.
• get_all_datasets: This method fetches all available datasets in your custom data source.
• close: Optional method for cleaning up resources (e.g., database connections).

Step 3: Configure the Connector

Once you've written your custom connector, you'll need to configure it by creating an appropriate configuration file. This file will typically be in YAML format and specify the details for your custom data source.

Example my_custom_data_source.yml configuration:

source:
  type: my_custom_data_source
  config:
    host: "mydata.example.com"
    port: 1234
    username: "user"
    password: "password"
    database: "mydatabase"

Step 4: Ingest Metadata Using the Custom Connector

To ingest metadata from your custom data source into DataHub, run the ingestion process and specify your custom data source as the source.

datahub ingest -c my_custom_data_source.yml

This will execute the ingestion process, extracting metadata from your custom data source and loading it into DataHub.

Step 5: Register the Custom Connector in DataHub

Once you've developed your custom data source, register it in your DataHub instance by updating the configuration to ensure DataHub is aware of your custom connector. You'll typically do this by placing your custom plugin in the appropriate directory or specifying it in your ingestion configuration.

---

3. Handle Custom Data Source Authentication

If your custom data source requires authentication (e.g., using API tokens or database credentials), make sure to handle these aspects securely. You can use environment variables or a secret management system to avoid hardcoding credentials in your scripts or configuration files.

---

4. Optional: Implementing Data Sync and Incremental Updates

If you want to implement incremental updates or sync the metadata regularly from your custom data source, you can integrate with the DataHub Ingestion State system. This will ensure that only the changed or new data is fetched during each ingestion cycle.

class MyCustomDataSourceWithState(MetadataSource):
    def __init__(self, config: dict, state: SourceState):
        super().__init__(config)
        self.state = state  # Use the state object to keep track of previously ingested data

    def get_snapshot(self, dataset_name: str):
        # Fetch metadata and handle incremental updates based on state
        pass

5. Testing and Debugging

Test your custom connector thoroughly by running the ingestion process in a development or staging environment. Check the DataHub logs for errors or warnings, and validate that the metadata is being ingested correctly into the system.

• Check DataHub UI to ensure the datasets are appearing as expected.
• Validate that your custom data source's metadata, schema, and other aspects are correctly populated.

Conclusion:

Adding a custom data source to DataHub involves:

1. Creating a custom plugin or connector class.
2. Implementing necessary methods (such as fetching datasets and metadata).
3. Configuring and running the ingestion process.

By following this approach, you can integrate nearly any data source into DataHub, enhancing your metadata management and data discovery capabilities. If you run into specific issues or need more detailed assistance, feel free to ask!

 

How to do Step 5: Register the Custom Connector in DataHub?

To register a custom connector in DataHub, you will need to ensure that your custom data source is recognized by the DataHub ingestion framework and can be used during metadata ingestion. DataHub’s architecture is highly extensible, allowing you to add custom sources, but registering them properly is a key step to making sure they work.

Here’s a detailed explanation of how to register your custom connector in DataHub:

1. Place Your Custom Connector Code in the Correct Location

When you develop a custom connector, DataHub expects the connector code to be structured in a way that it can easily load.

• For custom connectors, place your code in a Python package that is included in your DataHub project or installable via pip.
• A typical directory structure might look like:

datahub/
    ├── datahub-connector-custom/
    │    ├── __init__.py
    │    ├── my_custom_data_source.py
    │    └── setup.py
    ├── datahub-ingestion/
    ├── datahub-frontend/
    ├── docker-compose.yml
    └── ingestion-scripts/

If you are adding a standalone custom connector, place it in a subfolder within the main project directory, e.g., datahub-connector-custom.

2. Update the setup.py (if applicable)

If your custom connector is in a separate Python package (i.e., you have made it a distributable package), you will need to define the package so that it can be installed in DataHub’s environment. Ensure that your setup.py file has the proper dependencies and package details.

Example of setup.py for a custom package:

from setuptools import setup, find_packages

setup(
    name="datahub-connector-custom",
    version="0.1",
    packages=find_packages(),
    install_requires=[
        "datahub",
        "requests",  # or other dependencies needed for your custom connector
    ],
    entry_points={
        'datahub.ingest': [
            'my_custom_data_source = datahub_connector_custom.my_custom_data_source:MyCustomDataSource',
        ],
    },
)

In this case:

• The entry point under datahub.ingest tells DataHub where to find the custom connector (my_custom_data_source.py) and which class to use (MyCustomDataSource).
• This allows DataHub to load and use your custom connector when running an ingestion job.

3. Add the Custom Connector to the DataHub Configuration File

DataHub’s ingestion jobs rely on YAML configuration files. To register your custom connector, you need to modify the ingestion configuration to specify that your custom data source should be used.

The configuration file will define the source for ingestion, and you will specify your custom connector in this configuration.

Here is an example of how you might set up your ingestion configuration file (my_custom_data_source.yml) to use the custom connector:

source:
  type: my_custom_data_source
  config:
    host: "mydata.example.com"
    port: 5236
    username: "user"
    password: "password"
    database: "mydatabase"

sink:
  type: datahub-rest
  config:
    server: http://localhost:8080

• type: my_custom_data_source tells DataHub to use your custom connector.
• config contains the specific connection parameters for your custom data source (these can vary depending on the implementation).

Ensure the source.type matches the class name or identifier you used in the entry_points of your setup.py file.

4. Install the Custom Connector

If you packaged your custom connector as a Python package, you need to install it into your DataHub environment.

You can install it via pip:

pip install /path/to/your/datahub-connector-custom

Or, if it’s hosted on a package repository (e.g., a private PyPI server or GitHub), you can install it via:

pip install git+https://github.com/your-username/datahub-connector-custom.git

Alternatively, if you're working within the local DataHub repo, simply running:

python setup.py install

will install the custom connector.

5. Configure and Run Ingestion

Now that your custom connector is registered and installed, you can run the DataHub ingestion job to start ingesting metadata from your custom data source.

Use the following command to run the ingestion process:

datahub ingest -c my_custom_data_source.yml

This command will:

• Load the my_custom_data_source.yml configuration.
• Use your custom connector to extract metadata from the specified data source.
• Send the extracted metadata to the configured sink (e.g., DataHub’s REST API).

6. Verify Registration and Logs

Once you run the ingestion, verify that:

1. The ingestion process picks up and uses your custom connector.
2. The metadata is correctly ingested into DataHub.
3. No errors are shown in the logs related to the custom connector.

If any issues arise, check the logs for detailed error messages. You can also add more logging in your custom connector code to help with debugging.

Example Log Checking:

DataHub logs will be located in the directory where you run the ingestion. If you are running DataHub in Docker, you can check the logs using:

docker-compose logs

Look for any messages related to your custom connector to verify if it’s being invoked correctly.

7. Ensure DataHub UI Shows the Metadata

After a successful ingestion, navigate to the DataHub UI and verify that the metadata from your custom data source is visible.

• Check that the datasets, schema, and other metadata aspects appear as expected.
• If needed, test querying for the ingested metadata to ensure that it's usable for search, discovery, and other DataHub features.

Conclusion

To register a custom connector in DataHub:

1. Implement the connector code.
2. Use setup.py to define the entry point for your custom connector.
3. Modify the ingestion YAML configuration to use your custom source.
4. Install your custom connector package.
5. Run the ingestion job and verify that the custom source is correctly ingested into DataHub.

This process allows DataHub to extend its native connectors to support custom data sources for metadata ingestion, helping you integrate nearly any data system into your DataHub environment.