Source Catalog Agent: Workflow & Artifacts

The Source Catalog Agent serves as the foundational data librarian for the Datapunkt ecosystem. It connects to your diverse databases, files, APIs, and event streams to discover schemas, profile distributions, and generate machine-readable artifacts. By automating metadata ingestion and organization, this agent eliminates the manual overhead of updating data catalogs and maintains a live, centralized source of truth.

Supported Formats & Protocols

To ensure seamless compatibility across modern enterprise data architectures, the Source Catalog Agent provides out-of-the-box support for a wide range of database connectivity protocols, file structures, messaging systems, and metadata specifications:

• JDBC / ODBC: Establishes secure, standard connections to relational databases (PostgreSQL, MySQL, Microsoft SQL Server, Oracle) and enterprise data warehouses (Snowflake, BigQuery, Redshift).
• APACHE KAFKA: Connects to broker clusters to inspect active topics, partition structures, and payload formats.
• AMQP (Advanced Message Queuing Protocol): Integrates with message brokers such as RabbitMQ and ActiveMQ to inspect queues, exchanges, and message payloads.
• REST API: Queries custom endpoints, registry applications, and developer endpoints to fetch dynamic data structures and technical payload designs.
• JSON / AVRO: Decodes semi-structured application logs, document store records, and payload formats, parsing complex nested arrays and key-value records.
• PARQUET: Inspects columnar binary files on cloud object storage (AWS S3, Google Cloud Storage, Azure Blob Storage) to extract schema details and metadata.
• OPEN LINEAGE: Ingests lineage events and run-time metadata from processing engines (Apache Spark, Airflow) to trace operational context.

Step-by-Step Discovery & Profiling Workflow

The Source Catalog Agent executes a structured, repeatable workflow to continuously capture, analyze, and document your technical assets. This process operates autonomously based on cron schedules or webhooks, keeping your schemas aligned with physical data structures.

Step 1: Connection & Session Initialization

First, the agent initializes a secure, read-only session with the target source using the configured credentials:

• For databases, it establishes a pool using JDBC or ODBC drivers.
• For streaming brokers, it connects to Apache Kafka bootstrap servers or AMQP exchange queues.
• For object stores, it authenticates with cloud service accounts to scan folders containing Parquet, Avro, or JSON files.
• For APIs, it triggers REST requests utilizing OAuth tokens or API keys.

Step 2: Schema Discovery & System Inspection

Once authenticated, the agent scans the source's structural directory without reading the actual row-level business records:

• In relational environments, it queries system tables (such as information_schema.columns or pg_catalog) to extract table names, column layouts, primary keys, and index configurations.
• For message queues (Kafka / AMQP), it queries the Schema Registry or reads the latest message headers to extract the payload structure.
• For files (Parquet / Avro / JSON), it reads file headers and schema blocks to extract nested layouts.
• For lineage data, it intercepts Open Lineage events to map target dataset addresses.

Step 3: Out-of-Band Data Profiling

To understand the quality, freshness, and shape of the data, the agent performs light metadata profiling:

• It analyzes value distributions to identify null-value percentages, unique constraint metrics, and minimum/maximum boundaries.
• It calculates mathematical standard deviations to flag outliers.
• It monitors timestamps to detect when tables or partitions were last written, mapping data freshness.
• This execution runs inside your secure environment using minimal resources to prevent performance degradation on production systems.

Step 4: Semantic Relationship Mapping

The agent runs an inference engine over the gathered metadata:

• It matches key patterns (e.g. comparing user_id across different systems) to automatically map foreign key relationships.
• It groups tables and topics into logical domain boundaries.
• It records any structural modifications (e.g. added columns or changed types) to log schema drift.

Step 5: Artifact Generation & Publishing

Finally, the agent compiles its findings into standardized, machine-readable developer artifacts:

• Master Schemas: Clean JSON or YAML files detailing the exact type, description, and source mapping of every field.
• Profiling Reports: Comprehensive data health summaries tracking null rates and structural validity.
• Data Catalogs: Glossary entries ready to be integrated into downstream platforms.

Benefits: What Makes It Good?

• Eliminates Manual Cataloging: Traditional data dictionary documentation is obsolete the day it is written. The Source Catalog Agent automates this documentation, executing schema updates in real time.
• Accelerates Engineering Workflows: Developers and analysts no longer waste hours searching for data locations or asking what a column means. The master schemas act as clear developer APIs.
• Prevents Production Breakage: By profiling structure and formats (like Parquet or Avro schemas), the agent flags schema changes before they break downstream ETL, ELT, or business intelligence pipelines.
• Source Agnostic Integration: Connects modern stream-based architectures (Kafka, AMQP) and legacy relational setups (JDBC/ODBC) under a single management system.