Meshly Data Stack / Architecture
Every layer of the platform is wired to the next.
The stack runs 32 to 37 self-hosted data services, depending on the configuration, and they come pre-wired to work together on infrastructure you control.
Each layer talks to the layers beside it through connections that are already in place. Services collect their credentials from OpenBao at startup, and Keycloak gives every interface the same single sign-on. Queries run through Trino, and events travel through Kafka into the lakehouse.
Detail 01, layers
Ten layers run from security up to management.
Security sits underneath every other layer. Data comes in, moves across the event bus, gets processed, stored and queried, and then reaches dashboards and agents. Observability and management span the whole stack.
Every call through the MCP gateway carries two credentials: one belongs to the person who asks, and the other belongs to the agent itself. Together they decide which data the answer may use and whether each field comes back in full, masked or empty.
| Layer | Purpose and components |
|---|---|
| Layer: 00 / Security | SSO, access and secretsKeycloak SSO, OAuth2 Proxy, OPA policies, OpenBao secrets, Vault Agents, tenant federation, per-service tokens, dynamic credentials |
| Layer: 01 / Ingestion | Bring data inApache NiFi, Kafka Connect with Debezium, REST and API connectors, IoT bridges, external CDC sources |
| Layer: 02 / Messaging | Real-time event busApache Kafka, Schema Registry, Avro and JSON, 3 listener contexts |
| Layer: 03 / Processing | Stream and batchApache Flink, Apache Spark, Faust Streams, MLlib, CEP |
| Layer: 04 / Storage | Lakehouse, relational and vectorsMinIO (S3), PostgreSQL, Apache Iceberg, Nessie versioning, Qdrant vectors, Valkey cache |
| Layer: 05 / Query | One SQL surfaceTrino federation, Iceberg catalog, PostgreSQL catalog, Kafka catalog |
| Layer: 06 / Orchestration and catalog | Pipelines, lineage, qualityApache Airflow, OpenMetadata, data quality tests, lineage |
| Layer: 07 / Analytics and AI | Dashboards and agentsApache Superset, Grafana, Meshly Build (operations MCP), MDS-BI, MDS Data API, LangFlow, LangGraph (customer agents) |
| Layer: 08 / Observability | Metrics, logs, healthPrometheus, Grafana, Fluent Bit, Fluent Bit Mapper, OpenSearch logs, health checks |
| Layer: 09 / Management | Operate and self-serveMDS Dashboard, customer portal, per-service restart, audit log |
Detail 02, services
The services arrive already wired together.
All services share one network namespace, and their credentials are generated at random and stored in OpenBao. Trino, Kafka, Iceberg, Airflow, Superset, Keycloak and the other components below were chosen and tuned to work together.
- Network
- Every service in the pod shares one network namespace.
- Containers
- The stack runs 32 to 37 containers, depending on the tier.See the deployment sheet
- Credentials
- The installer generates every credential at random and stores it in OpenBao.
- Optional
- NiFi, Flink and Spark come with the Enterprise tier, or you can add them from the marketplace.See Detail 05
| Service | What it does | Category |
|---|---|---|
| PostgreSQL | It is the shared metadata database for Airflow, Superset, OpenMetadata, Grafana, LangFlow and LangGraph. | Category: Storage |
| MinIO | It provides S3-compatible object storage and holds the Iceberg lakehouse and intermediate ETL outputs. | Category: Storage |
| Nessie | It is a Git-like catalogue for Iceberg tables, with branching and time travel, and it is backed by PostgreSQL. | Category: Catalog |
| Apache Kafka | It runs as a single-node KRaft broker, and its three listener addresses cover laptop, in-pod and external clients. | Category: Streaming |
| Schema Registry | It manages Avro and JSON schemas for Kafka topics and enforces data contracts across producers. | Category: Streaming |
| Kafka Connect | It runs Debezium PostgreSQL CDC with pluggable sinks, including PostgreSQL replicas, the Iceberg lakehouse, OpenSearch and S3. | Category: CDC |
| OAuth2 Proxy | It sits as an auth sidecar in front of the Kafka Connect REST API and enforces Keycloak SSO on every call. | Category: Auth proxy |
| Trino | It federates distributed SQL across MinIO, PostgreSQL, Iceberg and Kafka, and it enforces OPA policies. | Category: Query |
| OpenSearch | It is a distributed search and analytics engine that indexes logs and gives OpenMetadata full-text search. | Category: Search |
| Fluent Bit | It tails the stdout of every container through a read-only Podman log mount and ships the records to OpenSearch. | Category: Logging |
| Fluent Bit Mapper | It polls the Podman API every 30 seconds and rewrites log records, so OpenSearch indices carry service names in place of container IDs. | Category: Sidecar |
| Apache Airflow | It orchestrates DAG-based pipelines, with a native OpenBao secrets backend and Keycloak SSO. | Category: Orchestration |
| OpenMetadata | It handles the data catalogue, lineage, governance and quality tests, runs on OpenSearch and signs users in through Keycloak. | Category: Catalog |
| Apache Superset | It offers self-service dashboards on Trino federation, with row-level security and Keycloak SSO. | Category: BI |
| Grafana | It shows time-series dashboards and alerts from Prometheus, with a PostgreSQL backend and Keycloak OAuth. | Category: Observability |
| Prometheus | It collects metrics from every container in the pod, with retention and multi-target support. | Category: Observability |
| Keycloak | It is the OIDC identity provider, which hosts the mds realm and federates SSO across every stack interface. | Category: Identity |
| Open Policy Agent | It holds dynamic RBAC policies for Trino, Kafka Connect and other resources, and it reloads them without a restart. | Category: Authz |
| OpenBao | It is a Vault-compatible secrets store with TLS, auto-unseal and full audit logging. | Category: Secrets |
| Vault Agent for Superset | This OpenBao agent renders secret templates into the Superset configuration at startup. | Category: Sidecar |
| Vault Agent for Grafana | This OpenBao agent renders secret templates for the Grafana environment and dashboard provisioning. | Category: Sidecar |
| Valkey | It is a Redis-compatible in-memory cache for sessions, pub/sub, Faust state and LangGraph job queues. | Category: Cache |
| Faust Streams | It runs stateful stream processing in Python for real-time aggregations and transformations on Kafka topics. | Category: Streaming |
| Qdrant | It provides vector similarity search for embeddings and RAG, with REST and gRPC APIs. | Category: Vectors |
| MDS MCP Server | It exposes the stack as 99 MCP tools for Meshly Build and Claude Code agents. | Category: AI, agents |
| LangFlow | It is a visual builder for LangChain workflows, where you prototype AI flows by dragging and dropping nodes. | Category: AI, agents |
| LangGraph | It is the customer agentic runtime, serving FastAPI REST and SSE on PostgreSQL-checkpointed state with Keycloak-validated identity. It ships with 2 worker replicas that pull from Valkey Streams, so runs can proceed concurrently. | Category: AI, agents |
| MDS Dashboard | It handles container monitoring, logs, credential retrieval and quick restarts, behind Keycloak SSO. | Category: Management |
| MDS-BI | It is an AI-operated dashboard builder in React and MUI, with a build, preview and approve flow. | Category: AI, BI |
| MDS Data API | It serves AI-managed REST endpoints on Trino and PostgreSQL, with auth, caching and CORS. | Category: AI, API |
Detail 03, connect
Four surfaces lead into the same stack.
You can connect from code on the host, from inside the pod or from an external container, and operators and agents drive the stack through Meshly CLI from any workstation. Every surface is documented, and every path stays the same as your deployment grows.
| Surface | Who and how |
|---|---|
| Surface: 01 / Host machine | Developing on your laptopYou reach every service through forwarded localhost ports, which is the development path. |
| Surface: 02 / Pod-internal | Containers inside the podAll containers share one network namespace, so services reach each other on the shared localhost. Kafka producers stay on the in-pod listener. |
| Surface: 03 / External container | Sidecars and external appsThese reach the stack through the host bridge on host-published ports, and Kafka has its own cross-pod bridge listener. |
| Surface: 04 / Meshly CLI | Operators and agents, anywhereOne CLI talks to both the Data Stack and Meshly Build, whether over the network, behind a VPN or fully airgapped. You can sign in, run Trino queries and run agents from a workstation. |
Detail 04, MDS Dashboard
One view shows every container.
The MDS Dashboard ships with the stack and shows live container metrics and structured logs. You can use it to inspect PostgreSQL schemas and queries, browse OpenBao secrets, manage the lifecycle of Data API consumer keys, install modules and restart services, and all of it sits behind Keycloak SSO.
- Stack health
- The dashboard tracks CPU, RAM, disk and network use across all containers.It counts running, stopped, paused and total containers
- Per container
- Each container card shows uptime, CPU and RAM trends and published ports.You can start, stop and restart the container from its card
- Views
- Overview, Data Quality, Iceberg Catalog, S3 Storage, Sentinel, Documentation, Settings, Shortcuts.
- Access
- Sign-in goes through Keycloak SSO.
Detail 05, marketplace
You install domain modules from your portal.
Alongside the services in your stack, you can browse domain modules in the Meshly catalogue and install them into your stack from the customer portal. Every module is tier-locked and signature-verified, and we add new modules regularly.
- Source
- Modules come from the Meshly catalogue in the customer portal.
- Integrity
- Each module is signature-verified before it installs.
- Access
- Modules are tier-locked, and each one names the tier it needs.
- More modules
- FinOps Suite, ITxPT Ingestion, OPC-UA Bridge, Fleet Analytics, Milvus, LangFlow, MDS Scraper.
| Module | Category | Tier |
|---|---|---|
| Apache NiFiIt provides visual data flow programming for complex ingestion pipelines. | Category: Ingestion | Tier: Enterprise |
| Apache FlinkIt runs stateful stream processing at scale, with a jobmanager and taskmanager. | Category: Streaming | Tier: Enterprise |
| Apache SparkIt handles distributed batch and structured streaming, with a master, worker and history server. | Category: Compute | Tier: Enterprise |
| HL7 / FHIR BridgeIt is a healthcare interoperability adapter for HL7v2 and FHIR R4. | Category: Domain | Tier: Enterprise |
| GTFS-RT FeedIt ingests public-transit positions and alerts in real time. | Category: Domain | Tier: Enterprise |
| GDPR Compliance PackIt adds retention policies, right-to-erasure tooling and audit reports. | Category: Domain | Tier: Enterprise |
Detail 06, operate
The stack is production-grade from the first install.
We shaped the Data Stack around what you only learn by running platforms in production, such as secrets rotation, blue-green deploys, structured logging and backup retention.
Nobody hardcodes a password. Every service password is generated at install and sealed in OpenBao, and per-service tokens grant least-privilege reads. One script rotates any credential.
One Keycloak realm serves every interface. Superset, Grafana, Airflow, OpenMetadata, NiFi and MDS-BI all federate through the
mdsrealm, and you can bring your own Entra ID, Google Workspace or LDAP.Backups are versioned and testable. PostgreSQL, Keycloak, Grafana, MinIO buckets, the OpenBao Raft snapshot, Trino catalogs and OpenSearch are backed up on a schedule with retention, and you can run a restore in dry-run mode before you commit to it.
The stack is built for OpenShift. Red Hat OpenShift is the production target, with Kubernetes 1.19+ supported alongside it, and the same install runs on macOS, Linux and WSL2 for development. Platforms and server mode.
Logs and metrics arrive structured. Services write Pino logs, and the stdout of every container streams to OpenSearch with no per-app configuration, because Fluent Bit tails Podman log storage, parses each line and writes it to logs-<service>-YYYY.MM.DD indices. Prometheus scrapes every container, and Grafana ships with default dashboards. Health checks run across every container, with automatic recovery through MDS Sentinel.
Agents are first-class users of the stack. There are two AI surfaces. On the operations surface, MDS MCP Server with 99 tools lets Meshly Build and Claude Code inspect, query and manage the stack. On the customer surface, LangGraph hosts your own agentic flows on PostgreSQL-checkpointed state with Keycloak-validated per-user identity. MDS MCP Server is also the MCP gateway: when an agent reaches data through it, the call carries both the asking person's credential and the agent's own, and together they decide which data the answer may use and whether each field comes back in full, masked or empty. Both surfaces sit behind audit trails.
Components install and uninstall like packages. The installer has explicit flags to install, uninstall and purge each component. The component registry rehydrates on stack updates, so your customisations survive upgrades without a manual reinstall. Component runtime configuration lives in OpenBao and is managed through MCP tools, and every write is audited.
If you want local models on your own hardware, see Meshly AI.
Related sheets
See how it fits your infrastructure.
The Data Stack is for customers, and the Meshly team onboards every deployment. Tell us about your data setup, and we will walk through the architecture with you.
One level deeper
Pick your footprint.
The deployment sheet covers configuration tiers, hardware sizing, platforms, onboarding and the customer portal.
Open the deployment sheet