Meshly Data Stack / Architecture

Every layer of the platform is wired to the next.

The stack runs 32 to 37 self-hosted data services, depending on the configuration, and they come pre-wired to work together on infrastructure you control.

Each layer talks to the layers beside it through connections that are already in place. Services collect their credentials from OpenBao at startup, and Keycloak gives every interface the same single sign-on. Queries run through Trino, and events travel through Kafka into the lakehouse.

32 to 37 services10 layersKeycloak SSOOPA RBACOpenBao secrets

Detail 01, layers

Ten layers run from security up to management.

Security sits underneath every other layer. Data comes in, moves across the event bus, gets processed, stored and queried, and then reaches dashboards and agents. Observability and management span the whole stack.

Every call through the MCP gateway carries two credentials: one belongs to the person who asks, and the other belongs to the agent itself. Together they decide which data the answer may use and whether each field comes back in full, masked or empty.

LayerPurpose and components
Layer: 00 / SecuritySSO, access and secretsKeycloak SSO, OAuth2 Proxy, OPA policies, OpenBao secrets, Vault Agents, tenant federation, per-service tokens, dynamic credentials
Layer: 01 / IngestionBring data inApache NiFi, Kafka Connect with Debezium, REST and API connectors, IoT bridges, external CDC sources
Layer: 02 / MessagingReal-time event busApache Kafka, Schema Registry, Avro and JSON, 3 listener contexts
Layer: 03 / ProcessingStream and batchApache Flink, Apache Spark, Faust Streams, MLlib, CEP
Layer: 04 / StorageLakehouse, relational and vectorsMinIO (S3), PostgreSQL, Apache Iceberg, Nessie versioning, Qdrant vectors, Valkey cache
Layer: 05 / QueryOne SQL surfaceTrino federation, Iceberg catalog, PostgreSQL catalog, Kafka catalog
Layer: 06 / Orchestration and catalogPipelines, lineage, qualityApache Airflow, OpenMetadata, data quality tests, lineage
Layer: 07 / Analytics and AIDashboards and agentsApache Superset, Grafana, Meshly Build (operations MCP), MDS-BI, MDS Data API, LangFlow, LangGraph (customer agents)
Layer: 08 / ObservabilityMetrics, logs, healthPrometheus, Grafana, Fluent Bit, Fluent Bit Mapper, OpenSearch logs, health checks
Layer: 09 / ManagementOperate and self-serveMDS Dashboard, customer portal, per-service restart, audit log

Detail 02, services

The services arrive already wired together.

All services share one network namespace, and their credentials are generated at random and stored in OpenBao. Trino, Kafka, Iceberg, Airflow, Superset, Keycloak and the other components below were chosen and tuned to work together.

Network
Every service in the pod shares one network namespace.
Containers
The stack runs 32 to 37 containers, depending on the tier.See the deployment sheet
Credentials
The installer generates every credential at random and stores it in OpenBao.
Optional
NiFi, Flink and Spark come with the Enterprise tier, or you can add them from the marketplace.See Detail 05
ServiceWhat it doesCategory
PostgreSQLIt is the shared metadata database for Airflow, Superset, OpenMetadata, Grafana, LangFlow and LangGraph.Category: Storage
MinIOIt provides S3-compatible object storage and holds the Iceberg lakehouse and intermediate ETL outputs.Category: Storage
NessieIt is a Git-like catalogue for Iceberg tables, with branching and time travel, and it is backed by PostgreSQL.Category: Catalog
Apache KafkaIt runs as a single-node KRaft broker, and its three listener addresses cover laptop, in-pod and external clients.Category: Streaming
Schema RegistryIt manages Avro and JSON schemas for Kafka topics and enforces data contracts across producers.Category: Streaming
Kafka ConnectIt runs Debezium PostgreSQL CDC with pluggable sinks, including PostgreSQL replicas, the Iceberg lakehouse, OpenSearch and S3.Category: CDC
OAuth2 ProxyIt sits as an auth sidecar in front of the Kafka Connect REST API and enforces Keycloak SSO on every call.Category: Auth proxy
TrinoIt federates distributed SQL across MinIO, PostgreSQL, Iceberg and Kafka, and it enforces OPA policies.Category: Query
OpenSearchIt is a distributed search and analytics engine that indexes logs and gives OpenMetadata full-text search.Category: Search
Fluent BitIt tails the stdout of every container through a read-only Podman log mount and ships the records to OpenSearch.Category: Logging
Fluent Bit MapperIt polls the Podman API every 30 seconds and rewrites log records, so OpenSearch indices carry service names in place of container IDs.Category: Sidecar
Apache AirflowIt orchestrates DAG-based pipelines, with a native OpenBao secrets backend and Keycloak SSO.Category: Orchestration
OpenMetadataIt handles the data catalogue, lineage, governance and quality tests, runs on OpenSearch and signs users in through Keycloak.Category: Catalog
Apache SupersetIt offers self-service dashboards on Trino federation, with row-level security and Keycloak SSO.Category: BI
GrafanaIt shows time-series dashboards and alerts from Prometheus, with a PostgreSQL backend and Keycloak OAuth.Category: Observability
PrometheusIt collects metrics from every container in the pod, with retention and multi-target support.Category: Observability
KeycloakIt is the OIDC identity provider, which hosts the mds realm and federates SSO across every stack interface.Category: Identity
Open Policy AgentIt holds dynamic RBAC policies for Trino, Kafka Connect and other resources, and it reloads them without a restart.Category: Authz
OpenBaoIt is a Vault-compatible secrets store with TLS, auto-unseal and full audit logging.Category: Secrets
Vault Agent for SupersetThis OpenBao agent renders secret templates into the Superset configuration at startup.Category: Sidecar
Vault Agent for GrafanaThis OpenBao agent renders secret templates for the Grafana environment and dashboard provisioning.Category: Sidecar
ValkeyIt is a Redis-compatible in-memory cache for sessions, pub/sub, Faust state and LangGraph job queues.Category: Cache
Faust StreamsIt runs stateful stream processing in Python for real-time aggregations and transformations on Kafka topics.Category: Streaming
QdrantIt provides vector similarity search for embeddings and RAG, with REST and gRPC APIs.Category: Vectors
MDS MCP ServerIt exposes the stack as 99 MCP tools for Meshly Build and Claude Code agents.Category: AI, agents
LangFlowIt is a visual builder for LangChain workflows, where you prototype AI flows by dragging and dropping nodes.Category: AI, agents
LangGraphIt is the customer agentic runtime, serving FastAPI REST and SSE on PostgreSQL-checkpointed state with Keycloak-validated identity. It ships with 2 worker replicas that pull from Valkey Streams, so runs can proceed concurrently.Category: AI, agents
MDS DashboardIt handles container monitoring, logs, credential retrieval and quick restarts, behind Keycloak SSO.Category: Management
MDS-BIIt is an AI-operated dashboard builder in React and MUI, with a build, preview and approve flow.Category: AI, BI
MDS Data APIIt serves AI-managed REST endpoints on Trino and PostgreSQL, with auth, caching and CORS.Category: AI, API

Detail 03, connect

Four surfaces lead into the same stack.

You can connect from code on the host, from inside the pod or from an external container, and operators and agents drive the stack through Meshly CLI from any workstation. Every surface is documented, and every path stays the same as your deployment grows.

SurfaceWho and how
Surface: 01 / Host machineDeveloping on your laptopYou reach every service through forwarded localhost ports, which is the development path.
Surface: 02 / Pod-internalContainers inside the podAll containers share one network namespace, so services reach each other on the shared localhost. Kafka producers stay on the in-pod listener.
Surface: 03 / External containerSidecars and external appsThese reach the stack through the host bridge on host-published ports, and Kafka has its own cross-pod bridge listener.
Surface: 04 / Meshly CLIOperators and agents, anywhereOne CLI talks to both the Data Stack and Meshly Build, whether over the network, behind a VPN or fully airgapped. You can sign in, run Trino queries and run agents from a workstation.

Detail 04, MDS Dashboard

One view shows every container.

The MDS Dashboard ships with the stack and shows live container metrics and structured logs. You can use it to inspect PostgreSQL schemas and queries, browse OpenBao secrets, manage the lifecycle of Data API consumer keys, install modules and restart services, and all of it sits behind Keycloak SSO.

Stack health
The dashboard tracks CPU, RAM, disk and network use across all containers.It counts running, stopped, paused and total containers
Per container
Each container card shows uptime, CPU and RAM trends and published ports.You can start, stop and restart the container from its card
Views
Overview, Data Quality, Iceberg Catalog, S3 Storage, Sentinel, Documentation, Settings, Shortcuts.
Access
Sign-in goes through Keycloak SSO.

Detail 05, marketplace

You install domain modules from your portal.

Alongside the services in your stack, you can browse domain modules in the Meshly catalogue and install them into your stack from the customer portal. Every module is tier-locked and signature-verified, and we add new modules regularly.

Source
Modules come from the Meshly catalogue in the customer portal.
Integrity
Each module is signature-verified before it installs.
Access
Modules are tier-locked, and each one names the tier it needs.
More modules
FinOps Suite, ITxPT Ingestion, OPC-UA Bridge, Fleet Analytics, Milvus, LangFlow, MDS Scraper.
ModuleCategoryTier
Apache NiFiIt provides visual data flow programming for complex ingestion pipelines.Category: IngestionTier: Enterprise
Apache FlinkIt runs stateful stream processing at scale, with a jobmanager and taskmanager.Category: StreamingTier: Enterprise
Apache SparkIt handles distributed batch and structured streaming, with a master, worker and history server.Category: ComputeTier: Enterprise
HL7 / FHIR BridgeIt is a healthcare interoperability adapter for HL7v2 and FHIR R4.Category: DomainTier: Enterprise
GTFS-RT FeedIt ingests public-transit positions and alerts in real time.Category: DomainTier: Enterprise
GDPR Compliance PackIt adds retention policies, right-to-erasure tooling and audit reports.Category: DomainTier: Enterprise

Detail 06, operate

The stack is production-grade from the first install.

We shaped the Data Stack around what you only learn by running platforms in production, such as secrets rotation, blue-green deploys, structured logging and backup retention.

  1. Nobody hardcodes a password. Every service password is generated at install and sealed in OpenBao, and per-service tokens grant least-privilege reads. One script rotates any credential.

  2. One Keycloak realm serves every interface. Superset, Grafana, Airflow, OpenMetadata, NiFi and MDS-BI all federate through the mds realm, and you can bring your own Entra ID, Google Workspace or LDAP.

  3. Backups are versioned and testable. PostgreSQL, Keycloak, Grafana, MinIO buckets, the OpenBao Raft snapshot, Trino catalogs and OpenSearch are backed up on a schedule with retention, and you can run a restore in dry-run mode before you commit to it.

  4. The stack is built for OpenShift. Red Hat OpenShift is the production target, with Kubernetes 1.19+ supported alongside it, and the same install runs on macOS, Linux and WSL2 for development. Platforms and server mode.

  5. Logs and metrics arrive structured. Services write Pino logs, and the stdout of every container streams to OpenSearch with no per-app configuration, because Fluent Bit tails Podman log storage, parses each line and writes it to logs-<service>-YYYY.MM.DD indices. Prometheus scrapes every container, and Grafana ships with default dashboards. Health checks run across every container, with automatic recovery through MDS Sentinel.

  6. Agents are first-class users of the stack. There are two AI surfaces. On the operations surface, MDS MCP Server with 99 tools lets Meshly Build and Claude Code inspect, query and manage the stack. On the customer surface, LangGraph hosts your own agentic flows on PostgreSQL-checkpointed state with Keycloak-validated per-user identity. MDS MCP Server is also the MCP gateway: when an agent reaches data through it, the call carries both the asking person's credential and the agent's own, and together they decide which data the answer may use and whether each field comes back in full, masked or empty. Both surfaces sit behind audit trails.

  7. Components install and uninstall like packages. The installer has explicit flags to install, uninstall and purge each component. The component registry rehydrates on stack updates, so your customisations survive upgrades without a manual reinstall. Component runtime configuration lives in OpenBao and is managed through MCP tools, and every write is audited.

If you want local models on your own hardware, see Meshly AI.

Related sheets

See how it fits your infrastructure.

The Data Stack is for customers, and the Meshly team onboards every deployment. Tell us about your data setup, and we will walk through the architecture with you.

One level deeper

Pick your footprint.

The deployment sheet covers configuration tiers, hardware sizing, platforms, onboarding and the customer portal.

Open the deployment sheet