One pipeline for every
log, metric, event, and trace.
NimbusPulse ingests data from AWS, Azure, GCP, Kubernetes, system logs, security tools, databases, and custom sources — through declarative connectors defined in JSON or YAML. Every button, modal, chart, and the AI assistant below is fully interactive on static data + localStorage.
How data moves through NimbusPulse
Five logical stages. Sources push, or are pulled by connectors, normalized into a common event schema, enriched, then routed to storage and the dashboard. Every stage is horizontally scalable.
Real-time signals from every source
Tiles update every 2 seconds from the in-memory event simulator. Click any tile to see what the metric measures and how it's computed.
22 built-in connectors, plus your own
Each connector is a declarative spec: protocol, endpoint, auth, schedule, and field schema. Toggle a connector to start or stop ingestion. Custom connectors you add are saved to localStorage and appear below alongside the built-ins.
Add a custom connector
Paste a JSON or YAML schema below. It will be validated, saved, and appear in the grid above.
Every event, as it arrives
A continuous feed of normalized events. Click any row to see the raw payload, run an explanation, or take action (acknowledge, mute, investigate, replay).
Search & drill into raw log lines
Use the search to filter by message text, source, or severity. Click a row to see structured fields, the original raw line, and an AI explanation of what likely happened.
Pick the right wire format per source
Different sources speak different protocols. The platform exposes five gateways; connectors pick whichever fits the source. Below is each protocol with a flow example.
What we'd actually build, behind this UI
The simulation above is the front-end of the system. Below is the production reference architecture this UI is designed against: Apache NiFi as the dataflow orchestrator, Kafka as the durable bus, Flink for stream processing, and a polyglot storage layer feeding the dashboard, alerts, and AI assistant.
NiFi gives a visual, auditable, drag-and-drop dataflow with built-in back-pressure, provenance, and 300+ processors. Connectors defined as YAML in this UI compile down to NiFi process groups. Schema validation, enrichment, masking, and routing all happen here before anything hits Kafka.
Kafka is the durable, replayable bus — every downstream system (storage, alerts, AI) is a consumer. Flink reads from Kafka to compute windowed metrics, sessions, and anomaly scores, writing results back to a metrics topic that fans out to TimescaleDB and the dashboard.
No single store is good at everything. Elasticsearch for fast text search, ClickHouse for analytics over billions of rows, S3 + Iceberg for cheap long-term retention with query, Redis for hot counters, TimescaleDB for metrics. The query API hides this from callers.
Ask anything about your data
In production, this assistant runs over your indexed events using RAG (retrieval over Elasticsearch + vector store) and a guard-railed LLM. Here, it's a static pattern-matched simulation — try the suggested prompts or type your own.
Upload anything. Watch the pipeline.
Drop a file below to simulate the full Extract → Transform → Load pipeline.
In production this writes to MinIO (raw + images) and Elasticsearch (searchable text), and emits a Kafka event for each stage.
Here, everything stays in your browser — fake extraction, plausible metadata, persisted to localStorage.
- received file accepted by ETL API —
- extract parse text + images per file type —
- transform PII redaction · language detect · SHA-256 —
- load.minio raw → np-cold-archive · images → np-cold-events —
- load.elasticsearch indexed → np-documents —
-
complete
event emitted to Kafka
etl.events—
etl.eventsLibrary · processed documents
Click any card to see extracted text + images. Persists in localStorage.