monster-mq

Imported from vogler75/monster-mq (AGENTS.md). Install upstream with npx skills add vogler75/monster-mq. Copyright stays with the author.

AGENTS.md

This file provides guidance to AI coding assistants working with code in this repository.

Project Overview

MonsterMQ Core is the Kotlin/Vert.X MQTT broker and core platform with data persistence through PostgreSQL, CrateDB, MongoDB, or SQLite. It features a web dashboard, GraphQL API, MCP (Model Context Protocol) Server for AI integration, REST API, Prometheus metrics, I3X API for manufacturing data, AI agents with GenAI providers, a flow engine for automation workflows, and device bridging for OPC UA, PLC4X, Kafka, NATS, Redis, Telegram, WinCC OA/Unified, Neo4j, JDBC loggers, and SparkplugB.

MonsterMQ Edge Broker

MonsterMQ also has an edge broker variant (MonsterMQ Edge) implemented in Go for a lightweight footprint. The edge broker exposes the same GraphQL interface shape as MonsterMQ Core where supported, but currently implements only a limited subset of the full broker functionality. More GraphQL-backed functionality will be added over time, so new dashboard and API work should consider compatibility with both MonsterMQ Core and MonsterMQ Edge when practical.

Build and Run Commands

Building the Project

cd broker
mvn clean package

Running Locally

cd broker
# Run the broker
java -classpath target/classes:target/dependencies/* at.rocworks.MonsterKt [-cluster] [-log INFO|FINE|FINER|FINEST|ALL]
# Or use the convenience script (Linux/macOS or Windows PowerShell)
./run.sh [-cluster] [-log INFO|FINE|FINER|FINEST|ALL]
.\run.ps1 [-cluster] [-log INFO|FINE|FINER|FINEST|ALL]

iX Dashboard (monster-mq-dashboard)

The dashboard using Siemens iX lives in its own repository at monster-mq-dashboard (and locally at ../dashboard or ../monster-mq-dashboard). It is built with Vite and outputs to dist/.

cd ../dashboard        # or within the dashboard checkout
npm install
npm run dev          # Vite dev server on http://localhost:5173, proxies /graphql to :4000
npm run build        # Outputs to dist/

Running Tests

Kotlin Unit Tests

cd broker
mvn test

Python Integration Tests

cd tests
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Run all tests
pytest

# Run by category
pytest pytest_tests/mqtt3/     # MQTT v3.1.1 tests
pytest pytest_tests/mqtt5/     # MQTT v5 tests
pytest pytest_tests/graphql/   # GraphQL API tests
pytest pytest_tests/opcua/     # OPC UA tests
pytest pytest_tests/database/  # Database backend tests

# Run single test
pytest pytest_tests/mqtt3/test_basic_pubsub.py::test_basic_pubsub_qos0 -v

Test configuration via environment variables:

MQTT_BROKER=localhost  MQTT_PORT=1883  MQTT_USERNAME=Test  MQTT_PASSWORD=Test
GRAPHQL_URL=http://localhost:4000/graphql

Docker Commands

# Build Docker image
cd docker
./build.sh

# Run with Docker
docker run -v ./log:/app/log -v ./config.yaml:/app/config.yaml rocworks/monstermq [-cluster] [-log INFO|FINE|FINER|FINEST|ALL]

Code Architecture

Core Components

• Main Entry Point: broker/src/main/kotlin/Main.kt -> Monster.kt
• MQTT Server: MqttServer.kt - Handles MQTT protocol implementation (v3.1.1 and v5)
• MQTT Client Handler: MqttClient.kt - Manages individual client connections and session state
• Message Bus: bus/ directory - Abstraction for message distribution (Vertx EventBus or Kafka)
• Storage Layer: stores/ directory - Modular storage implementations for sessions, retained messages, and archives
• GraphQL Server: graphql/GraphQLServer.kt - GraphQL API for dashboard and external integrations
• Flow Engine: flowengine/ directory - Automation workflow engine with script execution
• Auth System: auth/ directory - User management, password encoding, ACL caching

Storage Architecture

The system uses different store types for different purposes:

• SessionStore: Persistent client sessions (PostgreSQL, CrateDB, MongoDB, SQLite)
• QueueStore: Queued messages for offline clients (PostgreSQL, MongoDB, SQLite). V1 uses a two-table design, V2 uses a single-table PGMQ-inspired design. Defaults to matching SessionStoreType if not set.
• RetainedStore: Retained MQTT messages (Memory, Hazelcast, PostgreSQL, CrateDB)
• MessageArchive: Historical message storage (PostgreSQL, CrateDB, MongoDB, SQLite, Kafka)
• LastValueStore: Current value cache for topics (Memory, Hazelcast, PostgreSQL, CrateDB)
• MetricsStore: Broker metrics history (PostgreSQL, CrateDB, MongoDB, SQLite)

Device Bridging Architecture

Devices follow a pattern of Extension (cluster-aware coordinator) + Connector (per-device verticle):

• devices/mqttclient/ - MQTT client bridge
• devices/kafkaclient/ - Kafka client bridge
• devices/natsclient/ - NATS client bridge
• devices/redisclient/ - Redis client bridge
• devices/telegramclient/ - Telegram bot client bridge
• devices/opcua/ - OPC UA client
• devices/opcuaserver/ - OPC UA server
• devices/plc4x/ - PLC4X protocol (industrial PLCs, supports JSON payload extraction via jsonPath)
• devices/winccoa/ - WinCC OA integration
• devices/winccua/ - WinCC Unified integration
• devices/neo4j/ - Neo4j graph database bridge
• devices/sparkplugb/ - SparkplugB decoder

For device integration guidance, see dev/plans/plan-device-integration.md.

Key Directories

• broker/src/main/kotlin/bus/ - Message bus implementations (Vertx, Kafka)
• broker/src/main/kotlin/stores/ - Storage layer implementations
  • dbs/postgres/ - PostgreSQL storage implementations
  • dbs/cratedb/ - CrateDB storage implementations
  • dbs/mongodb/ - MongoDB storage implementations
  • dbs/sqlite/ - SQLite storage implementations
  • devices/ - Device configuration data classes
• broker/src/main/kotlin/devices/ - Device connector/extension implementations
• broker/src/main/kotlin/handlers/ - Message and subscription handlers
• broker/src/main/kotlin/extensions/ - Extensions (MCP Server, I3X API, Sparkplug)
• broker/src/main/kotlin/graphql/ - GraphQL resolvers, queries, and mutations
• broker/src/main/kotlin/flowengine/ - Flow engine for automation workflows
• broker/src/main/kotlin/auth/ - Authentication and authorization
• broker/src/main/kotlin/data/ - Data models and codecs
• broker/src/main/kotlin/genai/ - GenAI provider abstraction
• broker/src/main/kotlin/logging/ - Syslog and in-memory log handling

Web Dashboard

iX Dashboard (Siemens iX design system):

• dashboard/ - Vite project root
  • package.json - Dependencies: @siemens/ix, @siemens/ix-icons, @siemens/ix-echarts, echarts, vite
  • vite.config.js - Build config, outputs to dashboard/dist/
  • src/ - Dashboard source files
    • js/ix-init.js - iX component registration + classic-dark theme (ES module)
    • js/sidebar.js - SidebarManager using ix-menu + ix-menu-category + ix-menu-item (add new pages here) + SPA router
    • js/ui.js - Shared modals, confirms, toasts, loading/error/empty states (window.ui)
    • js/storage.js - localStorage wrapper
    • js/graphql-client.js - GraphQLDashboardClient class
    • js/log-viewer.js - Global log viewer component
    • assets/components.css - The design system. Every recurring UI element, defined once
    • assets/monster-theme.css - Brand tokens, scrollbars, auth indicator, standalone login page
    • assets/ix-app.css - iX layout overrides (layout only)
    • pages/ - HTML pages (one per view), use <ix-application> + <ix-menu> shell
    • js/ - Page-specific JavaScript

Read dashboard/DESIGN.md before adding or editing a dashboard page. It defines the two page shapes (list / detail) and the rules that keep them consistent. The critical one: pages must not redefine shared components in a local <style> block — the SPA router injects page styles after the shared stylesheets, so a local copy silently overrides the design system and the component changes appearance as you navigate. Use window.ui instead of alert()/confirm()/hand-rolled toasts, and never load scripts from a CDN (air-gapped deployments — add to VENDOR_BUNDLES in vite.config.js instead).

• dist/ - Build output (gitignored)

IMPORTANT: Always edit dashboard files in the monster-mq-dashboard repository (src/), NOT directly in broker/src/main/resources/dashboard/. The build script or Maven downloads and packages the built output into broker/src/main/resources/dashboard/ or the JAR classpath, so any direct edits there will be overwritten.

HTTP Services and Ports

Configuration

Configuration is done via YAML file (config.yaml). The schema is defined in broker/yaml-json-schema.json.

Key configuration sections:

• Network ports (TCP, TCPS, WS, WSS, HTTP)
• Storage backends (SessionStoreType, QueueStoreType, RetainedStoreType, LastValueStoreType)
• Archive groups with topic filters
• Database connections (PostgreSQL, CrateDB, MongoDB)
• Kafka configuration
• Auth/User management (UserStoreType, JwtSecret)
• MCP Server requires an ArchiveGroup named "Default"

Feature Flags

Optional broker features are toggled via the top-level Features block in config.yaml. The single source of truth is the Features object at broker/src/main/kotlin/Features.kt — every flag is declared as a const val, and Features.all is the list iterated at startup to compute enabledFeatures (see Monster.kt).

When you add a new feature, you MUST update all four of these:

1. broker/src/main/kotlin/Features.kt — add a const val MyFeature = "MyFeature" and append it to Features.all.
2. broker/yaml-json-schema.json — add an entry under properties.Features.properties so the YAML config validates and IDEs autocomplete it.
3. broker/src/main/kotlin/Monster.kt — gate the verticle deployment with if (Monster.isFeatureEnabled(Features.MyFeature)) { ... } next to the other deployments around line 1223+.
4. GraphQL resolvers — gate every public DataFetcher method that backs the feature with if (!Monster.isFeatureEnabled(Features.MyFeature)) return@DataFetcher .... Use the existing pattern in FlowMutations.kt:30-31 for mutation results, return emptyList()/null/false for queries based on declared return type. Without this, disabled features still respond to GraphQL calls (the verticle is gone but the resolver is wired).
5. Dashboard sidebar — if the feature has a UI page, add feature: 'MyFeature' to the sidebar item in dashboard/src/js/sidebar.js so it auto-hides when the broker reports the feature as disabled.

The Broker.enabledFeatures GraphQL field returns the active set; the dashboard reads it once and hides any sidebar entry whose feature: flag isn't in the set.

Extension Points

1. MCP Server (extensions/McpServer.kt, extensions/McpHandler.kt): Model Context Protocol integration for AI models
2. I3X API (extensions/I3xServer.kt): i3X v1 API for manufacturing data — mounted under /i3x/v1. See dev/done/plan-i3x-spec.md for the v1 envelope and endpoints.
3. REST API (extensions/RestApiServer.kt): REST API endpoints for external integrations
4. Prometheus (extensions/PrometheusServer.kt): Prometheus metrics exporter
5. OA4J Bridge (extensions/Oa4jBridge.kt): WinCC OA Java API bridge for datapoint subscriptions
6. SparkplugB Decoder: Handled by devices/sparkplugb/ device connectors (configured via dashboard)

Technology Stack

• Language: Kotlin (JVM 21)
• Framework: Vert.X 5.0.7 (async/reactive)
• Clustering: Hazelcast
• Databases: PostgreSQL, CrateDB, MongoDB, SQLite
• Message Bus: Kafka (optional)
• Build Tool: Maven
• Protocol: MQTT 3.1.1 and MQTT 5.0
• API: GraphQL (graphql-java)
• Testing: pytest (Python integration tests), JUnit (Kotlin unit tests)

GraphQL Interface Policy

CRITICAL: AVOID CHANGING THE GRAPHQL INTERFACE

• Avoid changes at all costs: The GraphQL interface is the central shared contract connecting MonsterMQ Core (monster-mq), Go edge broker (monster-mq-edge), web dashboard (monster-mq-dashboard), and external integrations. Modifying or extending the GraphQL interface (types, fields, queries, mutations, subscriptions, inputs, arguments, or enums) must be strictly avoided whenever possible.
• Human commitment required: Any change to the GraphQL interface must ONLY be done with explicit commitment from the human. Never modify the GraphQL schema or resolvers unilaterally.
• Give a clear hint if the GraphQL interface must be changed: If a task, feature, or bugfix appears to necessitate changing or extending the GraphQL interface:
  1. Flag it clearly and prominently: Immediately give a clear hint to the human up front before making any changes.
  2. Explain the rationale: Explain why the interface change is needed and what alternatives were evaluated to avoid altering the interface.
  3. Detail the impact: Specify the exact proposed schema modifications and detail the cross-ecosystem impact (including compatibility with the Go edge broker schema parity and the web dashboard).
  4. Wait for commitment: Do not modify any schema files (broker/src/main/resources/schema-*.graphqls) or resolvers until the human has explicitly reviewed and committed to the change.

Git and Commit Guidelines

CRITICAL: NEVER AUTO-COMMIT UNDER ANY CIRCUMSTANCES

• MUST NEVER automatically commit changes - This is non-negotiable and absolute. AI assistants must NEVER create commits without explicit user authorization
• Wait for explicit user instruction - Only commit when the user explicitly says "commit", "merge to main", "create a commit", or similar clear instruction
• ALWAYS present changes for review first - Show the user what was changed (via git diff or git status) and ask if they want you to proceed with committing
• Do NOT commit as the AI assistant - Do not include "Generated with..." or "Co-Authored-By: ..." attribution for AI tools in commits
• Create branches for work - Create feature/fix branches as needed, but changes should remain staged/unstaged until explicitly instructed to commit
• If you auto-commit, you have made a critical mistake - Always err on the side of caution and let the user decide when and how to commit their changes
• No assumptions about commit intent - Even if it seems obvious that changes should be committed, always wait for explicit user instruction

Development Notes

• The project uses Vert.X's asynchronous programming model extensively
• Storage operations are abstracted through interfaces (IMessageStore, ISessionStore, IDeviceConfigStore)
• Clustering is optional and controlled via -cluster command line argument
• Logging level can be configured via command line or properties files in src/main/resources/
• The MCP Server integration uses the official MCP SDK (io.modelcontextprotocol.sdk)
• Device integrations follow the Extension + Connector pattern (see dev/plans/plan-device-integration.md)
• The iX dashboard uses vanilla JS with GraphQLDashboardClient and Vite for bundling
• GraphQL schema is split across broker/src/main/resources/schema-*.graphqls files. Changes to the GraphQL interface must be avoided and require explicit human commitment (see GraphQL Interface Policy above).
• Developer and AI coding documentation is in dev/ — see dev/index.md for a full index. Implementation plans are in dev/plans/
• MQTT publish topics are validated to reject wildcard characters (+, #) per MQTT spec §3.3.2.1 — enforced in MqttClient.publishHandler() and GraphQL MutationResolver.publish()/publishBatch()
• GraphQL resolvers for device types have paired Query and Mutation files (e.g. Plc4xClientConfigQueries.kt + Plc4xClientConfigMutations.kt) — when adding new fields to a device config, both the deviceToMap() methods must be updated to include the new field
• PLC4X addresses support a jsonPath field for extracting values from JSON MQTT payloads when writing to PLC registers (dot-notation path, e.g. data.temperature.value)