Imported from hemapriyan-rk/vimes-industrial-intrusion-detection (
AGENTS.md). Install upstream withnpx skills add hemapriyan-rk/vimes-industrial-intrusion-detection. Copyright stays with the author.
VIMES — Dual & Multi-Camera Vision Safety System: Agent Guide
This document provides complete instructions, architecture documentation, configuration specifications, and operational workflows for any coding agent working on the VIMES (Vision-based Industrial Monitoring & Emergency Safety System) repository.
1. System Overview & Architecture
VIMES is an industrial-grade edge vision and safety monitoring platform designed to prevent worker-machine collisions and enforce perimeter safety in real-time.
┌────────────────────────────────────────────────────────────────────────┐
│ VIMES DUAL-CAMERA SYSTEM │
└────────────────────────────────────────────────────────────────────────┘
│ │
[CAMERA 1: ALPHA] [CAMERA 2: BRAVO]
(e.g. 10.169.247.7:8080) (e.g. 10.169.247.187:8080)
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ YOLO11 Detection │ │ YOLO11 Detection │
│ ByteTrack Tracking │ │ ByteTrack Tracking │
└──────────┬──────────┘ └──────────┬──────────┘
│ Local Tracks │ Local Tracks
└─────────────────────┬─────────────────────┘
│
▼
┌───────────────────────────────────┐
│ CROSS-CAMERA RE-ID ENGINE │
│ - MobileNetV2 128-d Embeddings │
│ - Cosine Distance Metric │
│ - Spatial-Temporal Gating Matrix │
│ - Persistent Global ID Registry │
└─────────────────┬─────────────────┘
│ Global Tracks
▼
┌───────────────────────────────────┐
│ PER-CAMERA ANALYTICS PIPELINE │
│ - Ground-Plane Homography (m) │
│ - Metric Kinematics (v, a, deg) │
│ - Polygon Zone Classification │
│ - Proximity & TTC Analysis │
│ - Deterministic Risk Engine │
└─────────────────┬─────────────────┘
│
▼
┌───────────────────────────────────┐
│ UNIFIED GRID HUD / DASHBOARD │
│ - Side-by-Side Live Video Grid │
│ - Interactive Hazard Zone Editor │
│ - Real-Time Risk Alarm Banner │
│ - Structured Event Logger │
└───────────────────────────────────┘2. Directory Layout
d:\VIMES\
├── configs/ # Edge server global configurations
│ ├── hardware.yaml # ESP32 serial & sensor thresholds
│ ├── safety.yaml # Risk engine safety & TTC thresholds
│ └── zones.yaml # Global edge-server zones
├── hardware/ # ESP32 acquisition firmware & pinouts
│ ├── esp32/firmware/ # FreeRTOS firmware (temp, vibration, PIR)
│ ├── schematics/ # Hardware wiring diagrams
│ └── wiring/
├── software/
│ ├── analytics/ # Kinematics, trajectory prediction, proximity
│ ├── app/ # Edge application orchestrator
│ ├── fusion/ # Multi-modal sensor fusion engine
│ ├── hardware_interface/ # Serial parser & telemetry models
│ ├── perception/ # Deep learning detectors & ByteTrack
│ ├── safety/ # Deterministic risk engine & reason codes
│ ├── tests/ # Automated unit & integration tests
│ ├── visualization/ # Web Command Center & OpenCV renderers
│ └── vision/ # Dual & Multi-Camera Vision Engine
│ ├── camera_config.py # Camera loader & schema parser
│ ├── main.py # Single-camera vision runner
│ ├── multi_camera_main.py # Dual/Multi-camera vision runner
│ ├── config/
│ │ ├── cameras.json # Dual camera network definitions
│ │ ├── calibration.json # Homography 4-point ground plane
│ │ ├── zones.json # Default hazard zones
│ │ ├── zones_cam1.json # Sector Alpha custom hazard zones
│ │ └── zones_cam2.json # Sector Bravo custom hazard zones
│ ├── core/
│ │ ├── detector.py # YOLO11 inference wrapper
│ │ ├── homography.py # Perspective Ground-Plane Transformer
│ │ ├── kinematics.py # Velocity, acceleration, and bearing
│ │ ├── logger.py # Structured JSONL event logging
│ │ ├── proximity.py # Human-machine pairwise distance & TTC
│ │ ├── reid.py # MobileNetV2 Cross-Camera Re-ID
│ │ ├── risk_engine.py # Score (0-100) and Level (SAFE..CRITICAL)
│ │ ├── tracker.py # ByteTrack multi-object tracker
│ │ ├── visualizer.py # Real-time OpenCV HUD renderer
│ │ └── zones.py # Polygon point-in-polygon classifier
│ └── models/ # YOLO weights (.pt)
└── AGENTS.md # This specification guide3. Dual-Camera Configuration (cameras.json)
The primary configuration for network IP cameras is located at:
software/vision/config/cameras.json
{
"_doc": "Dual camera configuration for VIMES industrial monitoring.",
"layout": {
"columns": 2
},
"cameras": [
{
"name": "CAM_01_ALPHA",
"ip": "10.169.247.7",
"port": 8080,
"endpoint": "/video",
"source": "http://10.169.247.7:8080/video",
"zones_file": "config/zones_cam1.json",
"enabled": true,
"description": "Primary Sector Alpha Camera"
},
{
"name": "CAM_02_BRAVO",
"ip": "10.169.247.187",
"port": 8080,
"endpoint": "/video",
"source": "http://10.169.247.187:8080/video",
"zones_file": "config/zones_cam2.json",
"enabled": true,
"description": "Secondary Sector Bravo Camera"
},
{
"name": "CAM_03_CHARLIE",
"source": "synthetic",
"zones_file": "config/zones_cam3.json",
"enabled": false,
"description": "Synthetic test feed (Disabled)"
}
]
}Camera Parameters
name: Unique display string shown in HUD status badges and event logs.source: Can be:- Full HTTP stream URL:
http://10.169.247.7:8080/video - Full RTSP stream URL:
rtsp://username:password@10.169.247.7:554/live - Local webcam device index:
"0","1" - Video file:
"software/data/test_videos/forklift.mp4" - Synthetic simulation generator:
"synthetic"
- Full HTTP stream URL:
zones_file: Path to camera-specific hazard zones JSON (relative tosoftware/vision).enabled: Boolean (true/false) controlling whether the pipeline captures and renders the camera.
4. Execution Commands for Agents & Operators
A. Run Dual/Multi-Camera Pipeline with ESP32 Hardware (Live Network IP Feeds)
python software/vision/multi_camera_main.py --config software/vision/config/cameras.json --serial-port COM5- Integrates IP streams configured in
cameras.jsonwith physical ESP32 sensor telemetry onCOM5.
B. Run Single-Camera Pipeline (Primary IP Camera / Webcam + ESP32 Hardware)
python software/vision/main.py --serial-port COM5(Automatically connects to the first enabled camera from cameras.json and ESP32 on COM5)
C. Run Synthetic Dual-Camera Test with ESP32 Hardware (Offline / CI Verification)
To test cross-camera Re-ID, grid HUD, and live physical sensor telemetry without physical IP cameras:
python software/vision/multi_camera_main.py --sources synthetic,synthetic --serial-port COM55. Interactive Keyboard & Dashboard Controls
When running the OpenCV visualizer window:
A. View Mode Switching (When NOT in Zone Editor)
| Key | View Name | Description |
|---|---|---|
| W | Toggle Side-by-Side Dual Windows | Toggles between Single Combined Window and Dual Synchronized Windows (Window 1: Cameras Grid, Window 2: Hardware I/O & Manipulator Cockpit) |
| 1 | Master Unified Dashboard | Side-by-side video feeds + Top HUD banner + Bottom Worker Risk Cards ($P_1, P_2$) & Actuator Bar |
| 2 | Cameras Grid Only | Full-screen multi-camera video grid with 3D hazard zones |
| 3 | Hardware I/O & Decision Matrix | Dedicated instrument deck (Temp gauge with $dT/dt$, Vibration bar, PIR beacon, Machine state, Buzzer/Relay status, Worker Risk breakdown table) |
| 4 | Hardware Input Manipulator Deck | Interactive test bench: Override toggle, manual sensor injection controls, 5 scenario presets, and live actuator verification |
B. Hardware Input Manipulator & Actuator Controls
| Key | Action | Description |
|---|---|---|
| M | Toggle Manipulator Mode | Switch between Live ESP32 stream on COM5 and Custom Injection Mode |
| T | Temp +5°C | Increase manipulated temperature by $+5^\circ\text{C}$ |
| t | Temp -5°C | Decrease manipulated temperature by $-5^\circ\text{C}$ |
| V | Cycle Vibration | Cycle vibration intensity (0.05 G $\to$ 0.50 G $\to$ 0.88 G) |
| P | Toggle PIR Motion | Toggle PIR motion presence (ACTIVE $\leftrightarrow$ CLEAR) |
| O | Cycle Scenario Preset | Cycle Presets 1–5 (NORMAL_IDLE, MACHINE_RUNNING, MACHINE_OVERHEAT, VIBRATION_FAULT, EMERGENCY_CRITICAL) |
| R | Reset Emergency Stop | Manually reset latched Emergency Stop in Safety Engine and unlatch relay |
C. Multi-Camera Focus & Interactive Zone Editor
| Key | Action | Description |
|---|---|---|
| Tab | Cycle Camera Focus | Switches active camera focus between CAM 1 and CAM 2 in the grid |
| Click | Focus Camera | Left-clicking any camera pane sets it as the active editing target |
| Z | Enter Zone Editor | Starts interactive polygon hazard zone definition for the active camera |
| 1 | Select SAFE Zone | Green pedestrian safe corridor (in zone editor) |
| 2 | Select MODERATE Zone | Yellow caution / approach lane (in zone editor) |
| 3 | Select HAZARD Zone | Red machine exclusion zone (in zone editor) |
| Left Click | Add Vertex | Places a vertex point for the polygon on the live video |
| Right Click | Undo Vertex | Removes the last placed vertex point |
| Enter | Save Zone | Saves polygon to the active camera's zones_camX.json |
| Esc | Cancel Editor | Discards in-progress polygon edits |
| C | Clear Zones | Deletes all zones for the active camera |
| S | Save Screenshot | Captures timestamped frame to data/events/ |
| Q | Quit Application | Gracefully releases camera captures and closes windows |
6. Cross-Camera Re-ID & Multi-Object Tracking
How Cross-Camera Association Works (software/vision/core/reid.py)
- Local Tracking: ByteTrack assigns local track IDs within each camera stream.
- Feature Extraction: Worker crops ($128 \times 256$) are passed through a MobileNetV2 feature extractor generating 128-dimensional $L_2$-normalized visual embeddings.
- Similarity Gating: Cosine distance matrix is evaluated against active global identities: $$\text{Similarity}(u, v) = \frac{u \cdot v}{|u|_2 |v|_2}$$
- Spatial-Temporal Constraints: A worker cannot jump across non-adjacent cameras in sub-second intervals. Spatial-temporal gating prevents false cross-camera identity switches.
- Global Identity Assignment: If similarity $\ge 0.72$ within spatial-temporal bounds, the existing
Global_IDis maintained across camera views.
7. Testing & Quality Assurance
To execute the test suite across all vision, kinematics, Re-ID, hardware manipulator, and safety modules:
python -m pytest software/tests -vKey Test Modules:
software/tests/safety/test_fusion_decision_engine.py: Tests full 20-Case Sensor-Vision Decision Matrix and temporal hysteresis.software/tests/unit/test_sensor_interpreter.py: Tests thermal derivative $dT/dt$, vibration hysteresis bands, and machine operational states.software/tests/unit/test_hardware_manipulator.py: Tests live serial multiplexing, custom sensor injection, and actuator dispatch.software/tests/vision/test_multi_dashboard_views.py: Tests rendering across all 4 Dashboard Views.software/tests/vision/test_multicam_config.py: Validates camera configuration loading and fallback resolution.software/tests/vision/test_reid_multicam.py: Tests cross-camera feature extraction and global ID matching.software/tests/vision/test_reid_optimizations.py: Tests spatial-temporal gating and embedding caches.software/tests/vision/test_kinematics_homography.py: Tests ground plane perspective transforms and metric velocities.software/tests/vision/test_zones_proximity.py: Tests polygon intersection and TTC calculations.software/tests/vision/test_risk_engine.py: Validates deterministic safety scoring (0–100).