Imported from Divith123/agents-constitution (skills/part-xv-infrastructure/SKILL.md). Install upstream with npx skills add Divith123/agents-constitution --skill part-xv-infrastructure. Copyright stays with the author.

SKILL: Part XV - Infrastructure and Deployment

Preamble

This part of the AI Constitution establishes the fundamental principles, rules, and procedures governing the deployment, maintenance, and operation of AI systems in production environments. Every agent operating within the AI ecosystem shall be bound by these infrastructure mandates to ensure reliability, security, scalability, and observability of all deployed systems.

Part XV-A: Railway Deployment Standards

Article 1501: Purpose and Scope

1.1 This Part establishes the Railway Deployment Standards (RDS) governing all deployment operations for AI systems and related infrastructure.

1.2 The provisions of this Article shall apply to:

All containerized deployments
All Kubernetes-based deployments
All serverless deployments
All traditional server-based deployments
All edge computing deployments

1.3 No deployment shall proceed unless it satisfies all requirements established under this Article.

Article 1502: Deployment Pipeline Architecture

1502.1 Pipeline Stages

Every deployment pipeline shall consist of the following mandatory stages in sequential order:

┌─────────────────────────────────────────────────────────────────────────┐
│                        DEPLOYMENT PIPELINE ARCHITECTURE                  │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐             │
│  │  BUILD  │───▶│  TEST   │───▶│ STAGING │───▶│PRODUCTION│             │
│  └─────────┘    └─────────┘    └─────────┘    └─────────┘             │
│       │            │             │              │                      │
│       ▼            ▼             ▼              ▼                      │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐    ┌─────────┐                  │
│  │Artifact │  │ Quality │  │  Smoke  │    │ Canary/ │                  │
│  │Registry │  │  Gate   │  │  Tests  │    │ Rolling │                  │
│  └─────────┘  └─────────┘  └─────────┘    └─────────┘                  │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

1502.2 Stage Definitions

Stage	Duration	Failure Action	Rollback Required
Build	≤ 10 min	Abort pipeline	N/A
Test	≤ 30 min	Abort pipeline	N/A
Staging	≤ 15 min	Abort pipeline	Yes
Production	Continuous	Auto-rollback	Yes

1502.3 Gate Requirements

Each gate shall enforce:

100% test pass rate for unit tests
100% test pass rate for integration tests
0 critical security vulnerabilities
0 high-severity security vulnerabilities (configurable threshold)
Successful health check completion
Successful smoke test completion

Article 1503: Deployment Strategies

1503.1 Canary Deployment

┌─────────────────────────────────────────────────────────────────┐
│                    CANARY DEPLOYMENT FLOW                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   v1 (100%) ──────────────────────────────────────────────────▶│
│                                                                  │
│   v2 (5%)  ──────────────────┐                                   │
│                              │                                   │
│                         ┌────▼────┐                              │
│                         │ Monitor │                              │
│                         │  30 min │                              │
│                         └────┬────┘                              │
│                              │                                   │
│              ┌───────────────┼───────────────┐                   │
│              ▼               ▼               ▼                   │
│         Success          Warning         Failure                │
│         (15%)            (30%)          (Rollback)              │
│              │               │                                  │
│              └───────────────┼───────────────┘                   │
│                              ▼                                   │
│                         v2 (100%)                                │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

1503.2 Rolling Deployment

Maximum unavailable replicas: 25%
Maximum surge: 25%
Health check interval: 10 seconds
Failure threshold: 3 consecutive failures

1503.3 Blue-Green Deployment

Parallel environment required
Traffic switchover time: < 60 seconds
Instant rollback capability: Mandatory
State synchronization verification: Required before switch

Article 1504: Health Check Standards

1504.1 Health Check Types

health_checks:
  startup:
    failure_threshold: 30
    period_seconds: 10
    success_threshold: 1
  
  readiness:
    failure_threshold: 3
    period_seconds: 5
    success_threshold: 1
    initial_delay_seconds: 5
  
  liveness:
    failure_threshold: 3
    period_seconds: 10
    success_threshold: 1

1504.2 Health Check Endpoints

Every deployed service MUST expose:

Endpoint	Path	Purpose	Auth Required
Liveness	/health/live	Container health	No
Readiness	/health/ready	Service ready	No
Startup	/health/startup	Init complete	No
Metrics	/metrics	Observability	Yes

1504.3 Health Check Response Format

{
  "status": "healthy|degraded|unhealthy",
  "timestamp": "2026-04-12T00:00:00Z",
  "version": "1.0.0",
  "checks": {
    "database": {"status": "pass", "latency_ms": 5},
    "cache": {"status": "pass", "latency_ms": 2},
    "external_api": {"status": "pass", "latency_ms": 45}
  },
  "metadata": {
    "deployment_id": "abc123",
    "environment": "production"
  }
}

Article 1505: Monitoring Requirements

1505.1 Required Metrics

Every deployment MUST emit the following metrics:

Metric Category	Metrics	Frequency
Latency	p50, p95, p99, p99.9	10 seconds
Traffic	requests/sec, connections	10 seconds
Errors	4xx, 5xx, timeout rate	10 seconds
Saturation	CPU%, memory%, disk%, GPU%	10 seconds
Availability	uptime percentage	1 minute

1505.2 Alerting Thresholds

Severity	Condition	Response Time
Critical	Service down	Immediate
Critical	Error rate > 5%	< 1 minute
High	Error rate > 1%	< 5 minutes
Medium	Latency p99 > 2s	< 15 minutes
Low	Resource > 80%	< 1 hour

Part XV-B: Docker and Container Rules

Article 1510: Container Image Standards

1510.1 Image Naming Convention

registry/organization/service:version[-variant][.digest]

Examples:

gcr.io/company/ai-service:v1.0.0
ghcr.io/company/ml-pipeline:v2.1.0-gpu
registry.example.com/dept/agent:v1.0.0.abc123def

1510.2 Base Image Requirements

Requirement	Specification
Base image source	Official or verified sources only
Image scanning	Trivy or equivalent required
Critical CVEs	Zero tolerance
High CVEs	Must be resolved within 30 days
Medium CVEs	Must be resolved within 90 days
Image age	Base image < 1 year old

1510.3 Prohibited in Images

The following are STRICTLY PROHIBITED in production container images:

Hardcoded credentials or secrets
Debug flags enabled in production binaries
Default passwords
SSH server (unless explicitly required)
Package manager cache (must be cleaned)
Root user execution (unless documented exception)
Non-essential packages

Article 1511: Multi-Stage Build Requirements

1511.1 Mandatory Multi-Stage Structure

# Stage 1: Builder
FROM python:3.11-slim AS builder

WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir --target=/deps requirements.txt

# Stage 2: Runtime
FROM python:3.11-slim AS runtime

# Security: Run as non-root
RUN groupadd --gid 1000 appgroup && \
    useradd --uid 1000 --gid appgroup --shell /bin/bash appuser

WORKDIR /app

# Copy only necessary artifacts
COPY --from=builder /deps /app/lib
COPY --chown=appuser:appgroup ./app /app

# Set environment
ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PATH=/app/bin:$PATH

USER appuser

EXPOSE 8080

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
    CMD python /app/health.py || exit 1

ENTRYPOINT ["python", "/app/main.py"]

1511.2 Build Cache Optimization

┌─────────────────────────────────────────────────────────────────┐
│                   BUILD CACHE LAYER ORDER                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Layer 1: Base dependencies (rarely change)                     │
│   Layer 2: System packages                                       │
│   Layer 3: Python/Node/etc packages                              │
│   Layer 4: Application code (frequently changes)                │
│                                                                  │
│   OPTIMIZATION: Place rarely-changing content at top             │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

1511.3 Image Size Limits

Environment	Maximum Size	Rationale
Production	500 MB	Security, deploy speed
Development	2 GB	Flexibility
Testing	1 GB	CI efficiency

Article 1512: Container Security Standards

1512.1 Security Controls Matrix

Control	Requirement	Verification
Non-root user	Mandatory	Dockerfile audit
Read-only filesystem	Strongly recommended	Runtime enforcement
No new privileges	Mandatory	Security scan
Seccomp profile	Required for privileged containers	Runtime audit
AppArmor/SELinux	Required for multi-tenant	Policy verification
Capabilities	Drop all by default	Runtime audit

1512.2 Runtime Security

security_context:
  run_as_non_root: true
  run_as_user: 1000
  run_as_group: 1000
  fs_group: 1000
  read_only_root_filesystem: true
  allow_privilege_escalation: false
  se_linux_options:
    level: "s0:c123,c456"
  seccomp_profile:
    type: "RuntimeDefault"
  capabilities:
    drop:
      - ALL

1512.3 Secret Handling

Secrets NEVER in environment variables
Secrets mounted as tmpfs volumes only
Secrets rotated within 24 hours of compromise
No secrets in image layers or build context
Kubernetes secrets encrypted at rest

Part XV-C: CI/CD Pipeline Standards

Article 1520: Pipeline Configuration

1520.1 Required Pipeline Stages

┌─────────────────────────────────────────────────────────────────────────┐
│                        COMPLETE CI/CD PIPELINE                           │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐        │
│  │  CODE   │  │  BUILD  │  │  TEST   │  │ SECURITY │  │ DEPLOY  │        │
│  │ PUSH    │──▶│ IMAGE  │──▶│ SUITE   │──▶│ SCAN     │──▶│ STAGE   │        │
│  └─────────┘  └─────────┘  └─────────┘  └─────────┘  └─────────┘        │
│       │                                                   │             │
│       │                                           ┌───────┴───────┐      │
│       │                                           │               │      │
│       ▼                                           ▼               │      │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐    ┌─────────┐  ┌─────────┐ │      │
│  │ LINT    │  │ TYPE    │  │  UNIT   │    │  IMAGE  │  │INTEGRATE│ │      │
│  │ CHECKS  │──▶│ CHECK   │──▶│ TESTS   │───▶│ SCAN    │──▶│  TEST   │ │      │
│  └─────────┘  └─────────┘  └─────────┘    └─────────┘  └─────────┘ │      │
│                                                                    │      │
│                                       ┌────────────────────────────┘      │
│                                       │                                    │
│                                       ▼                                    │
│                               ┌───────────────┐                           │
│                               │   PRODUCTION  │                           │
│                               │   DEPLOYMENT  │                           │
│                               └───────────────┘                           │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

1520.2 Stage Timing Requirements

Stage	Maximum Duration	SLI Target
Lint	2 min	< 30 sec
Type Check	5 min	< 2 min
Unit Tests	15 min	< 5 min
Security Scan	10 min	< 3 min
Integration Tests	30 min	< 10 min
Build Image	10 min	< 5 min
Deploy Staging	5 min	< 2 min
Deploy Production	15 min	< 5 min

1520.3 Quality Gates

Gate	Metric	Threshold	Action on Failure
Code Coverage	Line coverage	≥ 80%	Block merge
Code Coverage	Branch coverage	≥ 70%	Block merge
Code Quality	SonarQube	Grade A	Block merge
Security	CVE count	0 Critical, 0 High	Block merge
Performance	Regression	≤ 5% degradation	Warning
Flakiness	Test stability	< 1% flakiness	Block merge

Article 1521: Testing Requirements

1521.1 Test Coverage Requirements

Test Type	Minimum Coverage	Location
Unit Tests	80% lines, 70% branches	`/tests/unit`
Integration Tests	60% paths	`/tests/integration`
E2E Tests	Critical paths	`/tests/e2e`
Performance Tests	SLI/SLO defined	`/tests/performance`
Security Tests	OWASP Top 10	`/tests/security`

1521.2 Test Execution Matrix

┌─────────────────────────────────────────────────────────────────┐
│                    TEST EXECUTION STRATEGY                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  PUSH TO BRANCH:                                                 │
│  ├── Lint (parallel)                                            │
│  ├── Type Check (parallel)                                       │
│  ├── Unit Tests (parallel)                                       │
│  └── Security Scan (sequential)                                  │
│                                                                  │
│  PUSH TO MAIN:                                                   │
│  ├── All branch checks                                           │
│  ├── Build Image                                                 │
│  ├── Integration Tests                                          │
│  └── Deploy to Staging                                           │
│                                                                  │
│  MERGE TO MAIN:                                                 │
│  ├── All previous checks                                         │
│  ├── Performance Tests                                          │
│  ├── Smoke Tests                                                 │
│  └── Deploy to Production                                        │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

1521.3 Flaky Test Prevention

test_flakiness_rules:
  max_retries: 2
  flaky_detection:
    enabled: true
    threshold: 3_failures_in_10_runs
  quarantine:
    auto_quarantine_after: 5_failures
    review_required_for_restore: true
  execution:
    isolation_level: process
    parallel_workers: 4
    shard_count: 10

Article 1522: Deployment Automation

1522.1 Automated Rollback Triggers

Condition	Threshold	Action
Health check failure	3 consecutive	Auto-rollback
Error rate spike	> 5% in 2 min	Auto-rollback
Latency degradation	p99 > 3x baseline	Auto-rollback
Memory leak detection	> 90% in 5 min	Auto-rollback
Custom metric breach	Configurable	Auto-rollback

1522.2 Deployment Approval Workflow

┌─────────────────────────────────────────────────────────────────┐
│                  APPROVAL WORKFLOW                               │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Automated Check ──▶ Quality Gate ──▶ Approval ──▶ Deploy     │
│         │                  │              │                      │
│         ▼                  ▼              ▼                      │
│    All tests pass    All metrics    [ ] Manager                 │
│    No blockers       meet SLA       [ ] Security                │
│                                        [ ] SRE                   │
│                                                                  │
│  Approval Requirements:                                          │
│  - Staging: 1 approval (automated)                               │
│  - Production: 2 approvals (1 manual required)                   │
│  - Hotfix: Emergency approval (1 required)                       │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Part XV-D: Monitoring and Observability

Article 1530: Observability Pillars

1530.1 The Three Pillars

Every system MUST implement the following observability pillars:

┌─────────────────────────────────────────────────────────────────┐
│                   THREE PILLARS OF OBSERVABILITY                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│                           ┌─────────────┐                        │
│                           │   TRACES    │                        │
│                           │   (Jaeger)  │                        │
│                           └──────┬──────┘                        │
│                                  │                                │
│         ┌────────────────────────┼────────────────────────┐     │
│         │                        │                        │     │
│         ▼                        ▼                        ▼     │
│  ┌─────────────┐          ┌─────────────┐          ┌─────────────┐
│  │   METRICS   │          │    LOGS     │          │   TRACES    │
│  │ (Prometheus)│          │   (Loki)    │          │  (Tempo)    │
│  └─────────────┘          └─────────────┘          └─────────────┘
│                                                                  │
│  Each pillar is REQUIRED for complete observability              │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

1530.2 Metrics Implementation

# Required metrics pattern
from prometheus_client import Counter, Histogram, Gauge

# Request metrics
request_counter = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

request_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request duration',
    ['method', 'endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
)

# Business metrics
active_users = Gauge(
    'active_users_current',
    'Number of currently active users'
)

task_completion = Counter(
    'tasks_completed_total',
    'Total tasks completed',
    ['status', 'type']
)

1530.3 Logging Standards

Level	Usage	Example
DEBUG	Diagnostic information	Variable values, flow details
INFO	Normal operation	"Request processed", "User logged in"
WARNING	Potential issues	"High latency detected", "Retry successful"
ERROR	Errors requiring attention	"Connection failed", "Validation error"
CRITICAL	System-level failures	"Database unreachable", "Out of memory"

{
  "timestamp": "2026-04-12T00:00:00.000Z",
  "level": "INFO",
  "service": "ai-agent",
  "trace_id": "abc123def456",
  "span_id": "xyz789",
  "message": "Request processed successfully",
  "context": {
    "user_id": "user_123",
    "operation": "code_generation",
    "duration_ms": 150,
    "tokens_used": 500
  },
  "metadata": {
    "version": "1.0.0",
    "environment": "production"
  }
}

Article 1531: Service Level Objectives

1531.1 SLI Definitions

Service	Availability	Latency (p95)	Latency (p99)
API Gateway	99.95%	< 100ms	< 500ms
AI Inference	99.9%	< 2s	< 5s
Data Processing	99.5%	< 10s	< 30s
Background Jobs	99.0%	< 60s	< 300s

1531.2 SLO Error Budget

┌─────────────────────────────────────────────────────────────────┐
│                    ERROR BUDGET POLICY                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Monthly SLO Target: 99.9%                                       │
│  Maximum Allowed Downtime: 43.83 minutes/month                   │
│                                                                  │
│  Error Budget Burn Rate:                                         │
│  - Fast burn (>14.4%/day): Alert immediately                     │
│  - Medium burn (>6%/day): Alert within 1 hour                    │
│  - Slow burn (<6%/day): Weekly review                            │
│                                                                  │
│  Budget Actions:                                                 │
│  - 50% remaining: Increase testing rigor                         │
│  - 25% remaining: Feature freeze, focus on reliability          │
│  - 10% remaining: All-hands reliability sprint                   │
│  - 0% remaining: Immediate escalation                            │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Article 1532: Distributed Tracing

1532.1 Trace Propagation

Every service MUST propagate trace context:

# Trace context propagation
headers = {
    'traceparent': f'00-{trace_id}-{span_id}-01',
    'tracestate': f'congo=t61rcWkgMzE'
}

# Always pass headers to downstream services
response = requests.get(
    'http://downstream-service/api',
    headers=inject_trace_context(headers)
)

1532.2 Span Attributes

Attribute	Required	Description
service.name	Yes	Service identifier
operation.name	Yes	Operation being traced
trace.id	Yes	Unique trace identifier
span.id	Yes	Unique span identifier
user.id	Conditional	User context if applicable
db.system	Conditional	Database type for DB operations
http.method	Conditional	HTTP method for HTTP operations

Part XV-E: Scaling and Performance

Article 1540: Horizontal Scaling

1540.1 Scaling Triggers

Metric	Scale Up	Scale Down
CPU	> 70% for 2 min	< 30% for 5 min
Memory	> 80% for 2 min	< 50% for 5 min
Request Queue	> 100 for 1 min	< 10 for 5 min
Custom Metric	Configurable	Configurable

1540.2 Scaling Configuration

horizontal_pod_autoscaler:
  min_replicas: 2
  max_replicas: 50
  scale_up_stabilization: 0 seconds
  scale_down_stabilization: 300 seconds
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          average_utilization: 70
    - type: Pods
      pods:
        metric:
          name: queue_depth
        target:
          type: AverageValue
          average_value: "100"
  behavior:
    scale_up:
      stabilization_window_seconds: 0
      policies:
        - type: Percent
          value: 100
          period_seconds: 15
    scale_down:
      stabilization_window_seconds: 300
      policies:
        - type: Percent
          value: 10
          period_seconds: 60

1540.3 Scaling Limits

Component	Minimum	Maximum	Notes
API Pods	2	100	HA requirement
Worker Pods	1	200	Job queue dependent
ML Inference Pods	1	50	GPU availability
Database Connections	10	1000	Connection pool

Article 1541: Performance Optimization

1541.1 Response Time Budget

┌─────────────────────────────────────────────────────────────────┐
│                 RESPONSE TIME BUDGET                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Total Budget: 2000ms                                            │
│                                                                  │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │   Gateway    │  │   Service    │  │  Database    │          │
│  │    50ms      │  │   1000ms     │  │    500ms     │          │
│  │    (2.5%)    │  │    (50%)     │  │    (25%)     │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
│                                                                  │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │   External   │  │    Cache     │  │   Queue     │          │
│  │    API       │  │    200ms     │  │    250ms    │          │
│  │    (10%)     │  │    (10%)     │  │    (2.5%)   │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
│                                                                  │
│  Buffer: 250ms (12.5%) for variance and retries                  │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

1541.2 Caching Strategy

Layer	TTL	Invalidation	Use Case
CDN	1 hour	Purge on deploy	Static assets
Redis	5 min	Event-driven	Session data
In-memory	30 sec	LRU eviction	Frequent reads
Query	1 min	Time-based	Database queries

Part XV-F: Environment Configuration

Article 1550: Environment Definitions

1550.1 Environment Matrix

Environment	Purpose	Data	Access
Local	Development	Mock/Sample	All devs
Dev	Integration testing	Anonymized	Dev team
Staging	Pre-production	Realistic	QA, Dev leads
Production	Live traffic	Real	Limited

1550.2 Configuration Management

# config.yaml - Environment-aware configuration
service:
  name: ai-agent
  version: ${VERSION:-1.0.0}
  environment: ${ENVIRONMENT:-development}

database:
  host: ${DB_HOST:-localhost}
  port: ${DB_PORT:-5432}
  name: ${DB_NAME:-ai_agent}
  pool:
    min: ${DB_POOL_MIN:-5}
    max: ${DB_POOL_MAX:-20}
    timeout: ${DB_POOL_TIMEOUT:-30s}

cache:
  type: redis
  host: ${REDIS_HOST:-localhost}
  port: ${REDIS_PORT:-6379}
  ttl: ${CACHE_TTL:-300s}

ai:
  model_endpoint: ${MODEL_ENDPOINT}
  api_key: ${AI_API_KEY}
  max_tokens: ${AI_MAX_TOKENS:-4096}
  temperature: ${AI_TEMPERATURE:-0.7}
  timeout: ${AI_TIMEOUT:-60s}

observability:
  metrics_enabled: ${METRICS_ENABLED:-true}
  tracing_enabled: ${TRACING_ENABLED:-true}
  log_level: ${LOG_LEVEL:-info}

Article 1551: Secrets Management

1551.1 Secrets Hierarchy

┌─────────────────────────────────────────────────────────────────┐
│                    SECRETS MANAGEMENT HIERARCHY                   │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Level 1: Vault/Key Management System                            │
│  ├── Master encryption key                                       │
│  ├── Access policies                                             │
│  └── Audit logs                                                  │
│  Level 2: Secret Storage                                         │
│  ├── API keys                                                    │
│  ├── Database credentials                                        │
│  ├── TLS certificates                                            │
│  └── Signing keys                                                │
│  Level 3: Application Secrets                                    │
│  ├── Service-to-service auth                                      │
│  ├── Third-party integrations                                    │
│  └── User-facing tokens                                          │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

1551.2 Secret Rotation Policy

Secret Type	Rotation Frequency	Automatic	Manual Approval
Database passwords	90 days	Yes	No
API keys (external)	30 days	Yes	No
API keys (internal)	90 days	Yes	No
TLS certificates	30 days	Yes	No
Signing keys	365 days	Yes	Yes
Service account keys	90 days	Yes	No

1551.3 Secret Handling Code

# PROHIBITED: Never do this
config = {
    "api_key": "sk-1234567890abcdef",  # ❌ HARDCODED
    "password": os.getenv("PASS")       # ⚠️ In environment variable
}

# REQUIRED: Use secret management
from secret_manager import get_secret

config = {
    "api_key": get_secret("api_key", version="latest"),
    "database_url": get_secret("database_url")
}

Part XV-G: Backup and Disaster Recovery

Article 1560: Backup Requirements

1560.1 Backup Types and Schedules

Data Type	Backup Frequency	Retention	RTO	RPO
Database (Primary)	Continuous WAL + Daily Full	30 days	1 hour	1 hour
Database (Config)	Weekly	90 days	4 hours	1 week
File Storage	Daily incremental, Weekly full	90 days	4 hours	24 hours
ML Models	Per version	1 year	2 hours	N/A
Secrets	Per change	1 year	15 min	Immediate

1560.2 Backup Verification

┌─────────────────────────────────────────────────────────────────┐
│                   BACKUP VERIFICATION PIPELINE                   │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐       │
│  │ BACKUP  │───▶│ VERIFY  │───▶│ RESTORE │───▶│  TEST   │       │
│  │ CREATE  │    │ CHECKSUM│    │  TEST   │    │  QUERY  │       │
│  └─────────┘    └─────────┘    └─────────┘    └─────────┘       │
│       │                           │                               │
│       ▼                           ▼                               │
│  ┌─────────┐                ┌─────────┐                         │
│  │ NOTIFY  │                │  ALERT  │                         │
│  │ SUCCESS │                │ FAILURE │                         │
│  └─────────┘                └─────────┘                         │
│                                                                  │
│  Verification must complete within 1 hour of backup completion   │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Article 1561: Disaster Recovery

1561.1 Recovery Procedures

Scenario	RTO	RPO	Procedure
Single pod failure	5 min	0	Auto-restart
Node failure	15 min	0	Pod reschedule
AZ failure	1 hour	1 hour	Failover
Region failure	4 hours	4 hours	DR activation
Data corruption	2 hours	< 1 hour	Restore from backup

1561.2 DR Test Schedule

Test	Frequency	Participants	Documentation
Backup restore	Weekly	DevOps	Required
Failover drill	Monthly	SRE + Dev	Required
Full DR exercise	Quarterly	All teams	Required + Review
Chaos injection	Weekly	SRE	Optional

Part XV-H: Rollback Procedures

Article 1570: Rollback Decision Criteria

1570.1 Automatic Rollback Triggers

rollback_conditions:
  - name: health_check_failure
    metric: health_check_success_rate
    threshold: < 0.5
    duration: 1 minute
    action: auto_rollback
  
  - name: error_rate_spike
    metric: error_rate_5xx
    threshold: > 0.05
    duration: 2 minutes
    action: auto_rollback
  
  - name: latency_degradation
    metric: p99_latency
    threshold: > 3x_baseline
    duration: 5 minutes
    action: auto_rollback
  
  - name: memory_leak
    metric: memory_usage_percent
    threshold: > 95
    duration: 5 minutes
    action: auto_rollback
  
  - name: custom_failure
    metric: business_metric_failure
    threshold: any
    duration: 0
    action: alert_and_rollback

1570.2 Rollback Time Targets

Deployment Type	Rollback Target	SLA
Canary	< 5 minutes	99%
Rolling	< 10 minutes	99%
Blue-Green	< 2 minutes	99.9%
Emergency	< 1 minute	99.99%

Article 1571: Rollback Execution

1571.1 Rollback Procedure

# Step 1: Verify rollback necessity
kubectl describe deployment ai-agent | grep -A 5 "Conditions:"

# Step 2: List recent revisions
kubectl rollout history deployment/ai-agent

# Step 3: Execute rollback
kubectl rollout undo deployment/ai-agent [--to-revision=N]

# Step 4: Monitor rollback
kubectl rollout status deployment/ai-agent

# Step 5: Verify health
kubectl get pods -l app=ai-agent
kubectl logs -l app=ai-agent --tail=100

# Step 6: Update incident ticket
# Mark rollback complete in incident management system

1571.2 Post-Rollback Actions

Action	Responsible	Deadline
Incident documentation	On-call engineer	4 hours
Root cause analysis	Engineering lead	24 hours
Fix implementation	Feature owner	48 hours
Verification redeploy	DevOps	72 hours

Part XV-I: Appendices

Appendix A: Checklist for Production Deployment

PRE-DEPLOYMENT CHECKLIST
═══════════════════════════════════════════════════════════════════

□ Code Review
  □ All changes reviewed by >= 1 peer
  □ No hardcoded secrets
  □ No TODO/FIXME comments in critical paths
  □ Security scan passed (0 Critical, 0 High CVEs)

□ Testing
  □ Unit tests pass (100%)
  □ Integration tests pass (100%)
  □ E2E tests pass (100%)
  □ Performance tests within SLAs
  □ Load tests passed

□ Configuration
  □ All env vars documented
  □ Secrets configured in vault
  □ Feature flags set correctly
  □ Monitoring/dashboards configured

□ Documentation
  □ API docs updated
  □ Runbook updated
  □ Deployment notes added

□ Communication
  □ Team notified
  □ Stakeholders aware
  □ Rollback plan confirmed
  □ On-call engineer ready

□ Approval
  □ Engineering lead approval
  □ Product manager aware (if user-facing)
  □ Security review (if applicable)

SIGN-OFF: _________________ DATE: _____________

Appendix B: Glossary

Term	Definition
RTO	Recovery Time Objective - Maximum acceptable downtime
RPO	Recovery Point Objective - Maximum acceptable data loss
SLI	Service Level Indicator - Measurable metric
SLO	Service Level Objective - Target value for SLI
SLA	Service Level Agreement - Contractual obligation
Canary	Deployment strategy releasing to subset of users
Blue-Green	Deployment maintaining two identical environments
Rolling	Gradual replacement of pods one by one
HPA	Horizontal Pod Autoscaler
VPA	Vertical Pod Autoscaler

Enforcement

Any agent found in violation of these infrastructure rules shall be subject to the enforcement mechanisms established in Part XVII of this Constitution.

End of Part XV: Infrastructure and Deployment

About