Instruction file imported from sachin-khiladi/devops-templates (.github/instructions/deployment-and-supply-chain-security.instructions.md). Copyright stays with the author.

Container Deployment & Supply Chain Security Best Practices

Overview

This guide ensures all container deployments to Azure Container Apps (ACA) and Azure Kubernetes Service (AKS) follow industry-standard supply chain security practices to prevent attacks at deployment time.

Supply Chain Security Pillars

1. Image Integrity Verification

What it does: Validates that the image being deployed is exactly what was built and scanned.

How it works:

• Image digest verification (SHA256 hash)
• Trivy vulnerability scanning (CRITICAL/HIGH severity)
• SBOM (Software Bill of Materials) validation

Tools:

• templates/common/verify-image-v1.0.0.sh - Comprehensive image verification
• templates/common/generate-sbom-v1.0.0.sh - SBOM generation

Best Practice:

# Strict mode: verification MUST pass (production)
IMAGE_VERIFY_MODE=strict verify-image-v1.0.0.sh <image-ref>

# Standard mode: validation with warnings (non-production)
IMAGE_VERIFY_MODE=standard verify-image-v1.0.0.sh <image-ref>

2. Access Control & Approvals

What it does: Ensures only authorized personnel can deploy to production.

Implementation:

• GitHub: Use GitHub Environments with required reviewers
• Azure DevOps: Use Deployment Job environments with approval checks

Production Requirements:

• Minimum 2 approvers for production deployments
• Main branch only policy (enforced by templates)
• Approval metadata captured in deployment attestation

Setup: See .github/instructions/platform-approval-gates-setup.md for step-by-step configuration.

3. Artifact Attestation

What it does: Creates an immutable audit trail of what was deployed.

Captured Details:

{
  "deployment_type": "aks-helm",
  "container_image": "registry/app:digest",
  "release_environment": "prd",
  "deployed_by": "user@org.com",
  "git_commit": "abc123def456",
  "git_branch": "main",
  "image_verified": true,
  "timestamp": "2024-03-26T14:30:00Z"
}

Location:

• ADO: Published as pipeline artifact: deployment-attestation
• GitHub: Uploaded as workflow artifact: deployment-attestation

4. Helm Chart Security (AKS Only)

What it does: Validates Helm chart structure, values, and deployment.

Security Features:

• helm lint validates syntax and structure
• Values validation prevents injection attacks
• Atomic deployments: automatic rollback on failure
• Namespace isolation with labels

Best Practice:

helm upgrade --install <release-name> <chart-path> \
  --validate \
  --atomic \
  --timeout 15m

5. Container Image Build & Push

What it does: Builds, scans, and pushes images securely.

Build Job Features:

• Multi-stage Docker builds
• Security scanning (Trivy) with configurable severity
• SBOM generation (Syft/Trivy)
• Release metadata stamping

Job Parameters:

• containerRegistryUrl - ACR endpoint
• containerImageRepository - Image path (e.g., myapp)
• generateSbom - Enable SBOM generation (recommended: true)
• signImage - Enable image signing with cosign (optional)

Usage:

# ADO
- template: templates/ado/python/job-build-v1.0.0.yml
  parameters:
    containerRegistryUrl: $(containerRegistryUrl)
    containerImageRepository: myapp
    generateSbom: true

# GitHub
- uses: your-org/templates/github/python/workflow-build-v1.0.0.yml@v1
  with:
    container_registry_url: ${{ env.REGISTRY }}
    container_image_repository: myapp
    generate_sbom: true
  secrets:
    registry_username: ${{ secrets.ACR_USERNAME }}
    registry_password: ${{ secrets.ACR_PASSWORD }}

Deployment Templates

Azure Container Apps (ACA) Deployment

Job Template: templates/ado/deployment/job-deploy-aca-v1.0.0.yml
Workflow Template: templates/github/deployment/workflow-deploy-aca-v1.0.0.yml

Features:

• Image integrity verification
• Release environment enforcement (prd=main only)
• Deployment attestation

Required Parameters:

resourceGroupName: 'rg-myapp-dev'
containerAppName: 'ca-myapp-dev'
containerImageRef: 'registry.azurecr.io/myapp:v1.0.0'
releaseEnvironment: 'dev'  # or 'prd'
verifyImage: true

Example ADO Deployment:

- template: templates/ado/deployment/job-deploy-aca-v1.0.0.yml
  parameters:
    resourceGroupName: 'rg-myapp-prd'
    containerAppName: 'ca-myapp-prd'
    containerImageRef: '$(containerRegistry)/myapp:$(Build.BuildId)'
    releaseEnvironment: 'prd'
    verifyImage: true
    deploymentEnvironment: 'aca-prd'

Example GitHub Deployment:

- uses: your-org/templates/github/deployment/workflow-deploy-aca-v1.0.0.yml@v1
  with:
    resource_group_name: rg-myapp-prd
    container_app_name: ca-myapp-prd
    container_image_ref: ${{ env.IMAGE_REF }}
    release_environment: prd
    verify_image: true
    deployment_environment: aca-prd
  secrets:
    azure_credentials: ${{ secrets.AZURE_CREDENTIALS }}

Azure Kubernetes Service (AKS) Helm Deployment

Job Template: templates/ado/deployment/job-deploy-aks-helm-v1.0.0.yml
Workflow Template: templates/github/deployment/workflow-deploy-aks-helm-v1.0.0.yml

Features:

• Image integrity verification
• Helm chart validation and linting
• Kubernetes namespace preparation
• Release environment enforcement
• Rollout status monitoring
• Atomic deployments with auto-rollback

Required Parameters:

resourceGroupName: 'rg-myapp-aks-dev'
kubernetesClusterName: 'aks-myapp-dev'
helmChartPath: 'templates/helm/aks-app-helm-chart'
helmReleaseName: 'myapp-dev'
helmNamespace: 'default'
containerImageRef: 'registry.azurecr.io/myapp:v1.0.0'
releaseEnvironment: 'dev'  # or 'prd'
verifyImage: true
validateHelmChart: true

Example ADO Deployment:

- template: templates/ado/deployment/job-deploy-aks-helm-v1.0.0.yml
  parameters:
    resourceGroupName: 'rg-myapp-aks-prd'
    kubernetesClusterName: 'aks-myapp-prd'
    helmChartPath: 'templates/helm/aks-app-helm-chart'
    helmReleaseName: 'myapp-prd'
    helmNamespace: 'myapp'
    containerImageRef: '$(containerRegistry)/myapp:$(Build.BuildId)'
    releaseEnvironment: 'prd'
    verifyImage: true
    validateHelmChart: true
    deploymentEnvironment: 'aks-prd'
    additionalHelmArgs: '--atomic --timeout 15m'

Example GitHub Deployment:

- uses: your-org/templates/github/deployment/workflow-deploy-aks-helm-v1.0.0.yml@v1
  with:
    resource_group_name: rg-myapp-aks-prd
    kubernetes_cluster_name: aks-myapp-prd
    helm_chart_path: templates/helm/aks-app-helm-chart
    helm_release_name: myapp-prd
    helm_namespace: myapp
    container_image_ref: ${{ env.IMAGE_REF }}
    release_environment: prd
    verify_image: true
    validate_helm_chart: true
    deployment_environment: aks-prd
    additional_helm_args: --atomic --timeout 15m
  secrets:
    azure_credentials: ${{ secrets.AZURE_CREDENTIALS }}

Helm Chart Best Practices

Chart Structure

templates/helm/aks-app-helm-chart/
├── Chart.yaml                 # Chart metadata
├── values.yaml               # Default configuration
├── templates/
│   ├── deployment.yaml       # Kubernetes Deployment
│   ├── service.yaml          # Kubernetes Service
│   ├── _helpers.tpl          # Helm template helpers
│   └── notes.txt             # Post-install notes

Security in values.yaml

# Pod security context (non-root execution)
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 1000
  capabilities:
    drop:
      - ALL

# Resource limits (prevent DoS)
resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi

# Health checks
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Deployment Metadata

Pass CI/CD metadata to Helm values for compliance tracking:

# In deployment template
helm upgrade --install myapp ./chart \
  --set image.repository=$(IMAGE_REPO) \
  --set image.tag=$(IMAGE_TAG) \
  --set releaseEnvironment=prd \
  --set deploymentId=$(BUILD_ID) \
  --set gitCommit=$(GIT_COMMIT) \
  --set gitBranch=$(GIT_BRANCH)

Production Deployment Checklist

Before deploying to production, ensure:

• Image passes Trivy vulnerability scan (no CRITICAL/HIGH findings)
• SBOM generated and retained for compliance
• Deployment environment has approval requirements configured
• Production resource group and cluster are separate from dev/staging
• Helm chart validated with helm lint
• Kubernetes namespace labels applied for policy enforcement
• Database migrations (if any) completed and rolled back tested
• Feature flags configured for safe rollout
• Monitoring and alerting rules active
• Team aware of deployment (Slack notification sent)

Troubleshooting

Image verification fails

Error: Image signature verification failed

Solution: Ensure cosign key is provided if verifySignature=true, or disable signature verification for non-signed images.

Helm deployment times out

Error: Helm upgrade timed out

Solution: Increase timeout: --timeout 20m (default is 10m)

Pod fails to start

kubectl logs -n <namespace> <pod-name>
kubectl describe pod -n <namespace> <pod-name>

Rollout monitoring shows incomplete

kubectl rollout status deployment/<release-name> -n <namespace>

Compliance & Audit

Deployment Attestation provides compliance evidence:

• What was deployed (image, chart, namespace)
• When it was deployed (timestamp)
• Who deployed it (approver)
• What code version (git commit, branch)
• Environment context (dev/staging/prd)

Retention: Store attestation artifacts for compliance period (typically 1-7 years)

Related Documentation

• Platform Approval Gates Setup
• Azure DevOps Template Best Practices
• GitHub Workflow Best Practices
• Terraform Template Best Practices

References

• Helm Best Practices
• Kubernetes Security Policies
• SLSA Framework - Supply chain Levels for Software Artifacts
• Sigstore/Cosign - Container image signing and verification