Imported from GuiguiBlitz/audio-effects-gen (AGENTS.md). Install upstream with npx skills add GuiguiBlitz/audio-effects-gen. Copyright stays with the author.

AGENTS.md - Contribution Guide for AI Models & Developers

This document provides guidance for AI models and developers contributing to the audio-effects-gen project. It covers the project architecture, setup, common tasks, and best practices.

Project Overview

audio-effects-gen is a PyTorch-based audio effect generation tool that uses Stability AI's Stable Audio Open model to generate high-quality audio effects from text prompts.

Key Features:

• Text-to-audio generation using Stable Audio Open 1.0
• ROCm support for AMD GPUs (GFX11 architecture)
• NVIDIA CUDA support
• CPU fallback mode
• Half-precision inference for reduced VRAM usage

Tech Stack:

• Python 3.11
• PyTorch 2.5.1
• TorchAudio 2.5.1
• TorchVision (for stable-audio-tools)
• einops (for tensor operations)
• Hugging Face Hub (for model downloads)
• stable-audio-tools (for inference utilities)
• uv (Python package manager)

Environment Setup for Contributors

Prerequisites

• Linux OS (Ubuntu 24.04+ recommended)
• Python 3.11
• Git
• Either NVIDIA GPU with CUDA or AMD GPU with ROCm

Initial Setup

1. Clone and navigate to the project:

cd /workspaces/audio-effects-gen

2. Install base dependencies:

uv sync

3. Install PyTorch with GPU support:

For AMD ROCm (GFX11):

uv pip uninstall torch torchaudio torchvision pytorch-triton-rocm
uv pip install torch==2.5.1+rocm6.2 torchaudio==2.5.1+rocm6.2 torchvision --index-url https://download.pytorch.org/whl/rocm6.2

For NVIDIA CUDA:

uv pip install torch==2.5.1+cu118 torchaudio==2.5.1+cu118 torchvision

4. Authenticate with Hugging Face:

source .venv/bin/activate && python -c "from huggingface_hub import login; login()"

5. Verify setup:

export HSA_OVERRIDE_GFX_VERSION=11.0.0  # AMD only
/workspaces/audio-effects-gen/.venv/bin/python -c "import torch; print(f'GPU: {torch.cuda.is_available()}')"

Running the Application

For AMD ROCm users:

export HSA_OVERRIDE_GFX_VERSION=11.0.0
export HSA_ENABLE_SDMA=0
/workspaces/audio-effects-gen/.venv/bin/python app.py

For NVIDIA CUDA users:

/workspaces/audio-effects-gen/.venv/bin/python app.py

⚠️ IMPORTANT: Never use uv run - it reinstalls dependencies from the lockfile and breaks GPU support. Always use the direct Python path.

Project Architecture

Core Files

app.py - Main application entry point

• Hardware detection (AMD/NVIDIA/CPU)
• Model initialization and loading
• Text-to-audio inference
• Audio processing and export

pyproject.toml - Project configuration

• Dependency declarations (torch/torchaudio/torchvision installed separately via uv pip)
• Python version requirements (3.11.x)
• Project metadata

README.md - User documentation

• Installation instructions
• Usage examples
• Troubleshooting guide

Workflow

1. Hardware Detection → Detects GPU type and applies appropriate optimizations
2. Model Loading → Downloads Stable Audio Open 1.0 from Hugging Face (requires authentication)
3. Conditioning Setup → Defines audio generation prompt and duration
4. Inference → Runs diffusion model with mixed precision
5. Audio Processing → Trims, normalizes, and converts to 16-bit WAV
6. Export → Saves output as output.wav

Common Contribution Tasks

1. Modifying Audio Generation Parameters

File: app.py (lines 35-50)

Parameters:

conditioning = [{
    "prompt": "Your text prompt here",  # The audio description
    "seconds_start": 0,                 # Start time in generated audio
    "seconds_total": duration_seconds   # Total duration to generate
}]

output = generate_diffusion_cond(
    model,
    steps=50,           # More steps = better quality, slower
    cfg_scale=7,        # Guidance scale (higher = more faithful to prompt)
    sigma_min=0.3,      # Noise schedule minimum
    sigma_max=500,      # Noise schedule maximum
    sampler_type="dpmpp-3m-sde",  # Sampling algorithm
    device=device
)

Common Modifications:

• Adjust duration_seconds for longer/shorter audio
• Modify steps for quality vs speed tradeoff
• Change cfg_scale for prompt adherence vs creativity
• Experiment with different sampler_type values

2. Adding Multi-Prompt Support

To generate multiple audio segments from different prompts:

conditioning = [
    {
        "prompt": "First effect",
        "seconds_start": 0,
        "seconds_total": 2
    },
    {
        "prompt": "Second effect",
        "seconds_start": 2,
        "seconds_total": 3
    }
]

3. Improving Memory Efficiency

Current optimizations:

• Half-precision (float16) inference
• torch.inference_mode() for reduced gradients
• torch.autocast() for automatic mixed precision

For limited VRAM:

• Reduce steps (20-30 instead of 50)
• Lower sample_size in model_config
• Use CPU for non-inference tasks

4. Adding Batch Processing

To process multiple prompts efficiently:

conditioning_list = [
    {"prompt": "Effect 1", "seconds_start": 0, "seconds_total": 3},
    {"prompt": "Effect 2", "seconds_start": 0, "seconds_total": 3},
]

for i, conditioning in enumerate(conditioning_list):
    output = generate_diffusion_cond(model, conditioning=[conditioning], ...)
    torchaudio.save(f"output_{i}.wav", output, sample_rate)

5. GPU Compatibility Enhancements

For new hardware support:

1. Update hardware detection in app.py (lines 8-20)
2. Test precision modes (float16 vs float32)
3. Document platform-specific environment variables
4. Add to README.md with test results

Supported precisions:

• torch.float16 - Most GPUs (lower VRAM)
• torch.float32 - Universal but slower
• torch.bfloat16 - Newer GPUs (H100, MI300)

6. Adding Error Handling & Logging

Template for robust error handling:

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

try:
    model, model_config = get_pretrained_model("stabilityai/stable-audio-open-1.0")
except Exception as e:
    logger.error(f"Failed to load model: {e}")
    # Fallback or exit

Testing & Validation

Verification Checklist

• Environment variables set correctly
• GPU detected (torch.cuda.is_available() == True)
• Model downloads successfully (requires HF token)
• Inference completes without memory errors
• Output WAV file has correct duration
• Audio quality is acceptable (no artifacts)

Performance Benchmarking

import time

start = time.time()
output = generate_diffusion_cond(...)
elapsed = time.time() - start
print(f"Generation time: {elapsed:.2f}s for {duration_seconds}s audio")
print(f"Speed ratio: {duration_seconds / elapsed:.2f}x")

Testing on Different GPUs

For AMD ROCm:

• GFX90A (MI250X) - Uses default ROCm
• GFX908 (MI100) - May need HSA_OVERRIDE_GFX_VERSION
• GFX906 (MI50) - Limited support, test float32

For NVIDIA CUDA:

• A100, H100 - Optimal performance
• RTX 3090, 4090 - Good compatibility
• Older cards - May need reduced batch sizes

Code Style & Standards

Python Standards

• Python 3.11+ syntax
• Type hints encouraged but not required
• Max line length: 100 characters
• Follow PEP 8

Documentation

• Add docstrings for functions/classes
• Comment non-obvious logic
• Update README.md for user-facing changes
• Update this AGENTS.md for developer-facing changes

Git Practices

• Descriptive commit messages
• One feature per commit
• Test before committing
• Reference issues when applicable

Common Issues & Solutions

Issue: uv run reinstalls wrong PyTorch

Solution: Use direct Python path instead:

/workspaces/audio-effects-gen/.venv/bin/python app.py

Issue: "Cannot access gated repo" from Hugging Face

Solution: Authenticate before running:

source .venv/bin/activate && python -c "from huggingface_hub import login; login()"

Issue: Out of Memory (OOM)

Solution:

1. Reduce steps to 20-30
2. Lower duration_seconds
3. Use torch.float32 → torch.float16
4. Close other applications

Issue: ROCm not detected on AMD GPU

Solution:

1. Set HSA_OVERRIDE_GFX_VERSION=11.0.0
2. Verify with rocm-smi
3. Reinstall torch ROCm version
4. Check torch.cuda.is_available()

Issue: CUDA not detected on NVIDIA GPU

Solution:

1. Verify CUDA installation: nvidia-smi
2. Reinstall PyTorch with CUDA: uv pip install torch --index-url https://download.pytorch.org/whl/cu118
3. Check torch.cuda.is_available()

Future Enhancement Opportunities

1. CLI Interface - Add argparse for command-line parameters
2. Batch Processing - Process multiple prompts in parallel
3. Real-time Audio - Stream generation instead of offline
4. Config Files - YAML/JSON for prompt management
5. Web API - FastAPI/Flask server wrapper
6. Progress Tracking - Progress bars for long generations
7. Audio Editing - Post-processing effects chain
8. Model Comparison - Support multiple model versions
9. Caching - Cache common generations
10. Monitoring - GPU utilization tracking

Resources

• Stable Audio Docs: https://github.com/Stability-AI/stable-audio-tools
• PyTorch Docs: https://pytorch.org/docs/stable/index.html
• ROCm Docs: https://rocmdocs.amd.com/
• Hugging Face Hub: https://huggingface.co/
• einops: https://einops.readthedocs.io/

Contact & Questions

For questions or issues:

1. Check README.md for setup troubleshooting
2. Review app.py comments for implementation details
3. Consult upstream library documentation
4. File issues with environment/hardware details

Last Updated: January 28, 2026 Tested On: Ubuntu 24.04.3 LTS, Python 3.11, AMD ROCm 6.2