Imported from Yuzhaoxin946/LLMs (AGENTS.md). Install upstream with npx skills add Yuzhaoxin946/LLMs. Copyright stays with the author.

Repository Guidelines

Project Structure & Module Organization

This repository is a small Python package for working with a local vLLM OpenAI-compatible server.

• vllm_api.py contains VLLM_API, an async wrapper around openai.AsyncOpenAI.
• start_vllm.sh starts the local vLLM server with environment-variable configuration.
• test/test_vllm.py is an interactive multi-turn chat smoke test.
• __init__.py exposes the package namespace when importing from the parent directory.

Keep source modules at the repository root unless the package grows enough to justify subpackages. Put tests and manual smoke checks under test/.

Build, Test, and Development Commands

• bash start_vllm.sh starts vLLM using defaults such as MODEL_PATH=/u01/yuzhaoxin/plms/Qwen3-32B and PORT=8331.
• PORT=8332 TENSOR_PARALLEL_SIZE=4 CUDA_VISIBLE_DEVICES=0,1,2,3 bash start_vllm.sh runs the server with explicit hardware settings.
• python test/test_vllm.py Qwen3-32B 8331 starts an interactive client against the served model.
• python -m pytest test runs pytest tests if non-interactive tests are added.

There is no project manifest yet. Use the existing vllm Conda environment unless a future requirements.txt or pyproject.toml is added.

Coding Style & Naming Conventions

Use Python 3.12-compatible syntax, type annotations, and async APIs for network calls. Follow the existing style: 4-space indentation, snake_case functions and variables, uppercase constants or environment names, and short docstrings for public classes or scripts. Prefer explicit mappings and serializable dictionaries at API boundaries.

Shell scripts should use #!/usr/bin/env bash plus set -euo pipefail, and should read configuration from environment variables with sensible defaults.

Testing Guidelines

Favor fast unit tests for request-building helpers and error handling. Use pytest naming conventions: files named test_*.py, test functions named test_*. Keep interactive or GPU-dependent checks clearly separated from automated tests, as test/test_vllm.py requires a running vLLM service.

Commit & Pull Request Guidelines

This directory does not currently contain Git history, so use concise imperative commit messages such as Add vLLM client timeout handling or Document server launch settings. Pull requests should describe the behavior change, list any environment variables or model assumptions, and include the command used to verify the change. For user-visible CLI or chat behavior changes, include a short terminal transcript or screenshot when helpful.

Security & Configuration Tips

Do not commit real API keys, private model paths, or machine-specific secrets. The default vLLM key is EMPTY; keep sensitive overrides in the shell environment. Check GPU-related variables before launching long jobs, especially CUDA_VISIBLE_DEVICES, TENSOR_PARALLEL_SIZE, and GPU_MEMORY_UTILIZATION.