Imported from wieslawsoltes/MediaPlayer (
AGENTS.md). Install upstream withnpx skills add wieslawsoltes/MediaPlayer. Copyright stays with the author.
AGENTS.md
Mission
Implement and evolve a production-grade, no-airspace, GPU-rendered media player control for Avalonia on:
- Windows
- macOS
- Linux
The control must be embeddable in normal Avalonia layouts and support overlays (controls/subtitles/effects) without native child window clipping issues.
Non-Negotiable Requirements
- No airspace gap.
- Rendering remains inside Avalonia visual/compositor tree.
- Use hardware decode where available and prefer native platform decode APIs.
- Keep backend architecture swappable (do not hardwire UI to one media engine).
- Keep API surface simple for app consumers (
Source,Play,Pause,Stop,Seek,Volume,Mute,Loop).
Current Architecture
Projects
src/MediaPlayer.Controls- Reusable control library.
- Contains backend abstraction, LibVLC backend, OpenGL renderer, and
GpuMediaPlayer.
src/MediaPlayer.Demo- Desktop demo app to validate playback and integration.
Key Files
src/MediaPlayer.Controls/GpuMediaPlayer.cs- Public Avalonia control.
- Inherits
OpenGlControlBase. - Exposes control properties and transport methods.
src/MediaPlayer.Controls/Backends/IMediaBackend.cs- Backend contract.
src/MediaPlayer.Controls/Backends/MacOsNativeMediaBackend.cs- macOS native AVFoundation helper backend (non-FFmpeg/non-VLC).
src/MediaPlayer.Controls/Backends/WindowsNativeMediaBackend.cs- Windows native MediaFoundation helper backend (non-FFmpeg/non-VLC).
src/MediaPlayer.Controls/Backends/NativeFramePumpMediaBackend.cs- Shared native helper frame-pump backend base (probe/play/frame process orchestration).
src/MediaPlayer.Controls/Backends/MacOsFfmpegProfileMediaBackend.cs- macOS FFmpeg profile fallback backend (VideoToolbox hint path).
src/MediaPlayer.Controls/Backends/WindowsFfmpegProfileMediaBackend.cs- Windows FFmpeg profile fallback backend (D3D11VA hint path).
src/MediaPlayer.Controls/Backends/LibVlcMediaBackend.cs- Platform-aware LibVLC backend implementation.
src/MediaPlayer.Controls/Backends/FfmpegMediaBackend.cs- Generic FFmpeg fallback backend.
src/MediaPlayer.Controls/Rendering/OpenGlVideoRenderer.cs- GPU draw path for video frames.
src/MediaPlayer.Demo/MainWindow.axaml- Demo shell + transport UI.
Platform Strategy
Backend Selection Order
- macOS:
MacOsNativeMediaBackendMacOsFfmpegProfileMediaBackendLibVlcMediaBackendFfmpegMediaBackend
- Windows:
WindowsNativeMediaBackendWindowsFfmpegProfileMediaBackendLibVlcMediaBackendFfmpegMediaBackend
- Linux:
LibVlcMediaBackendFfmpegMediaBackend
Decode API Preference
- Windows: Media Foundation + D3D11VA
- macOS: AVFoundation / VideoToolbox
- Linux: VAAPI / VDPAU (depends on system drivers)
These are expressed via native helper backends first, then FFmpeg/LibVLC fallback profiles.
Render Strategy
- Decode callbacks deliver frames to control-owned buffers.
- Control uploads frames to an OpenGL texture.
- Texture is rendered by
OpenGlControlBaseinto Avalonia composition surface. - No platform child window embedding is used.
No-Airspace Policy
Do not introduce:
libvlc_media_player_set_hwnd,set_nsobject,set_xwindow.- Native child-host controls for video output.
- Any approach that creates a separate platform surface layered above/below Avalonia content.
Always keep video as texture content rendered by Avalonia-owned composition.
Quality Bar
dotnet build MediaPlayer.slnmust pass.- Control must render within normal Avalonia layout containers.
- Basic playback operations must function in demo:
- open source
- play/pause/stop
- seek
- volume/mute/loop
- Errors must surface via
LastError.
Coding Rules for Contributors
- Preserve public API compatibility unless intentional and documented.
- Keep backend-specific logic out of
GpuMediaPlayerwhen possible. - Add platform-specific behavior in backend/profile layer, not UI layer.
- Maintain thread-safety around frame buffer ownership.
- Never block UI thread in decode callback paths.
- Prefer deterministic disposal and explicit event unsubscription.
Native Runtime Notes
- macOS default native backend:
- Builds a small AVFoundation frame-pump helper with
xcrun swiftcon first run. - Caches helper under
~/Library/Application Support/MediaPlayer/native-helpers/macos. - HTTP/HTTPS sources are cached locally before AVAssetReader playback.
- Builds a small AVFoundation frame-pump helper with
- Windows default native backend:
- Builds a MediaFoundation/WPF frame-pump helper with
dotnet publishon first run. - Caches helper under
%LOCALAPPDATA%/MediaPlayer/native-helpers/windows.
- Builds a MediaFoundation/WPF frame-pump helper with
- LibVLC fallbacks:
- Windows native libs come from
VideoLAN.LibVLC.Windows. - macOS native libs come from
VideoLAN.LibVLC.Mac. - Linux relies on system LibVLC installation.
- Windows native libs come from
macOS Apple Silicon Note
VideoLAN.LibVLC.Mac may still fail on some arm64 hosts due runtime distribution differences.
The default native AVFoundation backend avoids that dependency and is selected first.
Build and Run
dotnet restore MediaPlayer.sln
dotnet build MediaPlayer.sln
dotnet run --project src/MediaPlayer.Demo/MediaPlayer.Demo.csproj
Renderer Selection (Demo)
- Renderer preference can be selected with:
- CLI:
--renderer=auto|opengl|vulkan|metal|software - Environment:
MEDIAPLAYER_RENDERER=auto|opengl|vulkan|metal|software - macOS menu:
View -> Renderer(saved for next launch)
- CLI:
- Platform mapping:
- Windows: Vulkan mode is supported (
Win32RenderingMode.Vulkan). - Linux/X11: Vulkan mode is supported (
X11RenderingMode.Vulkan). - macOS: Avalonia.Native does not expose a Vulkan renderer mode; Vulkan preference maps to Metal/OpenGL/Software fallback order.
- Windows: Vulkan mode is supported (
- macOS
Autoprefers Metal first, then falls back to OpenGL and Software. - Current playback-surface compatibility:
GpuMediaPlayerusesOpenGlControlBase, so runtime renderer is coerced to an OpenGL-capable mode for reliable video rendering. Preferences are still stored and shown for future renderer-path work.
Known Technical Tradeoff (Current)
Current callback-based integration copies decoded frames into control-owned buffers before GPU upload. This preserves no-airspace behavior and broad compatibility, but is not a full zero-copy path.
Audio controls in FFmpeg-based backends are limited (volume/mute are not wired in-process).
Future Extension Path
- Add optional zero-copy GPU interop where backend and platform support it.
- Replace helper-process transport with in-process native API bindings for lower copy overhead.
- Add benchmark and diagnostics hooks for frame timing and drop analysis.
Definition of Done for Feature Work
- Builds cleanly.
- No airspace policy preserved.
- Playback control behavior validated in demo.
- Docs updated (
PLAN.mdand thisAGENTS.mdwhen architecture changes).
