vstest

Imported from microsoft/vstest (AGENTS.md). Install upstream with npx skills add microsoft/vstest. Copyright stays with the author.

AGENTS.md — vstest

This file contains project-specific knowledge for AI agents working on the VSTest test platform.

Repository Overview

VSTest is the test platform that powers dotnet test, Visual Studio Test Explorer, and Azure DevOps test tasks. It discovers and executes tests written with MSTest, xUnit, NUnit, and other frameworks.

Architecture

vstest.console (entry point)
  ├── TestRequestManager (orchestration)
  │   ├── ProxyDiscoveryManager ──IPC──► testhost (discovery)
  │   ├── ProxyExecutionManager ──IPC──► testhost (execution)
  │   └── ProxyOperationManager (shared IPC logic)
  ├── Translation Layer (bridges old/new handler interfaces for VS/AzDO)
  └── Data Collectors (code coverage, blame, etc.)

Key IPC boundary: vstest.console and testhost communicate via JSON-RPC over stdin/stdout. Wire format changes must be backward-compatible.

Key Directories

Build

CI builds use -c Release. Always build with Release config before submitting PRs.

-projects is resolved with Resolve-Path, so it takes a path or a glob. A bare nickname such as smoke or htmllogger fails with Cannot find path. Test categories are switches instead: -smokeTest, -integrationTest, -compatibilityTest, -performanceTest.

Filtering by test name is a Windows-only wrapper parameter. test.cmd goes through eng/build.ps1, which owns -filter and the category switches; test.sh calls Arcade's eng/common/build.sh directly and has neither, so on Linux/macOS the filter is passed as an MSBuild property. Do not pass the filter as /p:TestRunnerAdditionalArguments="--filter ..." on Windows — eng/build.ps1 throws "Use --filter instead of passing filter as an additional argument to TestRunnerAdditionalArguments." See CONTRIBUTING.md for the Linux/macOS quoting rules.

test.cmd does not set DOTNET_ROOT, so test executables built against the repo's preview TFM fail with "You must install or update .NET to run this application." This hits unit tests too, not only integration and smoke tests. Set it first:

$env:DOTNET_ROOT = "$PWD\.dotnet"

test.cmd does not restore either. On a fresh clone, build once (./build.cmd -c Release) before running it, or pass -restore -build, otherwise it stops with Toolset version <version> has not been restored.

The .cmd wrappers pass arguments to PowerShell literally. See Wrapper scripts pass arguments literally before changing one.

.github/skills/vstest-build-test/SKILL.md is the fuller build and test reference. Keep it and this section in agreement.

Test Structure

Test projects mirror source projects under test/:

src/Microsoft.TestPlatform.ObjectModel/     → test/Microsoft.TestPlatform.ObjectModel.UnitTests/
src/Microsoft.TestPlatform.CrossPlatEngine/ → test/Microsoft.TestPlatform.CrossPlatEngine.UnitTests/
src/vstest.console/                         → test/vstest.console.UnitTests/

Test categories: Unit (fast, default), Smoke (P0 e2e), Acceptance (full e2e with --integrationTest).

Known Gotchas

Wrapper scripts pass arguments literally

build.cmd, test.cmd, restore.cmd, open-vs.cmd, open-code.cmd, and eng/RestoreInternal.cmd invoke PowerShell with -File, so everything after the script path reaches the target script as a literal argument. The first five call eng/build.ps1, eng/RestoreInternal.cmd calls eng/common/build.ps1.

They previously used the form -command "& """<script>""" %*", which spliced %* into a string that PowerShell then parsed as source code. That caused two problems, both fixed:

• ; in an argument became a statement separator. ./test.cmd -projects "test\A\A.csproj;test\B\B.csproj" ran Build.ps1 -projects test\A\A.csproj and then executed test\B\B.csproj as its own statement. On Windows .csproj is file-associated with Visual Studio, so every entry after the first opened a full IDE. Three agents ran this form at the same time and opened eighteen instances of Visual Studio.
• Exit codes collapsed to 1. eng/build.ps1 ends with exit $LastExitCode to forward the real code, but -command discarded it, so 8 (filter matched no tests) and every other code arrived as 1.

Use -File in any new .cmd wrapper. Do not switch back to -command.

Because arguments are no longer re-parsed, one level of quoting is enough instead of two, for wrapper parameters and /p: MSBuild properties alike. A single pair of quotes now reaches the target script intact:

./test.cmd -c Release -projects "test\Microsoft.TestPlatform.CrossPlatEngine.UnitTests\Microsoft.TestPlatform.CrossPlatEngine.UnitTests.csproj" -filter "FullyQualifiedName~MtpProxyExecutionManagerTests"

eng/common/* comes from Arcade and still uses -command. Do not edit those files here; fix them in the Arcade repository.

Independent of the wrappers: never run a .csproj or .sln path as a command. The path is always an argument, as in dotnet test <path>.csproj.

Binding Redirects

Bumping a netstandard2.0 package cascades: transitive deps need binding redirects in ALL three app.configs (vstest.console/app.config, testhost.x86/app.config, datacollector/app.config). Miss one and you get FileLoadException in net462 DTA hosts.

Package Verification

After packaging changes, regenerate eng/expected-nupkg-file-counts.json and eng/expected-dll-frameworks.json from a clean Release build. Never hand-edit these files.

Assert.Contains

Assert.Contains(expected, actual) — first param is the needle. This is the opposite of old StringAssert. This has been a recurring mistake.

Localization

*.xlf files must be manually edited to match .resx changes. They are NOT auto-generated by the build.

CI

• CI runs on Azure DevOps, not GitHub Actions
• DOTNET_ROLL_FORWARD=LatestMajor masks version mismatches — don't rely on it
• Doc-only PRs skip CI builds
• Windows builds finish first (~15 min), macOS/Ubuntu take longer

Git Workflow

• Never commit to main
• Never force-push PR branches — squash-merge at the end
• Push to fork remote, PR against microsoft/vstest
• Don't create draft PRs — undrafting forces a rebuild

Agentic Workflows (gh-aw)

• Use gh aw secrets set to manage secrets, NOT gh secret set. Plain gh secret set creates the repo secret but gh-aw can't see it.
• Auth is company-token first — no long-lived personal PATs. Copilot inference uses the copilot-requests: write permission (billed to the org Copilot subscription), so COPILOT_GITHUB_TOKEN is no longer referenced by any compiled workflow. Write-backs use an org-owned GitHub App (APP_ID variable + APP_PRIVATE_KEY secret) with ignore-if-missing: true, falling back to GITHUB_TOKEN until an org admin provisions it. See .github/workflows/README.md for the full secrets table and rationale (the Microsoft OSS enterprise now 403s fine-grained PATs older than 8 days).
• lockdown: has been removed repo-wide (deprecated upstream); workflows keep min-integrity: none and use the default GITHUB_TOKEN for MCP reads.
• Workflow source files are .md in .github/workflows/. Compiled .lock.yml files are generated — don't hand-edit them.
• To recompile after editing a workflow: gh aw compile from the repo root.
• .github/* and AGENTS.md are excluded from CI path triggers — editing workflows won't trigger a full build.