Custom agent imported from sombaner/agent-buildeer-ai-toolkit (.github/agents/ADOAssementFoundry-agent.agent.md). Copyright stays with the author.

ADO Assement Foundry agent

Plan: Azure DevOps Health Assessment Agent with SharePoint Knowledge Integration

An AI agent that browses Azure DevOps organizations and projects, assesses work item health in Azure Boards, cross-references with SharePoint knowledge repositories, and identifies missing requirements without corresponding work items.

Steps

1. Create Azure DevOps MCP Server Integration

File: src/mcp/azure_devops_integration.py

Implement MCPStdioTool pattern to connect to Azure DevOps, exposing tools for:

• Listing projects/organizations
• Querying work items (bugs, user stories, tasks)
• Fetching work item details with relationships
• Retrieving Azure Boards configurations

Implementation approach: MCPStdioTool wrapping either existing @azure/devops-mcp package or custom Node.js server using azure-devops npm package.

2. Implement SharePoint MCP Server Integration

File: src/mcp/sharepoint_integration.py

Create custom Node.js MCP server in mcp-servers/sharepoint-mcp/ using:

• @microsoft/microsoft-graph-client and msal-node for authentication

Exposing tools for:

• Searching documents
• Reading document content with metadata
• Listing SharePoint lists/libraries
• Extracting requirements from Word/PDF documents with pattern matching

3. Build Work Item Health Assessment Agent

File: src/agent/devops_health_agent.py

Extend existing agent patterns with:

• Multi-MCP tool initialization (Azure DevOps + SharePoint + optional Playwright)
• Assessment logic for:
  • Work item completeness (missing descriptions, acceptance criteria, priorities)
  • Work item staleness (inactive > 30 days)
  • Blocked dependencies
  • Orphaned work items
• Generate health score per work item/project

4. Implement Requirements Gap Analysis Module

File: src/reports/requirements_analyzer.py

Features:

• Extract requirements from SharePoint documents using NLP/keyword matching
• Create requirement fingerprints for deduplication
• Query Azure DevOps for matching work items using semantic similarity
• Identify missing work items for documented requirements
• Generate gap report with confidence scores and suggested work item mappings

5. Extend Configuration Schema

File: src/config.py

Add:

AzureDevOpsConfig:

• organization_url
• project_name
• pat_token
• work_item_types
• area_path

SharePointConfig:

• site_url
• tenant_id
• client_id
• client_secret
• document_libraries
• requirement_patterns (regex for identifying requirements in documents)

6. Create Enhanced Report Generator

File: src/reports/devops_health_report.py

Produce comprehensive health assessment reports (JSON/HTML/Markdown) with sections for:

• Work item health summary with charts
• Requirements gap analysis with missing items list
• Recommendations for action items
• SharePoint documents analyzed metadata
• Exportable to Azure DevOps as new work items or uploaded to SharePoint

Further Considerations

1. MCP Server Implementation Strategy

Options:

• Option A: Search for existing @azure/devops-mcp npm package
• Option B: Build custom Node.js MCP servers for both Azure DevOps and SharePoint following MCP SDK specification
• Option C: Hybrid approach using existing packages where available + custom implementations

Recommendation: Option B for full control over tool definitions and authentication flows.

2. SharePoint Authentication Approach

Options:

• Option A: Service Principal with application permissions (daemon)
• Option B: Delegated permissions with user context
• Option C: Certificate-based authentication

Recommendation: Option A for automated agent scenarios with appropriate Graph API permissions (Sites.Read.All, Files.Read.All).

3. Requirements Extraction Method

Options:

• Option A: Simple keyword/regex patterns for "shall", "must", "requirement"
• Option B: Azure AI Document Intelligence for structured extraction
• Option C: LLM-based extraction using the agent's GPT-4 model

Recommendation: Option C leveraging agent's LLM to understand document context and extract semantic requirements with better accuracy.

4. Work Item Matching Algorithm

How to determine if a SharePoint requirement has a corresponding Azure DevOps work item?

Options:

• Option A: String similarity (fuzzy matching)
• Option B: Embedding-based semantic similarity
• Option C: LLM-powered comparison asking GPT-4 to assess matches

Recommendation: Option B with Option C fallback for high-confidence automated matching with LLM verification for borderline cases.

5. Multi-Project Scope

Question: Should the agent assess a single Azure DevOps project or browse entire organization?

If organization-wide:

• Implement pagination and rate limiting for large-scale analysis
• Consider adding project filtering configuration

6. Report Action Integration

Question: Should the agent automatically create work items for missing requirements or only report them?

If automated creation desired:

• Add confirmation workflow
• Implement dry-run mode for safety

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                 Azure AI Foundry Agent                       │
│                (devops_health_agent.py)                      │
└───────────────────┬─────────────────────────────────────────┘
                    │
        ┌───────────┼───────────┐
        │           │           │
        ▼           ▼           ▼
┌──────────┐  ┌──────────┐  ┌──────────┐
│ Playwright│  │ Azure    │  │SharePoint│
│   MCP    │  │DevOps MCP│  │   MCP    │
└──────────┘  └──────────┘  └──────────┘
     │             │              │
     │             │              │
     ▼             ▼              ▼
┌─────────┐  ┌──────────┐  ┌──────────┐
│ Browser │  │Azure     │  │SharePoint│
│Automation│  │DevOps API│  │Graph API │
└─────────┘  └──────────┘  └──────────┘
                    │              │
                    └──────┬───────┘
                           ▼
                  ┌─────────────────┐
                  │ Requirements    │
                  │ Gap Analyzer    │
                  └────────┬────────┘
                           ▼
                  ┌─────────────────┐
                  │ Health Report   │
                  │   Generator     │
                  └─────────────────┘

Key Dependencies to Add

Python Packages (requirements-agent.txt)

# Azure DevOps
azure-devops>=7.1.0b3

# SharePoint / Microsoft Graph
msal>=1.25.0
msgraph-sdk>=1.2.0
office365-rest-python-client>=2.5.0

# Text Processing for Requirements Extraction
python-docx>=1.0.0
PyPDF2>=3.0.0
beautifulsoup4>=4.12.0

# Semantic Similarity for Matching
sentence-transformers>=2.2.0
scikit-learn>=1.3.0

Node.js Packages (for MCP Servers)

@azure/devops-node-api
@microsoft/microsoft-graph-client
@azure/msal-node
@modelcontextprotocol/sdk

Environment Variables Required

# Azure AI Foundry (Existing)
AZURE_AI_PROJECT_ENDPOINT=
AZURE_AI_MODEL_DEPLOYMENT=

# Azure DevOps (New)
AZURE_DEVOPS_ORG_URL=https://dev.azure.com/your-org
AZURE_DEVOPS_PROJECT=YourProject
AZURE_DEVOPS_PAT=your-pat-token

# SharePoint (New)
SHAREPOINT_SITE_URL=https://yourtenant.sharepoint.com/sites/yoursite
SHAREPOINT_TENANT_ID=your-tenant-id
SHAREPOINT_CLIENT_ID=your-client-id
SHAREPOINT_CLIENT_SECRET=your-client-secret
SHAREPOINT_DOCUMENT_LIBRARY=Documents

# Optional
AZURE_DEVOPS_AREA_PATH=
AZURE_DEVOPS_ITERATION_PATH=
REQUIREMENT_KEYWORDS=shall,must,should,requirement

Success Criteria

1. ✅ Agent successfully connects to Azure DevOps and lists all projects in organization
2. ✅ Agent retrieves all work items from specified project(s)
3. ✅ Agent calculates health metrics for each work item (completeness, staleness, dependencies)
4. ✅ Agent connects to SharePoint and searches requirement documents
5. ✅ Agent extracts requirements from Word/PDF documents using LLM
6. ✅ Agent matches requirements to existing work items with confidence scores
7. ✅ Agent generates comprehensive gap analysis report
8. ✅ Report identifies missing work items with high confidence
9. ✅ Report provides actionable recommendations
10. ✅ Agent can optionally create work items for missing requirements (with confirmation)

Timeline Estimate

• Step 1-2 (MCP Servers): 2-3 days - Custom Node.js MCP server development
• Step 3 (Health Agent): 1-2 days - Agent integration and health assessment logic
• Step 4 (Gap Analysis): 2-3 days - Requirements extraction and matching algorithms
• Step 5 (Configuration): 0.5 days - Configuration schema extension
• Step 6 (Reporting): 1-2 days - Report generation with visualizations
• Testing & Integration: 2-3 days - End-to-end testing and refinement

Total: 9-14 days

Next Actions

1. Confirm implementation strategy choices for the 6 considerations above
2. Set up Azure DevOps test organization/project for development
3. Set up SharePoint site with sample requirement documents
4. Create service principal with appropriate permissions
5. Begin with MCP server implementations (foundational layer)
6. Iteratively build and test each component
7. Create comprehensive integration tests
8. Document usage and deployment procedures