teams-ui-automation

name	teams-ui-automation
description	Hybrid Teams automation using Playwright MCP (primary), keyboard shortcuts (secondary), and UIA window management (tertiary). Handles Teams operations not available via Graph API - app installation, tab management, connector setup, UI navigation.
triggers	["teams ui","install teams app","add teams tab","teams connector","teams desktop","teams automation","ui automation teams","teams playwright"]
confidence	medium

Teams UI Automation Skill (Hybrid Approach)

Overview

A hybrid three-layer automation skill designed to overcome the limitation that new Microsoft Teams (ms-teams.exe) runs inside an Edge WebView, making traditional UI Automation unable to access the actual Teams UI elements.

Architecture: Three Layers

Layer 1 (Primary): Playwright MCP on teams.microsoft.com

• Agents use existing Playwright MCP tools to automate teams.microsoft.com
• Full DOM access with self-healing selectors (data-tid, aria-label, CSS, text)
• Web version has identical features to desktop
• Most reliable approach for complex operations

Layer 2 (Secondary): Keyboard Shortcuts via PowerShell

• Quick navigation using documented Teams shortcuts
• SendInput API for reliable keyboard input
• PowerShell functions in Teams-UIA.ps1
• Good for simple navigation when Playwright overhead is unnecessary

Layer 3 (Tertiary): UIA for Window Management

• Process detection and window handle retrieval
• Window state (minimized, maximized, focused)
• Version detection for cache validation
• NOT used for in-app element discovery (proven not to work with WebView)

When to Use

✅ USE THIS SKILL FOR:

• Installing Teams apps to specific teams/channels
• Adding tabs to channels (Wiki, Planner, custom apps, website tabs)
• Configuring connectors that lack Graph API support
• UI-based operations not exposed via API
• Navigating Teams UI programmatically
• Testing Teams UI workflows

❌ DO NOT USE FOR:

• Sending messages (use Teams MCP PostMessage)
• Reading messages (use Teams MCP ListChatMessages)
• Creating teams/channels (use Teams MCP CreateChannel)
• Managing members (use Teams MCP)
• Any operation where Graph API or Teams MCP provides a reliable alternative

Prerequisites

• Windows OS (for Layer 2 & 3 - keyboard shortcuts and window management)
• Microsoft Teams Desktop App OR Web Browser (for Layer 1 - Playwright)
• PowerShell 5.1 or later (for Layer 2 & 3)
• Teams must be signed in before automation begins
• User must have appropriate permissions for operations (e.g., team owner to install apps)
• Playwright MCP tools available in agent's toolset

---

Layer 1: Playwright MCP Automation (Primary)

How Agents Use This Layer

Agents have access to Playwright MCP tools and should call them directly to automate teams.microsoft.com. Below are proven recipes for common operations.

Known Selectors & Fallback Patterns

Teams uses multiple selector strategies. Always try in this order:

1. data-tid attributes (most stable - test IDs)
2. aria-label attributes (accessibility - stable)
3. CSS structural selectors (position-based)
4. Text content matching (least stable - localization issues)

Recipe: Navigate to teams.microsoft.com and Login

1. playwright-browser_navigate
   url: https://teams.microsoft.com

2. Wait for login redirect or Teams UI to load
   playwright-browser_wait_for
   text: "Teams"  # or wait for known element
   
3. If login required, handle Microsoft auth flow
   (user may need to complete MFA manually)
   
4. Wait for main Teams UI
   playwright-browser_snapshot  # verify loaded state

Recipe: Install App to a Team

1. Navigate to Apps
   Primary: playwright-browser_click
     ref: [data-tid="app-bar-apps"]
     element: "Apps button in sidebar"
   Fallback 1: [aria-label="Apps"]
   Fallback 2: button:has-text("Apps")
   
2. Wait for Apps marketplace to load
   playwright-browser_wait_for
   text: "Built for your org"  # or "Apps" heading
   
3. Search for the app
   playwright-browser_type
     ref: [data-tid="search-input"] or input[placeholder*="Search"]
     text: "Planner"
     element: "App search box"
     
4. Click the app card from results
   playwright-browser_click
     ref: [data-tid*="app-card"] or [aria-label*="Planner"]
     element: "Planner app card"
     
5. Click "Add to a team" button
   playwright-browser_click
     ref: button:has-text("Add to a team")
     element: "Add to a team button"
     
6. Select team from dropdown
   playwright-browser_click
     ref: [data-tid="team-picker"] or [role="combobox"]
     element: "Team selection dropdown"
   Then type team name or click from list
   
7. Click confirm/install button
   playwright-browser_click
     ref: button:has-text("Set up") or button:has-text("Install")
     element: "Install confirmation button"
     
8. Wait for success confirmation
   playwright-browser_wait_for
   text: "added"  # or success message

Fallback Selectors for Each Step:

Apps Button:

• [data-tid="app-bar-apps"]
• [aria-label="Apps"]
• button:has-text("Apps")
• .app-bar button:nth-of-type(6) (position-based)

Search Box:

• [data-tid="search-input"]
• input[placeholder*="Search"]
• .search-box input

App Card:

• [data-tid*="app-card-{appname}"]
• [aria-label*="{AppName}"]
• button:has-text("{AppName}")

Add Button:

• button:has-text("Add to a team")
• button:has-text("Add")
• [data-tid="add-button"]

Recipe: Add Tab to a Channel

1. Navigate to the team (use Teams MCP or Playwright navigation)
   playwright-browser_click
     ref: [data-tid="team-{teamId}"] or button:has-text("{TeamName}")
     element: "Team in sidebar"
     
2. Navigate to the channel
   playwright-browser_click
     ref: [data-tid="channel-{channelId}"] or [aria-label*="{ChannelName}"]
     element: "Channel in team list"
     
3. Click the "+" button to add a tab
   playwright-browser_click
     ref: [data-tid="add-tab-button"] or button:has-text("+")
     element: "Add tab button in channel tabs bar"
     
4. Search for tab type (e.g., Wiki, Planner, Website)
   playwright-browser_type
     ref: input[placeholder*="Search"]
     text: "Wiki"
     element: "Tab type search"
     
5. Click the tab type from results
   playwright-browser_click
     ref: [data-tid*="app-{tabtype}"] or button:has-text("Wiki")
     element: "Wiki tab type"
     
6. Configure tab settings (varies by tab type)
   For Wiki:
     - Enter tab name
     - Click Save/Add
   For Website:
     - Enter tab name
     - Enter URL
     - Click Save
     
7. Click Add/Save button
   playwright-browser_click
     ref: button:has-text("Add") or button:has-text("Save")
     element: "Add tab confirmation"
     
8. Verify tab appears in tabs bar
   playwright-browser_snapshot  # check for new tab

Fallback Selectors:

Add Tab Button:

• [data-tid="add-tab-button"]
• button[aria-label="Add a tab"]
• button:has-text("+")
• .tabs-container button:last-of-type

Tab Type Card:

• [data-tid*="app-wiki"]
• [aria-label*="Wiki"]
• button:has-text("Wiki")

Tab Name Input:

• [data-tid="tab-name-input"]
• input[placeholder*="Tab name"]
• input[type="text"]:visible

Recipe: Navigate to Team/Channel

1. Click the Teams icon (Ctrl+3 shortcut or Playwright)
   playwright-browser_click
     ref: [data-tid="app-bar-teams"] or [aria-label="Teams"]
     element: "Teams button in sidebar"
     
2. Locate team in list
   playwright-browser_click
     ref: [data-tid="team-{teamId}"] or button:has-text("{TeamName}")
     element: "Team: {TeamName}"
     
3. If team is collapsed, expand it
   playwright-browser_click
     ref: [data-tid="team-expand-{teamId}"]
     element: "Expand team button"
     
4. Click channel
   playwright-browser_click
     ref: [data-tid="channel-{channelId}"] or [aria-label*="{ChannelName}"]
     element: "Channel: {ChannelName}"
     
5. Wait for channel to load
   playwright-browser_wait_for
   text: "{ChannelName}"  # or channel content loads

Recipe: Open Teams Settings

1. Click profile/settings menu (top right)
   playwright-browser_click
     ref: [data-tid="profile-menu"] or button[aria-label*="Profile"]
     element: "Profile menu button"
     
2. Click Settings from dropdown
   playwright-browser_click
     ref: button:has-text("Settings") or [data-tid="settings-button"]
     element: "Settings menu item"
     
3. Navigate to desired settings section
   playwright-browser_click
     ref: button:has-text("General") or button:has-text("Privacy")
     element: "Settings section"
     
4. Interact with settings controls as needed

Fallback Selectors:

Profile Menu:

• [data-tid="profile-menu"]
• button[aria-label*="Profile"]
• .top-bar button:last-of-type

Settings Item:

• button:has-text("Settings")
• [data-tid="settings-menu-item"]
• [aria-label="Settings"]

Recipe: Manage Connectors / Workflows

1. Navigate to team and channel (see recipe above)

2. Click channel options menu (three dots)
   playwright-browser_click
     ref: [data-tid="channel-options"] or button[aria-label="More options"]
     element: "Channel options menu"
     
3. Click "Connectors" or "Workflows"
   playwright-browser_click
     ref: button:has-text("Connectors") or button:has-text("Workflows")
     element: "Connectors menu item"
     
4. Search for connector (e.g., "GitHub", "Trello")
   playwright-browser_type
     ref: input[placeholder*="Search"]
     text: "GitHub"
     element: "Connector search box"
     
5. Click Configure on desired connector
   playwright-browser_click
     ref: button:has-text("Configure")
     element: "Configure connector button"
     
6. Follow connector-specific setup flow
   (varies by connector - may require external auth)
   
7. Save configuration
   playwright-browser_click
     ref: button:has-text("Save") or button:has-text("Done")
     element: "Save connector button"

Self-Healing Strategy

When a Playwright selector fails:

1. Try next fallback in chain (data-tid → aria-label → CSS → text)
2. Use playwright-browser_snapshot to inspect current page state
3. Update selector based on snapshot and retry
4. Cache successful selector in element-cache.json for future use
5. Log failure in cache.failureLog with context

Example cache entry after successful operation:

{
  "recipes": {
    "openApps": {
      "lastSuccess": "2025-01-15T10:30:00Z",
      "workingSelector": "[data-tid='app-bar-apps']",
      "fallbacksAttempted": 0
    }
  }
}

---

Layer 2: Keyboard Shortcuts (Secondary)

PowerShell functions for quick navigation. Load the script first:

. C:\temp\tamresearch1\.squad\skills\teams-ui-automation\Teams-UIA.ps1
Initialize-TeamsAutomation -Verbose

Available Functions

Navigation:

• Open-TeamsApps - Ctrl+Shift+6 - Opens Apps marketplace
• Open-TeamsChat - Ctrl+2 - Opens Chat view
• Open-TeamsTeams - Ctrl+3 - Opens Teams view
• Open-TeamsCalendar - Ctrl+4 - Opens Calendar
• Open-TeamsActivity - Ctrl+1 - Opens Activity feed
• Open-TeamsSearch - Ctrl+E - Opens search bar
• Open-TeamsSettings - Ctrl+, - Opens Settings dialog
• Start-TeamsNewChat - Ctrl+N - Starts new chat
• Open-TeamsNotifications - Ctrl+Shift+O - Opens notification panel

Custom Shortcuts:

• Send-TeamsShortcut - Send any key combination

Example: Quick Navigation

# Navigate to Teams view
Open-TeamsTeams

# Wait for view to load
Start-Sleep -Seconds 1

# Open search
Open-TeamsSearch

# Now use Playwright to interact with search box

When to Use Keyboard Shortcuts

✅ Good for:

• Quick view switching (Chat ↔ Teams ↔ Calendar)
• Opening search/settings dialogs
• Complementing Playwright workflows (e.g., switch to Apps view, then use Playwright to interact)

❌ Not good for:

• Complex interactions requiring element inspection
• Operations that need verification of results
• When precise element targeting is needed

---

Layer 3: UIA Window Management (Tertiary)

PowerShell functions for window-level operations only.

Available Functions

Window Detection:

• Get-TeamsWindow - Find Teams window handle and process info
• Test-TeamsRunning - Check if Teams process is running
• Focus-TeamsWindow - Bring Teams to foreground (required for keyboard shortcuts)

Version Info:

• Get-TeamsVersion - Get Teams version from process

Cache Management:

• Get-ElementCache - Load element cache
• Save-ElementCache - Persist cache to disk

Example: Check Teams Status

# Load module
. C:\temp\tamresearch1\.squad\skills\teams-ui-automation\Teams-UIA.ps1

# Check if Teams is running
if (Test-TeamsRunning) {
    $teams = Get-TeamsWindow
    Write-Host "Teams found: $($teams.Title)"
    Write-Host "Version: $(Get-TeamsVersion)"
} else {
    Write-Host "Teams is not running"
}

# Focus Teams before sending shortcuts
Focus-TeamsWindow

When to Use UIA

✅ Good for:

• Verifying Teams is running before automation
• Getting Teams version for cache validation
• Bringing Teams window to foreground
• Process management

❌ Not good for:

• Element discovery within Teams UI (WebView blocks UIA)
• Clicking buttons or typing in Teams UI
• Reading content from Teams

---

Complete Workflow Example: Install App

This example shows how to combine all three layers:

# === Layer 3: Verify Teams is Running ===
. C:\temp\tamresearch1\.squad\skills\teams-ui-automation\Teams-UIA.ps1

if (-not (Test-TeamsRunning)) {
    Write-Error "Teams is not running. Please start Teams first."
    exit
}

Initialize-TeamsAutomation -Verbose

# === Layer 2: Navigate to Apps (optional - can use Playwright instead) ===
# Focus window and use keyboard shortcut
Open-TeamsApps
Start-Sleep -Seconds 2  # Wait for Apps view to load

# === Layer 1: Use Playwright MCP to Install App ===
# Now agent calls Playwright MCP tools:

# 1. Take snapshot to verify we're in Apps view
#    playwright-browser_snapshot

# 2. Search for app
#    playwright-browser_type
#      ref: [data-tid="search-input"]
#      text: "Planner"

# 3. Click app card
#    playwright-browser_click
#      ref: [aria-label*="Planner"]

# 4. Click Add to Team
#    playwright-browser_click
#      ref: button:has-text("Add to a team")

# 5. Select team and confirm
#    (continue with Playwright recipe from Layer 1 section)

# === Update Cache with Success ===
# Cache is automatically updated by keyboard shortcut functions
# For Playwright selectors, manually update if needed:

$cache = Get-ElementCache
$cache.recipes["installApp"] = @{
    lastSuccess = Get-Date -Format "o"
    workingSelector = "[data-tid='search-input']"
}
Save-ElementCache

---

Cache Structure

File: .squad/skills/teams-ui-automation/element-cache.json

{
  "version": "2.0",
  "teamsVersion": "26043.2011.4461.9586",
  "approach": "hybrid",
  "layers": {
    "playwright": {
      "baseUrl": "https://teams.microsoft.com",
      "selectors": {
        "appsButton": "[data-tid='app-bar-apps']",
        "searchBox": "[data-tid='search-input']",
        "addTabButton": "[data-tid='add-tab-button']"
      }
    },
    "keyboard": {
      "shortcuts": {
        "OpenApps": {
          "shortcut": "Ctrl+Shift+6",
          "successCount": 15,
          "failureCount": 0,
          "lastUsed": "2025-01-15T14:30:00Z"
        },
        "OpenChat": {
          "shortcut": "Ctrl+2",
          "successCount": 23,
          "failureCount": 1,
          "lastUsed": "2025-01-15T12:15:00Z"
        }
      }
    },
    "uia": {
      "windowClass": "TeamsWebView",
      "processName": "ms-teams"
    }
  },
  "recipes": {
    "installApp": {
      "lastSuccess": "2025-01-15T10:30:00Z",
      "workingSelectors": [
        "[data-tid='app-bar-apps']",
        "[data-tid='search-input']",
        "[aria-label*='Add to a team']"
      ],
      "failureCount": 0
    }
  },
  "failureLog": [
    {
      "timestamp": "2025-01-14T09:00:00Z",
      "operation": "addTab",
      "selector": "[data-tid='old-add-button']",
      "error": "Element not found",
      "resolved": true,
      "resolution": "Updated to button:has-text('+')"
    }
  ]
}

Auto-Invalidation:

• Cache is version-checked on initialization
• If Teams version changes, cache is preserved but flagged for validation
• Failed selectors are logged and alternative selectors are tried
• Successful fallbacks update the cache

---

Troubleshooting

Playwright Selector Not Found

1. Take snapshot: playwright-browser_snapshot to see current page state
2. Try fallback selectors in order (data-tid → aria-label → CSS → text)
3. Check if Teams UI changed: Microsoft updates Teams frequently
4. Update cache with new working selector
5. Report issue if all fallbacks fail

Keyboard Shortcut Not Working

1. Verify Teams is focused: Call Focus-TeamsWindow explicitly
2. Check shortcut is enabled: Some shortcuts may be disabled by admin policy
3. Try Playwright instead: More reliable for complex operations

Teams Not Detected

1. Check process name: Verify it's "ms-teams" (not old "Teams" process)
2. Restart Teams: Close from system tray and reopen
3. Check window state: Teams may be minimized to system tray

UIA Can't See Elements

This is expected behavior. New Teams runs in a WebView that UIA cannot access. Use Playwright (Layer 1) instead.

---

Performance Notes

Playwright Operations:

• Navigation: ~500-1000ms
• Element interaction: ~100-300ms
• Search/type: ~200-500ms
• Total for complex operation (install app): ~5-10 seconds

Keyboard Shortcuts:

• Focus window: ~300-500ms
• Send shortcut: ~100-200ms
• Total: ~500-700ms per shortcut

UIA Window Management:

• Process detection: ~50-100ms
• Window focus: ~300-500ms
• Version check: ~100ms

Recommendation: Use Layer 2 (keyboard) for quick navigation, then Layer 1 (Playwright) for interaction.

---

Limitations

• Playwright requires web login: If using desktop Teams, Playwright automates teams.microsoft.com separately
• Keyboard shortcuts require focus: May fail if another app steals focus mid-operation
• UIA doesn't work for in-app elements: Only for window management
• Timing-dependent: Async UI rendering requires waits between operations
• Language-dependent: Text-based selectors may fail in non-English Teams
• Admin restrictions: Some operations may be blocked by organization policies

---

Best Practices

1. Prefer Playwright for complex operations - Most reliable and inspectable
2. Use keyboard shortcuts for quick navigation - Faster than Playwright for simple view changes
3. Always verify state with snapshots - Check page loaded before interacting
4. Use fallback selector chains - Don't rely on a single selector
5. Cache successful selectors - Improves performance on repeated operations
6. Log failures with context - Helps identify patterns and update selectors
7. Combine layers strategically - e.g., keyboard to Apps view, then Playwright to install
8. Handle waits appropriately - Teams UI animations take 500-1000ms

---

Confidence Level

Medium (Proven Hybrid Approach)

This hybrid approach is based on:

• Proven limitation: UIA cannot access Teams WebView (confirmed via testing)
• Working solution: Playwright successfully automates teams.microsoft.com
• Stable shortcuts: Teams keyboard shortcuts are well-documented and rarely change
• Production use: Similar hybrid approaches used in enterprise Teams automation

Confidence will increase as:

• More operations are tested and recipes are refined
• Selector patterns are validated across Teams updates
• Cache effectiveness is measured over time

---

Version: 2.0
Created: 2025
Last Updated: 2025
Maintainer: Data (Code Expert)