Labs in CI

itr labs run starts a Labs prompt red-team run against one agent (the same attacks, plan feature and rate limit as Labs → Red Team), waits for the grade, prints every attack that got through and writes SARIF 2.1.0. The GitHub Action itr-labs wraps it and uploads the SARIF to code scanning, so each attack that got through is an alert on the repository, and the job fails when one is at or over the severity you choose.

The alert names the scenario (its id, its public title, its category and severity) and links to the run. The attack's text, the agent's replies and the agent's instructions stay in Intertrace.

Create an org API key in Settings → API keys with access Labs: run and read (scope labs:run) and save it as a repository secret. A Labs: read results key (labs:read) can only read runs. Either is refused on every other route (403 KEY_SCOPE); runtime keys (itr_rt_…) are never accepted.

Copy .github/actions/itr-labs/ from the Intertrace repository into yours, then:

name: Red-team the refund agent
on:
  pull_request:
  schedule:
    - cron: "17 6 * * 1"
permissions:
  contents: read
  security-events: write   # the SARIF upload
jobs:
  labs:
    # Forks get no secrets: run only for pull requests from this repository.
    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<commit sha> # v4
      - uses: ./.github/actions/itr-labs   # copied from the Intertrace repository
        with:
          agent: Refund bot                # the agent's id or name
          packs: agentic,prompt_injection  # categories, a preset or a custom pack id
          fail-on: high                    # high | medium | any | none
          api-key: ${{ secrets.INTERTRACE_LABS_KEY }}
          itr-version: <the itr release you reviewed>

export INTERTRACE_API_KEY=itr_…            # scope labs:run
itr labs run --agent "Refund bot" --packs agentic --fail-on high --sarif labs.sarif
itr labs results <run-id> --json             # scope labs:read is enough

--packs takes categories (prompt_injection, jailbreak, data_exfiltration, system_prompt_extraction, pii_handling, tool_abuse, indirect_injection, excessive_agency), a preset (agentic, data, core, all) or one custom pack's id. Severity is the Work finding's: critical when nothing at the gateway stopped an attack that compromised the agent, high when the gateway flagged it, medium for a partial the gateway passed, low for any other partial.

Exit codes: 0 nothing at or over --fail-on, 2 attacks got through at or over it, 5 no grade (the run failed because most attacks never reached the agent, is still running, or didn't finish in --timeout), 1 any other error.

POST /api/control-plane/labs/runs        { "agent": "<id or name>", "packs": ["agentic"], "recon": false }
                                          → 202 { "run_id", "total", "dashboard_path" }
GET  /api/control-plane/labs/runs/:id    → { status, progress, grade, resistance_score, counts,
                                              results: [{ attack_id, title, category, outcome, severity, ... }] }

On the run in Labs → Red Team, each attack that got through shows the change that fixes it: block an attack kind or a kind of data for the agent, turn on an organization guardrail (every agent follows it), or block or ask a person for the agent's own tools that do what the attack coerced. Apply fix shows before → after and saves through the change log (read back, with Undo on Work → Proposed actions); nothing is saved any other way. Where no change is safe to make automatically (an agent that repeats its own instructions, argument injection in a tool you haven't named), you get a link to the page where a person makes it.

Re-test this attack runs that one scenario again. The attack reads Fixed and re-tested only when a re-test started after the fix ran held, and the fix hasn't been undone since.