fix: use new agent config (#7)

Signed-off-by: Calum Murray <[email protected]>

Author: Cali0707

01-getting-started/README.md

@@ -35,18 +35,6 @@ The server uses **streamable HTTP transport** with `mcp.run(transport="streamabl
 
 ## Quick Start
 
-### 0. Configure the Judge LLM
-
-MCPChecker uses an LLM to verify test results. Set these environment variables before running tests, for instance with OpenAI:
-
-```bash
-export JUDGE_BASE_URL="https://api.openai.com/v1"
-export JUDGE_API_KEY="sk-your-key-here"
-export JUDGE_MODEL_NAME="gpt-4o-mini"
-```
-
-The judge LLM evaluates whether the agent completed tasks correctly by analyzing the agent's output.
-
 ### 1. Install Prerequisites
 
 **Install Claude Code** (AI agent):
@@ -60,6 +48,14 @@ curl -fsSL https://claude.ai/install.sh | bash
 
 See the [official installation guide](https://github.com/anthropics/claude-code?tab=readme-ov-file#get-started) for Windows and other installation methods.
 
+**Install the Claude ACP agent adapter:**
+
+```bash
+npm install -g @agentclientprotocol/claude-agent-acp
+```
+
+This provides the `claude-agent-acp` command used by MCPChecker to run Claude Code as an ACP-compatible agent.
+
 **Install uv** (Python package manager):
 ```bash
 curl -LsSf https://astral.sh/uv/install.sh | sh
@@ -116,6 +112,20 @@ You should see:
 
 This quickstart includes a complete evaluation setup. Let's look at what gets tested and how it's defined:
 
+### The Agent Configuration (`evals/agent.yaml`)
+
+```yaml
+kind: Agent
+metadata:
+  name: "claude-code-acp"
+acp:
+  cmd: "claude-agent-acp"
+```
+
+**What this does:**
+- Defines an ACP (Agent Client Protocol) agent that uses Claude Code via the `claude-agent-acp` adapter
+- This agent configuration is referenced by both the eval runner and the LLM judge
+
 ### The Main Eval Configuration (`evals/eval.yaml`)
 
 ```yaml
@@ -124,19 +134,19 @@ metadata:
   name: "demo-server-test"
 
 config:
-  # Use Claude Code as the AI agent
+  # Use Claude Code as the AI agent (via ACP)
   agent:
-    type: "builtin.claude-code"
+    type: file
+    path: agent.yaml
 
   # MCP server configuration
   mcpConfigFile: mcp-config.yaml
 
-  # LLM judge configuration
+  # LLM judge configuration (reuses the same agent)
   llmJudge:
-    env:
-      baseUrlKey: JUDGE_BASE_URL
-      apiKeyKey: JUDGE_API_KEY
-      modelNameKey: JUDGE_MODEL_NAME
+    ref:
+      type: file
+      path: agent.yaml
 
   # Test tasks
   taskSets:
@@ -150,9 +160,9 @@ config:
 ```
 
 **What this does:**
-- Configures **Claude Code** as the agent that will attempt the tasks
+- Configures **Claude Code** as the agent via the ACP adapter defined in **agent.yaml**
 - Points to **mcp-config.yaml** to connect to your MCP server
-- Defines the **judge LLM** settings (using your environment variables)
+- The **LLM judge** also references **agent.yaml**, so no separate judge configuration is needed
 - Loads tasks from **tasks/add.yaml** and asserts the `add` tool must be used
 
 ### The Task Definition (`evals/tasks/add.yaml`)

01-getting-started/evals/agent.yaml

@@ -0,0 +1,5 @@
+kind: Agent
+metadata:
+  name: "claude-code-acp"
+acp:
+  cmd: "claude-agent-acp"

01-getting-started/evals/eval.yaml

@@ -5,17 +5,17 @@ metadata:
 config:
   # Use Claude Code as the AI agent
   agent:
-    type: "builtin.claude-code"
+    type: "file"
+    path: agent.yaml
 
   # MCP server configuration
   mcpConfigFile: mcp-config.yaml
 
   # LLM judge configuration
   llmJudge:
-    env:
-      baseUrlKey: JUDGE_BASE_URL
-      apiKeyKey: JUDGE_API_KEY
-      modelNameKey: JUDGE_MODEL_NAME
+    ref:
+      type: file
+      path: agent.yaml
 
   # Test tasks
   taskSets:

02-linux-mcp-server/README.md

@@ -33,7 +33,15 @@ curl -fsSL https://anthropic.com/install-claude-code | sh
 
 For more installation options, see the [official installation guide](https://github.com/anthropics/claude-code).
 
-### 2. Install MCPChecker
+### 2. Install the Claude ACP Agent Adapter
+
+```bash
+npm install -g @agentclientprotocol/claude-agent-acp
+```
+
+This provides the `claude-agent-acp` command used by MCPChecker to run Claude Code as an ACP-compatible agent. It is also used as the LLM judge.
+
+### 3. Install MCPChecker
 
 Download the latest release:
 
@@ -44,7 +52,7 @@ chmod +x mcpchecker-linux-amd64
 sudo mv mcpchecker-linux-amd64 /usr/local/bin/mcpchecker
 ```
 
-### 3. Install Linux MCP Server
+### 4. Install Linux MCP Server
 
 ```bash
 pip install --user linux-mcp-server
@@ -62,22 +70,6 @@ which linux-mcp-server
 
 For more installation options and documentation, see the [Linux MCP Server documentation](https://rhel-lightspeed.github.io/linux-mcp-server/).
 
-### 4. Configure Judge LLM
-
-MCPChecker uses an LLM to verify test results. Set these environment variables:
-
-```bash
-export JUDGE_BASE_URL="https://api.openai.com/v1"
-export JUDGE_API_KEY="sk-your-key-here"
-export JUDGE_MODEL_NAME="gpt-4o-mini"
-```
-
-**Why a judge LLM?** Testing AI agents requires flexible verification. Instead of exact string matching, we use an LLM to verify if the output is semantically correct.
-
-For example, instead of checking for the exact string "Fedora Linux 43", the judge checks if the output "contains information about the operating system". This allows the test to pass even if the formatting varies, as long as the required information is present.
-
-Each verification step includes a `reason` explaining what the judge is checking, which helps with debugging when tests fail.
-
 ## What Gets Tested
 
 This quickstart tests two diagnostic tools from the Linux MCP Server:

02-linux-mcp-server/evals/agent.yaml

@@ -0,0 +1,5 @@
+kind: Agent
+metadata:
+  name: "claude-code-acp"
+acp:
+  cmd: "claude-agent-acp"

02-linux-mcp-server/evals/eval.yaml

@@ -5,17 +5,17 @@ metadata:
 config:
   # Use Claude Code as the AI agent
   agent:
-    type: "builtin.claude-code"
+    type: file
+    path: agent.yaml
 
   # MCP server configuration
   mcpConfigFile: mcp-config.yaml
 
   # LLM judge configuration
   llmJudge:
-    env:
-      baseUrlKey: JUDGE_BASE_URL
-      apiKeyKey: JUDGE_API_KEY
-      modelNameKey: JUDGE_MODEL_NAME
+    ref:
+      type: file
+      path: agent.yaml
 
   # Test tasks
   taskSets:

03-evolution-case-study/README.md

@@ -181,18 +181,12 @@ Since the **code is identical**, differences in test results prove documentation
 
 ### Prerequisites
 
-See [getting-started](../getting-started/) for installation of:
+See [getting-started](../01-getting-started/) for installation of:
 - Claude Code
+- `@agentclientprotocol/claude-agent-acp` (Claude ACP agent adapter)
 - mcpchecker
 - uv (Python package manager)
 
-Set judge LLM environment variables:
-```bash
-export JUDGE_BASE_URL="https://api.openai.com/v1"
-export JUDGE_API_KEY="sk-your-key-here"
-export JUDGE_MODEL_NAME="gpt-4o-mini"
-```
-
 ### Run Both Iterations
 
 **Iteration 1 - Bad Documentation:**