Skip to content

fix: use new agent config - #7

Merged
Cali0707 merged 1 commit into
mcpchecker:mainfrom
Cali0707:llm-judge-fixes
Apr 2, 2026
Merged

fix: use new agent config#7
Cali0707 merged 1 commit into
mcpchecker:mainfrom
Cali0707:llm-judge-fixes

Conversation

@Cali0707

@Cali0707 Cali0707 commented Apr 2, 2026

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

Release Notes

  • New Features

    • Introduced Claude ACP Agent Adapter npm package (@agentclientprotocol/claude-agent-acp) for integrated agent and LLM judge support.
  • Documentation

    • Updated installation guides with new package requirements.
    • Simplified configuration approach: agent and judge settings now use external YAML files instead of manual environment variable setup.

Signed-off-by: Calum Murray <[email protected]>
@Cali0707
Cali0707 merged commit dc00bb3 into mcpchecker:main Apr 2, 2026
@coderabbitai

coderabbitai Bot commented Apr 2, 2026

Copy link
Copy Markdown

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 06576a19-663f-438a-803f-c6b52ef2ba58

📥 Commits

Reviewing files that changed from the base of the PR and between d473dca and 527f392.

📒 Files selected for processing (7)
  • 01-getting-started/README.md
  • 01-getting-started/evals/agent.yaml
  • 01-getting-started/evals/eval.yaml
  • 02-linux-mcp-server/README.md
  • 02-linux-mcp-server/evals/agent.yaml
  • 02-linux-mcp-server/evals/eval.yaml
  • 03-evolution-case-study/README.md

📝 Walkthrough

Walkthrough

This PR updates quickstart documentation and configuration files across three directories to use the Claude ACP Agent Adapter (claude-agent-acp) npm package. It adds new agent.yaml manifest files defining agents via ACP command and updates eval.yaml files to reference external agent configurations instead of built-in types and inline judge environment variables.

Changes

Cohort / File(s) Summary
Documentation Updates
01-getting-started/README.md, 02-linux-mcp-server/README.md, 03-evolution-case-study/README.md
Removed "Configure Judge LLM" sections containing environment variable setup instructions. Added installation steps for @agentclientprotocol/claude-agent-acp npm package and updated descriptions to reflect agent and judge configuration via agent.yaml.
Agent Manifest Files
01-getting-started/evals/agent.yaml, 02-linux-mcp-server/evals/agent.yaml
New files defining kind: Agent resources with metadata.name: "claude-code-acp" and acp.cmd: "claude-agent-acp" configuration, establishing ACP integration for Claude Code agent.
Evaluation Configuration
01-getting-started/evals/eval.yaml, 02-linux-mcp-server/evals/eval.yaml
Updated config.agent from type: "builtin.claude-code" to external file reference (type: file, path: agent.yaml). Changed config.llmJudge from environment variable keys to ref pointing to agent.yaml.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • mcpchecker/quickstarts#6: Introduces identical code-level changes across quickstart directories—adding agent.yaml manifests with acp.cmd and updating eval.yaml to load agent and llmJudge via file references instead of builtin/env-based configuration.

Poem

🐰 A rabbit hops through config files with glee,
Agent YAML blooms where env vars used to be,
ACP commands bound, judge configs set free,
From builtin constraints to flexibility!
Three quickstarts now dance in harmony.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant