Debuly
skill-optimizer logo

skill-optimizer

Docker workbench and agent skill that runs repeatable, graded evals on agent skills across OpenRouter models.

Open source
Visit github.com skill-optimizer (official site)Compare developer platforms

What is skill-optimizer?

skill-optimizer is an open-source repository that does two jobs for people who write agent skills. First, it ships a skill and plugin you install into your coding agent so the agent can author and debug eval suites for you. Second, it provides a local command-line workbench that runs those evals inside Docker against models served through OpenRouter. Each case pairs a user-style task with graders, and a suite crosses cases with a list of models. The agent only sees an isolated work directory, never the graders or hidden answers, and graders then inspect files, command logs, SQL and generated artifacts to decide pass or fail. It installs as a Claude Code or Codex plugin, and for Cursor, OpenCode and Gemini CLI too. The tool itself is free; real runs need Node.js 20 or later, Docker and an OpenRouter API key, so model usage is billed by OpenRouter.

Use cases

  • Test whether an agent skill works
  • Compare models on one skill
  • Write eval suites with an agent
  • Grade agent output deterministically
  • Debug a failing skill
  • Check a skill is not triggered by unrelated tasks

Who it's for

  • Skill authors
  • Agent developers
  • AI engineers

Why use skill-optimizer

  • Graders decide the outcome from files and logs, so results do not depend on another model's opinion.
  • The agent cannot see graders or answers, which keeps the test honest.
  • One suite can compare several models on the same skill with a repeatable setup.
  • Installing the skill lets your own coding agent draft and fix eval cases for you.
  • A negative-case pattern checks that a skill is left alone on tasks it should not handle.

skill-optimizer features

  • Deterministic graders that inspect files, logs and SQL
  • Cases and suites run across OpenRouter models
  • Isolated Docker workspace hidden from graders
  • Installable skill for authoring and debugging evals
  • Plugins for Claude Code, Codex, Cursor, OpenCode and Gemini CLI
  • run-case and run-suite CLI commands
  • Example PDF and MCP suites included
  • Hidden Docker services for MCP server tests

skill-optimizer integrations

  • OpenRouter
  • Docker
  • Claude Code
  • OpenAI Codex
  • Cursor
  • OpenCode
  • Gemini CLI

How to get started with skill-optimizer

  1. Install the skill-optimizer plugin into your agent, or clone the repo and run npm install and npm run build.
  2. Set OPENROUTER_API_KEY and make sure Docker is running.
  3. Write a case file with a task and graders, and a suite file listing cases and OpenRouter models.
  4. Run npx tsx src/cli.ts run-suite on the suite, or run-case for one case.
  5. Read the grader results and trace under the results folder, then adjust the skill and rerun.

skill-optimizer limitations

  • Only openrouter/... model references are supported
  • Real model runs need an OpenRouter API key
  • Requires Node.js 20+ and Docker
  • Best for evals that can be verified from local files, logs or other state

skill-optimizer pricing

skill-optimizer is open source, so you can use and self-host it for free.

Prices can change and may vary by country or exclude tax. Check the official pricing page.

skill-optimizer alternatives

The main skill-optimizer alternatives listed on Debuly are Promptfoo, Vals AI, Agent Development Kit (ADK), Agentprobe, BAML and BenchGen. The table compares pricing and platforms; order is A–Z.

skill-optimizer compared with its alternatives
ToolPricingStarting pricePlatforms
skill-optimizerOpen source—CLI, Self-hosted, Claude Code plugin, Codex plugin
Agent Development Kit (ADK)Open source—CLI, Self-hosted, API
AgentprobeFree—Web, API, GitHub Action
BAMLOpen source—CLI, macOS, Linux, Windows
BenchGenFreemium$49/moWeb, API, Self-hosted
PromptfooOpen source—CLI, Web, API, Self-hosted
Vals AIPricing unknown—Web, Python SDK, CLI

  1. Agent Development Kit (ADK)

    Google's open-source, code-first Python framework for building, testing and deploying AI agents and multi-agent workflows.

    Developer platformsAutomation and agentsCoding
    Open sourceVisit Agent Development Kit (ADK) (official site)

  2. Agentprobe

    Free scanner that scores a seller website on agentic commerce readiness, showing whether an AI agent can find, quote and buy.

    Developer platformsMarketing and SEOAutomation and agents
    FreeVisit Agentprobe (official site)

  3. BAML

    Open-source, TypeScript-like programming language for agents, with typed LLM functions, built-in tests and SDK generation.

    Developer platformsCoding
    Open sourceVisit BAML (official site)

  4. BenchGen

    Agent benchmarking platform that runs AI agents through simulated environments, scores each decision and exports training data.

    Developer platformsAutomation and agents
    FreemiumVisit BenchGen (official site)

  5. Promptfoo

    Open-source LLM evaluation and red-teaming tool that finds prompt injections, jailbreaks and data leaks in AI apps and agents.

    Developer platformsCodingAutomation and agents
    Open sourceVisit Promptfoo (official site)

  6. Vals AI

    Evaluation platform for testing LLM applications with test suites, plus custom benchmarks built from your GitHub repo.

    Developer platformsCoding
    Visit Vals AI (official site)

skill-optimizer FAQ

How much does skill-optimizer cost?

The tool is free to use from its repository and lists no paid plans. You pay for model usage through OpenRouter when you run real evals.

Is skill-optimizer open source?

It is published as a public GitHub repository that you clone, build and run locally. See the repository for the license terms.

What do I need to run it?

Node.js 20 or later, Docker, and an OPENROUTER_API_KEY for real model runs. Only openrouter model references are supported.

Which agents can install the skill?

The repository documents installation for Claude Code, OpenAI Codex CLI and app, Cursor, OpenCode and Gemini CLI, plus a skill-only install through the skills CLI.

How does grading work?

Graders run after the agent finishes and can inspect workspace files, command logs, answer.json, trace.jsonl and other local state. The agent never sees them.

What are alternatives to skill-optimizer?

Other ways to evaluate agent behavior include NVIDIA SkillEvaluator, Promptfoo, Inspect AI and OpenAI Evals.

Sources & verification

Facts on this page come from the official sources below, each with the date we read it. Written by Debuly; the vendor did not review this page.

SourceTypeAccessed
github.com/fastxyz/skill-optimizerofficial

Image credits

  • Logo: from avatars.githubusercontent.com · fetched · Vendor trademark, used only to identify the product; removed on request
  • Cover: none collected yet, so a generated tile is shown.

Spot an error? Report an issue with this listing. We also remove a logo or image on the owner's request.