context-evaluator

AI agent documentation quality analyzer for AGENTS.md and CLAUDE.md files.

This tool evaluates your AI agent instruction files using 17 specialized evaluators to identify issues and improvement opportunities, and can automatically remediate them using AI. It helps ensure your documentation provides clear, actionable guidance for AI coding assistants.

An experimental project from Packmind.

How to Use

Option 1: Online (No Install)

Visit https://context-evaluator.ai and paste your repository URL.

Option 2: Binary Release (No Dependencies)

Download a standalone executable from GitHub Releases. No runtime dependencies needed.

Available platforms: macOS (Apple Silicon), macOS (Intel), Linux (x64), Windows (x64).

# Make executable (macOS/Linux)
chmod +x ./context-evaluator-<platform>

# Start the web UI
./context-evaluator-<platform> api

# Or run a CLI evaluation
./context-evaluator-<platform> cli evaluate --url https://github.com/user/repo

# Or evaluate a local directory
./context-evaluator-<platform> cli evaluate --path /path/to/project

Then open http://localhost:3000 in your browser.

Option 3: From Source (requires Node 22+ and Bun)

# Clone and install
git clone https://github.com/PackmindHub/context-evaluator.git
cd context-evaluator
bun install

# Start the application
bun run dev

Then open http://localhost:3000 in your browser.

Local Scanning Notes

Git clone operations run on your local machine
Private repositories may work if your git credentials are configured (SSH keys, credential helpers)
The homepage auto-detects which AI agents you have installed

How it Works

Input (Git URL or Local Path)
    ↓
Clone Repository (if remote)
    ↓
Analyze Codebase (languages, frameworks, patterns)
    ↓
Find Documentation (AGENTS.md, CLAUDE.md, linked files)
    ↓
Run 17 Evaluators via AI
    ↓
Rank by Impact
    ↓
Calculate Score & Grade
    ↓
Return Results

Processing time: 1-3 minutes depending on codebase size and AI provider.

Cost display: Shows API costs when supported by the provider.

Understanding Results

Results are categorized into two types:

Errors (13 evaluators): Issues with existing content that need fixing
Suggestions (4 evaluators): Opportunities for new content based on codebase analysis

Each issue includes:

Severity level (Critical, High, Medium, Low)
Location in your documentation
Problem description
Recommended fix

Remediation

AI-powered automated fix for evaluation issues — available in the web UI only.

How it works

Select issues — Add issues from the Errors or Suggestions tabs using the + button on each card
Configure — Pick a target agent format and AI provider (auto-detected from installed CLI tools)
Execute — Runs a 4-phase pipeline: plan error fixes → execute error fixes → plan suggestions → execute suggestions. Live progress shows step labels, cost, and token tracking in real time.
Review — Inspect per-file diffs, view the action summary (fixed / added / skipped), and download a .patch file. No changes are committed — apply the patch manually with git apply remediation.patch

Errors are processed first (sorted by severity), then suggestions are layered on top.

Target agents

The target format determines which files and directory structure the remediation produces:

Target	Main file	Rules / Instructions	Skills
AGENTS.md	`AGENTS.md`	`.agents/rules/`	`.agents/skills/`
Claude Code	`CLAUDE.md`	`.claude/rules/`	`.claude/skills/`
GitHub Copilot	`.github/copilot-instructions.md`	`.github/instructions/`	`.github/skills/`
Cursor	`.cursor/rules/*.mdc`	(integrated)	`.cursor/skills/`

Output types

The AI decides per-issue which output type to use:

Rule — Short declarative constraint, always loaded by the agent
Skill — Procedural workflow loaded on demand via a trigger phrase
Context update — Direct edit to the main file (project structure, setup steps, env vars, etc.)

Remediation history & impact evaluation

Past remediations are saved and visible in the Remediate tab. Click Evaluate Impact to re-run the full evaluation on the patched repository and see a before → after score comparison.

Alternative: Generate Prompts

If you prefer manual control, click Generate Prompts to get copy-paste-ready prompts for your own CLI agent.

Evaluators

#	Evaluator	Type	Description
01	Content Quality	Error	Detects human-focused, irrelevant, or vague content
02	Structure & Formatting	Error	Identifies poor organization and inconsistent formatting
03	Command Completeness	Error	Finds incomplete commands and missing prerequisites
04	Testing Guidance	Error	Detects absent or unclear testing instructions
05	Code Style Clarity	Error	Identifies missing or conflicting style guidelines
06	Language Clarity	Error	Finds ambiguous language and undefined jargon
07	Workflow Integration	Error	Detects missing git/CI workflow documentation
08	Project Structure	Error	Identifies missing codebase organization explanations
09	Security Awareness	Error	Finds exposed credentials and security risks
10	Completeness & Balance	Error	Detects skeletal or over-detailed content
11	Subdirectory Coverage	Suggestion	Recommends separate AGENTS.md for subdirectories
12	Context Gaps	Suggestion	Discovers undocumented framework/tool patterns
13	Contradictory Instructions	Error	Detects conflicting instructions across files
14	Test Patterns Coverage	Suggestion	Discovers undocumented testing conventions
15	Database Patterns Coverage	Suggestion	Discovers undocumented database/ORM patterns
17	Markdown Validity	Error	Checks markdown syntax and link validity
19	Outdated Documentation	Error	Verifies documented paths and files exist

AI Providers

The tool supports multiple AI providers:

Provider	CLI Flag	Setup
Claude Code	`--agent claude` (default)	claude.ai/code
Cursor Agent	`--agent cursor`	cursor.com
OpenCode	`--agent opencode`	github.com/opencode-ai/opencode
GitHub Copilot	`--agent github-copilot`	docs.github.com/copilot
OpenAI Codex	`--agent codex`	docs.openai.com/codex

CLI Reference

Basic Usage

# Evaluate current directory
bun run evaluate

# Evaluate a remote repository
bun run evaluate --url https://github.com/user/repo

# Evaluate a local directory
bun run evaluate --path /path/to/project

Evaluate Command Options

Option	Description	Default
`--url <github-url>`	GitHub repository URL to clone and evaluate	-
`--path <directory>`	Local directory path (absolute or relative)	Current directory
`--agent <name>`	AI provider: `claude`, `cursor`, `opencode`, `github-copilot`, `codex`	`claude`
`-o, --output <file>`	Output file path for results	`evaluator-results.json`
`--report <mode>`	Output format: `terminal`, `raw`, `json`	`terminal`

Evaluation Scope:

Option	Description	Default
`--evaluators <number>`	Number of evaluators to run	`12`
`--evaluator-filter <type>`	Filter: `all` (17), `errors` (13), `suggestions` (4)	`all`
`--depth <integer>`	Limit directory depth for context file search (0 = root only)	Unlimited
`--concurrency <number>`	Number of evaluators to run concurrently	`3`

Evaluation Mode:

Option	Description
`--unified`	All files evaluated together (better cross-file detection)
`--independent`	Each file evaluated separately
`--max-tokens <number>`	Maximum combined tokens for unified mode (default: 100000)

Results:

Option	Description	Default
`--no-curation`	Show all issues without impact prioritization	Curation enabled
`--top-n <number>`	Number of top issues to curate	`20`

Debug:

Option	Description
`-v, --verbose`	Enable verbose output
`--debug`	Save prompts/responses to `debug-output/` directory
`--preserve-debug-output`	Keep debug files after successful evaluation

Advanced:

Option	Description	Default
`--timeout <ms>`	Timeout per evaluator in milliseconds	`600000`
`--linked-docs-concurrency <number>`	Parallel AI calls for linked doc summarization	`3`
`--enable-assessment-features`	Enable assessment features (feedback, selection basket)	Disabled

Examples

# Run all error evaluators only
bun run evaluate --evaluator-filter errors

# Evaluate with verbose output and top 10 issues
bun run evaluate -v --top-n 10

# Evaluate remote repo with JSON output
bun run evaluate --url https://github.com/user/repo --report json -o report.json

# Use Cursor agent with unified mode
bun run evaluate --agent cursor --unified

Contributing

See CONTRIBUTING.md for development setup, architecture details, API reference, and contribution guidelines.

About

Built with Bun, React, Tailwind CSS, and TypeScript.

License: MIT

Issues & Feedback: GitHub Issues

Name	pattern-discovery-capture
Description	Automatically capture recurring patterns discovered in the codebase during exploration. This skill logs implicit conventions, architectural patterns, and coding practices that are followed consistently but not formally documented. Use this proactively when you observe patterns that could become explicit standards.

pattern-discovery-capture

SKILL.md