SKILL.md

When to Activate

User asks to run end-to-end tests or audit the full Codi installation
User wants to validate artifacts, presets, hooks, and verification in a clean project
User is preparing a release and needs a full feature sweep

Skip When

User wants to test a single skill in isolation — use codi-test-suite
User wants e2e tests for a consumer web app — use codi-webapp-testing
User wants to generate new tests from scratch — use codi-tdd
User wants to run the existing test suite (not a full feature sweep) — use codi-test-suite
User is not working on the Codi source repo — this skill is contributor-only

This skill guides systematic validation of ALL Codi features in a test project. Each step is labeled:

[SYSTEM] — run this CLI command
[HUMAN] — STOP and ask the human to perform this action
[CODING AGENT] — the AI agent performs this

Full details: see docs/guides/testing-guide.md and docs/guides/user-flows.md.

Suite 1: Setup

[SYSTEM] Create a test project and install codi:

mkdir /tmp/codi-validation && cd /tmp/codi-validation
git init
npm init -y && npm install codi-cli
npx codi --version

Expected: Version prints (e.g., 0.3.1). Git repo initialized (needed for hooks).

Suite 2: Initialization

[SYSTEM] Non-interactive init:

npx codi init --agents claude-code cursor codex --preset balanced --json

Expected: .codi/ created with codi.yaml, flags.yaml (21 flags), rules/, skills/, frameworks/.

[CODING AGENT] Verify hooks were installed:

ls .git/hooks/pre-commit .git/hooks/commit-msg

Expected: Both files exist (pre-commit runner + commit message validator).

[HUMAN] Interactive wizard: run npx codi init --force in terminal. Select agents, rules, skills, preset. Verify output matches.

Suite 3: Artifacts (all 4 types)

[SYSTEM] Add all templates:

npx codi add rule --all --json
npx codi add skill --all --json
npx codi add agent --all --json

Expected: 28 rules, 63 skills, 21 agents. All managed_by: codi.

[SYSTEM] Add custom artifacts:

npx codi add rule my-custom --json
npx codi add skill my-custom --json

Expected: managed_by: user.

Suite 4: Generation & Drift

[SYSTEM] Generate and check:

npx codi generate --json
npx codi status --json

Expected: Files generated. hasDrift: false. Hooks re-installed.

[SYSTEM] Inject and fix drift:

echo "edit" >> CLAUDE.md
npx codi status --json
npx codi generate --json

Expected: hasDrift true, then false after regenerate.

[SYSTEM] Verify per-agent: .claude/rules/, .cursor/rules/.mdc, AGENTS.md, .codex/agents/.toml, .windsurfrules, .clinerules, .github/copilot-instructions.md, .claude/skills/, .windsurf/skills/, .github/skills/.

Suite 5: Verification & Compliance

[SYSTEM]

npx codi verify --json
npx codi compliance --json
npx codi doctor --ci --json
npx codi ci --json
npx codi docs-update --json

Expected: 12-char token deterministic. All checks pass. docs-update reports fixed files (if any).

Suite 6: Update & Presets

[SYSTEM] Preset switching:

npx codi update --preset strict --json
npx codi update --preset balanced --json

Expected: Strict shows restricted instructions. Balanced restores defaults.

[SYSTEM] Artifact refresh:

npx codi update --rules --skills --agents --dry-run --json

Expected: Managed artifacts listed. Custom (managed_by: user) skipped.

Suite 7: Preset Management

7a: Create & List

[SYSTEM]

npx codi preset create test-preset --json
npx codi preset list --json

Expected: Preset directory created at .codi/presets/test-preset/ with preset.yaml and subdirs (rules/, skills/, agents/). Visible in list.

7b: Built-in Presets

[SYSTEM]

npx codi preset list --builtin --json

Expected: Shows 6 built-in presets: minimal, balanced, strict, fullstack, development, power-user. Each with [builtin] source tag.

7c: Validate

[SYSTEM]

npx codi preset validate test-preset --json

Expected: Preset “test-preset” is valid. Reports version, artifact counts (rules: 0, skills: 0, agents: 0).

7d: Export as ZIP

[SYSTEM]

npx codi preset export test-preset --format zip --output /tmp/codi-validation/
ls /tmp/codi-validation/test-preset.zip

Expected: ZIP file created. File exists at /tmp/codi-validation/test-preset.zip.

7e: Remove

[SYSTEM]

npx codi preset remove test-preset --json
npx codi preset list --json

Expected: Preset removed. No longer visible in list. Lock file entry cleared.

7f: Install from ZIP

[SYSTEM]

npx codi preset install /tmp/codi-validation/test-preset.zip --json
npx codi preset list --json
npx codi preset validate test-preset --json

Expected: Preset installed from ZIP. Visible in list with [zip] source. Validation passes. Lock file has source: “zip:…” entry.

7g: Cleanup

[SYSTEM]

npx codi preset remove test-preset --json
rm -f /tmp/codi-validation/test-preset.zip

Expected: Preset and ZIP cleaned up.

7h: GitHub Preset Install

[HUMAN] Provide a GitHub repository URL that contains a valid preset (a repo with preset.yaml at the root). This can be:

A public preset repo (e.g., github:my-org/my-preset)
A private repo you have access to (SSH keys or GH_TOKEN configured)
If none exists, create a small test repo: add a preset.yaml with name: test-gh-preset, version: "1.0.0", push to GitHub, and provide the org/repo path.

[SYSTEM] Install the GitHub preset (substitute the real org/repo):

npx codi preset install github:<org>/<repo> --json

Expected: Preset cloned, validated, and installed to .codi/presets//.

[CODING AGENT] Verify the installation:

Run npx codi preset list --json — preset must appear with sourceType: "github"
Read .codi/preset-lock.json — entry must have source: "github:...", sourceType: "github", and a commit hash
Verify .codi/presets/<name>/preset.yaml exists

[SYSTEM] Validate and clean up:

npx codi preset validate <name> --json
npx codi preset remove <name> --json

Expected: Validation passes. Preset removed. Lock file entry cleared.

7i: Preset Registry Search

[HUMAN] Check if a preset registry is configured in .codi/codi.yaml under presetRegistry.url. If not configured, either:

Add a presetRegistry section pointing to a real registry repo, OR
Confirm this test should be skipped (the agent will note it as skipped)

[SYSTEM]

npx codi preset search test --json

[CODING AGENT] Verify the result:

If registry is configured: results are returned or “No presets found matching” message (both valid)
If registry is not configured or unreachable: a clear error message is reported (not a crash or unhandled exception)

7j: Preset Update

Prerequisite: A GitHub-sourced preset must be installed (from 7h). If 7h was skipped, skip this test too.

[SYSTEM]

npx codi preset update --dry-run --json

[CODING AGENT] Verify:

GitHub-sourced presets: reports “up to date” or lists available version changes
ZIP-sourced presets (if any): skipped with info message (ZIP presets require manual re-install)
No presets tracked: reports “Nothing to update”

Suite 8: MCP

[SYSTEM] Configure and verify:

cat > .codi/mcp.yaml << 'EOF'
servers:
  test-api:
    type: http
    url: "https://example.com/mcp"
EOF
npx codi generate --json

Expected: .mcp.json, .cursor/mcp.json, .codex/config.toml contain test-api.

Suite 9: Backup & Revert

[SYSTEM]

npx codi revert --list --json
npx codi revert --last --json

Expected: Backups from prior generates. Restore succeeds.

Suite 10: Clean & Reinstall

[SYSTEM]

npx codi clean --json
npx codi generate --json
npx codi clean --all --json

Expected: Clean removes generated. Regenerate works. Clean —all removes .codi/.

Suite 12: Agent Integration

[HUMAN] Open Claude Code in the test project. [HUMAN] Ask: “verify codi” [HUMAN] Copy response. [SYSTEM] npx codi verify --check "<response>" Expected: tokenMatch: true.

Suite 13: Pre-Commit Hooks

[SYSTEM] Verify hook scripts exist:

ls .git/hooks/pre-commit .git/hooks/commit-msg .git/hooks/codi-secret-scan.mjs .git/hooks/codi-file-size-check.mjs

Expected: All 4 files present.

[SYSTEM] Test file size check (800 LOC limit):

python3 -c "print('\n'.join(['line ' + str(i) for i in range(801)]))" > big-file.txt
git add big-file.txt
git commit -m "test: should fail"

Expected: Commit blocked — “big-file.txt: 802 lines (max: 800)”.

[SYSTEM] Test secret scan:

printf 'const key = "sk-%s"\n' "abc123def456ghi789jkl012mno345pqr" > secret-test.js
git add secret-test.js
git commit -m "test: should fail"

Expected: Commit blocked — “Potential secret found”.

[SYSTEM] Test commit-msg validation:

echo "hello" > valid-file.txt
git add valid-file.txt
git commit -m "bad message without type"

Expected: Commit blocked — “Invalid commit message format”.

[SYSTEM] Valid commit:

git commit -m "test: validate pre-commit hooks work"

Expected: Commit succeeds (proper format, small file, no secrets).

Suite 14: Documentation Sync

[CODING AGENT] Stale a count to test detection:

# Temporarily edit STATUS.md to say "Rule templates | 9 |"

[SYSTEM]

npx codi doctor --json

Expected: W_DOCS_STALE warning — “STATUS.md says Rule templates: 9 but 28 exist — run: codi docs-update”.

[SYSTEM]

npx codi docs-update --json

Expected: STATUS.md fixed. Count restored.

[SYSTEM]

npx codi doctor --json

Expected: No doc-sync warnings.

Suite 15: Commit Workflow

[SYSTEM] Add commit skill:

npx codi add skill commit --template commit --json
npx codi generate --json

Expected: Commit skill created. Generated for all agents.

[CODING AGENT] Verify the commit skill content includes:

Conventional commits format (feat, fix, docs, refactor, test, chore)
Pre-commit checks (formatting, linting, type checking, secret scan, file size)
Troubleshooting section (tool not found, hooks not installed, message rejected)

Suite 16: Watch Mode

[SYSTEM] Test auto-regeneration:

npx codi watch --once &
sleep 1
echo "# trigger" >> .codi/rules/my-custom.md
sleep 2
npx codi status --json

Expected: Watch detects change and regenerates. Status shows no drift.

Suite 17: Hook Dependencies & Version Check

[SYSTEM] Check hook dependency detection:

npx codi doctor --json

[CODING AGENT] Verify doctor output includes warnings for any missing hook tools (eslint, prettier, ruff, etc.) with install hints.

[SYSTEM] Test version check hook (if requiredVersion set in codi.yaml):

ls .git/hooks/codi-version-check.mjs

Expected: File exists when manifest has codi.requiredVersion.

Suite 18: Update —from Source Pull

[SYSTEM]

npx codi update --from codi/team-config --dry-run --json

Note: May fail if repo doesn’t exist. Verifies the —from flag is accepted and attempts a pull.

Cleanup

[SYSTEM] rm -rf /tmp/codi-validation

References

docs/guides/testing-guide.md — full testing procedure
docs/guides/user-flows.md — all 30 user flows
docs/project/troubleshooting.md — common issues (including hook troubleshooting)

Available Agents

For specialized analysis during e2e validation, delegate to these agents:

codi-test-generator — Generate automated tests from e2e findings. Prompt at ${CLAUDE_SKILL_DIR}[[/agents/test-generator.md]]
codi-security-analyzer — Security validation of auth flows and data handling. Prompt at ${CLAUDE_SKILL_DIR}[[/agents/security-analyzer.md]]

codi-dev-operations — Day-to-day Codi management (not full e2e validation)
codi-security-scan — Dedicated security audit for the test project

SKILL.md

When to Activate

Skip When

Suite 1: Setup

Suite 2: Initialization

Suite 3: Artifacts (all 4 types)

Suite 4: Generation & Drift

Suite 5: Verification & Compliance

Suite 6: Update & Presets

Suite 7: Preset Management

7a: Create & List

7b: Built-in Presets

7c: Validate

7d: Export as ZIP

7e: Remove

7f: Install from ZIP

7g: Cleanup

7h: GitHub Preset Install

7i: Preset Registry Search

7j: Preset Update

Suite 8: MCP

Suite 9: Backup & Revert

Suite 10: Clean & Reinstall

Suite 12: Agent Integration

Suite 13: Pre-Commit Hooks

Suite 14: Documentation Sync

Suite 15: Commit Workflow

Suite 16: Watch Mode

Suite 17: Hook Dependencies & Version Check

Suite 18: Update —from Source Pull

Cleanup

References

Available Agents

Related Skills