output-eval-validate-judge

Design diverse eval datasets using dimension-based variation. Use when bootstrapping eval datasets, when real traces are sparse, or when existing datasets miss edge cases.

Upgrade a project between versions of the Output framework. Use when the user asks to upgrade, migrate, or move to a newer Output version. Detects the current @outputai/* version in the project, fetches the matching migration guide from docs.output.ai, applies the changes, and verifies the project still type-checks.

Comprehensive guide to Output.ai Framework for building durable, LLM-powered workflows orchestrated by Temporal. Covers project structure, workflow patterns, steps, LLM integration, HTTP clients, CLI commands, and the full inventory of available agents and skills.

Pre-flight validation checks for Output SDK workflow operations. Ensures conventions are followed, requirements are gathered, and quality gates are passed before workflow execution.

Post-flight validation for Output SDK workflow operations. Systematic verification of step completion, convention compliance, quality validation, and deliverable verification.

Design effective LLM judge .prompt files for evaluators. Use when creating judgeVerdict/judgeScore/judgeLabel prompts, or when existing judges produce unreliable results.

Systematically review workflow traces to identify failure modes before building evaluators. Use when starting an eval project, after significant pipeline changes, or when production quality drops.

Handle errors caught in Output workflows without losing failure context. Use when adding workflow try/catch logic, fallbacks, partial-failure handling, custom Error classes, or typed checks for errors thrown by steps and evaluators.

Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests.

Grill the user relentlessly about a plan, decision, or idea. Use when the user wants to stress-test their thinking, or uses any 'grill' trigger phrases.

Migrate test files from `as` type assertions to @total-typescript/shoehorn. Use when user mentions shoehorn, wants to replace `as` in tests, or needs partial test data.

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

Use when implementing any feature or bugfix, before writing implementation code

Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.

Find, build and launch A/B tests on a Noibu-monitored store: ranked test ideas from 30 days of Noibu data and the live site, flag-gated variation code, a pull request or Shopify preview theme, automated QA, and a started test. Use for "what should we A/B test", CRO or experiment ideas, building a specific test, implementing a test already found, or taking a Noibu draft test live. Requires the Noibu MCP connector.

Check connectivity and permissions on both ends before any migration work. Verifies Snowflake is reachable and readable (list databases, read INFORMATION_SCHEMA) and, when AIDP coordinates are supplied, that the target catalog is readable and optionally writable. Use before the first assessment, when credentials change, or whenever a stage fails with an authorization error and you need to know which end is at fault.