creating-online-evaluations
Skill from PostHog/ai-plugin
Install
npx skills add PostHog/ai-plugin --skill creating-online-evaluations
Runs the open skills CLI and installs the skill for your agent (Claude Code, Codex, Cursor …). Alternatively copy the skill folder into .claude/skills/ or upload it in Claude.ai under Settings → Skills.
As a Claude Code plugin
/plugin marketplace add anthropics/claude-plugins-official
/plugin install posthog@posthog
This skill is part of the plugin “posthog”. In Claude Code the plugin installs it together with the plugin’s other skills, commands and hooks.
What does the creating-online-evaluations skill do?
Description from the SKILL.md in the source repository.
Questions about creating-online-evaluations
What does the creating-online-evaluations skill do?
Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified. Use when the user wants evaluations that automatically score new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn these failures into evals", or after reading traces finds a recurring failure. Covers letting the explored data decide how many evals to create, proposing that set for the user to pick, choosing the target and eval type (hog / llm_judge / sentiment), configuring an llm_judge's provider and model, scoping which generations trigger it via conditions, creating disabled, verifying scope, enabling, and backfilling past data. Proposes a sentiment eval when no failure mode is worth catching. Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.
How do I install creating-online-evaluations?
In a terminal: npx skills add PostHog/ai-plugin --skill creating-online-evaluations. The skills CLI asks which agents to install it for. In Claude.ai you can upload the skill folder as a ZIP under Settings → Skills.
Which agents can use creating-online-evaluations?
Skills use the open SKILL.md format, which Claude Code, Claude.ai, the Claude API, OpenAI Codex, Cursor, Gemini CLI and other agents support.
Is creating-online-evaluations safe to use?
Our automatic check found no risky patterns (piped shell scripts, hidden characters, secret access, prompt overrides). It is not a full audit – read the SKILL.md and any scripts before installing.
More skills from PostHog/ai-plugin
Write Streamlit app source code that runs well in a PostHog sandbox — the posthog_apps.query() bridge for reading PostHog data, the packages baked into the sandbox image, caching and session state across …
Write English that a reader cannot misread: one meaning per word, active voice, simple tenses, one idea per sentence, short sentences.
Read and use comments attached to the current PostHog task, its artifacts, and any canvas through the PostHog MCP exec dispatcher.
Best practices for agents managing PostHog skills via the MCP `skill-*` tools — how to discover, read, create, update, and refactor skills efficiently, especially large skills with many bundled files.
Work with PostHog Signals scouts: scheduled agents that monitor a project and write reports into the Signals inbox.
Use PostHog Autoresearch, agent-driven predictive models: ask "who will do X in the next N days?", let an agent train a model, then score people daily into a person property and `autoresearch_prediction` …
Validate and publish a canvas source project safely: the source-project shape, declared capabilities, reading the current version pointer, iterating on validation diagnostics, guarded publishing with expected_current_version_id, staging a draft build and promoting …
Explains PostHog billing usage and spend from the customer's visible Billing MCP tools. Use when the user asks why usage or spend is high, which product or project is driving usage, what …
Related skills in Development
Scan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.
Diagnosis loop for hard bugs and performance regressions. Use when the user says "diagnose"/"debug this", or reports something broken/throwing/failing/slow.
Review the changes since a fixed point (commit, branch, tag, or merge-base) along two axes: Standards (does the code follow this repo's documented coding standards?) and Spec (does the code match what …
Investigate a question against high-trust primary sources and capture the findings as a Markdown file in the repo.
Guidance for instrumenting webapps with Azure Application Insights. Provides telemetry patterns, SDK setup, and configuration references.
Analyze Azure resource groups and generate detailed Mermaid architecture diagrams showing the relationships between individual resources. WHEN: create architecture diagram, visualize Azure resources, show resource relationships, generate Mermaid diagram, analyze resource group …
Configure Azure API Management as an AI Gateway for AI models, MCP tools, and agents. WHEN: semantic caching, token limit, content safety, load balancing, AI model governance, MCP rate limiting, jailbreak detection …
Troubleshoot and resolve issues with Azure Messaging SDKs for Event Hubs and Service Bus. Covers connection failures, authentication errors, message processing issues, and SDK configuration problems.