AI tools for “llm”

Benchmark Jetson LLM/VLM serving performance across vLLM, llama.cpp, and Ollama with structured JSON output.

Checklist for code that calls a language model or exposes a system to one. Use when writing or reviewing code that calls a model API (Anthropic, OpenAI, Bedrock, Gemini, Perplexity), a prompt or prompt template,…

…that doesn't come from database rows (presence, typing, LLM streams), use…

…Google Gemini, TypeSafe (Jev), or OpenRouter-hosted models (xAI/DeepSeek/Meta/Mistral/Qwen) from Netlify Functions or Edge Functions without managing provider accounts or API keys. Reach for this when adding an…

Jump to content Main menu Main menu move to sidebar hide Navigation Main pageCommunity portalProject chatCreate a new ItemRecent changesRandom ItemQuery ServiceNearbyHelpSpecial pages Lexicographical data Create a new…

…/ agent tracking / trackAgent, mentions capturing LLM or chatbot conversations for Pendo, or asks how to get their AI agent's data into Pendo — even if they don't name the specific…

…user asks about top spenders, expensive users, per-user LLM cost, user-level cost drivers, or patterns behind high AI observability…

…up anomaly detection, or use the AI anomaly detector (the llm detector type). Covers choosing between threshold, relative change, statistical detector, and AI detector, previewing with alert-simulate before saving,…

…uncovered surface; record structured output from scheduled LLM scoring; or give feedback through notes. Covers SKILL.md structure, report requirements, follow-up report checks, structured output, duplicate detection,…

…— schedule a free-text prompt to run on a cron, with the LLM-synthesized markdown delivered to email, Slack, or Microsoft Teams on each tick. Use when the user wants a recurring AI summary of X on any cadence (daily,…

…the user to pick, choosing the target and eval type (hog / llm_judge / sentiment), configuring an llm_judge's provider and model, scoping which generations trigger it via conditions, creating disabled, verifying scope,…

Find where an AI/LLM application is failing in production and surface the failure patterns, working from real traces. Use when someone wants to understand what's going wrong with an AI feature, find and categorize…

Investigate AI observability clusters — understand usage patterns in AI/LLM traffic, compare cluster behavior, compute cost/latency metrics, and drill into individual traces within clusters.

…evaluations — `hog` (deterministic code-based), `llm_judge` (LLM-prompt-based), and `sentiment` (user-message sentiment). Find existing evaluations, inspect their configuration, run them against specific…

Investigate LLM spend in PostHog — total cost over time, cost by model, provider, user, trace, or custom dimension, token and cache-hit economics, and cost regressions. Use when the user asks "how much are we spending…

Add PostHog AI Observability to trace AI model usage. Use after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. Also handles initial PostHog SDK…

Set up an LLM-judge evaluation that extracts canonical use cases for a PostHog feature at scale and streams the results to a Slack channel as a live feed. Use when someone wants to understand how users are actually…

Debug and inspect LLM/AI agent traces using PostHog's MCP tools. Use when the user pastes a trace or session URL (e.g. /ai-observability/traces/<id> or /ai-observability/sessions/<id>), asks to debug a trace, figure out…

…web analytics, error tracking, logs, sessions, and LLM traces. Before you calculate a governed business or telemetry measure, check system.information_schema.metrics for an approved definition. Examples include…

Signals scout for PostHog AI observability. Watches LLM traces for cost, latency, error, volume, and eval-performance regressions.