Evaluation

Sibyl 反对者 agent - 挑战主流假设，寻找反面证据和盲点

Select the correct Ollama base model (and adapters) based on task type, resource fit, and registry availability. Use to translate Modelfile FROM/ADAPTER decisions into agent behavior.

Assess organizational AI maturity. TRIGGERS - Use when user needs help with ai-maturity-model related tasks.

Interpret results and build arguments

解析赛题PDF/Word，抽取任务与数据条件并给出模型选型与验证路线。Invoke when用户提供赛题文档或题目文本，需要确定采用何模型/方法。

Execute the complete experimental workflow - model optimization followed by evaluation - for all runs in a scaffolded experiment. Use after scaffold-experiment to submit jobs to SLURM.

Universal skill diagnosis and optimization tool. Detect and fix skill execution issues including context explosion, long-tail forgetting, data flow disruption, and agent coordination failures. Supports Gemini CLI for deep analysis. Triggers on "skill tuning", "tune skill", "skill diagnosis", "optimize skill", "skill debug".

Guidance for extracting weight matrices from black-box ReLU neural networks using only input-output queries. This skill applies when tasks involve model extraction attacks, recovering hidden layer weights from neural networks, or reverse-engineering ReLU network parameters from query access.

Manage learned patterns - list, view, archive, boost or penalize confidence. Use when you want to see what patterns Claude has learned, review pattern effectiveness, or manage the pattern library.

This skill should be used when the user asks to "analyze session", "세션 분석", "evaluate skill execution", "스킬 실행 검증", "check session logs", "로그 분석", provides a session ID with a skill path, or wants to verify that a skill executed correctly in a past session. Post-hoc analysis of Claude Code sessions to validate skill/agent/hook behavior against SKILL.md specifications.

Profile JAX training using xprof/TensorBoard/Perfetto and analyze hotspots. Use when asked to profile, benchmark, or optimize training throughput.

Agent skill for production-validator - invoke with $agent-production-validator

Research latest LLM benchmarks and update primary/fallback provider configurations if better models are available.

Run an independent review of a committee deliberation transcript. Evaluates against the five core rubrics, cites specific transcript passages, and produces actionable feedback. Use '/review' after a committee run or with a pasted transcript; use '/review --situation <path>' to review that situation's deliberation and write transcript_review to 04-evaluation-1.md (or 06-evaluation-2.md, 08-evaluation-3.md when the feedback loop has run).

Consistent agent behavior after restart — progressive identity loading, working context reconstruction from persistent artifacts, fresh-vs-continuation detection, calibration through centering and attunement, and identity verification for coherence. Addresses the cold-start problem where an agent must reconstruct who it is and what it was doing from evidence rather than memory. Use at the start of every new session, after a session interruption or crash, when agent behavior feels inconsistent with prior sessions, or when persistent memory and current context appear contradictory.

Apply optimization techniques to extend effective context capacity. Use when context limits constrain agent performance, optimizing for cost or latency, implementing long-running agent systems, agents exhaust memory, or when designing conversation summarization strategies.

Scaffolds evaluation suites for the Axiom AI SDK. Generates eval files, scorers, flag schemas, and config from natural-language descriptions. Use when creating evals, writing scorers, setting up flag schemas, or configuring axiom.config.ts.

Design effective prompts and optimize context for AI models and agents. Covers prompt engineering foundations (clarity, examples, chain of thought, XML structure), the four context engineering strategies (Write, Select, Compress, Isolate), agent memory and session patterns, and before/after examples showing measurable improvement. Use when writing system prompts, designing agent workflows, optimizing context windows, building prompt templates, structuring few-shot examples, reducing context rot, or improving AI output quality.

Gemma Domain Trainer (Prototype)

Comprehensive validation of FTD analysis output files. Checks JSONL format, field completeness, score validity, and structural requirements for TALD scale evaluations.

Comprehensive guide to creating, managing, and maintaining ground truth datasets for AI evaluation including annotation, quality control, and versioning

Use when investigating unexpected metric changes - systematically narrows root cause through 4D segmentation, intrinsic vs extrinsic factor analysis, hypothesis testing, and North Star impact assessment

Expert technology assessment specialist focused on evaluating, testing, and recommending tools, software, and platforms for business use and productivity optimization

LangSmith 통합 및 LLM Observability 가이드. 토큰 추적, 비용 계산, Run Tree, Tracing 데코레이터, OTEL 연동 구현 시 참조. "langsmith", "tracing", "observability", "token usage", "cost tracking" 키워드로 트리거.

Domain-validated decision logic for optogenetic stimulation parameter selection, including opsin choice, light delivery, pulse protocols, fiber placement, and control conditions

Domain-validated decision logic for selecting neuropsychological test batteries matched to suspected cognitive deficit profiles

DAG and potential outcomes frameworks for causal mediation identification

Systematic research protocol for discovering novel AI-native software businesses in the synthetic workforce era. Maps capability trajectories, analyzes segment-problem spaces, generates business models, and calculates inevitability scores across 3-24 month time horizons. Use when exploring AI business opportunities, conducting market research, or identifying automation-native ventures.

Evaluate AI systems for fairness using demographic parity, equalized odds, and bias detection techniques with mitigation strategies.

Performance Glycemic Intelligence System (PGIS) - integrates CGM, HRV, heart rate, sleep, and training data to provide daily readiness assessments, training prescriptions, post-workout analysis, fueling strategies, and clinical performance audits. Use for analyzing training data, generating readiness reports, creating performance audit presentations, and producing audio summaries for endurance athletes with Type 2 diabetes.

Use when a task needs multi-model perspectives, brainstorming, or stress-testing. Supports two modes: collaborative (default — agents build on each other's ideas) and adversarial (agents debate to find the strongest answer). Triggers: council, siege, swarm, multi-agent, debate, brainstorm.

Three-layer verification pipeline for AI output. Extracts verifiable claims, finds supporting or contradicting sources via web search, runs adversarial review for hallucination patterns, and produces a structured verification report with source links for human review.

Analyze raw prompts, identify intent and gaps, match ECC components (skills/commands/agents/hooks), and output a ready-to-paste optimized prompt. Advisory role only — never executes the task itself. TRIGGER when: user says "optimize prompt", "improve my prompt", "how to write a prompt for", "help me prompt", "rewrite this prompt", or explicitly asks to enhance prompt quality. Also triggers on Chinese equivalents: "优化prompt", "改进prompt", "怎么写prompt", "帮我优化这个指令". DO NOT TRIGGER when: user wants the task executed directly, or says "just do it" / "直接做". DO NOT TRIGGER when user says "优化代码", "优化性能", "optimize performance", "optimize this code" — those are refactoring/performance tasks, not prompt optimization.

MLflow — tracking, Model Registry, GenAI evaluation, tracing, S3/RDS backend, framework integrations. Use when setting up experiment tracking or model management. NOT for W&B.

This skill should be used when the user asks to "start an LLM project", "design batch pipeline", "evaluate task-model fit", "structure agent project", or mentions pipeline architecture, agent-assisted development, cost estimation, or choosing between LLM and traditional approaches.

QA harness for agentic systems: scenario suites, determinism controls, tool sandboxing, scoring rubrics, and regression protocols covering success, safety, latency, and cost.

Expert in tenant creditworthiness assessment and financial statement analysis. Use when evaluating tenant credit quality, analyzing financial ratios, assessing default risk, or structuring security requirements. Key terms include DSCR, current ratio, debt-to-equity, working capital, liquidity analysis, credit scoring, personal guarantee, security deposit, financial covenants

Operational prompt engineering for production LLM apps: structured outputs (JSON/schema), deterministic extractors, RAG grounding/citations, tool/agent workflows, prompt safety (injection/exfiltration), and prompt evaluation/regression testing. Use when designing, debugging, or standardizing prompts for Codex CLI, Claude Code, and OpenAI/Anthropic/Gemini APIs.

Quick "first look" overview of SAE features - top tokens, activation stats, weapons, families, sample contexts

Analiza ejercicios matemáticos tipo ICFES según 6 dimensiones oficiales (dificultad, competencia, componente, pensamiento, contenido, eje). Usa cuando tengas imagen de problema ICFES, pregunta matemática para clasificar, o necesites decidir si requiere gráficos complejos. Detecta automáticamente si el ejercicio necesita Graficador Experto para replicación visual.

Production prompt engineering frameworks for building, testing, versioning, and evaluating prompts. Covers chain-of-thought, few-shot design, system prompt architecture, prompt regression testing, and evaluation rubrics. Use when designing prompts for production systems, running A/B tests on prompts, building prompt libraries, or debugging prompt quality degradation.

Skill-aware agent routing with explicit competence/cost modeling. +22.5% accuracy, 700x cheaper than RL routers. Based on arXiv:2602.19672.

Remove refusal behaviors from open-weight LLMs using OBLITERATUS — mechanistic interpretability techniques (diff-in-means, SVD, whitened SVD, LEACE, SAE decomposition, etc.) to excise guardrails while preserving reasoning. 9 CLI methods, 28 analysis modules, 116 model presets across 5 compute tiers, tournament evaluation, and telemetry-driven recommendations. Use when a user wants to uncensor, abliterate, or remove refusal from an LLM.

This skill should be used when the user asks to "diagnose context problems", "fix lost-in-middle issues", "debug agent failures", "understand context poisoning", or mentions context degradation, attention patterns, context clash, context confusion, or agent performance...

Open-source AI observability platform for tracing, evaluating, and improving LLM applications with OpenTelemetry integration

Generate optimized indirect prompt injection, H-CoT, and multi-layer attack payloads for AI security testing and CTF competitions with automated family selection and success rate optimization

Standard datasets and benchmarks for evaluating AI security, robustness, and safety

Evaluate LLM systems using automated metrics, LLM-as-judge, and benchmarks. Use when testing prompt quality, validating RAG pipelines, measuring safety (hallucinations, bias), or comparing models for production deployment.

Adaptive multi-model AI roundtable. Runs up to 4 AI models (configurable) in 2 debate rounds with cross-critique and formal consensus scoring. Requires a configured Anthropic provider (Claude Opus recommended). Optionally adds GPT-5.3 Codex (OpenAI), Grok 4, and Gemini 3.1 Pro via Blockrun proxy. Works with Claude-only fallback if optional providers are unavailable. Writes results to local filesystem. Debate panel agents are persistent thread sessions; meta-panel and synthesis agents are one-shot.

Assess UK Government AI Playbook compliance for responsible AI deployment