Skip to content

Analytical AI Handbook

Search the guide

Overview

Primitives

Classifiers

Types of Classifiers

Abstention

Are judges classifiers?

Extractors

Extraction Verification

Good Extractor Design

Judges

Judge Terminology

Judge Anatomy

Types of LLM Judges

Judges in Evals: Flip Your Intuition

Good Task Design Is All You Need

Patterns

Consistency

Determinism

Task Specificity

Fine-tuning and RL

Parallel Sampling

Confidence Scores

Ensembles

Temperature

Context

Expert Annotation

Why Annotations Matter

What Annotations Capture

Finding Good Candidates for Annotation

Evals

Evals as Outer Loop

Eval Approaches

Where to Start

Static Evals vs. Judges

Deployment

Batch vs. Real Time Inference

Model Selection

Open vs. Closed

Performance Tradeoffs

Routers & Ensembles

Architectures

Handbook/Architectures

Architectures

Coming soon

View full guide Markdown

Previous Routers and Ensembles