Experiments and evaluations

Define prompt variants in code, assign sessions to them, and measure the results using human feedback, application checks, or a separate evaluator agent.

Use the console

Open an agent and choose Edit. The Experiment section lets you name an experiment and revision, select an assignment strategy, and add weighted variants. Each variant can inherit or replace the prompt, model, tools, and sub-agent roster. The Evaluators section attaches a registered judge at a specific version, with a trigger, sample rate, idle window, and time limit. Saving creates a new agent version; existing sessions keep their original configuration.

Start session shows the experiment being used. Identity experiments accept a stable assignment key or identity. Custom experiments also require a variant. The session pins the exact agent version shown in the dialog.

In a session, open Evaluations to record thumbs-up/down, numeric or categorical feedback with notes, or run an attached evaluator manually. Choose a transcript boundary to rate a particular answer. The history shows pending, completed, skipped, and failed evaluations, with links to evaluator sessions. The details panel displays the immutable experiment assignment and exposure status.

The agent's Experiments panel compares allocation, exposure, token usage, and scores. Select a historical experiment/revision or agent version, refresh results, and use View sessions to inspect a variant's sessions. Human, application, and judge scores remain separate. Session-list filters can also match an experiment, revision, and variant directly.

Define variants

import { defineAgent, tool } from "@trycereal/sdk"

const search = tool({
  name: "search",
  handler: (_input, ctx) => {
    const concise = ctx.session?.experiment?.variant === "concise"
    return { instructions: concise ? "Return three results." : "Return ten results." }
  }
})

export const research = defineAgent({
  agent: {
    slug: "research",
    model: { provider: "my-provider", model: "my-model" },
    system: "Research the question. Verify claims against source documents.",
    experiment: {
      name: "research-prompt",
      revision: "v1",
      strategy: "identity",
      variants: [
        { name: "control", weight: 50 },
        {
          name: "concise",
          weight: 50,
          system: "Write a concise research brief. Verify claims against source documents."
        }
      ]
    }
  },
  tools: [search]
})

An experiment has 2–20 uniquely named variants whose weights total 100. A variant can replace system, model, tools, or subagents. Unspecified fields inherit the base agent. Objects and arrays replace the whole field.

sync stores the experiment inside the agent version. Creating a session selects one variant and retains it for the entire conversation, including retries and identity changes. A subsequent agent version cannot change that conversation.

Choose the assignment unit

const session = await research.sessions.start({
  identity: { scope: "user", id: "user-42" },
  message: "What are the launch risks? Cite your sources.",
  idempotencyKey: "request-123"
})

console.log(session.experiment?.variant)

• identity: deterministic assignment using the asserted identity. Pass experiment: { key: "account-42" } to group by account, device, or another stable key instead. Missing identity/key is an error.
• session: deterministic assignment using the new session's id. It remains fixed within the conversation; a new conversation may receive a different variant.
• custom: your application chooses a declared variant and supplies a stable key.

Identity assignment repeats across sessions when allocation rules remain unchanged. Changing weights or their order can move new sessions between variants. Use a new revision for changed treatments or allocation rules. Existing sessions keep both their version and assignment.

For custom routing, set the strategy to custom and supply a resolver:

const research = defineAgent({
  agent: {
    slug: "research",
    model: { provider: "my-provider", model: "my-model" },
    system: "Research the question.",
    experiment: {
      name: "research-prompt",
      revision: "v1",
      strategy: "custom",
      variants: [
        { name: "control", weight: 50 },
        { name: "concise", weight: 50, system: "Write a concise research brief." }
      ]
    }
  },
  tools: [],
  resolveExperiment: async ({ identity }) => ({
    key: identity!.id,
    variant: identity!.id === "internal-tester" ? "concise" : "control"
  })
})

The hook can call your existing experimentation service. It runs in your application when research.sessions.start is called. Low-level clients supply its result directly as client.sessions.create({ agentId, experiment: { key, variant } }). Custom routing owns its own assignment consistency; Cereal validates the variant and persists it.

Tools read the stored selection through ctx.session.experiment. The same context reaches descendants and hooks, covered by the existing body signature. Caller-owned metadata.experiment does not override it. Referenced sub-agent versions are pinned when the parent is assigned; external MCP implementation deployments still need versioned endpoints or application deployment metadata.

Collect human feedback and application checks

await research.sessions.feedback({
  id: session.id,
  name: "helpful",
  result: { value: true, notes: "Answered the question clearly." },
  rater: "user-42",
  idempotencyKey: "vote-123"
})

await research.sessions.feedback({
  id: session.id,
  name: "sources-cited",
  source: "code",
  result: { value: true },
  idempotencyKey: "citation-check-123"
})

Values can be boolean, numeric, or categorical strings. Source defaults to human. The optional rater is your application's assertion; Cereal separately records the authenticated producer as author. Submit these calls through your server, where you authorize the person rating the session.

Pass throughSeq to associate a rating with a particular answer or transcript boundary. Omission uses the current event-log head. Repeating an idempotency key returns the original result. Corrections use a new key and preserve the old rating.

Attach an evaluator agent

First register a judge with a focused rubric. Use the version returned by the API:

const judge = await client.agents.upsert({
  slug: "research-judge",
  definition: {
    name: "Research outcome evaluator",
    model: { provider: "my-provider", model: "my-judge-model" },
    system: `Evaluate whether the research brief answers the question and cites sources that support its claims.
Use tool results as evidence. A claim that an action succeeded is insufficient.
Return value true for success, false for failure, or "unknown" if the evidence is
insufficient. Explain the finding briefly in notes and cite event sequence numbers.`,
    tools: [],
    policies: {},
    limits: { maxTurns: 4 }
  }
})

Add this to the subject agent's definition, alongside its experiment if it has one:

evaluators: [{
  name: "task-success",
  agent: "research-judge",
  version: judge.version,
  trigger: "turn",
  sampleRate: 0.1,
  maxSeconds: 120
}]

Experiments and evaluators are independent: either works without the other. Automatic evaluators run on root sessions and inspect the recorded parent and sub-agent activity. Three trigger choices are available:

The default sample rate is 1. A turn completing means the agent stopped generating, not necessarily that the customer's entire task is over. Choose idle or an explicit application signal for conversation-level outcomes.

Each evaluator gets a durable session of its own, with its own usage and a link back to the subject. It receives an immutable transcript snapshot and returns a JSON object with value and optional notes. It cannot automatically trigger more evaluators or wake the parent. It uses its own tools and does not inherit the parent's credentials or sandbox. Evaluation charges count against the environment's spend cap.

const evaluation = await research.sessions.evaluate({
  id: session.id,
  evaluator: "task-success"
})

const results = await research.sessions.evaluations({ id: session.id })

Manual evaluation bypasses sampling and idle delay. It can also target an individual sub-agent session. throughSeq selects an older boundary. Repeating the same request returns the same job; provide a new idempotencyKey to deliberately rerun it.

Results distinguish pending, running, completed, error, and skipped. Malformed judge output and timeouts produce an error without a task score. Use evaluatorSessionId to inspect the judge's events and usage. Sampled-out evaluations and superseded idle windows are recorded as skipped.

Compare variants

const results = await client.agents.experimentResults({
  id: agentId,
  experiment: "research-prompt",
  revision: "v1"
})

const sessions = await client.sessions.list({
  filter: { experiment: { experiment: "research-prompt", revision: "v1", variant: "concise" } }
})

Comparison rows separate agent versions and report assigned sessions, exposed sessions, distinct exposed assignment subjects, token usage, and score groups. Exposure means a session actually reached a model request. Human, code, and agent scores remain separate; agent score groups also identify the judge version.

subjectMean averages boolean/numeric ratings within each exposed assignment subject, then gives each rated subject equal weight. Unrated subjects do not contribute a zero. Categorical values, evaluator errors, skipped jobs, and feedback coverage remain visible. These descriptive results do not establish statistical significance or correct for feedback selection bias. Calibrate judges against human examples before relying on their scores, and use actual application outcomes where available.