Training and run labels

Add objective and training.labeler to an outcomeci.workflow/v1 file to annotate completed runs. These fields were introduced before CLI v0.71.0 and remain part of v1. The block creates evaluation records; it does not itself fine-tune a model or change the workflow's execution result.

A complete example

apiVersion: outcomeci.workflow/v1
name: summarize-request
objective: Produce a concise, actionable summary of the supplied request.
trigger: manual
secrets:
  coder: vault:agents/codex
  labeler: vault:models/anthropic
reasoning:
  default:
    agent: codex
    auth: secrets.coder
training:
  labeler:
    mode: llm
    model: anthropic/claude-haiku-4-5
    auth: secrets.labeler
    review_below: 0.7
steps:
  - summarize:
      reason: Summarize the request in trigger.payload in one actionable sentence.
      with: [trigger]
      returns: {summary: string}

Store and grant both Vault credentials before a cloud run. The agent login and the labeler's model API key are separate credentials: a Claude or Codex agent subscription does not supply a direct model API key. For local runs, store the same paths in the local Vault.

objective must be a non-empty string. It is independent of training: you can declare an objective without a labeler. A labeler can also operate without an explicit objective, using the run record to judge the result.

Labeler settings

training accepts only labeler, whose settings are:

The model credential uses the same provider matching and cloud platform-key rules as model profiles. Declare fallback credentials in secrets and grant them too. In local runs, provide the required local model credentials. The labeler is a model profile, not an agent profile: agent and runner are not labeler settings.

Choose a decision model

Use mode: decision to evaluate a fixed rubric through a decision model:

secrets:
  labeler: vault:models/typesafe
training:
  labeler:
    mode: decision
    model: typesafe/jev-latest
    auth: secrets.labeler
    review_below: 0.7

Decision mode accepts typesafe/<model> or openai/gpt-6-luna. Use the model ID available to your account and a matching Vault provider. It does not support fallback. TypeSafe models and openai/gpt-6-luna require decision mode; they cannot be used as an LLM labeler or its fallback.

The rubric asks two questions: whether the outcome is success, partial, or failure, and the primary failure category: none, missing_output, incorrect_output, incomplete_execution, insufficient_evidence, or other. The annotation confidence is the lower of the two answer confidences. Decision mode preserves the answers under decisions; explanation and evidence are empty because the decision model does not generate those texts. Choose LLM mode when you want generated explanations and evidence references.

What gets recorded

The labeler runs after a workflow completes successfully. It sees the objective, run status and summary, step states, skip reasons, and bounded step outputs. Credentials and agent transcripts are not part of that input. A failed or interrupted workflow is not labeled by this completion hook.

For local runs, the annotation is written beneath the run directory:

.outcomeci/outcomes/<run-id>/training/annotation.json

It records the labeler model and rubric version together with the outcome, confidence, labels, explanation, evidence, and any failure category. The labeler's own transcript and status are also recorded. These records are separate from the step's returns JSON and do not replace it. See Run artifacts.

When confidence is below review_below, the annotation is marked as requiring human review. Omitting the threshold disables that routing. This flag does not pause a completed workflow or send a Slack approval request; use workflow human-approval steps when execution must wait for a person.

A labeler failure is recorded separately and does not turn a completed run into a failed run. Labels describe the evidence available to the labeler, so an execution that completed can still receive a partial or failure assessment when its outputs do not satisfy the objective.