From brief to delivery

The workflow connects a buyer's brief to funded assignments, evaluated annotations and accepted delivery. THOR coordinates the order, workers produce the annotations, authorised reviewers evaluate submissions and the buyer accepts the delivered batches. The operating model below describes the mechanism; each job specifies its units, review conditions and payment terms.

1. 01

   Agree brief

   

   Buyer specification, acceptance criteria and delivery terms.

2. 02

   Fund / assign

   • Budget
   • Source
   • Spec

   Reserve worker rewards, then lease batched units.

3. 03

   Annotate

   

   Workers label actions or correct supplied pre-labels.

4. 04

   Verify

   

   Run gates, VLM scoring, gold tasks and cross-checks.

5. 05

   Buyer review

   • Accept
   • Rework
   • Reject

   Accept units, request rework or reject against the spec.

6. 06

   Deliver

   

   Package accepted output for the buyer.

• Steps 01–02Agreed taskSpecification and approved source material
• Steps 03–04Review recordAnnotation and verification result
• Steps 05–06Buyer deliveryAccepted output with provenance

Publish payment-linked delivery evidence without exposing customer data.

Agree the brief

Before production begins, the buyer and THOR agree the source material, permitted use, annotation schema, delivery format and acceptance criteria. THOR will publish the applicable acceptance criteria, payment terms and dispute process before work starts, while keeping sensitive customer material restricted. The specification and review rubric are fixed for assigned work. Scope changes require an agreed update, with a clear record of which specification applies to each assignment.

A useful brief answers five questions:

• Purpose: what training or evaluation task should the annotations support?
• Source and rights: which material may be used, and who may access it?
• Output: which actions, temporal boundaries, schema and delivery format are required?
• Acceptance: which checks, review criteria and correction process apply?
• Commercial scope: what volume, budget, delivery schedule and payment terms are agreed?

Calibrate and reserve funding

A small calibration batch lets the buyer check the task and output before main production. Before the main work is assigned, THOR reserves the full budgeted worker reward from treasury funds already held. A job waits if sufficient funding is unavailable. Economics explains the reserve and staged payments.

Produce and review

A project is divided into batches and task-specific units. Workers receive leases: defined assignments with a completion window. Related units can be grouped to preserve context. Planned cross-checks are assigned separately and included in the reward budget.

A worker annotates the approved material or corrects model-generated pre-labels where the task provides them. Review checks the submission against the fixed rubric. Failed submissions can return for rework under the job's terms, and unfinished assignments can be reassigned when their leases expire. Quality and verification explains the checks.

Buyer review and rework

Batches that pass verification go to the buyer with their annotations and review records. The buyer can accept a batch, accept part of it or return units for correction within the agreed review window. Rejections must cite a requirement in the specification. Disputed or unusually high rejection rates receive sample review through the agreed dispute process before disputed buyer-linked payments are finalised.

The buyer pays for the units finally accepted. Accepted work unlocks the acceptance-linked worker payment described in Economics.

The verified unit

An hour of footage, an episode and a trajectory describe different things. The unit of delivery must be defined for the task.

An intended verified unit combines the accepted annotation with its source reference, applicable specification and verification record. This should let a buyer trace what was requested, what was produced and how it was judged. The delivery format should specify which records accompany the accepted work.

Deliver accepted output

The buyer receives accepted annotations in the agreed format, with the supporting verification and provenance records. Worker earnings and transfers are recorded under the job's staged payment terms. Economics explains how those payments are funded.

The public reporting plan connects payments with evaluated delivery. Data rights and access governs private material and any authorised reuse of review records.

See action captioning for an illustrative task and Quality and verification for the distinction between structural checks and annotation quality.