Documentation / THOR Data
13 / 17
Quality and verification
Structural checks, model-assisted review and hidden gold tasks assess each submission.
On this page
THOR's verification operating model combines task-specific gates, model-assisted scoring, hidden gold tasks and cross-checks between contributors. Verified batches then go to the buyer for acceptance. Each job defines the specification, scoring rubric and review parameters.
Acceptance before production
The buyer and THOR agree the required output, acceptance criteria and review process before production begins. The specification sets the checks, thresholds and procedures for correction, rejection and disputes. The rubric stays fixed for assigned work. From brief to delivery explains the terms published before work starts.
Workers complete task-specific qualification with test examples, including examples with known answers, before receiving production assignments.
Coverage
Required actions are labelled.
Gaps
Unlabelled intervals follow the task rules.
Segments
Actions use the requested level of detail.
Alignment
Labels match the action and its timing.
Coloured bars show labelled actions. Grey intervals show unlabelled time.
Acceptance criteria agreed before productionApply the same specification to review, rework and acceptance.
Structural validity and annotation quality
Structural gates check the required format, fields and coverage. For action captioning, the specification defines timing, gaps and segment boundaries. A task requiring object boxes also checks that all required objects are annotated. Mandatory gate failures make a submission ineligible to pass, regardless of its content score.
Content review assesses whether annotations match the visible evidence. A vision-language model (VLM) scores label accuracy, consistency and, for box-annotation tasks, box fit against the frozen rubric. The mechanism calls this content-quality score q. The job's pass threshold determines which scored submissions can proceed.
Verification methods
| Number | Layer | What it assesses |
|---|---|---|
| 01 | Structural gates | Required format, fields and annotation coverage |
| 02 | VLM content scoring | Labels, consistency and applicable box fit against the frozen rubric |
| 03 | Hidden gold / Ground truth | Accuracy against human-checked reference answers |
| 04 | Overlapping assignments | Agreement between independent annotations; disagreements sent for review |
| 05 | Duplicate and replay checks | Repeat submissions and unauthorised copies before additional payout |
Hidden gold and ground truth
Gold tasks are examples with human-checked reference answers used as ground truth for evaluation. They are mixed into assignments, with their gold status hidden from workers. Submissions on those tasks are assessed against the reviewed answers.
Gold accuracy provides a quality signal alongside the model-generated score. It helps expose workers who copy incorrect pre-labels, and known-answer examples also help assess checker reliability. The job defines its reference set and sampling level.
Overlapping assignments
Selected units are independently assigned to a second worker as a cross-check. Reviewers compare the annotations and investigate disagreements. Both workers can earn payment when their authorised assignments pass; cross-check rewards are included in the job reserve.
Authorised validators will have access to samples under the project's permissions. Sensitive source material and submissions stay restricted to authorised participants.
Checking each submission
The mechanism runs structural gates first, then VLM content checks against the fixed rubric. It records the quality result for the unit. A checker outage leaves the unit pending until evaluation resumes.
Worker notes are treated as task data. The scorer follows the frozen rubric when evaluating the submission. Failed submissions follow the correction process specified for the job.
Unit and submission fingerprints identify repeat submissions and unauthorised copying. Repeated submissions receive no additional payout, and a correction replaces the earlier submission. Content identified as an unauthorised copy is flagged and excluded from payout. Governance and transparency explains the distinction between duplicates and commissioned cross-checks.
The action-captioning example remains the initial task described in these docs. Other tasks define their own required labels and annotation fields.
Allocating future work
Gold-task accuracy and buyer acceptance history determine future work allocation, subject to declared worker capacity. The first round is shared equally among qualified workers. Subsequent rounds also reserve opportunities for newcomers to establish a history. These decisions affect future assignments while preserving rewards already earned.
Job rules specify scoring thresholds, sampling levels and allocation parameters.
Reliability and cost
During beta, THOR will measure review consistency, correction effort and the total cost of gold-reference preparation, review, rework and dispute handling. Those results will guide refinements to the checks and delivery budgets.
Buyer acceptance assesses delivery against the specification. For projects that include downstream evaluation, held-out robotics tests separately measure the data's effect on model performance. That assessment sits alongside delivery verification.
Verification results, buyer acceptance and payment remain separate records. Worker payments explains how verification and acceptance release the two payment stages. From brief to delivery describes the records that accompany accepted work.