Documentation / THOR Data
12 / 17
Initial task: action captioning
Describe visible actions and when they occur, according to the buyer's brief.
On this page
The initial task focuses on action captioning on first-person video. Annotators identify the visible action and mark its beginning and end according to the buyer's schema and timing requirements. The result is a sequence of labelled behaviour through time.
- 0.0–2.0 secReach for drawer
- 2.0–4.6 secOpen drawer
- 4.6–8.0 secRemove cup
The brief sets the label vocabulary, temporal boundaries and required output format. These timings are illustrative.
One clip becomes a time-aligned action record
A concrete example
A clip might show a person reaching for a drawer, opening it and removing a cup. The annotations describe those actions with the required labels and temporal boundaries. The buyer's brief sets the label vocabulary and submission format.
For the illustrative eight-second clip above, the action segments are:
- 0.0–2.0 seconds: reach for drawer.
- 2.0–4.6 seconds: open drawer.
- 4.6–8.0 seconds: remove cup.
These timings illustrate the format; the buyer's brief sets the required temporal granularity. For obscured or ambiguous actions, annotators follow the brief's uncertainty policy and label only what the evidence supports.
The brief controls the detail
The buyer's brief determines the camera view, task scope, label granularity, timing requirements and how uncertainty should be handled. Approved source material can carry different access and reuse conditions.
Frame-level labels and action segments are possible representations. The required format and any merging of adjacent labels are agreed for the task.
What the pilot should establish
This bounded task gives THOR a way to assess annotation usefulness, verification reliability, correction effort and total delivery cost. Broader task types should be introduced when there is a defined need and a credible way to assess the output.
Quality and verification explains how review should distinguish a correctly formatted submission from a correct description of the action.