How to build a reliable video-to-prompt pipeline

A video-to-prompt tool looks simple from the outside: upload a clip, wait a moment, and copy the result. The hard part is not generating text. It is preserving enough of the source video's structure that the prompt remains useful when another model interprets it. I learned this while working on a small video analysis workflow. Early versions produced fluent paragraphs, but they often dropped a camera move, merged two events, or placed dialogue in the wrong shot. The output sounded good and still failed as a production prompt. The fix was to stop treating the result as one block of prose. Start with an intermediate representation I now treat the prompt as the last stage of a compiler. The video is first converted into a structured record, and only then rendered for a specific video model. A minimal record might look like this: { "duration_seconds" : 12.4 , "shots" : [ { "start" : 0.0 , "end" : 3.8 , "subject" : "a cyclist waiting at a red light" , "action" : "looks over the left shoulder" , "camera" : { "shot_size" : "medium" , "movement" : "slow push-in" , "angle" : "eye level" }, "dialogue" : null } ] } This structure is deliberately boring. That is useful. A typed record makes missing data visible and gives you something concrete to validate before you ask a language model to write polished prose. Normalize the input first Video files arrive with different frame rates, codecs, orientations, and audio layouts. Links from social platforms add another layer of inconsistency. If every downstream stage has to understand every input format, failures become difficult to reproduce. The ingestion stage should produce a canonical package: a timestamped frame stream at a known sampling rate a normalized audio track basic metadata such as duration, aspect ratio, and frame rate a stable internal time base Keep the original timestamps. Rounding everything to whole seconds is tempting, but it causes trouble in short clips where several actions happen in quick succession. Detect