Youtu-Parsing-Omni: Turn Documents and Media into Structured Data

A task prompt selects the output family. Document parsing includes reading order, bounding boxes and content such as text, LaTeX and tables. Audio and video tasks return timed segments; text-rich video can include OCR, ASR and a report. The result is structured text, so this guide belongs with text-output models even though inputs span several modalities.

Use a parser when layout and timestamps need to survive into downstream data. A general assistant such as MiMo can interpret a file and reason about it, while Youtu’s task prompt supplies a specific output contract. Validate required fields and compare them with the source before importing them into a search index or business record.

Use the youtu_parsing_omni subdirectory of Tencent’s repository. Its setup requires Python 3.10 or later and CUDA GPUs. The serving environment pins vLLM 0.19.0 and Transformers 5.2.0 with the model plugin; the separate Transformers mode uses 5.10.2. Install those modes in separate environments. The setup script removes PEFT from serving, and audio/video inputs need ffmpeg.

Review the repository scripts and pin a tested revision before installing them. The first command below prepares the serving environment; put a document page you are allowed to process in its media directory. Start the server in that environment, then use another terminal in the same environment for the parsing command.

The server binds to 127.0.0.1:8000/v1 with the served name Youtu-Parsing-Omni. Its DATA_ROOT is the allowed local-media directory. The reference configuration uses BF16, one concurrent sequence and a 65,536-token context. The gpu-memory-utilization value of 0.70 is an allocation setting, not a minimum VRAM requirement.

The inference example selects the matching task prompt, sets parsing on and thinking off, and resolves the stopping tokens before sending the request. It prints the returned content. Keep this task-aware client when starting out instead of copying a generic chat request that omits those controls.

Save the returned content and parse its JSON, then inspect a small labeled set. For a page, compare reading order, table rows and exact figures. For recordings, compare transcript and timestamps with playback. JSON that parses successfully can still contain a wrong value or missing element; count corrected and rejected outputs in the throughput measurement.

The official sources provide weights and a CUDA recipe, but no dependable lowest-VRAM configuration or original-vendor hosted API price. A 5B parameter count is not a complete memory budget: precision, the multimodal encoder, media resolution, sequence length, caches and concurrency also consume capacity. Use the recipe on a controlled sample and record peak device and host memory before committing to a rental.

For an assumed $1.50-per-hour GPU rented for 40 minutes, GPU time costs $1 before storage and transfer. If 200 pages pass review, that is $0.005 per accepted page; if only 100 pass, it is $0.01. These are arithmetic examples with a hypothetical rate, not a Youtu quote or measured page speed. Include startup, downloads, retries and idle time in the billable session.

For an owned machine, include electricity and the share of machine cost appropriate to the workload. For a hosted document service, compare its actual page/media price and retention conditions on the same accepted files. This page leaves API price and hardware presets unset rather than treating unavailable figures as zero.

Tencent’s release reports 96.96 Overall on OmniDocBench v1.6 and 75.08 average on OmniParsingBench. These are the model publisher’s benchmark tables, not an independent reproduction here. Overall document accuracy does not establish throughput or guarantee that your scanned table, language or recording will parse correctly.

Build a sample around the failure that costs you most: reordered columns, an omitted footnote, a mislabeled speaker or a chart value detached from its unit. Measure complete-file processing and review time, not only generated tokens. If the next step is retrieval, keep the extracted source and its locations with the vectors; EmbeddingGemma handles that separate retrieval task.

The checkpoint uses the custom youtu-parsing license. It explicitly says the model is not intended for use within the European Union, with that clause taking precedence in a conflict. Treat this as a deployment eligibility condition before downloading for an EU workflow; the model’s availability does not make it an ordinary Apache 2.0 release.

Read the exact checkpoint license for use and redistribution obligations, and review your intended deployment against it. The source repository, model materials and any hosted provider each have their own terms; an integration example does not expand the checkpoint’s permission.