What one interface buys you

The Gateway product page says the catalog currently contains 21 models. OCR, captioning, and multimodal chat use chat completions. VLM Run also says the service can take long PDFs or videos, then chunk, batch, and reassemble them. Its launch post documents document_dpi, document_pages, and stream: true. It says failed pages are retried or marked rather than killing the whole document. Output is normalized as Markdown with page markers, a unified bounding-box schema, and self-describing JSON, with usage.cost and an x-request-id on responses.

That contract removes a lot of glue code. A team can keep its client, request shape, authentication, and output envelope while changing only model. The official Python SDK repository includes CLI support and optional document and video dependencies. The Gateway also works with the regular OpenAI Python client.

A minimal comparison can look like this:

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai/",
    api_key="<VLM_RUN_API_KEY>",
)

models = ["rednote-hilab/dots.mocr", "paddlepaddle/paddleocr-vl-1.6"]

for model in models:
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": [{
            "type": "document_url",
            "document_url": {"url": "invoice.pdf"},
        }]}],
        extra_body={"method": "markdown", "document_dpi": 200},
    )
    print(model, response.usage, response.choices[0].message.content)

The models do not become equivalent. The plumbing simply stops moving while you examine their differences.

The endpoint can still conceal different tests

The Hugging Face launch note is unusually direct about serving variability. It says quantization and runtime settings can change OCR, small-text, and spatial accuracy even when the model ID looks identical. It also reports that more than 80% of the popular-router providers the team tested lacked video input support for video-native models. These are vendor-reported observations, not an independent audit. Still, they identify what should be tested: the deployed system, not a name in a catalog.

That is also the lesson from AgentArena's repository-specific evaluations: useful benchmarks resemble the work you plan to do. For document vision, keep the ugly inputs. Include skewed scans, faint faxes, dense tables, stamps, handwriting, non-Latin scripts, and multi-page failures. Averages can hide the one document class that matters most.

VLM Run publishes benchmark and pricing claims in its launch material. Use them to make a shortlist, then test that list yourself. The company selected the models, datasets, settings, and accounting. Your documents may reward a different choice.

Five controls for a useful model bake-off

Begin with a frozen corpus: original files, hashes, page ranges, and expected outputs. Put ordinary documents in one slice and difficult cases in others, including rotated pages, tiny print, mixed languages, and handwritten notes. An improved average means little if the new model mangles the regulated form your team processes all day.

Preprocessing needs its own record. Save document_dpi, page selection, video FPS, image resizing, color conversion, and any deskewing or denoising. The Gateway can centralize part of this work, but each model run must receive the same configuration.

Text accuracy and structural accuracy belong in separate columns. Character or word error rate cannot tell you whether table cells landed under the wrong headings. Depending on the application, measure field accuracy, reading order, table topology, bounding-box overlap, valid JSON, and page-level completion. NeoMME's compressed document embeddings make a related distinction: retrieval quality and extraction fidelity are different jobs.

Operational failures count too. Log timeouts, retries, blank pages, truncated outputs, malformed JSON, and partial documents. A system that reaches a high score after silent retries has a different cost and latency profile from one that finishes on the first attempt.

For cost, trust observed usage rather than a calculator alone. Store usage.cost, request IDs, page counts, output lengths, and retry counts. Report median and tail latency, not just throughput. Rerun the frozen corpus whenever a model revision or serving configuration changes. Without revision history, a benchmark can go stale while its chart still looks authoritative.

The open-source vlmbench CLI covers part of this process. It records model metadata, environment information, time to first token, time per output token, throughput, latency percentiles, and raw runs. It can sweep concurrency and supports Ollama, vLLM, and SGLang-oriented backends. It does not know what a correct invoice, form, or video result looks like for your application. Gold answers and failure categories remain your responsibility.

Keep the easy swap, distrust the easy verdict

VLM Run Gateway makes visual models replaceable without forcing developers to rebuild the surrounding document or video pipeline. That can make a bake-off cheaper and a fallback model realistic. It can also encourage a verdict after one edited line.

Keep the endpoint, but freeze the inputs and settings and test the deployed system. Teams routing language models through api.ish.chat face the same separation: compatibility is an integration feature; quality, latency, and cost are properties you measure. A shared endpoint lowers the cost of the experiment. It does not run the experiment for you.