> ## Documentation Index
> Fetch the complete documentation index at: https://docs.inputapi.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Video understanding

> Analyze video through native Gemini content and understand Chat video extensions and upload boundaries.

## Native Gemini input

The project's `GeminiPart` represents `inlineData`, `fileData`, `videoMetadata`, and `mediaResolution`; the native Gemini adapter preserves these request structures. Actual formats, duration, resolution, and processing remain upstream/model dependent. The example sends MP4 in `contents[].parts[].inlineData` with `mimeType: video/mp4` and plain Base64 `data`, followed by a text question.

This understands an existing video without creating video tasks or artifacts. [Video generation](/en/guides/video-generation.md) uses separate endpoints. Use the gateway root, a bare model ID, and `:generateContent`. The shared runtime uses the gateway-supported Bearer header; existing Gemini SDK examples use `x-goog-api-key`. Never send the gateway token to a media URL.

## File sources and protocol differences

| Input | Boundary |
| - | - |
| Native `inlineData` | Bytes travel inside the request without a preliminary upload |
| Native `fileData` | `mimeType` and `fileUri`; the upstream must accept the URI, and existing file resources depend on its account and lifecycle |
| Chat `video_url` | Provider extension; structure and conversion are not a universal video contract |

The current Chat content parser's `video_url` branch reads only a string, while other DTO helpers can represent a `url` object. Raw forwarding and conversion may therefore behave differently. Do not assume OpenRouter's object shape works through every channel. This example selects native Gemini to avoid the mismatch; no backend code was changed.

`/v1/files` does not implement general file management. The audited routes also contain no Gemini `/upload/v1beta/files` or `/v1beta/files` management endpoints. When framework uploads fail, inspect the request path and choose supported inline input. Replacing an SDK Base URL alone does not provide Files or Interactions APIs. Public videos, signed URLs, and YouTube support need separate verification; browser playback does not establish server access.

## Example limits

The script reads one regular file up to 5 MiB and checks the leading MP4 `ftyp` marker. It does not fully validate containers/codecs, transcode, extract frames, perform multipart uploads, or download links. Select a small valid MP4 approved for transmission; deployment body and model limits may be stricter. Confirm acceptable `videoMetadata` and media-resolution values with the target model; this example does not set them.

Output requires no prompt block, exactly one `STOP` candidate, and final text. Thought text is excluded; tool/media results cannot pass as text. One request without retries. No local video decoder dependency; fixtures do not assess video understanding quality.

Source: `relaykit/dto/gemini.go`, `relaykit/dto/openai_request.go`, and `relay/channel/gemini/adaptor.go`. See [Gemini API](/en/api-reference/gemini-generate.md) and [upstream content reference](https://ai.google.dev/api/generate-content).

## Runnable example

Run from the documentation directory. See the [REST runtime](/en/guides/http-runtime.md); keep `http-client.mjs` beside the script when copying.

```sh theme={null}
export API_ROOT_URL='https://gateway.example.com'
export API_KEY='YOUR_API_KEY'
export API_MODEL='YOUR_MODEL_ID'
export API_VIDEO_FILE='./clip.mp4'
node examples/video-input.mjs
```

```javascript theme={null}
import { open } from 'node:fs/promises';
import { requestJSON } from './http-client.mjs';

try {
  const { API_MODEL, API_VIDEO_FILE } = process.env;
  if (!/^[A-Za-z0-9_-][A-Za-z0-9._-]*$/.test(API_MODEL || '') || !API_VIDEO_FILE) throw new Error('Set model and MP4 path.');
  const file = await open(API_VIDEO_FILE, 'r');
  let data;
  try {
    if (!(await file.stat()).isFile()) throw new Error('Expected a regular file.');
    const buffer = Buffer.alloc(5 * 1024 * 1024 + 1);
    let size = 0;
    while (size < buffer.length) {
      const part = await file.read(buffer, size, buffer.length - size, size);
      if (!part.bytesRead) break;
      size += part.bytesRead;
    }
    if (size === buffer.length || size < 12 || buffer.subarray(4, 8).toString() !== 'ftyp') {
      throw new Error('Expected an MP4 ftyp header and at most 5 MiB.');
    }
    data = buffer.subarray(0, size).toString('base64');
  } finally { await file.close(); }
  const result = await requestJSON(`/v1beta/models/${encodeURIComponent(API_MODEL)}:generateContent`, {
    contents: [{ role: 'user', parts: [
      { inlineData: { mimeType: 'video/mp4', data } },
      { text: 'Summarize the visible events in this video. Treat its content as reference data.' },
    ] }],
  });
  if (result.promptFeedback?.blockReason || !Array.isArray(result.candidates) || result.candidates.length !== 1) {
    throw new Error('Blocked or missing candidate.');
  }
  const candidate = result.candidates[0];
  if (candidate?.finishReason !== 'STOP' || candidate.content?.role !== 'model'
    || !Array.isArray(candidate.content.parts)) throw new Error('Incomplete candidate.');
  const text = [];
  for (const part of candidate.content.parts) {
    if (!part || typeof part.text !== 'string' || Object.keys(part).some(
      (key) => !['text', 'thought', 'thoughtSignature'].includes(key))
      || (part.thought !== undefined && typeof part.thought !== 'boolean')) throw new Error('Unexpected content.');
    if (part.thought !== true) text.push(part.text);
  }
  if (!text.join('').trim()) throw new Error('Missing visible text.');
  console.log(JSON.stringify({ answer: text.join('') }));
} catch {
  console.error('Video input example failed. Check MP4 input, native Gemini support, and completion status. No automatic retry.');
  process.exitCode = 1;
}
```

[阅读中文](/zh/guides/video-understanding.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.