OLLM

OLLM Integration with the Vercel AI SDK

Step-by-step guide to integrating OLLM with the Vercel AI SDK. Configure the OpenAI-compatible provider for secure, TEE-backed LLM inference in your Next.js application.

The OLLM provider enables you to use OLLM models directly with the AI SDK through a unified, OpenAI-compatible interface.

All models accessed through this provider execute with confidential computing enabled by default. From the SDK's perspective, you interact with OLLM like any other model provider, but inference runs inside hardware-backed Trusted Execution Environments (TEEs) with zero data retention.

Using the provider gives you:

  • Verifiable privacy: Every model runs under confidential computing with cryptographic proofs of secure execution
  • One API key, hundreds of models: Claude, GPT, Gemini, Llama, GLM, Kimi, DeepSeek, Qwen, Mistral, Whisper, and more
  • Dynamic model discovery: Fetch the live catalog at runtime; no hardcoded IDs to go stale
  • Multimodal input: Text, images, PDFs (and other documents), plus speech-to-text via Whisper
  • OpenAI-compatible wire format: Drop-in for tools that already speak OpenAI
  • Unified API access: Works seamlessly with generateText, streamText, embed, embedMany, and experimental_transcribe

This page covers installation, initialization, and usage examples for integrating OLLM models into your AI SDK applications.

Setup

The OLLM provider is available in the @orgn/gateway module.

Install using your preferred package manager:

pnpm add @orgn/gateway
npm install @orgn/gateway
yarn add @orgn/gateway
bun add @orgn/gateway

Then grab an API key from the OLLM Dashboard and export it:

export OLLM_API_KEY="sk-ollm-..."

Create a Provider Instance

Import and initialize the provider using createOLLM.

provider-setup.ts
import { createOLLM } from '@orgn/gateway';

const ollm = createOLLM({
  apiKey: process.env.OLLM_API_KEY,
  // baseURL: 'https://api.ollm.com/v1',  // optional, defaults to this
  // headers: { 'X-Request-Id': '...' },  // optional extra headers
  // fetch: customFetch,                  // optional fetch override (testing, proxies)
});

OLLM_API_KEY is also picked up automatically if you omit apiKey. There's also a default ollm export if you don't need custom settings:

import { ollm } from '@orgn/gateway';

Discovering Available Models

OLLM's catalog changes frequently as new providers and model versions land. Instead of hardcoding IDs, ask the gateway directly:

list-models.ts
// All active models
const all = await ollm.listModels();

// Filter by modality
const chat       = await ollm.listModels({ inputModality: 'text',  outputModality: 'text' });
const embeddings = await ollm.listModels({ outputModality: 'embedding' });
const vision     = await ollm.listModels({ inputModality: 'image' });
const audio      = await ollm.listModels({ inputModality: 'audio' });

// Include inactive models in the result
const everything = await ollm.listModels({ activeOnly: false });

Each entry includes details such as:

{
  id: 'near_glm_5_1',
  display_name: 'GLM 5.1',
  owned_by: 'zai',
  is_active: true,
  input_modalities:  ['text'],
  output_modalities: ['text'],
  max_input_tokens:  202752,
  max_output_tokens: 131072,
  input_cost_per_token:  8.5e-7,
  output_cost_per_token: 3.3e-6,
  model_info: { TEE: true },
}

Use the id field directly as the argument to chatModel(), embeddingModel(), or transcriptionModel().

Language Models

All OLLM models run with confidential computing enabled by default. Use ollm.chatModel() to access chat-capable models:

specify-model.ts
const confidentialModel = ollm.chatModel('near_glm_5_1');

Refer to the OLLM Models page for the full catalog.

Examples

generateText

generate-text.ts
import { createOLLM } from '@orgn/gateway';
import { generateText } from 'ai';

const ollm = createOLLM({
  apiKey: process.env.OLLM_API_KEY,
});

const { text, usage } = await generateText({
  model: ollm.chatModel('near_glm_5_1'),
  prompt: 'What is OLLM?',
});

console.log(text);

streamText

stream-text.ts
import { createOLLM } from '@orgn/gateway';
import { streamText } from 'ai';

const ollm = createOLLM({
  apiKey: process.env.OLLM_API_KEY,
});

const result = streamText({
  model: ollm.chatModel('vercel_claude_sonnet_4_6'),
  prompt: 'Write a short story about secure AI.',
});

for await (const chunk of result.textStream) {
  process.stdout.write(chunk);
}

Using System Messages

system-messages.ts
import { createOLLM } from '@orgn/gateway';
import { generateText } from 'ai';

const ollm = createOLLM({
  apiKey: process.env.OLLM_API_KEY,
});

const { text } = await generateText({
  model: ollm.chatModel('near_glm_5_1'),
  system: 'You are a helpful assistant that responds concisely.',
  prompt: 'What is TypeScript in one sentence?',
});

console.log(text);

Multi-turn Conversations

Use messages instead of prompt whenever you have a real conversation:

multi-turn.ts
const { text } = await generateText({
  model: ollm.chatModel('near_glm_5_1'),
  messages: [
    { role: 'user',      content: 'What is a TEE?' },
    { role: 'assistant', content: '…' },
    { role: 'user',      content: 'How does that protect my prompt?' },
  ],
});

Provider Options

Chat calls accept OLLM-specific options under providerOptions.ollm:

provider-options.ts
const { text } = await generateText({
  model: ollm.chatModel('vercel_gpt_5'),
  prompt: 'Walk me through the proof.',
  providerOptions: {
    ollm: {
      reasoningEffort: 'high',  // 'low' | 'medium' | 'high', for reasoning models (o1, o3, gpt-5, …)
      user: 'user-1234',        // optional end-user identifier for abuse monitoring
    },
  },
});

Multimodal Input

Images

Pass image bytes or a URL as a file content part with an image/* media type. The provider serializes them as OpenAI-canonical image_url parts.

image-input.ts
import { readFile } from 'node:fs/promises';
import { generateText } from 'ai';

const image = await readFile('photo.jpg');

const { text } = await generateText({
  model: ollm.chatModel('vercel_claude_sonnet_4_6'),
  messages: [{
    role: 'user',
    content: [
      { type: 'text', text: 'Describe this image.' },
      { type: 'file', data: image, mediaType: 'image/jpeg' },
    ],
  }],
});

Supported media types include image/jpeg, image/png, image/webp, image/gif, and any other image/* value the underlying model accepts.

PDFs and Documents

Non-image file parts are passed through as inline base64, no S3 or Files API required. The SDK rewrites them into OpenAI's canonical type: "file" shape (file.file_data) before sending, so they work with any AI SDK call that takes messages:

pdf-input.ts
import { readFile } from 'node:fs/promises';
import { generateText } from 'ai';

const pdf = await readFile('report.pdf');

const { text } = await generateText({
  model: ollm.chatModel('vercel_claude_sonnet_4_6'),
  messages: [{
    role: 'user',
    content: [
      { type: 'text', text: 'Summarize this report in three bullets.' },
      { type: 'file', data: pdf, mediaType: 'application/pdf', filename: 'report.pdf' },
    ],
  }],
});

Common document types: application/pdf, text/plain, text/markdown, text/csv, application/json. Pick a model whose input_modalities from listModels() includes 'pdf' or 'text' (Claude 4.x, Gemini 2.5/3, GPT-4.1+, Kimi K2.6, etc.).

Embeddings

embeddings.ts
import { embed, embedMany } from 'ai';

// Single vector
const { embedding } = await embed({
  model: ollm.embeddingModel('near_qwen3_embedding_0_6b'),
  value: 'OLLM routes confidential LLM traffic.',
});

// Batch
const { embeddings } = await embedMany({
  model: ollm.embeddingModel('vercel_text_embedding_3_small'),
  values: [
    'Confidential computing protects data in use.',
    'TEEs use hardware-level encryption.',
  ],
});

Discover available embedding models with ollm.listModels({ outputModality: 'embedding' }).

Audio Transcription (Speech → Text)

OLLM exposes Whisper-style speech-to-text through transcriptionModel(), compatible with the AI SDK's experimental_transcribe() helper:

transcription.ts
import { experimental_transcribe as transcribe } from 'ai';
import { readFile } from 'node:fs/promises';

const audio = await readFile('meeting.mp3');

const result = await transcribe({
  model: ollm.transcriptionModel('near_whisper_large_v3'),
  audio,
});

console.log(result.text);                // full transcript
console.log(result.language);            // e.g. "english"
console.log(result.durationInSeconds);   // total audio length
for (const seg of result.segments) {
  console.log(`[${seg.startSecond}s – ${seg.endSecond}s] ${seg.text}`);
}

You can pass Whisper-specific options through providerOptions.ollm:

transcription-options.ts
await transcribe({
  model: ollm.transcriptionModel('near_whisper_large_v3'),
  audio,
  providerOptions: {
    ollm: {
      language: 'en',
      temperature: 0,
      prompt: 'OLLM, Whisper, TEE',  // hint for tricky vocabulary
    },
  },
});

Supported audio types: audio/mpeg, audio/wav, audio/mp4 (m4a), audio/webm, audio/flac, audio/ogg.

What is Not Supported

CapabilityStatusWhy
imageModel() (text → image)Throws NoSuchModelErrorThe OLLM gateway doesn't expose image-generation models through the AI SDK provider interface yet. Image-output models from listModels() (vercel_flux_*, vercel_seedream_*, etc.) are reachable via raw HTTP, but not wired up here.
completionModel() (legacy /v1/completions)RemovedThe OLLM gateway only serves /v1/chat/completions. Use chatModel(), every "completion" task can be expressed as a chat call.
Speech generation (TTS)Not availableNo TTS models in the OLLM catalog (zero models with audio in output_modalities).

Security & Confidential Computing

All models accessed through OLLM execute inside Trusted Execution Environments (TEEs). This provides:

Zero Data Retention (ZDR)

Prompts and completions are not stored or logged by providers.

Confidential Computing

Hardware-level encryption ensures data remains protected during processing.

Verifiable Privacy

Inference runs inside attested environments, enabling cryptographic verification of execution integrity.

Model Flexibility

OLLM provides access to multiple models from many providers under a single API key. All OLLM models run with confidential computing by default. Swap models with a one-line change; no per-provider SDKs required.

model-selection.ts
// Confidential computing chat models
const confidentialModel = ollm.chatModel('near_glm_5_1');

Cost & Usage Tracking

Usage and token accounting are available through the OLLM dashboard, enabling:

  • Real-time cost monitoring
  • Per-model usage visibility

Enterprise Features

For high-volume or regulated workloads, OLLM offers enterprise support with custom SLAs.

API Reference

import {
  createOLLM,
  ollm,
  type OLLMProvider,
  type OLLMProviderSettings,
  type OLLMChatModelId,             // alias for string
  type OLLMChatProviderOptions,     // providerOptions.ollm shape for chat
  type OLLMEmbeddingModelId,        // alias for string
  type OLLMEmbeddingProviderOptions,
  type OLLMTranscriptionModelId,    // alias for string
  type OLLMModel,                   // shape returned by listModels()
  type ListModelsOptions,
  type OLLMErrorData,
  VERSION,
} from '@orgn/gateway';

OLLMProvider exposes:

  • (modelId): shorthand for chatModel(modelId)
  • chatModel(modelId), languageModel(modelId): chat / text generation (V3)
  • embeddingModel(modelId), textEmbeddingModel(modelId) (deprecated alias): embeddings (V3)
  • transcriptionModel(modelId): speech-to-text (V3)
  • imageModel(modelId): currently throws NoSuchModelError
  • listModels(options?): fetch the live /v1/models catalog with optional modality/active filters

Tool Integrations

OLLM works with popular AI development environments and tools, including:

  • Cursor
  • Visual Studio Code
  • Replit
  • Windsurf
  • Cline
  • Roo Code
  • Origin

These tools can connect using OLLM's OpenAI-compatible interface.

Additional Resources

OLLM API Requests and Response Structure

How OLLM structures API requests and responses for confidential LLM inference. Covers the response envelope, success and error handling, usage metadata, attestation data, and production integration patterns.

Sandboxes

The OLLM Sandboxes dashboard gives you visibility into compute sandboxes running under your organization. Sandboxes are created and managed in ORGN, OLLM provides a read-only view for monitoring and tracking.

On this page

Setup
Create a Provider Instance
Discovering Available Models
Language Models
Examples
generateText
streamText
Using System Messages
Multi-turn Conversations
Provider Options
Multimodal Input
Images
PDFs and Documents
Embeddings
Audio Transcription (Speech → Text)
What is Not Supported
Security & Confidential Computing
Zero Data Retention (ZDR)
Confidential Computing
Verifiable Privacy
Model Flexibility
Cost & Usage Tracking
Enterprise Features
API Reference
Tool Integrations
Additional Resources