---
title: "LLM provider configuration"
description: "LLM provider configuration for TinyMCE AI on-premises service"
canonical_url: "https://www.tiny.cloud/docs/tinymce/latest/tinymceai-on-premises-providers/"
md_url: "https://www.tiny.cloud/docs/tinymce/latest/tinymceai-on-premises-providers/index.md"
version: "latest"
last_updated: "2026-11-12T01:44:43Z"
tokens: 12534
---
# LLM provider configuration

This page configures **how the AI service reaches LLM providers** and **which models are exposed to clients**. It sits between the [data layer setup](../tinymceai-on-premises-database/) (which must be complete first) and [JWT authentication](../tinymceai-on-premises-jwt/) (which gates per-user access to the models configured here). The environment variables documented below are set on the AI service container — in a `docker run` command, a Kubernetes Deployment manifest, or an ECS task definition.

Two environment variables and one JWT claim control provider and model configuration:

- `PROVIDERS`: tells the AI service how to authenticate with each LLM provider API.
- `MODELS`: tells the service which models to expose to clients and which features each model supports.
- `auth.ai.permissions` (JWT claim): gates per-user access to individual models.

This page documents every supported provider `type`, every required field, and every known production issue for both variables.

Start with the [Getting started guide](../tinymceai-on-premises-getting-started/) if the AI service container is not yet running. The following sections assume a running `ai-service` container.

## Provider and model architecture

The AI service uses two related environment variables:

| Variable | Type | What it does |
| --- | --- | --- |
| `PROVIDERS` | JSON object | Map of provider IDs to provider configurations. Each entry says how to authenticate with one LLM provider API. |
| `MODELS` | JSON array | List of models exposed to clients. Each model points at a `PROVIDERS` entry and declares which features it can serve. |
| JSON Web Token (JWT) `auth.ai.permissions` | string array | Per-user authorization list. Includes `ai:models:<provider-key>:<model-id>` entries to gate access to individual models. |

The `PROVIDERS` keys are arbitrary identifiers (for example `"openai"`, `"my-bedrock"`, `"team-azure"`). Each value object has a `type` field that picks the implementation:

| `type` | Provider |
| --- | --- |
| `openai` | OpenAI (api.openai.com) |
| `anthropic` | Anthropic (api.anthropic.com) |
| `google` | Google AI Studio / Gemini direct |
| `azure` | Azure OpenAI Service |
| `bedrock` | AWS Bedrock |
| `vertex` | Google Cloud Vertex AI |
| `openai-compatible` | Any OpenAI-compatible HTTP API (Ollama, vLLM, LM Studio, llama.cpp, LocalAI, OpenRouter, Together, Groq, Fireworks, and similar) |

The **key** (not the `type`) is what gets referenced from:

- `MODELS[].provider`; to wire a model to a provider
- JWT `auth.ai.permissions`; for per-provider or per-model access control (see [JWT authentication](../tinymceai-on-premises-jwt/))

```json
{
  "my-openai-key":   { "type": "openai",   "apiKeys": ["sk-proj-..."] },
  "my-bedrock":      { "type": "bedrock",  "region": "us-east-1", "credentials": { "accessKeyId": "...", "secretAccessKey": "..." } },
  "my-ollama":       { "type": "openai-compatible", "baseUrl": "http://host.docker.internal:11434/v1" }
}
```
`GET /v1/models/1` reports the provider `name`, not the provider key, in the `provider` field of each model.

### How the pieces fit together

![PROVIDERS JSON structure mapping provider keys to OpenAI Anthropic Azure Bedrock Vertex and self-hosted endpoints](../_images/tinymceai-on-premises/providers-guide-fig-1.svg)
The diagram reflects three stacked layers: **how to authenticate** with each provider (`PROVIDERS`), **what to expose** to clients (`MODELS`), and **who can use which model** (JWT permissions). The same provider key string ties all three together.

## Defining the model list

### Default model behavior (`agent-1`)

The OpenAI, Anthropic, and Google providers come with a default list of models. The default list includes `agent-1`, the recommended agent model. Set `MODELS` to replace the default list, and to declare the models of Azure OpenAI, Amazon Bedrock, Google Vertex AI, and openai-compatible providers, which have no default list.

> **Important:** Set `MODELS` explicitly for production. Image versions can add models to the default list, so a deployment without `MODELS` can offer different models after an upgrade.

<!-- <details> -->
<!-- <summary> -->
Startup validation
<!-- </summary> -->
The service validates `PROVIDERS` and `MODELS` when the container starts:

- The service does not start without at least one provider in `PROVIDERS`.
- The service does not start until each conversation, review, and action feature has at least one model to serve it. List `conversations`, `reviews`, and `actions` on at least one model each, or cover each of their sub-features.
- A `MODELS` entry that names a provider key that is not defined in `PROVIDERS` stops startup. See [Multiple providers coexistence](#multiple-providers-coexistence).
- Some features need a model that supports structured output. Models of the same class as the latest major releases from Anthropic, Google, or OpenAI give the best results.

<!-- </details> -->

<!-- <details> -->
<!-- <summary> -->
Details: `agent-1` behavior and limitations
<!-- </summary> -->

- The `/v1/models/1` endpoint reports `agent-1` with `allowed: false` unless the JWT includes the `ai:models:agent` permission.
- Setting `MODELS` explicitly routes each model to the correct provider.
- See [JWT authentication](../tinymceai-on-premises-jwt/) for permission syntax.

<!-- </details> -->
For the response format of the model list endpoint, see [Fetch all models for a compatibility version](https://tinymceai.api.tiny.cloud/v1/docs#tag/Models/operation/getAiModels) in the API reference. The endpoint only accepts the literal version `1`. Other values return `400 Bad Request`:

```bash
# Works
curl http://localhost:8000/v1/models/1

# All 400
curl http://localhost:8000/v1/models/v1
curl http://localhost:8000/v1/models/v2
curl http://localhost:8000/v1/models/latest
```
**Minimum `MODELS` entry** to make a model usable:

```json
[
  {
    "id": "gpt-5-mini",
    "provider": "my-openai-key",
    "description": "OpenAI GPT-5 mini",
    "features": ["conversations", "reviews", "actions"]
  }
]
```
The `features` array must include at least one of `conversations`, `reviews`, or `actions` for the model to be selectable for that feature. A model with no overlapping features is invisible to that part of the UI.

A full field reference for `MODELS` is at the end of this page.

## OpenAI

API key from [platform.openai.com](https://platform.openai.com/api-keys). The OpenAI provider comes with a default model list, so it works without an explicit `MODELS` definition, but defining `MODELS` is recommended for production (see [Defining the model list](#models-required)).

> **Note:** When `MODERATION` is not set, the AI service also uses the API key of the OpenAI provider in `PROVIDERS` for content moderation, and sends prompts and uploaded images to the OpenAI moderation API even when other providers serve the models. See [Content moderation](../tinymceai-on-premises-moderation/).

<!-- <details> -->
<!-- <summary> -->
Configuration details
<!-- </summary> -->
**JSON shape:**

```json
{
  "openai": {
    "type": "openai",
    "apiKeys": ["sk-proj-YOUR_KEY_HERE"]
  }
}
```
**Fields:**

| Field | Required | Notes |
| --- | --- | --- |
| `type` | Yes | Literal `"openai"` |
| `apiKeys` | Yes | Array of one or more API keys. Multiple keys allow zero-downtime rotation. |
| `baseUrl` | No | Override only if proxying through a private OpenAI gateway. |

**Full `docker run` example:**

```bash
docker run --init -d -p 8000:8000 \
  --name ai-service \
  -e LICENSE_KEY="$LICENSE_KEY" \
  -e ENVIRONMENTS_MANAGEMENT_SECRET_KEY="$MANAGEMENT_SECRET" \
  -e DATABASE_DRIVER='mysql' \
  -e DATABASE_HOST='mysql' \
  -e DATABASE_USER='root' \
  -e DATABASE_PASSWORD="$DB_PASSWORD" \
  -e DATABASE_DATABASE='ai_service' \
  -e REDIS_HOST='redis' \
  -e STORAGE_DRIVER='database' \
  -e PROVIDERS='{"openai":{"type":"openai","apiKeys":["sk-proj-YOUR_KEY_HERE"]}}' \
  registry.containers.tiny.cloud/ai-service-tiny:latest
```
**Verify:**

```bash
curl -s -H "Authorization: Bearer sk-proj-YOUR_KEY_HERE" \
  https://api.openai.com/v1/models | head -20
```
When that `curl` call succeeds from the host, the same key in `PROVIDERS` works inside the container.

> **Important:** **Project-scoped keys** (`sk-proj-…​`) only work for models the project has been granted access to. A 404 on `gpt-5.5` usually means the key’s project is restricted. **Org-level keys** require the org header, which the service does not send; use a project key instead.

<!-- </details> -->

## Anthropic

API key from [console.anthropic.com](https://console.anthropic.com/). Same shape as OpenAI.

<!-- <details> -->
<!-- <summary> -->
Configuration details
<!-- </summary> -->
**JSON shape:**

```json
{
  "anthropic": {
    "type": "anthropic",
    "apiKeys": ["sk-ant-YOUR_KEY_HERE"]
  }
}
```
**Fields:**

| Field | Required | Notes |
| --- | --- | --- |
| `type` | Yes | Literal `"anthropic"` |
| `apiKeys` | Yes | Array. Rotation behaves the same as OpenAI. |

**Reasoning models:**

Claude models support reasoning. To surface the reasoning toggle in the TinyMCE UI, add `capabilities.reasoning: true` to the model entry in `MODELS`:

```json
{
  "id": "claude-sonnet-5",
  "provider": "anthropic",
  "description": "Anthropic Claude Sonnet 5 with reasoning",
  "capabilities": { "reasoning": true },
  "features": ["conversations", "reviews", "actions"]
}
```
**Minimal example:**

```bash
-e PROVIDERS='{"anthropic":{"type":"anthropic","apiKeys":["sk-ant-YOUR_KEY_HERE"]}}'
```
**Verify:**

```bash
curl -s https://api.anthropic.com/v1/models \
  -H "x-api-key: sk-ant-YOUR_KEY_HERE" \
  -H "anthropic-version: 2023-06-01"
```

<!-- </details> -->

## Google (Gemini direct)

Direct Gemini access through [Google AI Studio](https://aistudio.google.com/app/apikey). Distinct from Vertex AI; different keys, different endpoints, different billing path.

<!-- <details> -->
<!-- <summary> -->
Configuration details
<!-- </summary> -->
**JSON shape:**

```json
{
  "google": {
    "type": "google",
    "apiKeys": ["AIza-YOUR_KEY_HERE"]
  }
}
```
**Fields:**

| Field | Required | Notes |
| --- | --- | --- |
| `type` | Yes | Literal `"google"` |
| `apiKeys` | Yes | Array of `AIza…​` keys from AI Studio. |

**Minimal example:**

```bash
-e PROVIDERS='{"google":{"type":"google","apiKeys":["AIza-YOUR_KEY_HERE"]}}'
```
**Verify:**

```bash
curl -s "https://generativelanguage.googleapis.com/v1beta/models?key=AIza-YOUR_KEY_HERE"
```

> **Important:** AI Studio `AIza` keys do **not** work against Vertex endpoints. For Vertex, see the [Google Vertex AI](#google-vertex-ai) section.

> **Tip:** Free-tier keys are heavily rate limited and return 429 responses under modest load. Move to a paid tier before load testing or production traffic.

<!-- </details> -->

## Azure OpenAI

Azure-hosted OpenAI models. Requires an Azure subscription, an Azure OpenAI resource, and at least one deployment. The `MODELS[].id` must match the Azure deployment name exactly.

<!-- <details> -->
<!-- <summary> -->
Configuration details
<!-- </summary> -->
**Prerequisites in the Azure portal:**

1. Create an Azure OpenAI resource. Note the **resource name**; this is the subdomain prefix in `https://<RESOURCE_NAME>.openai.azure.com`.
2. Apply for model access if required by the region.
3. In Azure AI Studio, create a **deployment** for each model to expose. Start each deployment name with the ID of the OpenAI model behind it, followed by any suffix (for example `gpt-5-mini-prod`, `gpt-4.1-mini-low-cost`).
4. Copy one of the two API keys from **Keys and Endpoint** in the resource overview.

**JSON shape:**

```json
{
  "azure": {
    "type": "azure",
    "resourceName": "YOUR_RESOURCE_NAME",
    "apiKeys": ["YOUR_AZURE_KEY_HERE"],
    "apiVersion": "2024-10-21"
  }
}
```
**Fields:**

| Field | Required | Notes |
| --- | --- | --- |
| `type` | Yes | Literal `"azure"` |
| `resourceName` | Yes, unless `baseUrl` is set | The `*.openai.azure.com` prefix only, not the full URL. |
| `apiKeys` | Yes, unless `authentication` is set (image version 1.1.0 and later) | Array. Azure issues two keys per resource for zero-downtime key rotation. Set either `apiKeys` or `authentication`, not both. |
| `apiVersion` | Yes | The Azure OpenAI API version, sent as the `api-version` query parameter. Always set it explicitly: omitting it produces an SDK error about a missing query string parameter. Refer to [Microsoft’s API version matrix](https://learn.microsoft.com/azure/ai-services/openai/reference) for current stable values. |

> **Important:** The `MODELS[].id` value must match the Azure **deployment name** exactly, and the deployment name must start with the ID of the OpenAI model it serves, for example `gpt-4.1-editorial` for a `gpt-4.1` deployment. The service reads that prefix to recognize the model family and choose the request options. A `MODELS[].id` that does not match the deployment name produces a `DeploymentNotFound` error.
**Authentication and URL options in image version 1.1.0 and later:**

In image version 1.1.0 and later, an Azure OpenAI provider can authenticate with a token instead of API keys. The following example defines a provider that authenticates with Microsoft Entra ID:

```json
{
  "azure-entra": {
    "type": "azure",
    "baseUrl": "https://YOUR_RESOURCE_NAME.openai.azure.com",
    "apiVersion": "2024-10-21",
    "authentication": {
      "type": "entra-id",
      "scope": "https://cognitiveservices.azure.com/.default"
    }
  }
}
```

| Field | Required | Notes |
| --- | --- | --- |
| `baseUrl` | No | Endpoint of the Azure resource, for example `https://YOUR_RESOURCE_NAME.openai.azure.com`. Use it instead of `resourceName`. |
| `useDeploymentBasedUrls` | No | Defaults to `false`, which sends requests to the Responses API URLs under `{baseUrl}/v1`. Set to `true` when the resource exposes only the deployment-based Chat Completions endpoint, `{baseUrl}/deployments/{model id}/chat/completions`. |
| `authentication.type` | If not using `apiKeys` | `bearer-token` or `entra-id`. |
| `authentication.tokenSecret` | For `bearer-token` | Token sent with every request to the Azure endpoint. |
| `authentication.scope` | For `entra-id` | Scope requested from Microsoft Entra ID, for example `https://cognitiveservices.azure.com/.default`. |

With `bearer-token`, the service sends the configured token. With `entra-id`, the service obtains and refreshes tokens through the Azure default credential chain, so the identity comes from the host: environment variables, a managed identity, or a workload identity.

**Two-deployment example:**

```bash
-e PROVIDERS='{
  "azure": {
    "type": "azure",
    "resourceName": "tinymce-ai",
    "apiKeys": ["YOUR_AZURE_KEY_HERE"],
    "apiVersion": "2024-10-21"
  }
}'

-e MODELS='[
  {
    "id": "gpt-5-mini-prod",
    "provider": "azure",
    "name": "GPT-5 mini (production)",
    "description": "Azure deployment of GPT-5 mini for production traffic",
    "recommended": true,
    "features": ["conversations", "reviews", "actions"]
  },
  {
    "id": "gpt-4.1-mini-low-cost",
    "provider": "azure",
    "name": "GPT-4.1 mini (low cost)",
    "description": "Azure deployment of GPT-4.1 mini for cheap actions",
    "features": ["reviews", "actions"]
  }
]'
```
**Verify:**

```bash
curl -s -H "api-key: YOUR_AZURE_KEY_HERE" \
  "https://YOUR_RESOURCE_NAME.openai.azure.com/openai/deployments?api-version=2024-10-21"
```
Expect a JSON list that includes the deployment IDs from `MODELS`.

> **Important:** **Wrong region quota:** each Azure region has independent quotas. Bursty workloads may throttle at modest QPS; split traffic across resources in different regions if needed. **Content filter false positives:** Azure’s safety filter is stricter than OpenAI direct. Reviews on certain technical content can return `content_filter` errors. Configure custom content filter policies in Azure AI Studio.

<!-- </details> -->

## AWS Bedrock

Amazon’s hosted-model marketplace (Anthropic, Meta, Mistral, Cohere, Amazon Titan).

In image versions earlier than 1.2.0, inline the credentials in the `PROVIDERS` JSON: the AWS SDK default credential chain is not used. Image version 1.2.0 and later can also use an assumed role or the credentials of the host. See [[bedrock-newer-credentials]](#bedrock-newer-credentials).

<!-- <details> -->
<!-- <summary> -->
Configuration details
<!-- </summary> -->

> **Warning:** The AWS SDK reads the `AWS_BEARER_TOKEN_BEDROCK` environment variable as a Bedrock API key, and it takes precedence over `assumedRole` and `credentials`. Leave it unset unless it is the intended identity.
**JSON shape:**

```json
{
  "bedrock": {
    "type": "bedrock",
    "region": "us-east-1",
    "credentials": {
      "accessKeyId": "AKIA_YOUR_KEY_HERE",
      "secretAccessKey": "YOUR_SECRET_HERE",
      "sessionToken": "OPTIONAL_FOR_STS"
    }
  }
}
```
The `sessionToken` field is optional but required for STS-issued short-lived credentials. Plan a rotation procedure when using temporary credentials.

Instead of `credentials`, a Bedrock provider can set `apiKeys` to a Bedrock API key issued in the AWS console.

> **Note:** Bedrock console-issued API keys have a separate billing entitlement check that can return `INVALID_PAYMENT_INSTRUMENT` on accounts where the IAM-credential path works fine. Prefer IAM credentials (the `credentials` block, or in image version 1.2.0 and later, an assumed role or the credentials of the host) for production.
**Assumed role and host credentials in image version 1.2.0 and later:**

The following example defines a provider that assumes a role and a provider that uses the credentials of the host:

```json
{
  "bedrock-role": {
    "type": "bedrock",
    "region": "us-east-1",
    "assumedRole": {
      "roleArn": "arn:aws:iam::123456789012:role/tinymce-ai-bedrock",
      "externalId": "YOUR_EXTERNAL_ID",
      "sessionDurationSeconds": 3600
    }
  },
  "bedrock-host": {
    "type": "bedrock",
    "region": "us-east-1"
  }
}
```

| Field | Required | Notes |
| --- | --- | --- |
| `assumedRole.roleArn` | Yes, when `assumedRole` is set | ARN of the IAM role that the service assumes through AWS Security Token Service (STS) before calling Bedrock. |
| `assumedRole.externalId` | No | External ID required by the trust policy of the role. |
| `assumedRole.sessionDurationSeconds` | No | Lifetime of the temporary credentials, from 900 to 43200 seconds, capped by the maximum session duration of the role. Defaults to the AWS default. |
| `assumedRole.roleSessionName` | No | Session name recorded in AWS CloudTrail. Must match `^[\w+=,.@-]{2,64}$`. |

The service authenticates with the first of the following that is configured:

1. `assumedRole`: the service calls `sts:AssumeRole` and signs Bedrock requests with the temporary credentials, renewing them before they expire. The `AssumeRole` call uses `credentials` when both are set, and the credentials of the host when `assumedRole` is set on its own. Use this for cross-account access, with the Bedrock permissions attached to the role.
2. `credentials`: the service signs requests with the configured access key. Temporary keys also need `sessionToken`, and the service does not renew them.
3. `apiKeys`: the service sends the Bedrock API key as a bearer token, with no request signing.
4. The credentials of the host: when none of the options above is set, or `apiKeys` resolves to an empty string, the AWS SDK resolves the identity of the machine in its standard order (environment variables, shared AWS configuration files, a web identity token file, the container credentials endpoint, and the instance metadata service). A container on EC2, ECS, or EKS can then reach Bedrock with no secret in `PROVIDERS`.

**Prerequisites checklist:**

1. **Enable model access.** Bedrock console → **Model access** → Manage model access. Each model must be approved per-region.
2. **Subscribe through AWS Marketplace** for non-Amazon models. Anthropic Claude on Bedrock requires a one-time Marketplace subscription.
3. **Create an IAM user or role** with the permissions below.
4. **Pick a region that has the model.** Not every model is in every region. Check Bedrock console > Model catalog before assuming.

**Required IAM permissions:**

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BedrockInvoke",
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream",
        "bedrock:Converse",
        "bedrock:ConverseStream"
      ],
      "Resource": "*"
    }
  ]
}
```
The service uses both `InvokeModel` and `Converse` paths depending on the model family, so include both.

**Claude cross-region inference profiles:**

Recent Claude models on Bedrock use **cross-region inference profiles** rather than per-region model IDs. The `MODELS[].id` must be the inference profile ID, prefixed with the regional group. For example, the US profile ID for Claude Sonnet 5 is `us.anthropic.claude-sonnet-5`.

The region groups and profile IDs differ for each model. Check the model card in the Amazon Bedrock documentation for the profiles of the model in use.

Using the bare model ID (`anthropic.claude-sonnet-5`) returns `ValidationException`.

**Minimal example:**

```bash
-e PROVIDERS='{
  "bedrock": {
    "type": "bedrock",
    "region": "us-east-1",
    "credentials": {
      "accessKeyId": "AKIA_YOUR_KEY_HERE",
      "secretAccessKey": "YOUR_SECRET_HERE"
    }
  }
}'

-e MODELS='[
  {
    "id": "us.anthropic.claude-sonnet-5",
    "provider": "bedrock",
    "name": "Claude Sonnet 5 (Bedrock)",
    "description": "Anthropic Claude Sonnet 5 through AWS Bedrock cross-region inference",
    "recommended": true,
    "capabilities": { "reasoning": true },
    "features": ["conversations", "reviews", "actions"]
  }
]'
```
**Verify before running the AI service:**

```bash
aws bedrock list-foundation-models --region us-east-1 \
  --query 'modelSummaries[?contains(modelId,`claude`)].[modelId,modelLifecycle.status]' \
  --output table
```

```bash
aws bedrock-runtime converse \
  --region us-east-1 \
  --model-id us.anthropic.claude-sonnet-5 \
  --messages '[{"role":"user","content":[{"text":"Say hello in five words."}]}]'
```
If either fails, the AI service fails with the same root cause; fix IAM and model access before continuing.

> **Important:** **`AccessDeniedException`** almost always means missing model access in the Bedrock console. **`ThrottlingException`** on the first call indicates low default quotas; request an increase through Service Quotas. **`ValidationException`** means a mismatch between regional model IDs and cross-region inference profiles. **`INVALID_PAYMENT_INSTRUMENT`** indicates a missing AWS Marketplace subscription.

<!-- </details> -->

## Google Vertex AI

Google’s enterprise model surface. Project-scoped, IAM-driven, GCP-billed.

In image versions earlier than 1.2.0, inline a service account key or an account-bound API key in the `PROVIDERS` JSON: Application Default Credentials are not used. Image version 1.2.0 and later can also use impersonation or the credentials of the host. See [[vertex-newer-credentials]](#vertex-newer-credentials).

<!-- <details> -->
<!-- <summary> -->
Configuration details
<!-- </summary> -->

> **Important:** An API key authenticates Gemini models only. Requests to Claude models on Vertex AI always need a Google OAuth token. Give a provider that serves Claude models `credentials`. In image version 1.2.0 and later, `impersonation` or a host with Application Default Credentials also work.
**JSON shape (service account):**

```json
{
  "vertex": {
    "type": "vertex",
    "project": "YOUR_GCP_PROJECT_ID",
    "location": "us-central1",
    "credentials": {
      "clientEmail": "ai-service@YOUR_GCP_PROJECT_ID.iam.gserviceaccount.com",
      "privateKey": "-----BEGIN PRIVATE KEY-----\nMIIE...\n-----END PRIVATE KEY-----\n"
    }
  }
}
```
**JSON shape (account-bound API key):**

```json
{
  "vertex": {
    "type": "vertex",
    "project": "YOUR_GCP_PROJECT_ID",
    "location": "us-central1",
    "apiKeys": ["YOUR_VERTEX_API_KEY"]
  }
}
```
**Fields:**

| Field | Required | Notes |
| --- | --- | --- |
| `type` | Yes | Literal `"vertex"` |
| `project` | Yes | GCP project ID, not the project number. Can also come from the `GOOGLE_VERTEX_PROJECT` environment variable. |
| `location` | Yes | Region, for example `us-central1`, `europe-west4`. Must support the model family. Can also come from the `GOOGLE_VERTEX_LOCATION` environment variable. |
| `credentials.clientEmail` | If using SA | The `client_email` field from the SA JSON key. |
| `credentials.privateKey` | If using SA | The `private_key` field from the SA JSON key. See the newline escaping note below. |
| `apiKeys` | If using account-bound key | Array of one or more Vertex-bound API keys. |

**Impersonation and host credentials in image version 1.2.0 and later:**

The following example defines a provider that impersonates a service account and a provider that uses the credentials of the host:

```json
{
  "vertex-impersonation": {
    "type": "vertex",
    "project": "YOUR_GCP_PROJECT_ID",
    "location": "us-central1",
    "impersonation": {
      "targetPrincipal": "ai-service@YOUR_GCP_PROJECT_ID.iam.gserviceaccount.com",
      "lifetimeSeconds": 3600
    }
  },
  "vertex-host": {
    "type": "vertex",
    "project": "YOUR_GCP_PROJECT_ID",
    "location": "us-central1"
  }
}
```

| Field | Required | Notes |
| --- | --- | --- |
| `impersonation.targetPrincipal` | Yes, when `impersonation` is set | Email of the service account to impersonate. |
| `impersonation.lifetimeSeconds` | No | Lifetime of the access tokens issued for that service account. Defaults to `3600`, maximum `43200`. |

The service authenticates with the first of the following that is configured:

1. `impersonation`: the service requests a short-lived access token for `targetPrincipal` from the IAM Credentials API, scoped to `https://www.googleapis.com/auth/cloud-platform`, and renews it as it expires. The token request uses `credentials` when both are set, and the credentials of the host when `impersonation` is set on its own. Grant the source identity `roles/iam.serviceAccountTokenCreator` on the target service account.
2. `credentials`: the service signs a JWT with the service account key and exchanges it for an access token.
3. `apiKeys`: the service calls Vertex AI in express mode with the API key (Gemini models only).
4. The credentials of the host: when none of the options above is set, the Google auth library resolves Application Default Credentials in its standard order (the file that `GOOGLE_APPLICATION_CREDENTIALS` points to, the file written by `gcloud auth application-default login`, and the service account attached to the host). A container on Google Cloud can then reach Vertex AI with no secret in `PROVIDERS`.

**`private_key` newline escaping:**

When copying the `private_key` from a GCP service account JSON key into a `.env` file and then expanding with `set -a && source .env`, the shell collapses the `\n` sequences and the key becomes invalid (Vertex returns `invalid_grant`). A reliable approach is to build the entire `PROVIDERS` value using a JSON serializer:

```bash
python3 - <<'EOF' > providers.json
import json
sa = json.load(open("/path/to/service-account.json"))
providers = {
    "vertex": {
        "type": "vertex",
        "project": sa["project_id"],
        "location": "us-central1",
        "credentials": {
            "clientEmail": sa["client_email"],
            "privateKey": sa["private_key"]
        }
    }
}
print(json.dumps(providers))
EOF

export PROVIDERS="$(cat providers.json)"
```
**Account-bound API keys still require a service account:**

The Vertex "account-bound API key" is bound to a Google Cloud project AND a service account inside that project. A service account must exist and have `roles/aiplatform.user` granted; the API key only avoids distributing the private key.

**GCP org policies that block setup:**

| Org policy | Blocks |
| --- | --- |
| `iam.disableServiceAccountCreation` | Creating the service account. Symptom: `403 Permission iam.serviceAccounts.create denied`. |
| `iam.disableServiceAccountKeyCreation` | Downloading a JSON key. Symptom: **Create key** greyed out, or `FAILED_PRECONDITION`. |
| `iam.allow.disabled` | Account-bound API key creation. Symptom: `API_KEY_INVALID`. |

When a GCP org enforces any of these, the security or cloud team must grant a project-level exception or pre-provision the credentials.

**Minimal example:**

```bash
-e PROVIDERS='{
  "vertex": {
    "type": "vertex",
    "project": "tinymce-ai",
    "location": "us-central1",
    "credentials": {
      "clientEmail": "ai-service@tinymce-ai.iam.gserviceaccount.com",
      "privateKey": "-----BEGIN PRIVATE KEY-----\nMIIEv...\n-----END PRIVATE KEY-----\n"
    }
  }
}'

-e MODELS='[
  {
    "id": "gemini-3.8-flash",
    "provider": "vertex",
    "name": "Gemini 3.8 Flash (Vertex)",
    "description": "Google Gemini 3.8 Flash through Vertex AI",
    "features": ["conversations", "reviews", "actions"]
  }
]'
```
The service account needs `roles/aiplatform.user` (or a custom role with `aiplatform.endpoints.predict` and `aiplatform.endpoints.streamGenerateContent`).

**Verify:**

```bash
gcloud auth activate-service-account \
  --key-file=/path/to/service-account.json

gcloud ai models list \
  --region=us-central1 \
  --project=YOUR_GCP_PROJECT_ID
```

> **Important:** **Region mismatch:** set `location` to a region that hosts the model. **`SERVICE_DISABLED`** means the Vertex AI API is not enabled; run `gcloud services enable aiplatform.googleapis.com`. **Quota:** new projects default to 5 QPS; request an increase before production traffic.

<!-- </details> -->

## OpenAI-compatible (Ollama, vLLM, LM Studio, and similar)

Use this provider type for any runtime or aggregator that is not listed as a native provider above but exposes an OpenAI-compatible `/v1/chat/completions` endpoint — including Ollama, vLLM, LM Studio, OpenRouter, Together, Groq, and Fireworks.

The `baseUrl` **must** include the OpenAI-compatible path prefix of the runtime, typically `/v1`. The AI service appends `/chat/completions` to the `baseUrl`.

### Configuration

```json
{
  "local-llm": {
    "type": "openai-compatible",
    "baseUrl": "http://host.docker.internal:11434/v1",
    "apiKeys": ["optional-bearer-token"],
    "headers": { "X-Custom-Header": "value" }
  }
}
```

| Field | Required | Notes |
| --- | --- | --- |
| `type` | Yes | Literal `"openai-compatible"` |
| `baseUrl` | Yes | **Must include the OpenAI-compatible path prefix of the runtime, typically `/v1`.** Without it, every request fails with a misleading "Not Found" Server-Sent Events (SSE) error. |
| `apiKeys` | No | Sent as `Authorization: Bearer <key>`. Most local runtimes ignore it. |
| `headers` | No | Additional headers such as auth tokens or tenant IDs. |

> **Note:** File uploads through this adapter are limited to `image/*` MIME types. PDFs and Office documents are not forwarded. To work with non-image files, route through an OpenAI, Anthropic, or Bedrock provider instead.

### Ollama networking

Ollama listens on `127.0.0.1:11434` by default, which is unreachable from inside a Docker container. Bind to all interfaces:

```bash
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```
On Linux with systemd, create an override file instead:

```ini
# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
```
Then reload and restart: `sudo systemctl daemon-reload && sudo systemctl restart ollama`.

On Linux, add the host gateway so `host.docker.internal` resolves:

```yaml
services:
  ai-service:
    image: registry.containers.tiny.cloud/ai-service-tiny:latest
    extra_hosts:
      - "host.docker.internal:host-gateway"
```
If Ollama returns "does not support tools", the model was built from a raw GGUF without a chat template. Use `ollama pull` for a Library model that includes a proper Modelfile, or author a custom one.

<!-- <details> -->
<!-- <summary> -->
Custom Modelfile example
<!-- </summary> -->

```none
FROM /path/to/your-model.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ range .Messages }}<|im_start|>{{ .Role }}
{{ .Content }}<|im_end|>
{{ end }}<|im_start|>assistant
"""

PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"
```
The exact template depends on the base model. Check the model card for the recommended chat template. Verify tool support with `ollama show <model>` before connecting to the AI service.


<!-- </details> -->
The reasoning toggle (`capabilities.reasoning: true`) is cosmetic for Ollama-backed models; the openai-compatible adapter does not translate it to the native Ollama API.

### Timeout

Large self-hosted models on consumer hardware can exceed the default 180-second timeout. Override with:

```bash
-e LLM_TIMEOUT_MS='600000'
```

### Example — Ollama

```bash
-e PROVIDERS='{
  "ollama": {
    "type": "openai-compatible",
    "baseUrl": "http://host.docker.internal:11434/v1"
  }
}'

-e MODELS='[
  {
    "id": "qwen3:0.6b",
    "provider": "ollama",
    "name": "Qwen3 0.6B (local)",
    "description": "Local Ollama model for fast actions",
    "features": ["conversations", "reviews", "actions"]
  }
]'
```

### Example — vLLM and LM Studio

The configuration pattern is identical to Ollama — only the `baseUrl`, provider key, and model ID differ.

**Runtime:** vLLM

**PROVIDERS entry:**

```json
{
  "vllm": {
    "type": "openai-compatible",
    "baseUrl": "http://vllm-host.internal:8001/v1",
    "apiKeys": ["YOUR_VLLM_TOKEN"]
  }
}
```
**MODELS entry:**

```json
{
  "id": "meta-llama/Llama-3.1-8B-Instruct",
  "provider": "vllm",
  "name": "Llama 3.1 8B (vLLM)",
  "description": "Self-hosted Llama model served by vLLM",
  "features": ["conversations", "reviews", "actions"]
}
```
**Runtime:** LM Studio

**PROVIDERS entry:**

```json
{
  "lmstudio": {
    "type": "openai-compatible",
    "baseUrl": "http://host.docker.internal:1234/v1"
  }
}
```
**MODELS entry:**

```json
{
  "id": "lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF",
  "provider": "lmstudio",
  "name": "Llama 3.1 8B (LM Studio)",
  "description": "Local Llama model served by LM Studio",
  "features": ["conversations", "reviews", "actions"]
}
```

### Verify

The AI service does not expose an OpenAI-compatible endpoint of its own. It appends `/chat/completions` to the `baseUrl` of the provider. To verify the setup, first confirm that the runtime answers at that path prefix. For the Ollama example, run on the host:

```bash
curl -s http://localhost:11434/v1/models
```
The response lists the models that the runtime serves. Each `id` in `MODELS` must match one of them.

Then open the editor, select the local model in Chat, and send a message. If the reply fails, the AI service log contains the URL that the service called on the runtime. A `404` response from that URL usually means that the `baseUrl` is missing the `/v1` path prefix.

## Multiple providers coexistence

`PROVIDERS` is a single JSON object that may contain any number of entries. Each entry is independent; the service maintains a separate client pool per provider. There is no limit beyond JSON-in-env-var size constraints (many shells allow 64 KB+ or more in a single variable).

A `MODELS` array routes individual models to specific providers using the `provider` field, which must match a `PROVIDERS` key.

### Example: three providers, mixed routing

<!-- <details> -->
<!-- <summary> -->
Click to expand: three-provider `PROVIDERS` and `MODELS` example
<!-- </summary> -->

```bash
-e PROVIDERS='{
  "openai":     { "type": "openai",    "apiKeys": ["sk-proj-YOUR_KEY_HERE"] },
  "bedrock-us": {
    "type": "bedrock",
    "region": "us-east-1",
    "credentials": {
      "accessKeyId": "AKIA_YOUR_KEY_HERE",
      "secretAccessKey": "YOUR_SECRET_HERE"
    }
  },
  "ollama":     { "type": "openai-compatible", "baseUrl": "http://host.docker.internal:11434/v1" }
}'

-e MODELS='[
  {
    "id": "gpt-5.5",
    "provider": "openai",
    "name": "GPT-5.5",
    "description": "OpenAI GPT-5.5 for conversations",
    "recommended": true,
    "features": ["conversations"]
  },
  {
    "id": "us.anthropic.claude-sonnet-5",
    "provider": "bedrock-us",
    "name": "Claude Sonnet 5 (Bedrock)",
    "description": "Claude on Bedrock for reviews",
    "capabilities": { "reasoning": true },
    "features": ["reviews"]
  },
  {
    "id": "qwen3:0.6b",
    "provider": "ollama",
    "name": "Qwen3 0.6B (local)",
    "description": "Local model for cheap quick actions",
    "features": ["actions"]
  }
]'
```

<!-- </details> -->
This wires conversations to OpenAI, reviews to Bedrock-hosted Claude, and quick actions to a local Ollama model. The AI service picks the model for system reviews, system actions, and conversation title generation from the models that declare each feature in `features`. Chat messages, custom actions, and custom reviews use the model that the request names. See [Model resolution order](#model-resolution-order).

A `MODELS` entry with a `provider` value that does not exist in `PROVIDERS` stops the service at startup with `[FATAL] Following models have invalid provider:` followed by the affected model IDs. Check the spelling of each `provider` field against the keys in `PROVIDERS`. See [Troubleshooting](../tinymceai-on-premises-troubleshooting/) for additional debugging steps.

### Model resolution order

Feature strings form a dot-separated hierarchy: in `reviews.correctness`, the parent feature is `reviews`. The service routes a request as follows:

- A request goes to a model that declares the most specific matching feature. A request for `reviews.correctness` goes to a model that declares `reviews.correctness`, even when a model that declares `reviews` comes earlier in `MODELS`.
- A feature with no exact match falls back to its parent. When models declare only `reviews` and `reviews.correctness`, a `reviews.clarity` request goes to the model that declares `reviews`.
- When several models declare the same feature, the order of `MODELS` ranks them. When the service picks the model for a feature, the first model serves the request, and the next one serves it when the first model is unavailable (see [Model availability](#model-availability)).

## `MODELS` field reference

### Top-level fields

| Field | Required | Type | Default | Notes |
| --- | --- | --- | --- | --- |
| `id` | Yes | string | - | Model identifier sent to the provider. For Azure, this must match the deployment name, which starts with the ID of the OpenAI model. For Bedrock, this must match the model ID or inference profile. Must be unique across all models. No model can use an `id` that starts with the reserved `agent-` prefix. |
| `type` | No | string | `standard` | `standard` or `agent`. See [Custom agent models](#agent-models). |
| `fallbackOrder` | For agent models | `string[]` | - | Ordered list of standard model IDs that an agent model routes to. See [Custom agent models](#agent-models). |
| `provider` | Yes (standard models) | string | - | Must match a key in `PROVIDERS`. The match is case-insensitive. Ignored for agent models. |
| `description` | Yes | string | - | Shown under the model name in the model picker. |
| `name` | No | string | value of `id` | Display name in the model picker. |
| `recommended` | No | boolean | `false` | If `true`, marks the model as recommended. Several models can be recommended. The TinyMCE AI plugin does not use this flag: it selects the model set in [`tinymceai_default_model`](../tinymceai/#tinymceai_default_model), or the first model returned. |
| `capabilities.webSearch` | No | boolean | `false` | Whether to allow the web search toggle for this model. Requires `WEBSEARCH_ENABLED`. |
| `capabilities.reasoning` | No | boolean | `false` | Whether to expose a reasoning toggle. Supported providers include Anthropic, Bedrock-Claude, and OpenAI o-series. Cosmetic for Ollama (see openai-compatible section). |
| `contextLimits` | No | object | see below | Per-model context constraints. |
| `features` | No | `string[]` | - | Which features the model serves. A standard model without `features` is not offered for any feature directly, but an agent model can list it in `fallbackOrder`. See [Startup validation](#startup-validation). |

### `contextLimits` defaults

| Field | Default | Unit |
| --- | --- | --- |
| `maxContextLength` | `256000` | characters |
| `maxFiles` | `100` | count |
| `maxFileSize` | `5242880` (Anthropic) / `7340032` (all others) | bytes |
| `maxTotalFileSize` | `31457280` | bytes |
| `maxTotalPdfFilePages` | `100` | pages |

Values set in `contextLimits` are in bytes, characters, or counts, as listed above. To read the limits that the running image applies, call `GET /v1/models/1` and check the `limits` object of each model. `contextLimits` caps only what the model accepts. When a model cannot accept file input, remove the file upload permission from the users who can select that model.

Override `contextLimits` only when necessary, for example when a model has a smaller real context window than the default 256K character budget, or when a deployment policy restricts file size.

### Available `features` strings

The full set of feature strings recognized by the service:

```text
conversations
conversations.titleGeneration
reviews
reviews.correctness
reviews.clarity
reviews.readability
reviews.make-longer
reviews.make-shorter
reviews.make-tone-casual
reviews.make-tone-direct
reviews.make-tone-friendly
reviews.make-tone-confident
reviews.make-tone-professional
reviews.translate
actions
actions.make-longer
actions.make-shorter
actions.continue
actions.make-tone-casual
actions.make-tone-direct
actions.make-tone-friendly
actions.make-tone-confident
actions.make-tone-professional
actions.translate
actions.fix-grammar
actions.improve-writing
```
The three umbrella values `conversations`, `reviews`, and `actions` enable every sub-feature of that feature. Use a specific sub-feature only when restricting a model to a subset; for example, a low-cost model that handles only `actions.fix-grammar`.

A model with no `features` entry, or with only sub-features the editor does not request, will be hidden from the picker.

## Custom agent models

`MODELS` can define custom agent models. A custom agent model is a virtual model that calls no provider itself. It lists standard models in `fallbackOrder`, and each request goes to the first model on that list that is available. When none of them is available, the request fails.

```bash
-e MODELS='[
  {
    "id": "claude-sonnet-5",
    "provider": "anthropic",
    "description": "Anthropic Claude Sonnet 5",
    "capabilities": { "reasoning": true }
  },
  {
    "id": "gpt-5-mini",
    "provider": "openai",
    "description": "OpenAI GPT-5 mini",
    "capabilities": { "reasoning": true }
  },
  {
    "type": "agent",
    "id": "writing-assistant",
    "name": "Writing assistant",
    "description": "Uses the first available model from two providers",
    "recommended": true,
    "capabilities": { "reasoning": true },
    "features": ["conversations", "reviews", "actions"],
    "fallbackOrder": ["claude-sonnet-5", "gpt-5-mini"]
  }
]'
```
An agent model accepts the same fields as a standard model (`id`, `name`, `description`, `recommended`, `capabilities`, `contextLimits`, and `features`), with these differences:

- `type` must be `agent`.
- `fallbackOrder` is required. It is a non-empty, ordered array of `id` values of standard models in the same `MODELS` array.
- The service ignores `provider`. Each request uses the provider of the standard model that serves it.

When an agent model sets `capabilities.webSearch` or `capabilities.reasoning` to `true`, every model in its `fallbackOrder` must set the same capability. Otherwise the service fails to start with a validation error.

Clients see an agent model as an ordinary model in the model list. The list never includes its `fallbackOrder` or the providers of the models behind it.

Agent models can replace or reorder the models behind a stable model ID, for example to roll out a newer model or retire an old one, without changing editor configuration or JWT permissions. They also keep the choice of models and providers private to the deployment, and continue to answer during a provider outage when they list models from more than one provider.

### Model availability

The service tracks the health of every model and takes an unhealthy model out of rotation for 60 seconds. All replicas share this state through Redis, so a model that is out of rotation is out of rotation on every replica.

- A model leaves rotation after three failed requests, or three responses that took more than 10 seconds to produce the first token, within 5 minutes. The service counts failures and slow responses separately, and one healthy response resets both counts.
- After 60 seconds, the service sends one probe request to the model. If it succeeds, the model returns to rotation. If it fails, the model stays out for another 60 seconds.
- The service picks the model before it sends a request, and does not switch models during a request.

A request to a standard model that is out of rotation fails, because there is nothing to fall back to. An agent model uses the first model in its `fallbackOrder` that is in rotation, and fails only when none is. To keep requests working during a provider outage, offer an agent model that lists models from at least two providers.

## Rotating API keys without downtime

The `apiKeys` field on every provider type that has one is an **array**. The service uses the first key, and if the provider rejects that key as invalid, the next one. To rotate a key without downtime, add the new key after the current one, restart, then remove the old key:

1. Append the new key to the array:

```json
"apiKeys": ["sk-proj-OLD_KEY", "sk-proj-NEW_KEY"]
```
2. Restart the AI service container so it picks up the new `PROVIDERS` value.
3. Verify the new key directly against the provider, with the **Verify** command in the provider section above. The service keeps using the first key until the provider rejects it.
4. Revoke the old key in the provider console.
5. Remove the old key from `apiKeys`:

```json
"apiKeys": ["sk-proj-NEW_KEY"]
```
6. Restart the AI service container again.

The same procedure works for `anthropic`, `google`, `azure`, and `openai-compatible` provider types. For Bedrock and Vertex, swap the `credentials` object atomically; there is no array of credential objects, so plan a short maintenance window or run two AI service replicas behind a load balancer for a no-downtime swap. In image version 1.2.0 and later, a Bedrock or Vertex AI provider that uses an assumed role, impersonation, or the credentials of the host has no static keys in `PROVIDERS` to rotate, unless the assumed role or impersonation also uses `credentials`.

## Quick troubleshooting index

| Symptom | Most likely cause | Section |
| --- | --- | --- |
| Requests return HTTP 403 `missing-permissions` listing `ai:models:agent` / `agent-1 allowed:false` | The token lacks `ai:models:agent` | [Defining the model list](#models-required) |
| `GET /v1/models/v1` returns 400 | Wrong compatibility version | [Defining the model list](#models-required) |
| Bedrock returns `NoValidApiKeysFoundError` | No credentials that the service can use. In image versions earlier than 1.2.0, relying on the AWS default credential chain | Bedrock |
| Bedrock returns `AccessDeniedException` | Model access not enabled in console | Bedrock prerequisites |
| Bedrock returns `ValidationException` on a Claude model | Bare model ID used instead of a cross-region inference profile | Bedrock |
| Bedrock returns `INVALID_PAYMENT_INSTRUMENT` | Missing AWS Marketplace subscription for the model family | Bedrock prerequisites |
| Vertex returns `invalid_grant` | `private_key` newline escaping mangled | Vertex |
| Vertex returns `SERVICE_DISABLED` | aiplatform.googleapis.com not enabled | Vertex |
| Azure returns `DeploymentNotFound` | `MODELS[].id` not equal to deployment name | Azure |
| Azure SDK errors about missing API version | `apiVersion` not set | Azure |
| Ollama: "Not Found" in SSE | `baseUrl` missing `/v1` | OpenAI-compatible |
| Ollama: "does not support tools" | Bare-GGUF Modelfile without chat template | OpenAI-compatible |
| Ollama: reasoning toggle has no effect | Not supported through the openai-compatible adapter | OpenAI-compatible |
| Self-hosted model times out at 180s | Default `LLM_TIMEOUT_MS` | OpenAI-compatible |
| AI request hangs forever in browser | Reverse proxy buffering SSE | See [Production deployment](../tinymceai-on-premises-production/) for the reverse proxy and TLS section |

More general troubleshooting (database, JWT, storage, networking) lives in [Troubleshooting](../tinymceai-on-premises-troubleshooting/).

## See also

- [MCP and web integrations](../tinymceai-on-premises-mcp/): Model Context Protocol tool calling and web scraping/search endpoints
- [Getting started](../tinymceai-on-premises-getting-started/): initial container bring-up and demo
- [JWT authentication](../tinymceai-on-premises-jwt/): per-model and per-provider JWT permissions
- [Database, Redis, and storage](../tinymceai-on-premises-database/): MySQL/Postgres configuration for the AI service
- [Troubleshooting](../tinymceai-on-premises-troubleshooting/): general troubleshooting beyond provider configuration
