Troubleshooting

Match the symptom to the fix below. If the symptom does not fit any section, escalate to Tiny Support with the output of docker logs ai-service --tail 200 and a redacted copy of the PROVIDERS value. In image version 1.3.0 and later, also attach an export from AI Diagnostics, which shows the prompt, response, and model of each failed AI request.

Quick triage

Work through this list to identify the symptom area:

  1. Is the container running? Run docker ps to check. If the container has exited or will not pull, see Container startup failures.

  2. Does /health return 200? Run curl -fsS http://localhost:8000/health. If it times out or returns 5xx, see Container startup failures.

  3. Does the API return an auth error? If the response is 401, allowed:false, or invalid-jwt-*, see API and JSON Web Token (JWT) authentication.

  4. Does the SSE stream carry an event: error from the LLM? If yes, see Large language model (LLM) provider errors.

  5. Is the editor side broken? Missing toolbar, token 401, or hanging stream? See Editor and front end.

  6. Slow, timing out, or failing under load? See Performance.

  7. Scaling, upgrades, or deployment questions? See Production deployment.

  8. Content rejected, or a hook error? See Content checks and hooks.

  9. Need the meaning of a service log message? See Log reference.

If none of the above match, see Diagnostic recipes and then escalate to Tiny Support.

Container startup failures

Run docker logs ai-service first. All entries below assume the log output is available.

Error / symptom Cause Fix

Wrong license key.

Key was truncated, contains a line break, or has surrounding whitespace

Paste the key as a single unbroken line. Verify the first and last eight characters against the original.

EACCES: permission denied, mkdir '/var/storage'

STORAGE_LOCATION points to a path the container user cannot write

Switch to STORAGE_DRIVER=database, or mount a writable volume and point STORAGE_LOCATION at it (for example /tmp/ai-storage).

Not enough permissions to access database. Missing privileges: …​

The MySQL user lacks the listed privileges on the AI service database. Privileges granted globally (ON *.*) or through a role, such as cloudsqlsuperuser on Cloud SQL, do not satisfy the start-up check.

Grant the privileges on the AI service database itself (ON ai_service.*). See Database user privileges for the GRANT statement.

schema "cs-on-premises" does not exist

Postgres schema not pre-created

Run CREATE SCHEMA "cs-on-premises"; (double quotes required), or set DATABASE_SCHEMA=public. See Database, Redis, and storage.

PostgreSQL Error: No permissions to connect to the database although the user and grants are correct

The managed PostgreSQL server requires TLS (rds.force_ssl=1 on Amazon RDS, require_secure_transport=ON on Azure Database for PostgreSQL Flexible Server), and the AI service connected without it. The server-side log shows no pg_hba.conf entry …​ no encryption.

See Managed database TLS.

duplicate key value violates unique constraint "pg_class_relname_nsp_index" on first start

Several replicas started at the same time against a new database and ran the schema migrations concurrently.

The replica restarts and recovers on its own. To avoid the error, start one replica, wait until it is healthy, then scale up. See Upgrade process.

[MY-000067] unknown variable 'default-authentication-plugin'

mysql:8 tag now points to the latest MySQL, which removed that variable

Pin mysql:8.0 in the compose file and run docker compose up -d --force-recreate mysql.

Following models have invalid provider: <model IDs>

A MODELS entry names a provider key that does not exist in PROVIDERS. The service stops at startup.

Correct the provider value, or add the missing key to PROVIDERS.

[FATAL] …​ is not a constructor with minified identifiers

A PROVIDERS entry has the wrong shape, for example a Vertex AI entry without the type and name fields and with client_email and private_key instead of clientEmail and privateKey.

Start from the provider example on the LLM providers page and add fields one at a time.

Container exits with no useful log

Missing required env var, or malformed JSON in PROVIDERS / MODELS

Run docker inspect ai-service | jq '.[0].Config.Env' and compare against the environment variable reference. Validate JSON with echo "$PROVIDERS" | jq .

Pods stay unready on Amazon EKS Auto Mode, restart after failed liveness probes, and kubectl logs shows no output for the current or the previous container

The database and Redis security groups allow inbound traffic from the node security group, but EKS Auto Mode attaches the cluster primary security group to the pods.

Allow inbound database and Redis traffic from the cluster primary security group. See Infrastructure recommendations.

/health times out despite successful boot

Port mapping missing

Add -p 8000:8000 to docker run, or ports: ["8000:8000"] in compose. Confirm with docker port ai-service.

getaddrinfo EAI_AGAIN mysql or getaddrinfo ENOTFOUND redis

AI service is on a different Docker network from the data layer

Use docker compose (shared network), or set DATABASE_HOST=host.docker.internal. On Linux, add extra_hosts: ["host.docker.internal:host-gateway"] to the AI service.

API and JSON Web Token (JWT) authentication

These assume the container is running and /health returns OK.

Error / symptom Cause Fix

invalid-jwt-signature

Token signed with the wrong key. Most commonly, signed with ENVIRONMENTS_MANAGEMENT_SECRET_KEY instead of the per-access-key API Secret

Re-copy the API Secret from the Management Panel at /panel/ and re-sign the token.

invalid-jwt-payload

aud claim does not match a known Environment ID, or aud is an array instead of a string

Copy the Environment ID from /panel/. Set aud to a string, not an array. Recreate environments through the Panel UI only.

invalid-jwt (expired)

Token is past its exp claim

Issue tokens with a reasonable lifetime (for example exp = now + 3600) and refresh before expiry. Synchronize clocks with Network Time Protocol (NTP).

Environment not found (shown in the editor as Invalid API key provided in JWT token (aud) claim)

Environment was not created through the Management Panel UI

Delete and recreate the environment through /panel/. Update AI_ENV_ID in .env.

JWT silently rejected

Token signed with RS256 instead of HS256

Re-sign with algorithm: 'HS256' and the API Secret. See JWT authentication.

409 conversation in use then 404 conversation does not exist

Stream abort left temporary state blocking the conversation

Start a new conversation or reload the page. Custom UIs should create a fresh conversation after cancel.

HTTP 500 error on every endpoint

auth.ai.permissions is not an array of strings, for example a single string, an object, or an array that contains a non-string value

Use an array of strings:

{
  "auth": {
    "ai": {
      "permissions": [
        "ai:conversations:*",
        "ai:models:agent",
        "ai:actions:system:*",
        "ai:reviews:system:*"
      ]
    }
  }
}

Common mistakes: "permissions": "ai:admin" (string shorthand), "permissions": "", "useAllFeatures": true, or a single permission as a string instead of an array. A [""] array is accepted but grants no permissions. A token with no auth claim, no permissions key, or an empty array returns HTTP 403 no-permissions-defined instead. See Permissions for every permission.

Large language model (LLM) provider errors

Most of these appear as event: error inside the SSE stream, and the HTTP response is still 200. NoValidApiKeysFoundError is the exception: the message request fails with HTTP 500, and the cause appears in the container log.

Cloud providers (OpenAI, Anthropic, Google)

Error Fix

Incorrect API key provided

Update the key in PROVIDERS, then docker compose up -d --force-recreate ai-service.

AWS Bedrock

Error Fix

NoValidApiKeysFoundError

The provider has no credentials that the service can use. In image versions earlier than 1.2.0, inline accessKeyId and secretAccessKey inside credentials in PROVIDERS, or set apiKeys: the AWS SDK default credential chain is not used. See AWS Bedrock.

AccessDeniedException

Enable model access in Bedrock console → Model access. Attach an IAM policy with bedrock:InvokeModel, bedrock:InvokeModelWithResponseStream, bedrock:Converse, and bedrock:ConverseStream.

INVALID_PAYMENT_INSTRUMENT

Complete the AWS Marketplace subscription for Anthropic in Bedrock console → Model access → Anthropic.

ValidationException (model invocation not supported)

Use the region-prefixed inference profile ID (for example us.anthropic.claude-sonnet-5). See LLM providers.

Google Vertex AI

Error Fix

NoValidApiKeysFoundError

The provider has no credentials that the service can use. In image versions earlier than 1.2.0, inline clientEmail and privateKey inside credentials in PROVIDERS, or set apiKeys to an account-bound API key: Application Default Credentials are not used. See Google Vertex AI.

Auth errors with a valid service account

private_key newlines were mangled during copy-paste. Build PROVIDERS with a script (json.dumps() on the SA JSON file) rather than hand-editing.

SERVICE_DISABLED

Run gcloud services enable aiplatform.googleapis.com --project=<PROJECT_ID>.

Blocked by GCP org policy

Check iam.disableServiceAccountCreation, iam.disableServiceAccountKeyCreation, and account-bound API key policies. Exempt the AI service project from all three.

Azure OpenAI

Error Fix

Model not found / DeploymentNotFound

MODELS[].id must match the Azure deployment name exactly.

API errors with no provider message

Set apiVersion explicitly. See Microsoft’s API version matrix.

OpenAI-compatible (Ollama, vLLM, LM Studio)

Error Fix

"Not Found" in SSE error

baseUrl is missing the /v1 suffix. Ollama default: http://host.docker.internal:11434/v1.

ECONNREFUSED on Linux

Start Ollama with OLLAMA_HOST=0.0.0.0:11434 ollama serve. Add extra_hosts: ["host.docker.internal:host-gateway"] to the AI service compose entry.

"does not support tools"

Use an official model (ollama pull qwen3:0.6b) rather than a bare GGUF. Custom models need a Modelfile with TEMPLATE and tool support. See LLM providers.

createMessage hangs ~180s then times out

Model is too slow for the default timeout. Set LLM_TIMEOUT_MS higher, use a lighter quantization, or use a smaller model.

Content checks and hooks

Content moderation, guardrails, and hooks return their own error codes: unsafe-content-detected, content-rejected, hook-failed, and hook-timeout. For the meaning of each code, see Error code reference.

To find the error code of a failed request in the editor, read the TinyMCE AI Error: entry in the browser console, then find the service log entry with the same trace ID. See Tracing a failed request. Some failures show only the status code in the console. The troubleshooting section of each feature explains how to find the log entry for those.

For symptoms and fixes, see:

Editor and front end

Confirm /health is OK and a direct curl to /v1/conversations works before investigating the editor.

Symptom Fix

No AI buttons in the toolbar

Ensure TinyMCE 8+ is loaded, plugins: 'tinymceai' is set, and the toolbar string includes the TinyMCE AI toolbar buttons, for example tinymceai-chat tinymceai-review tinymceai-quickactions. See Toolbar buttons. Verify the API key has the AI feature enabled.

Token fetch returns 401

The token endpoint’s own authentication middleware is rejecting the request. Check session cookies, Cross-Origin Resource Sharing (CORS) credentials, and bearer tokens in the browser network tab.

Token returned but rejected by the AI service

See API and JSON Web Token (JWT) authentication above: wrong secret, wrong aud, wrong algorithm (RS256 instead of HS256), or wrong permissions shape.

AI responses hang in the browser

The reverse proxy is buffering the SSE stream. Set proxy_buffering off; and proxy_cache off; in nginx (or the equivalent for the load balancer).

CORS error on /v1/conversations

Add the editor’s origin (scheme + host + port) to the ALLOWED_ORIGINS environment variable.

Editor renders then disappears (Next.js / Nuxt / SvelteKit)

TinyMCE references window at load time. Load the editor client-only: dynamic(() ⇒ import('./Editor'), { ssr: false }) in Next.js, <ClientOnly> in Nuxt, onMount in SvelteKit. See TinyMCE integration.

Editor shows an error, and the browser console logs TinyMCE AI Error: with the cause, for example Invalid JWT token provider result or Invalid JWT token payload

Token endpoint is returning an invalid JWT or non-JSON response. Test with curl -X POST http://localhost:3000/api/ai-token and verify the response is {"token":"eyJ..."}.

Performance

Symptom Fix

Self-hosted model is slow through the AI service compared with raw curl

Co-locate the inference server with the AI service. Use a smaller or more quantized model. Disable telemetry during development (LLM_TELEMETRY_ENABLED=false). This also stops AI Diagnostics recording. See AI Diagnostics.

Containers OOM or MySQL takes 60+ seconds to start (Colima)

Default Colima VM is too small. Run colima stop && colima start --cpu 4 --memory 8 --disk 100.

Log reference

Read the AI service log with docker logs ai-service, or with kubectl logs on Kubernetes. To collect the log from several instances in one place, see Distributed logging.

Log format

Error and fatal entries are JSON objects, one per line:

Field Content

level

The severity, as a number: 50 for errors, and 60 for fatal errors, which stop the AI service. See Log levels.

time

The time of the entry, in ISO 8601 format, in UTC.

msg

The message. Error and fatal messages start with [ERROR] or [FATAL].

data

Optional. Details of the entry, such as the error name and message.

stack

Optional. The stack trace of an error.

For example, a PostgreSQL deployment without the required schema stops with this entry:

{"level":60,"time":"2026-05-05T11:19:07.634Z","msg":"[FATAL] PostgreSQL Error: schema \"cs-on-premises\" does not exist","data":{"name":"PostgreSQL Error","message":"PostgreSQL Error: schema \"cs-on-premises\" does not exist"}}

To show only error and fatal entries, filter the log with jq. The fromjson? filter skips any line that is not JSON:

docker logs ai-service 2>&1 | jq -R -c 'fromjson? | select(.level >= 50)'

Log levels

The LOG_LEVEL environment variable sets the lowest level that the AI service writes. The default is 40, so the AI service does not write trace, debug, and info entries. Keep LOG_LEVEL at 40 or higher in production: lower levels write a large volume of entries, which can affect performance and memory use.

Level Name Meaning

10

Trace

The most detailed information about the processes of the AI service.

20

Debug

Diagnostic detail for cases that are hard to debug.

30

Info

A change to a resource, an audit entry, or a request summary line.

40

Warn

A recoverable problem: a step that failed and then succeeded on retry, a side effect of asynchronous event processing, or a minor problem with the behavior of a model.

50

Error

A request or a process that failed.

60

Fatal

An error that stops the AI service.

A small, steady number of warnings about model behavior is normal. These warnings typically report a tool that the model called with invalid input. The AI service returns the error to the model, and the model corrects the call.

With ENABLE_METRIC_LOGS=true, the AI service writes request summary lines at level 30, whatever the value of LOG_LEVEL. See Structured metric logs.

Notable log messages

Start-up failures are logged with the prefix [FATAL] and stop the AI service. For those messages and their fixes, see Container startup failures. For MCP and web search messages, see MCP troubleshooting.

Message Meaning Action

Server is listening on port 8000.

The AI service started and accepts requests.

No action.

[ERROR] The user did not create a message. Reason: No valid API keys found for provider '<provider>'

The message request failed with HTTP 500. The data.name field is NoValidApiKeysFoundError.

Add credentials that the service can use for the provider. See Large language model (LLM) provider errors.

[ERROR] The user did not upload a file. Reason: Invalid argument.

The file upload failed with HTTP 500. With Google Cloud Storage through the S3-compatible API, a bucket with uniform bucket-level access enabled causes this error.

Turn off uniform bucket-level access on the bucket. See GCP (GKE).

Tracing a failed request

Every error response from the AI service carries a traceId that identifies the request. For the error response shape, see Error responses.

The TinyMCE AI plugin prints each error in the browser console as a TinyMCE AI Error: entry, followed by the status code and, when the response carries them, a Trace ID: line and the error code.

Diagnostic recipes

Expand for copy-ready diagnostic commands

Tail logs:

docker logs ai-service --tail 200 -f

Liveness check:

curl -fsS http://localhost:8000/health

Decode a JWT (inspect payload without verifying):

python3 -c "import jwt,sys,json; print(json.dumps(jwt.decode(sys.argv[1], options={'verify_signature': False}), indent=2))" <token>

Recreate after an env change:

docker compose up -d --force-recreate ai-service

Inspect effective environment:

docker inspect ai-service | jq '.[0].Config.Env'

Validate PROVIDERS JSON:

echo "$PROVIDERS" | jq .

Test data layer connectivity from inside the container:

docker exec ai-service /bin/sh -c "nc -zv mysql 3306"
docker exec ai-service /bin/sh -c "nc -zv redis 6379"

Replace mysql and redis with the actual hostnames configured in DATABASE_HOST and REDIS_HOST. If the AI service was started with docker compose rather than standalone docker run, use docker compose exec ai-service instead.

End-to-end smoke test (token mint through streamed response):

TOKEN=$(curl -s -X POST http://localhost:3000/api/ai-token | jq -r '.token')

curl -s -X POST http://localhost:8000/v1/conversations \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"id":"smoke-1","title":"Smoke test"}'

# Replace 'agent-1' with a model ID from MODELS if explicitly configured
curl -N -X POST http://localhost:8000/v1/conversations/smoke-1/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Say hi in five words.","model":"agent-1"}'