LLM Providers

LLM Provider Reference

setup_llm parameters

Model groups

setup_llm can create different kinds of Cognigy LLM resources. The important distinction is the modelType:

• Chat models: used for AI Agents, Knowledge Search, and Answer Extraction. Examples: gpt-4o, gpt-4o-mini, gpt-4.1, claude-sonnet-4-0, gemini-2.0-flash, mistral-small-2503.
• Embedding models: used for knowledge-store vector indexing. Examples: text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002, luminous-embedding-128, amazon.titan-embed-text-v2:0, Pharia-1-Embedding-4608, gemini-embedding-001, custom-embedding-model.

Chat/completion models are not embedding models. gpt-4o-mini is a chat model, not a valid embedding-model choice for knowledge-store indexing.

When using list_resources { resourceType: "llm_model", projectId }, inspect the returned modelType and select the model by its exact role. Do not infer that all LLMs are interchangeable.

OpenAI-compatible providers (openAICompatible)

Use provider openAICompatible for ANY endpoint that speaks the OpenAI API but is not OpenAI itself: vLLM, Hugging Face, LiteLLM, Groq, Together AI, Azure AI Foundry (model router), or self-hosted deployments. Do NOT mislabel these as openAI — without a base URL the connection test hits api.openai.com and fails.

Required parameters:

• modelType: exactly custom-model (chat) or custom-embedding-model (embedding) — never the real model name
• customModel: the model name as known by the provider, e.g. llama-3.3-70b-instruct
• baseCustomUrl: the provider's OpenAI-compatible base URL, e.g. https://my-llm-host.example.com/v1
• apiKey (or a same-project connectionId)

Optional parameters:

• customAuthHeader: custom HTTP header name for authentication (e.g. Ocp-Apim-Subscription-Key). When set, the API key is sent in that header instead of Authorization: Bearer <key>.
• apiType: chatCompletion (default) or responses — only use responses if the provider supports OpenAI's Responses API.

Example:

{
  "projectId": "<projectId>",
  "provider": "openAICompatible",
  "modelType": "custom-model",
  "customModel": "llama-3.3-70b-instruct",
  "baseCustomUrl": "https://my-llm-host.example.com/v1",
  "apiKey": "<key>"
}

Notes:

• The Completions API for custom LLMs is deprecated (removal planned for Cognigy.AI 2026.24.0) — use Chat Completions or Responses.
• When inspecting existing models via list_resources / get_resource, openAI-compatible models show modelType: "custom-model"; the real model name and endpoint are in the openAICompatible object (customModel, baseCustomUrl, customAuthHeader).

AWS Bedrock (awsBedrock)

Use provider awsBedrock for models hosted on AWS Bedrock (Amazon Nova, Anthropic Claude on Bedrock, Titan embeddings).

Required parameters:

• region: AWS region of the Bedrock deployment, e.g. us-east-1
• modelType: a Bedrock model id from Cognigy's supported list — chat: amazon.nova-pro-v1:0, amazon.nova-lite-v1:0, amazon.nova-micro-v1:0, amazon.nova-2-lite-v1:0; embedding: amazon.titan-embed-text-v2:0. For any other Bedrock model (including Claude) use custom-model and put the model id — or an inference profile id like eu.anthropic.claude-sonnet-4-6 — in customModel. Cognigy's list changes with platform releases (anthropic.claude-3-5-sonnet-20240620-v1:0 was removed in 2026.10); check the deprecations table in the docs if a named id is rejected.
• Credentials — one of (NOT apiKey):
  • accessKeyId + secretAccessKey (access-key auth → AwsBedrockProvider connection) — the standard option, works everywhere
  • roleArn (IAM-role auth → AwsBedrockProviderIamRole connection) — feature-gated: only works when the Cognigy installation has IAM connections enabled for the organisation (FEATURE_ENABLE_IAM_AWS_CONNECTION_WHITELIST); otherwise connection creation is rejected with "This type is not enabled for your installation". Don't steer users here unless they asked for IAM-role auth — default to the key pair.
  • connectionId of an existing same-project connection

Optional parameters:

• location: inference-call routing — region (default; requests stay in the given AWS region), geo (routed within a geographic boundary; requires geo), or global (routed worldwide, highest throughput, no data-residency guarantee)
• geo: required when location is geo — the geographic boundary, e.g. us, eu, apac. The AWS region must lie inside it.

Example:

{
  "projectId": "<projectId>",
  "provider": "awsBedrock",
  "modelType": "amazon.nova-pro-v1:0",
  "region": "eu-central-1",
  "accessKeyId": "<AWS access key id>",
  "secretAccessKey": "<AWS secret access key>"
}

When inspecting existing models, the region, routing location/geo, and custom model id are in the awsBedrock object of the response.

Caveat: AWS Bedrock rejects requests with an empty or whitespace-only system prompt (HTTP 400). For AI Agents this applies only when persona, job description, AND instructions are all empty — make sure a Bedrock-backed agent has at least one of them filled.

Credential resolution

• Provide the provider's credentials — a Connection is auto-created, then the LLM resource is linked to it. For most providers that is apiKey; for awsBedrock it is accessKeyId + secretAccessKey or roleArn instead (apiKey is rejected there)
• Provide connectionId (UUID referenceId of an existing Connection in the SAME project, of a type matching the provider) to skip connection creation — never together with inline credentials
• Either the provider's credentials or connectionId is required
• If the only working connection lives in another project, transfer the LLM + connection via manage_packages instead of passing that connectionId directly

Connection validation

After creating the model, setup_llm automatically tests the connection by sending a minimal probe to the provider. This catches invalid API keys, wrong model types, and misconfigured providers before they can break downstream flows.

• Test passes: the response includes connectionTest.isCredentialsValid: true and the provider's confirmation message.
• Test fails: the model is automatically deleted to prevent broken references, and an error is returned with the provider's error message.
• Test endpoint unreachable: the model is kept but a warning is returned advising manual verification.

Common errors

• "Invalid provider": use exact camelCase strings from the provider column (openAI, not openai)
• "Authentication failed": verify the credentials are valid for that provider (apiKey; or AWS access keys / role ARN for awsBedrock)
• "Model not found": check exact modelType spelling (e.g. gpt-4o, not gpt4o)

Troubleshooting: dangerouslySkipConnectionTest

If the connection test cannot run in your environment (e.g., air-gapped setup, unsupported custom model provider), you can pass dangerouslySkipConnectionTest: true to skip validation. This is a last resort — it may leave a non-functional model reference that silently breaks agent conversations and knowledge stores. Always prefer fixing the root cause instead. Do not use this to work around a missing, invalid, or cross-project connection.