Skip to main content
The integration-api abstracts all LLM providers behind a single LargeLanguageCaller interface. Provider credentials are encrypted at rest using INTEGRATION_CRYPTO_KEY. The assistant-api calls integration-api via gRPC to execute LLM inference.

​
LargeLanguageCaller Interface

Every LLM provider implements: 
// api/integration-api/internal/caller/
type LargeLanguageCaller interface {
    // GetChatCompletion returns a complete response (non-streaming).
    GetChatCompletion(
        ctx         context.Context,
        messages    []ChatMessage,
        opts        ChatCompletionOptions,
        credentials map[string]interface{},
    ) (*ChatCompletion, error)

    // StreamChatCompletion streams tokens via callbacks.
    StreamChatCompletion(
        ctx         context.Context,
        messages    []ChatMessage,
        opts        ChatCompletionOptions,
        credentials map[string]interface{},
        onStream    func(token string),              // called per token
        onMetrics   func(metrics LLMMetrics),        // called on completion with token counts
        onError     func(err error),                 // called on error
    ) error
}

​
ChatCompletionOptions

Model behaviour is controlled via the ModelParameter map: 
type ChatCompletionOptions struct {
    ModelParameter map[string]interface{}  // keyed by "model.*"
    Tools          []ToolDefinition
}
KeyTypeDescription
model.namestringModel ID (e.g. gpt-4o, claude-sonnet-4-6)
model.temperaturefloatSampling temperature (0.0–2.0)
model.max_tokensintMaximum tokens in the response
model.top_pfloatNucleus sampling threshold
model.stop[]stringStop sequences
model.tool_choicestringTool usage: auto, none, required
model.frequency_penaltyfloatRepeat penalty (OpenAI / Gemini)
model.presence_penaltyfloatTopic penalty (OpenAI / Gemini)
model.seedintDeterministic sampling seed
model.response_formatstringOutput format (e.g. json_object)
Not all providers support all keys — unsupported keys are silently ignored.

​
Supported LLM Providers

ProviderDirectoryNotes
OpenAIcaller/openai/GPT-4o, GPT-4o-mini, GPT-4-turbo
Anthropiccaller/anthropic/Claude Opus/Sonnet/Haiku
Google Geminicaller/gemini/Gemini 2.0, 1.5 Pro
Azure OpenAIcaller/azure/Azure-hosted OpenAI deployments
Mistralcaller/mistral/Mistral Large, Small, Nemo
Coherecaller/cohere/Command R, Command R+
Vertex AIcaller/vertexai/Google Vertex AI models
HuggingFacecaller/huggingface/HuggingFace Inference API
Replicatecaller/replicate/Replicate hosted models
Voyage AIcaller/voyageai/Embedding and reranking

​
Credential Encryption

All vault credentials are encrypted with INTEGRATION_CRYPTO_KEY before storage. Never commit this key or expose it in logs. 
# docker/integration-api/.integration.env
INTEGRATION_CRYPTO_KEY=32-char-random-string

​
Provider Pages

OpenAI

GPT-4o, GPT-4o-mini

Anthropic

Claude Opus, Sonnet, Haiku

Gemini

Gemini 2.0 Flash, 1.5 Pro

Azure OpenAI

Azure-hosted OpenAI deployments

Configure Your Own

Implement the LargeLanguageCaller interface