AI-Service documentation

The OX AI-Service integrates 3rd party AIs in App Suite UI, currently ChatGPT from OpenAI, Anthropic Claude on AWS Bedrock or localAI compatible models.

Architecture

The service offers the API for different AI actions and a micro frontend, actually the App Suite UI integration in form of an UI plugin. The UI Middleware loads the plugin code from the service for App Suite UI. There is no dedicated UI source container needed.

Configuration

As the AI Service delivers the App Suite UI plugin, it needs to be added to the baseUrls on the UI Middleware.

Add the AI Service url (K8s url) to the list of baseUrls of the UI Middleware configuration. This is needed to ensure that the AI integration's code is send to the user's browser.

Example config map

baseUrls:
  - http://core-ui-source-pod/
  - http://ai-service-pod/
  - http://some-other-ui-pod/

Further reading: UI middleware Config map

Capability

Important: Starting with the AI-Service 2.0 and Core UI 8.28 the needed capability was renamed.

Capability: ai-service (was open-ai)

Feature toggle

Important: Starting with the AI-Service 2.0 and Core UI 8.28 the feature toggle was renamed.

Feature toggle: ai (was open-ai)

Requirements

A running Switchboard is needed to run the AI Service, otherwise the plugin won't even start

How to enable for users

By default, AI integration is disabled for all users. To enable the feature for users, add the following:

Add capability ai-service which indicates a running AI-Service in your deployment
Enable the feature toggle for users with core ui setting io.ox/core//features/ai to true

Upsell configuration

AI integration supports the App Suite UI upsell functionality. To enable upsell for the AI features, set the following frontend configuration:

io.ox/core//upsell/activated = true
io.ox/core//upsell/enabled/ai-service = true
io.ox/core//features/ai = true

Remove the capability ai-service for all users that should see the upsell triggers. When enabled, all AI buttons and dropdowns are shown in the UI but will trigger an upsell event on the ox object.

Services and Models

The models parameter is deprecated and should no longer be used. Use the services parameter instead.

The services parameter defines a list of AI services (openai, localai etc.) that are supported by the deployment. Each service has specific attributes, such as name, models, and an optional default flag.

name: The name of the AI service. Allowed values are: openai, localai, claude-aws
default (optional): A boolean flag that indicates whether this service is the default model. Only one model per service can be marked as the default.

The models attribute defines a list of AI models that are supported by the AI service. Each model has specific attributes, such as name, maxTokens, and an optional default flag.

name: The name of the AI model.
maxTokens (optional): The maximum number of tokens that the model can handle.
default (optional): A boolean flag that indicates whether this model is the default model. Only one model per service can be marked as the default.

example for services:

services:
  - name: "openai"
    default: true
    models:
      - name: "gpt-4o"
        maxTokens: 128000
        default: true
      - name: "gpt-4"
        maxTokens: 8192
      - name: "gpt-3.5-turbo"
        maxTokens: 16385
  - name: "localai"
    models:
      - name: "my ai model"
        maxTokens: 8192
        default: true

A bit more detailed:

LocalAI Configuration

LocalAI allows you to use a self-hosted LLM with the OX AI Service. It acts as a drop-in API replacement compatible with the OpenAI API. You can also use it to connect to an already API-compatible LLM on your side.

To use it, configure your service JSON accordingly. See the Services and Models example configuration.

Additionally, set the localaiBaseUrl to point to your LocalAI endpoint. If no localaiBaseUrl is set but an openaiBaseUrl is configured, the latter will be used as a fallback. Note: This should only be used for testing.

To use OpenAI ChatGPT and LocalAI in parallel, you can either run two separate AI Service instances with different configurations or configure a single AI Service instance to connect to both OpenAI and a LocalAI instance.

enforceDefaultModel

The enforceDefaultModel value determines if the default model is used for all requests.

If set to false, the client configuration will specify which model to use for each request.

To set the model for the client use the following core ui configuration:

io.ox/ai/model = "<model-name>"

enforceDefaultService

The enforceDefaultService value determines if the default AI service is used for all requests.

If set to false, the client configuration will specify which AI service to use for each request.

To set the AI service for the client use the following core ui configuration:

io.ox/ai/service = "<service-name>"

Accounting Feature

As AI usage relies on paid APIs, providers may want to charge their users for AI usage. The service supports a flexible way to apply different commercial models. This is done by counting API requests per user. When enabled, the Accounting feature ensures that all requests done by a user are counted in a database. This number is checked against a so called "plan" which is assigned to the user. If the AI quota is reached, no additional request can be made for the current time range.

For instance, a "free" plan might allow 10 requests per day or month, while a paid plan allows 1000 requests per month. The user and brand limits can be configured via config map on the Helm chart. A plan is an object with its key as the id for the plan and has the following properties:

name: The name of the plan
plan: The type of the plan (free, paid)
strategy: The strategy for the plan (duration, monthly)
duration: The duration if strategy is "duration" (format is 30d for 30 days)
limit: The limit of requests per user and brand
enabled: The plan is enabled or not

Activation

To activate the Accounting feature in a Kubernetes environment, modify the configuration file by setting the accounting.enabled value to true in your values file.

When enabled, it monitors user activity and enforces quotas according to the user's capabilities and associated plan.

Functionality

Once the Accounting feature is activated, the service will inspect each user's JSON Web Token (JWT) for specific "capabilities" that match a configured "plan."

How It Works

JWT Inspection: When a user makes a request, the service inspects the user's JWT for any "capabilities" that correspond to a configured plan.
Plan Matching: If a capability matches a predefined plan, the service begins tracking the user's usage according to the rules of that plan.
Usage Tracking: The service monitors the number of requests made by the user as specified by the plan's quota. For example, a plan may allow 100 requests per month.
Quota Enforcement: If the user reaches the quota limit (e.g., 100 requests per month), the service responds with a status code 402 Payment Required to indicate that the quota has been exhausted.
Default Plan: If no capability is found that matches a configured plan, the user is automatically assigned a default plan. The default plan allows up to 10,000 requests per month.

Response Codes

200 OK: Request was successful, and usage is within quota limits. The response will include information about the user's remaining quota. The responses body will contain the property remainingRequests with the number of remaining requests.
402 Payment Required: The user has reached the quota limit for their plan.

User Routes

The service provides a route for users to check their usage and remaining quota:

GET /api/user/quota: Get the quota for the current user. The responses body will contain the property remainingRequests with the number of remaining requests.

Admin routes

There are several routes available for administrators to monitor usage and reset quotas:

GET /api/admin/brands/:brand/usage/:year/:month: Get usage statistics for a specific brand in a given month. :brand is the brand ID, :year is the year, and :month is the month. Example /api/admin/brands/brand1/usage/2024/1
GET /api/admin/quotas/:cid: Get the quota for a specific user. :cid is the composite user ID which is a combination of the context ID and the numeric user ID. For example, :cid could be 1-123.
DELETE /api/admin/quotas/:cid/:strategy: Reset the quota for a specific user. :cid is the composite user ID, and :strategy is the strategy for the plan (e.g., monthly or duration).

To secure these routes, there are 3 options available for authentication:

Basic Auth: Use basic authentication with the basicAuth.user and basicAuth.password values in the configuration file. If you want to use the charts secret file you have to set basicAuth.enabled to true. Otherwise, you can provide your own secret with basicAuth.enabled set to false and overrides.basicAuth set to the secret name. The provided secret should contain the keys basic_auth_user and basic_auth_password. In this case you don't need to set basicAuth.user and basicAuth.password. The request header should contain the Authorization header with the value Basic base64(username:password).
Bearer Auth: Use bearer authentication with the bearerAuth.keys value in the configuration file. The bearerAuth.keys value should be a list of keys that are allowed to access the admin routes seperated by a comma. If you want to use the charts secret file you have to set bearerAuth.enabled to true. Otherwise, you can provide your own secret with bearerAuth.enabled set to false and overrides.bearerAuth set to the secret name. The provided secret should contain the key bearer_auth_keys. In this case you don't need to set bearerAuth.keys. The request header should contain the Authorization header with the value Bearer <key>.
OAuth2: Given that you have an OAuth2 server running and this server supports openid connect and JWKS, you can use the oauth2.domain configuration to secure the admin routes. Turn this on via oauth2.enabled set to true. The service will then discover the JWKS from the given domain and validate the JWTs with it. The request header should contain the Authorization header with the value Bearer <JWT>.

Take note that you can configure multiple authentication methods at the same time. The service will try to authenticate the request with the first method and if it fails, it will try the next one. If no auth is enabled (default), no admin route is exposed at all.

Example plan configuration:

plans:
  - premium:
      name: "AI Premium"
      plan: "paid"
      strategy: "monthly"
      limit: 100
      enabled: true
  - trial:
      name: "AI Trial"
      plan: "free"
      strategy: "duration"
      duration: "30d"
      limit: 25
      enabled: true
  - specialTrial:
      name: "AI Sneak Peak Week"
      plan: "free"
      strategy: "duration"
      limit: 7
      limit: 15
      enabled: true

Steps to Configure the AI Service

Enable Database Configuration:

Add the following configuration to enable the database for the AI service:

 database:
   enabled: true
   host: <your mysql host>
   name: <your db name>
   user: <your mysql user omit if you provide your own secret>
   password: <password of your mysql user omit if you provide your own secret>

Enable Database Secret:
To create a Kubernetes secret for the database, set the mysqlSecret.enabled to true.
If you want to use your own secret for the database, set the mysqlSecret.enabled to false and provide the secret name in overrides.mysqlSecret.
Prepare the Database:
The service expects an existing database to connect to and run initial migrations. Follow the steps below to set up the database correctly.

Database Schema Setup

To ensure the AI service operates correctly, you need to create the necessary tables and procedures in your database. Although the actual migration script will be executed automatically, it is essential to ensure that the database user has the appropriate permissions.

Minimal Required Grants

The minimal grants needed for the database user are:

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, TRIGGER
ON `your_database_name`.*
TO 'your_db_user'@'%';

Replace your_database_name with the name of your database and your_db_user with the name of your database user.

Example Configuration

Here's an example of how to grant the necessary permissions:

CREATE USER 'ai_service_user'@'%' IDENTIFIED BY 'secure_password';

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, TRIGGER
ON `ai_service_db`.*
TO 'ai_service_user'@'%';

FLUSH PRIVILEGES;

In this example:

ai_service_user is the database user for the AI service.
secure_password is the password for the database user.
ai_service_db is the database name for the AI service.

Complete configuration reference

Parameter	Description	Default
image.repository	The image to be used for the deployment	registry.open-xchange.com/core/ai-service
image.pullPolicy	The imagePullPolicy for the deployment	IfNotPresent
image.tag	The image tag, defaults to app version	""
hostname	hostname for the ai-service deployment	""
origins	Allowed origins for CORS	*
logLevel	specify log level for service	"info"
logJson	Logs in JSON format	false
exposeApiDocs	Expose API documentation via Swagger UI at /api-docs	false
ingress.enabled	Generate ingress resource	false
ingress.annotations	Map of key-value pairs that will be added as annotations to the ingress resource	{}
overrides.name	Name of the chart	"ai-service"
overrides.fullname	Full name of the chart installation	"RELEASE-NAME-ai-service"
jwtSecret.enabled	Enable the secret for JWT	true
jwt.sharedSecret	Shared secret for JWT verification. This must match the secret configured for switchboard	""
jwks.domain	Domain of JWKS issuer like example.com leave empty if you want to use sharedSecret	""
openaiSecret.enabled	Enable the secret for openai	true
openaiAPIKey	OpenAI API Key	""
localaiSecret.enabled	Enable the secret for localai. May be optional, depending on your used service and model.	false
localaiAPIKey	localAI API Key. May be optional, depending on your used service and model.	""
azureSecret.enabled	Enable the secret for Azure	false
azureAPIUrl	OpenAI Azure API Url (Internal use only)	""
azureAPIKey	OpenAI Azure API Key (Internal use only)	""
mysqlSecret.enabled	Create the kubernetes secret for mysql (enable if you want to use the DB or provide own secret)	false
overrides.mysqlSecret	If you provide your own secret for mysql put the secret name here	""
database.enabled	Use Database (mandatory for usage tracking and rate limiting)	false
database.host	SQL server hostname	RELEASE-NAME-ai-service-sql
database.name	Database name	RELEASE-NAME-ai-service
database.connections	Number of concurrent connections to the DB server	"10"
database.user	DB User with access rights to sqlDB	""
database.password	DB Password of swDBUser	""
database.rootPassword	Database root password to perform admin tasks	""
database.rollback	WARNING: This will roll back the migrations this version has rolled out	false
cron.cleanupDb	Database cleanup interval (Cron notation)	0 0 * * * *
azureAPIVersion	OpenAI Azure API Key (Internal use only)	""
openaiBaseUrl	Url of the OpenAI service (internal or localAI use only)	""
localaiBaseUrl	Url of the localAI service. Falls back to to openaiBaseUrl if configured.	""
accounting.enabled	Enable the accounting feature	false
plans	List of plans with limits for users and brands	see example above
models	List of models that the service supports	see example above
enforceDefaultModel	Always use the configured default model regardless of the clients request	false
disableAPI	Disable all API endpoints. Only UI source files will be delivered	false
moderationDisabled	Disabled moderation for testing purposes (internal or localAI use only)	false
awsSecret.enabled	Enable the secret for AWS	false
awsRegion	Region of the AWS user with Bedrock & Claude enabled	""
awsAccessKey	AWS IAM access key with Bedrock & Claude enabled	""
awsSecretKey	AWS IAM secret key with Bedrock & Claude enabled	""
awsBaseUrl	URL of the AWS service (internal use only)	""
basicAuth.enabled	Enable the secret for basic authentication (Admin routes)	false
basicAuth.user	Username for basic authentication (Admin routes)	""
basicAuth.password	Password for basic authentication (Admin routes)	""
overrides.basicAuthSecret	If you provide your own secret for basic auth put the secret name here	""
bearerAuth.enabled	Enable the secret for bearer authentication (Admin routes)	false
bearerAuth.keys	Comma seperated keys to use with bearer auth (Admin Routes)	""
overrides.bearerAuthSecret	If you provide your own secret for bearer auth put the secret name here	""
oauth2.enabled	Enable OAuth2 for admin routes	false
oauth2.domain	Domain of the openid connect capable OAuth2 server	""