Grafana Integration

Prerequisites

Using kube-prometheus and Grafana Operator

If you are running kube-prometheus and the Grafana Operator in your cluster, you can enable automated dashboard provisioning with minimal configuration.

To enable this integration, configure the following settings:

  1. Set monitoring.prometheus.kube-prometheus: true to enable ServiceMonitor creation.
  2. Set monitoring.grafana.enabled: true to enable dashboard ConfigMap creation.
  3. Set monitoring.grafana.grafanaOperator: true to enable the Grafana Operator integration.

The Helm chart will automatically:

  • Add the required label (grafana_dashboards: "1") to the dashboard ConfigMap
  • Skip the Node.js provisioning job to avoid conflicts
  • Make your dashboards available in Grafana via the Operator

Requirements: Your cluster must support ServiceMonitor resources (monitoring.coreos.com/v1/ServiceMonitor).

Example: Minimal values.yaml

monitoring:
  prometheus:
    kube-prometheus: true
  grafana:
    enabled: true
    grafanaOperator: true

Alternative: If you are not using the Grafana Operator, set grafanaOperator: false and the Node.js provisioning job will upload dashboards directly via the Grafana API.

Note: ServiceMonitor settings such as custom labels or intervals can be configured under monitoring.prometheus.serviceMonitor.* in your values.yaml file.

Using Manual maintained prometheus and Grafana

If you are using a manually maintained Prometheus and Grafana setup (not using kube-prometheus or Grafana Operator):

Pre-requisites:

  • The AI-Service exposes Prometheus-compatible metrics on the /metrics endpoint (default port: 9090).
  • You must have a Prometheus instance running in your cluster or environment that scrapes these metrics.
  • In Grafana, you need a data source of type prometheus configured and pointing to your Prometheus server. The name of the data source will be detected automatically. If no Prometheus data source exists, one will be created with the name Prometheus and the URL from monitoring.prometheus.url.

Note: The dashboards will use the detected Prometheus data source. If none exists, a new one named Prometheus will be created automatically.

How it works (Node.js Provisioning)

When enabled, the Helm chart will:

  • Create a Kubernetes Secret with your Grafana credentials/API token.
  • Create a ConfigMap containing all dashboards from helm/ai-service/dashboards/*.json.
  • Run a Kubernetes Job (grafana-init) that uploads the dashboards to your Grafana instance using the credentials provided.
  • The job runs as a Helm pre-install/upgrade hook, so dashboards are always up-to-date after each deployment.
  • Dashboards will be automatically overwritten on every update. If you want to customize a dashboard, please create a copy in Grafana.

Example: Minimal values.yaml (Node.js Provisioning)

Note: This example assumes you already have a valid Grafana API token and a Prometheus datasource is configured in your cluster.

monitoring:
  grafana:
    enabled: true
    url: "http://your-grafana-url"
    apiToken: "<your-grafana-api-token>"

Configuration Options for Provisioning

You have the following options to configure dashboard provisioning:

monitoring:
  prometheus:
    url: "http://prometheus.monitoring.svc.cluster.local"
    dataSource: ""         # Optional: override the Prometheus datasource name
    kube-prometheus: false # Enable if you want to use kube-prometheus/ServiceMonitor
    serviceMonitor:
      namespace: ''           # Optional: target namespace for ServiceMonitor
      interval: 30s           # Optional: scrape interval
      scrapeTimeout: ''       # Optional: scrape timeout
      relabelings: []         # Optional: relabeling config
      metricRelabelings: []   # Optional: metric relabeling config
      honorLabels: false      # Optional: honor labels
      additionalLabels: {}    # Optional: additional labels for ServiceMonitor
      annotations: {}         # Optional: annotations for ServiceMonitor
  grafana:
    enabled: true
    grafanaOperator: false  # Set to true to use Grafana Operator for dashboard provisioning
    url: "http://your-grafana-url"
    secretName: ""         # Optional: name of the Kubernetes secret for Grafana credentials
    username: ""           # Optional: Grafana API username
    password: ""           # Optional: Grafana API password
    apiToken: ""           # Recommended for automation
    folder: ""             # Optional: target folder in Grafana for dashboards
    ttlInitJob: 600         # (Optional) TTL for the init job in seconds

prometheus:

  • url: The URL of your Prometheus instance (must be reachable from the cluster).
  • dataSource: (Optional) Override the Prometheus datasource name in Grafana.
  • kube-prometheus: Set to true to enable ServiceMonitor creation for kube-prometheus environments.
  • serviceMonitor.namespace: (Optional) Target namespace for ServiceMonitor.
  • serviceMonitor.interval: (Optional) Scrape interval for ServiceMonitor.
  • serviceMonitor.scrapeTimeout: (Optional) Scrape timeout for ServiceMonitor.
  • serviceMonitor.relabelings: (Optional) Relabeling configuration for ServiceMonitor.
  • serviceMonitor.metricRelabelings: (Optional) Metric relabeling configuration for ServiceMonitor.
  • serviceMonitor.honorLabels: (Optional) Honor labels for ServiceMonitor.
  • serviceMonitor.additionalLabels: (Optional) Additional labels for ServiceMonitor.
  • serviceMonitor.annotations: (Optional) Annotations for ServiceMonitor.

grafana:

  • enabled: Set to true to activate the Grafana integration.
  • grafanaOperator: Set to true to enable Grafana Operator integration. This will add the label grafana_dashboards: "1" to the dashboard ConfigMap and disable the Node.js provisioning path.
  • url: The URL of your Grafana instance (must be reachable from the cluster).
  • username/password: Credentials for Grafana API access (if required).
  • apiToken: Grafana API token (preferred for automation and security).
  • secretName: (Optional) Name of the Kubernetes secret for Grafana credentials.
  • folder: (Optional) Target folder in Grafana for dashboards.
  • ttlInitJob: (Optional) How long the init job should be kept after completion (default: 600 seconds).

Important:

  • You can use either basic authentication (username/password) or an API token for Grafana access.
  • If an apiToken is provided, it will always take precedence over basic auth.

Security Notes (Node.js Provisioning)

  • It is recommended to use an API token with minimal permissions (dashboard write access).
  • Credentials and tokens are stored as Kubernetes Secrets.

Troubleshooting (Node.js Provisioning)

  • Check the logs of the grafana-init job for errors if dashboards are not visible in Grafana.
  • Ensure the Grafana URL is reachable from the cluster and the API token is valid.
  • The job will be deleted automatically after the configured TTL.

Uninstalling Dashboards (Node.js Provisioning)

  • Disabling the integration (enabled: false) will stop provisioning new dashboards, but will not remove existing dashboards from Grafana.
  • To remove dashboards, delete them manually in Grafana or via the API.
Last Updated::