Monitoring Temporal Cloud metrics with ClickStack

WarningOpenMetrics support in the Temporal platform is available in Public Preview. Refer to their documentation for more information.

Temporal offers an abstraction for building simple, sophisticated, resilient applications.

TL;DRMonitor Temporal Cloud metrics in ClickStack using the OTel Prometheus receiver. Includes a pre-built dashboard.

Integration with existing Temporal Cloud

This section covers configuring ClickStack by configuring the ClickStack OTel collector with the Prometheus receiver.

Prerequisites

ClickStack instance running
Existing Temporal Cloud account
HTTP network access from ClickStack to your Temporal Cloud

Create Temporal Cloud key

Ensure you have a Temporal Cloud API key. This can be created by following the Authentication guide in the Temporal documentation.

Key fileEnsure these credentials are stored in a file temporal.key in the same directory as the config file created below. This key should just be stored as text with no preceding or following spaces.

Create custom OTel collector configuration

ClickStack allows you to extend the base OpenTelemetry collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP.Create a file named temporal-metrics.yaml with the following configuration:

temporal-metrics.yaml

receivers:
  prometheus/temporal:
    config:
      scrape_configs:
      - job_name: 'temporal-cloud'
        scrape_interval: 60s
        scrape_timeout: 30s
        honor_timestamps: true
        scheme: https
        authorization:
          type: Bearer
          credentials_file: /etc/otelcol-contrib/temporal.key
        static_configs:
          - targets: ['metrics.temporal.io']
        metrics_path: '/v1/metrics'

processors:
  resource:
    attributes:
      - key: service.name
        value: "temporal"
        action: upsert

service:
  pipelines:
    metrics/temporal:
      receivers: [prometheus/temporal]
      processors:
        - resource
        - memory_limiter
        - batch
      exporters:
        - clickhouse

This configuration:

Connects to Temporal Cloud at metrics.temporal.io
Collects metrics every 60 seconds
Collects key performance metrics
Sets the required service.name resource attribute per OpenTelemetry semantic conventions
Routes metrics to the ClickHouse exporter via a dedicated pipeline

You only define new receivers, processors, and pipelines in the custom config
The memory_limiter and batch processors and clickhouse exporter are already defined in the base ClickStack configuration - you just reference them by name
The resource processor sets the required service.name attribute per OpenTelemetry semantic conventions
For multiple Temporal cloud accounts, customize service.name to distinguish them (e.g., "temporal-prod", "temporal-dev")

Configure ClickStack to load custom configuration

To enable custom collector configuration in your existing ClickStack deployment, you must:

Mount the custom config file at /etc/otelcol-contrib/custom.config.yaml
Set the environment variable CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml
Mount the temporal.key file at /etc/otelcol-contrib/temporal.key
Ensure network connectivity between ClickStack and Temporal

All commands assume they’re executed from the sample directory as where temporal-metrics.yaml and temporal.key are stored.Update your ClickStack deployment configuration:

services:
  clickstack:
    # ... existing configuration ...
    environment:
      - CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml
    volumes:
      - ./temporal-metrics.yaml:/etc/otelcol-contrib/custom.config.yaml:ro
      - ./temporal.key:/etc/otelcol-contrib/temporal.key:ro
      # ... other volumes ...

If using the all-in-one image with docker run:

docker run --name clickstack \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/temporal-metrics.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v "$(pwd)/temporal.key:/etc/otelcol-contrib/temporal.key:ro" \
  clickhouse/clickstack-all-in-one:latest

Verify metrics in HyperDX

Once configured, log into HyperDX and verify metrics are flowing:

Navigate to the Metrics explorer
Search for metrics starting with temporal (e.g., temporal_cloud_v1_workflow_success_count, temporal_cloud_v1_poll_timeout_count)
You should see metric data points appearing at your configured collection interval

Dashboards and visualization

To help you get started monitoring Temporal Cloud with ClickStack, we provide some example visualizations for Temporal Metrics.

the dashboard configuration

Import the pre-built dashboard

Open HyperDX and navigate to the Dashboards section
Click Import Dashboard in the upper right corner under the ellipses

Upload the temporal-metrics-dashboard.json file and click Finish Import

View the dashboard

The dashboard will be created with all visualizations pre-configured:

Troubleshooting

Custom config not loading

Verify the environment variable CUSTOM_OTELCOL_CONFIG_FILE is set correctly:

docker exec <container-name> printenv CUSTOM_OTELCOL_CONFIG_FILE

Check that the custom config file is mounted at /etc/otelcol-contrib/custom.config.yaml:

docker exec <container-name> ls -lh /etc/otelcol-contrib/custom.config.yaml
# usually, docker exec clickstack ls -lh /etc/otelcol-contrib/custom.config.yaml

View the custom config content to verify it’s readable:

docker exec <container-name> cat /etc/otelcol-contrib/custom.config.yaml
# usually, docker exec clickstack cat /etc/otelcol-contrib/custom.config.yaml

Confirm the temporal.key is mounted into the container:

docker exec <container-name> cat /etc/otelcol-contrib/temporal.key
# usually, docker exec clickstack cat /etc/otelcol-contrib/temporal.key
# This should output your temporal.key

No metrics appearing in HyperDX

Verify Temporal Cloud is accessible from the collector:

# From the ClickStack container
docker exec <container-name> curl -H "Authorization: Bearer <API_KEY>" https://metrics.temporal.io/v1/metrics

You should see a series of Prometheus metrics printed e.g.

temporal_cloud_v1_workflow_success_count{operation="CompletionStats",region="aws-us-east-2",temporal_account="l2c4n",temporal_namespace="clickpipes-aws-prd-apps-us-east-2.l2c4n",temporal_task_queue="clickpipes-svc-dc118d12-b397-4975-a33e-c2888ac12ac4-peer-flow-task-queue",temporal_workflow_type="QRepPartitionWorkflow"} 0.067 1765894320

Verify the effective config includes your Prometheus receiver:

docker exec <container> cat /etc/otel/supervisor-data/effective.yaml | grep -A 10 "Prometheus:"
## usually, docker exec clickstack cat /etc/otel/supervisor-data/effective.yaml | grep -A 10 "prometheus:"

Check for errors in the collector agent logs:

docker exec <container> cat /etc/otel/supervisor-data/agent.log | grep -i Prometheus
# Look for connection errors or authentication failures
# docker exec clickstack cat /etc/otel/supervisor-data/agent.log | grep -i Prometheus

Check in the collector logs:

docker exec <container> cat /var/log/otel-collector.log | grep -i error
# Look for config parsing errors - early supervisor.opamp-client can be ignored 
# docker exec clickstack cat /var/log/otel-collector.log | grep -i error

Authentication errors

If you see authentication errors in the logs check your API key.

Network connectivity issues

If ClickStack can’t reach Temporal Cloud ensure your Docker Compose file or docker run commands allow external networking.

Next steps

Set up alerts for critical metrics (workflow failure rates, task backlog growth, schedule-to-start latency)
Create additional dashboards for specific use cases (namespace-level monitoring, workflow type performance)
Monitor multiple Temporal Cloud accounts by duplicating the receiver configuration with different endpoints and service names

Going to production

This guide extends ClickStack’s built-in OpenTelemetry Collector for quick setup. For production deployments, we recommend running your own OTel Collector and sending data to ClickStack’s OTLP endpoint. See Sending OpenTelemetry data for production configuration.

​Integration with existing Temporal Cloud

​Prerequisites

​Create Temporal Cloud key

​Create custom OTel collector configuration

​Configure ClickStack to load custom configuration

​Verify metrics in HyperDX

​Dashboards and visualization

​ the dashboard configuration

​Import the pre-built dashboard

​View the dashboard

​Troubleshooting

​Custom config not loading

​No metrics appearing in HyperDX

​Authentication errors

​Network connectivity issues

​Next steps

​Going to production

Integration with existing Temporal Cloud

Prerequisites

Create Temporal Cloud key

Create custom OTel collector configuration

Configure ClickStack to load custom configuration

Verify metrics in HyperDX

Dashboards and visualization

the dashboard configuration

Import the pre-built dashboard

View the dashboard

Troubleshooting

Custom config not loading

No metrics appearing in HyperDX

Authentication errors

Network connectivity issues

Next steps

Going to production