Durable Task KEDA External Scaler for Azure Storage

A KEDA external scaler for Durable Task Framework (DTFx) and Azure Durable Function applications in Kubernetes that rely on the Azure Storage backend.

Trigger Specification

This specification describes the external trigger for applications that use the Durable Task Azure Storage provider.

  triggers:
    - type: external
      metadata:
        scalerAddress: dtfx-scaler.keda:4370
        connectionFromEnv: STORAGE_CONNECTIONSTRING_ENV_NAME
        maxActivitiesPerWorker: 5
        maxOrchestrationsPerWorker: 2
        taskHubName: mytaskhub

Parameter List

• accountName - Optional name of the Azure Storage account used by the Durable Task Framework (DTFx). This value is only required when useManagedIdentity is true
• clientId - Optional identity used when authenticating via managed identity. This value can only be specified when useManagedIdentity is true
• cloud - Optional name of the cloud environment that contains the Azure Storage account. Must be a known Azure cloud environment, or Private for Azure Stack Hub or air-gapped clouds. If 'Private' is specified, both endpointSuffix and entraEndpoint must be specified. Defaults to the 'AzurePublicCloud'. Possible values include:
  • AzurePublicCloud
  • AzureUSGovernmentCloud
  • AzureChinaCloud
  • Private
• connection - Optional connection string for the Azure Storage account that may be used as an alternative to connectionFromEnv
• connectionFromEnv - Optional name of the environment variable your deployment uses to get the connection string. Defaults to 'AzureWebJobsStorage'
• endpointSuffix - Optional suffix for the Azure Storage service URLs. This value is only required when cloud is 'Private'. Otherwise, the value is automatically derived for well-known cloud environments
• entraEndpoint - Optional host authority for Microsoft Entra. This value is only required when cloud is 'Private'. Otherwise, the value is automatically derived for well-known cloud environments
• maxActivitiesPerWorker - Optional maximum number of activity work items that a single worker may process at any time. This is equivalent to MaxConcurrentActivityFunctionsin Azure Durable Functions and MaxConcurrentTaskActivityWorkItems in the Durable Task Framework (DTFx). Must be greater than 0. Defaults to 10
• maxOrchestrationsPerWorker - Optional maximum number of orchestration work items that a single worker may process at any time. This is equivalent to MaxConcurrentOrchestratorFunctions in Azure Durable Functions and MaxConcurrentTaskOrchestrationWorkItems in the Durable Task Framework (DTFx). Must be greater than 0. Defaults to 5
• scalerAddress - Required address for the scaler service within the Kubernetes cluster. The format of the address is '<scaler-service-name>.<scaler-kubernetes-namespace>:<port>'. By default, the chart uses port 4370 while the service name and namespace are dependent on the Helm installation command. For example, an installation like helm install -n keda dtfx-scaler wsugarman/durabletask-azurestorage-scaler would use the address dtfx-scaler.keda:4370. For more details, please see the service template in the Helm chart
• taskHubName - Optional name of the Durable Task Framework (DTFx) task hub. This name is used when determining the name of blob containers, tables, and queues related to the application. Defaults to 'TestHubName'
• useManagedIdentity - Optionally indicates that Microsoft Entra Workload Identity should be used to authenticate between the scaler and the Azure Storage account. If true, Account must be specified, and the scaler deployment must also include a workload identity. Defaults to false
• useTablePartitionManagement - Optionally indicates that the task hub uses the newer Partition Manager V3 that relies on Azure Table Storage instead of the older Blob-based Partition Manager. Defaults to true

Authentication

The scaler supports authentication using either an Azure Storage connection string or Microsoft Entra Workload Identity.

Connection Strings

Connection strings may be specified using an environment variable exposed to the deployment using the parameter connectionFromEnv. By default, the scaler will look for an environment variable called AzureWebJobsStorage. For example:

  triggers:
    - type: external
      metadata:
        scalerAddress: dtfx-scaler.keda:4370 # Required. Address of the external scaler service
        connectionFromEnv: <variable> # Optional. By default 'AzureWebJobsStorage'

Connection strings may also be specified directly via the connection parameter:

  triggers:
    - type: external
      metadata:
        scalerAddress: dtfx-scaler.keda:4370 # Required. Address of the external scaler service
        connection: DefaultEndpointsProtocol=https;AccountName=<account-name>;AccountKey=<account-key> # Optional. Defaults to connectionFromEnv

Identity-Based Connection

To use an identity, the scaler deployment must be configured to use Azure Workload Identity. If there are multiple identities, be sure to specify the clientId parameter if it is not the default used by the deployment.

An example specification that uses an identity-based connection can be seen below:

  triggers:
    - type: external
      metadata:
        scalerAddress: dtfx-scaler.keda:4370 # Required. Address of the external scaler service
        accountName: <account-name> # Optional. Required for workload identity
        clientId: <client-id>       # Optional. Recommended if there are multiple identities
        cloud: <cloud>              # Optional. Defaults to AzurePublicCloud
        useManagedIdentity: true    # Optional. Must be true for workload identity. Defaults to false

Transport Layer Security (TLS) Protocol

The scaler optionally supports TLS. Because the KEDA and external scaler pods are seperate, both parties must be configured for mutual TLS. To configure connections from the KEDA pod to use TLS, the corresponding ScaledObject must include information about the client certificates using the field authenticationRef and a matching TriggerAuthentication object containing the certificate. The scaler pods on the other hand must be configured via the Helm chart using the tls* values to provide a certificate (and optionally verify the client's certificate). See the chart README for more details.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: dtfx-scaler-auth
  namespace: <namespace>
spec:
  secretTargetRef:
  - parameter: caCert
    name: <secret>
    key: tls.crt
  - parameter: tlsClientCert
    name: <name>
    key: tls.crt
  - parameter: tlsClientKey
    name: <name>
    key: tls.key
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: <name>
  namespace: <namespace>
spec:
  scaleTargetRef:
    name: <function app>
    kind: Deployment
  triggers:
    - type: external
      metadata:
        scalerAddress: dtfx-scaler.keda:4370 # Required. Address of the external scaler service
        accountName: <account-name> # Optional. Required for workload identity
      authenticationRef:
        name: dtfx-scaler-auth

Helm

The scaler is available as a Helm chart in the repository https://wsugarman.github.io/charts:

helm repo add wsugarman https://wsugarman.github.io/charts
helm repo update

helm install --namespace keda --create-namespace dtfx-scaler wsugarman/durabletask-azurestorage-scaler

For more information, see the chart README or visit Artifact Hub.

Licenses

The external scaler is licensed under the MIT license. The storm icon was created by Evon and is licensed royalty-free through The Noun Project.