bug: Syncing an ApisixRoute object fails due to an unknown plugin error related to opentelemetry.

[douglasqsantos]

Current Behavior

The sync process fails and retries repeatedly due to the unknown plugin [opentelemetry] error.

Expected Behavior

The opentelemetry plugin should be recognized and correctly handled during the sync process.

Error Logs

ingress-apisix-composite-deployment-657758f575-59zs6 ingress-apisix 2025-01-08T03:49:46+08:00 warn apisix/apisix_route.go:481 sync ApisixRoute failed, will retry {"object": {"Type":1,"Object":{"Key":"internal-services/webhook-site","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unknown plugin [opentelemetry]\n\n"}

Steps to Reproduce

Deploy an ApisixRoute object referencing the opentelemetry plugin.

---
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: apisixroute-test-discovery
  namespace: default
spec:
  discovery:
    serviceName: default/apisixroute-test-svc:http
    type: kubernetes
  loadbalancer:
    hashOn: vars
    key: remote_addr
    type: roundrobin
  scheme: http
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: apisixroute-test
  namespace: default
spec:
  http:
  - match:
      hosts:
      - apisix-test.domain.net
      methods:
      - GET
      - HEAD
      - PUT
      - POST
      - PATCH
      - DELETE
      - OPTIONS
      paths:
      - /*
    name: mainz
    plugins:
    - config:
        sampler:
          name: always_on
      enable: true
      name: opentelemetry
    upstreams:
    - name: apisixroute-test-discovery
      weight: 100
...

Monitor the logs of the ingress-apisix pod.

ingress-apisix-composite-deployment-657758f575-59zs6 ingress-apisix 2025-01-08T03:49:46+08:00	error	apisix/apisix_route.go:379	failed to sync ApisixRoute to apisix	{"error": "1 error occurred:\n\t* unknown plugin [opentelemetry]\n\n"}

Observe the reported error regarding the unknown plugin.

The current deployed composite.yaml

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

apiVersion: v1
kind: Namespace
metadata:
  name: apisix
  labels:
    name: apisix
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: apisix-view-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - "*"
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - persistentvolumeclaims
      - pods
      - replicationcontrollers
      - replicationcontrollers/scale
      - serviceaccounts
      - services
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - bindings
      - limitranges
      - namespaces/status
      - pods/log
      - pods/status
      - replicationcontrollers/status
      - resourcequotas
      - resourcequotas/status
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - namespaces
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - apps
    resources:
      - controllerrevisions
      - daemonsets
      - deployments
      - deployments/scale
      - replicasets
      - replicasets/scale
      - statefulsets
      - statefulsets/scale
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - autoscaling
    resources:
      - horizontalpodautoscalers
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - batch
    resources:
      - cronjobs
      - jobs
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - daemonsets
      - deployments
      - deployments/scale
      - ingresses
      - networkpolicies
      - replicasets
      - replicasets/scale
      - replicationcontrollers/scale
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - policy
    resources:
      - poddisruptionbudgets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
      - ingresses/status
      - networkpolicies
    verbs:
      - "*"
  - apiGroups:
      - metrics.k8s.io
    resources:
      - pods
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - apisix.apache.org
    resources:
      - apisixroutes
      - apisixroutes/status
      - apisixupstreams
      - apisixupstreams/status
      - apisixtlses
      - apisixtlses/status
      - apisixclusterconfigs
      - apisixclusterconfigs/status
      - apisixconsumers
      - apisixconsumers/status
      - apisixpluginconfigs
      - apisixpluginconfigs/status
      - apisixglobalrules
      - apisixglobalrules/status
    verbs:
      - "*"
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - "*"
  - apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - gateway.networking.k8s.io
    resources:
      - httproutes
      - tlsroutes
      - tcproutes
      - gateways
      - gatewayclasses
      - udproutes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - gateway.networking.k8s.io
    resources:
      - gateways/status
      - gatewayclasses/status
    verbs:
      - get
      - update
      - list

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: apisix-view-serviceaccount
  namespace: apisix

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: apisix-view-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: apisix-view-clusterrole
subjects:
  - kind: ServiceAccount
    name: apisix-view-serviceaccount
    namespace: apisix

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: apisix-gw-config.yaml
  namespace: apisix
data:
  config.yaml: >
    deployment:
      admin:
        allow_admin:
          - 127.0.0.0/24
          - 0.0.0.0/0
        admin_listen:
          ip: 0.0.0.0
          port: 9180
      etcd:
        host:
          - "http://127.0.0.1:12379"
        prefix: "/apisix"
        timeout: 60

    apisix:
      enable_control: true
      enable_reuseport: true

      stream_proxy:
        only: false
        tcp:
          - 9100
          - addr: 9110
            tls: true
        udp:
          - 9200


    plugins:                           
      - real-ip                        
      - ai                             
      - client-control                 
      - proxy-control                  
      - request-id                     
      - zipkin                         
      - opentelemetry                  
      - ext-plugin-pre-req             
      - fault-injection                
      - mocking                        
      - serverless-pre-function        
      - batch-requests                 
      - cors                           
      - ip-restriction                 
      - ua-restriction                 
      - referer-restriction            
      - csrf                           
      - uri-blocker                    
      - request-validation             
      - openid-connect                 
      - cas-auth                       
      - authz-casbin                   
      - authz-casdoor                  
      - wolf-rbac                      
      - ldap-auth                      
      - hmac-auth                      
      - basic-auth                     
      - jwt-auth                       
      - key-auth                       
      - consumer-restriction           
      - forward-auth                   
      - opa                            
      - authz-keycloak                 
      - proxy-mirror                   
      - proxy-cache                    
      - proxy-rewrite                  
      - workflow                       
      - api-breaker                    
      - limit-conn                     
      - limit-count                    
      - limit-req                      
      - node-status                    
      - gzip                           
      - traffic-split                  
      - redirect                       
      - response-rewrite               
      - kafka-proxy                    
      - grpc-transcode                 
      - grpc-web                       
      - public-api                     
      - prometheus                     
      - datadog                        
      - elasticsearch-logger           
      - echo                           
      - loggly                         
      - http-logger                    
      - splunk-hec-logging             
      - skywalking-logger              
      - google-cloud-logging           
      - sls-logger                     
      - tcp-logger                     
      - kafka-logger                   
      - rocketmq-logger                
      - syslog                         
      - udp-logger                     
      - file-logger                    
      - clickhouse-logger              
      - tencent-cloud-cls              
      - inspect                        
      - aws-lambda                     
      - azure-functions                
      - openwhisk                      
      - openfunction                   
      - serverless-post-function       
      - ext-plugin-post-req            
      - ext-plugin-post-resp           

    plugin_attr:
      prometheus:
        enable_export_server: false
      opentelemetry:
        resource:
          service.name: APISIX
        collector:
          address: 127.0.0.1:4318
        batch_span_processor:
          drop_on_queue_full: true
          batch_timeout: 10
        real-ip:
          source: http_x_forwarded_for
          trusted_addresses:
            # Broad representation of all internal AWS IPs
            - "10.0.0.0/8"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-apisix-composite-deployment
  namespace: apisix
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-apisix-composite-deployment
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-apisix-composite-deployment
    spec:
      volumes:
        - name: apisix-config-yaml-configmap
          configMap:
            name: apisix-gw-config.yaml
            defaultMode: 420
      containers:
        - livenessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 15
            timeoutSeconds: 2
            periodSeconds: 2
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 10
            timeoutSeconds: 2
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          name: ingress-apisix
          image: apache/apisix-ingress-controller:1.7.0
          command:
            - /ingress-apisix/apisix-ingress-controller
            - ingress
            - "--log-output"
            - stdout
            - "--apisix-resource-sync-interval"
            - 1h
            - "--apisix-resource-sync-comparison=true"
            - "--http-listen"
            - ":8080"
            - "--https-listen"
            - ":8443"
            - "--default-apisix-cluster-name"
            - default
            - "--default-apisix-cluster-base-url"
            - http://127.0.0.1:9180/apisix/admin
            - "--default-apisix-cluster-admin-key"
            - edd1c9f034335f136f87ad84b625c8f1
            - "--api-version"
            - apisix.apache.org/v2
            - "--ingress-status-address"
            - ""
            - "--disable-status-updates=false"
            - "--etcd-server-enabled=true"
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
            - name: https
              containerPort: 8443
              protocol: TCP
            - name: etcd
              containerPort: 12379
              protocol: TCP
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
        - livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 20
            periodSeconds: 2
            successThreshold: 1
            tcpSocket:
              port: 9080
            timeoutSeconds: 2
          readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 15
            periodSeconds: 2
            successThreshold: 1
            tcpSocket:
              port: 9080
            timeoutSeconds: 1
          name: apisix
          image: apache/apisix:3.2.2-centos
          ports:
            - name: http
              containerPort: 9080
              protocol: TCP
            - name: http-admin
              containerPort: 9180
              protocol: TCP
            - name: https
              containerPort: 9443
              protocol: TCP
          resources: {}
          volumeMounts:
            - name: apisix-config-yaml-configmap
              mountPath: /usr/local/apisix/conf/config.yaml
              subPath: config.yaml
      restartPolicy: Always
      terminationGracePeriodSeconds: 0
      dnsPolicy: ClusterFirst
      serviceAccountName: apisix-view-serviceaccount
      serviceAccount: apisix-view-serviceaccount
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 50%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

---
apiVersion: v1
kind: Service
metadata:
  name: ingress-apisix-gateway
  namespace: apisix
spec:
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 9080
    - name: https
      protocol: TCP
      port: 443
      targetPort: 9443
  selector:
    app.kubernetes.io/name: ingress-apisix-composite-deployment
  type: NodePort

Environment

• Ingress APISIX Version: apache/apisix:3.2.2-centos, apache/apisix:3.11.0-redhat
• Kubernetes Version: v1.30.2-eks-1552ad0
• APISIX Plugins Configured: opentelemetry
• Deployment Method: Composite Architecture

[douglasqsantosaudibene]

We are currently investigating the issue and noticed that the OpenTelemetry configuration is missing from the apisix-schema.json file.

Could you please confirm if this is an intentional omission or if OpenTelemetry support is expected to be included in this schema? Additionally, any guidance on the recommended approach for adding OpenTelemetry configuration would be appreciated.

Thank you for your assistance!