Authenticate issue with EKS cluster using latest version v32.0.0 #2333

alyssa1303 · 2025-01-24T04:04:45Z

What happened (please include outputs or screenshots):
Error when trying to list EKS nodes or pods

    return self.api.list_node(label_selector=label_selector, field_selector=field_selector).items
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 17150, in list_node
    return self.list_node_with_http_info(**kwargs)  # noqa: E501
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 17261, in list_node_with_http_info
    return self.api_client.call_api(
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 244, in GET
    return self.request("GET", url,
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '33e59936-e514-4ff3-8b96-34a0af79a236', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '5c784c43-9cd3-4a50-9593-03c15141a761', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'c62bb07a-ffc7-4c7b-a8ba-fc542fd3c6d1', 'Date': 'Fri, 24 Jan 2025 00:32:24 GMT', 'Content-Length': '256'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodes is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"nodes"},"code":403}

What you expected to happen:
Should be able to list node just fine with function api.list_node()

How to reproduce it (as minimally and precisely as possible):
Create EKS cluster
Install kubernetes package version v32.0.0
Run following

from kubernetes import client, config
client = client.CoreV1Api()
client.list_node(label_selector=label_selector, field_selector=field_selector).items

Anything else we need to know?:
It was working fine with version v31.0.0 but failed after upgrading to version v32.0.0

Environment:

Kubernetes version (kubectl version): v1.32.0
OS (e.g., MacOS 10.13.6): Ubuntu 2204
Python version (python --version) Python 3.10.12
Python client version (pip list | grep kubernetes) v32.0.0

The text was updated successfully, but these errors were encountered:

66li · 2025-01-24T08:23:41Z

This can be avoided by downgrading to 31.0.0

cyliu0 · 2025-01-24T11:04:46Z

Downgrading to 31.0.0 also works for me.

diseku · 2025-01-24T11:17:05Z

same! broke our CI pipes

Ottovsky · 2025-01-24T13:58:52Z

Same with AKS, downgrading to 31.0.0 helps.

romilbhardwaj · 2025-01-24T22:22:54Z

Same on GKE v1.31.4-gke.1183000

  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 13441, in delete_namespaced_service
    return self.delete_namespaced_service_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File "/Users/romilb/tools/anaconda3/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 13552, in delete_namespaced_service_with_http_info
    return self.api_client.call_api(
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 415, in request
    return self.rest_client.DELETE(url,
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/rest.py", line 270, in DELETE
    return self.request("DELETE", url,
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '13b6bd19-4596-425a-aae0-d03b518a53a5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '965a2a6c-7d5c-4983-bedd-0f6df5807afb', 'X-Kubernetes-Pf-Prioritylevel-Uid': '95309311-518d-477e-b551-637d8218a96c', 'Date': 'Fri, 24 Jan 2025 22:16:19 GMT', 'Content-Length': '334'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services \"test-2ea4--skypilot-lb\" is forbidden: User \"system:anonymous\" cannot delete resource \"services\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"test-2ea4--skypilot-lb","kind":"services"},"code":403}

bagerard · 2025-01-27T11:23:13Z

Observing a slightly different error but root cause may be similar, I'm observing a 401 unauthorized when connecting to AKS with SP client/secret.

I've debugged it to see where was the difference with the previous version and it turned out to be the shell=True additional param added to process = subprocess.Popen done here .
2dfa782

In my case it's failing when calling kubelogin -get-token, the kubelogin calls returns the command help instead of the token, it logs a parsing error but proceeds and subsequent usage of the token fails badly. Executing the same subprocess.Popen call withshell=Falseworks.

This switches the project to use PEP 440 compatible metadata instead of the Poetry specific configuration. As part of this the version of the Kubernetes library was limited to 31.* since the most recent 32.0.0 release seems to break authentication: - kubernetes-client/python#2333 - kubernetes-client/python#2334

akhilputhiry · 2025-01-27T15:56:31Z

I just came across this issue :) Here is a workaround in case some one wants

from kubernetes.config import kube_config as k8config
from kubernetes.config.exec_provider import ExecProvider


class AWSExecProvider(ExecProvider):

    def __init__(self, exec_config, cwd, cluster=None) -> None:
        super().__init__(exec_config, cwd, cluster)
        if isinstance(self.args, list):
            self.args = " ".join(self.args)


k8config.ExecProvider = AWSExecProvider


api_client = k8config.new_client_from_config_dict()
client = k8client.CoreV1Api(api_client)
client.list_namespaced_pods("default")

cesarqdt · 2025-01-27T17:07:01Z

Same here, AKS kubernetes version 1.28.15 using this code:

config.load_kube_config()
api = client.CoreV1Api()
pods = api.list_namespaced_pod(namespace)

Getting the following error:

2025-01-27T16:36:00.6511896Z         if not 200 <= r.status <= 299:
2025-01-27T16:36:00.6512295Z >           raise ApiException(http_resp=r)
2025-01-27T16:36:00.6512712Z E           kubernetes.client.exceptions.ApiException: (401)
2025-01-27T16:36:00.6513132Z E           Reason: Unauthorized
2025-01-27T16:36:00.6513832Z E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '98668c59-8242-46dd-bcb3-2b43e08729bc', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 27 Jan 2025 16:35:59 GMT', 'Content-Length': '129'})
2025-01-27T16:36:00.6514912Z E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

The solution to downgrade to version 26 pip install kubernetes==26.1.0

rubix-git · 2025-01-27T23:14:54Z

I don't think shell=True works when passing a list to Popen()
https://github.com/kubernetes-client/python/blob/release-32.0/kubernetes/base/config/exec_provider.py#L85

alyssa1303 added the kind/bug Categorizes issue or PR as related to a bug. label Jan 24, 2025

Ethanlm mentioned this issue Jan 24, 2025

Pin kubernetes pip version to 31.0.0 to fix client authentication error apple/axlearn#946

Merged

romilbhardwaj mentioned this issue Jan 24, 2025

[k8s] Pin kubernetes-python versions to avoid auth errors skypilot-org/skypilot#4604

Merged

Tenzer mentioned this issue Jan 27, 2025

Bump kubernetes from 31.0.0 to 32.0.0 Memrise/copypod#30

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Authenticate issue with EKS cluster using latest version v32.0.0 #2333

Authenticate issue with EKS cluster using latest version v32.0.0 #2333

alyssa1303 commented Jan 24, 2025

66li commented Jan 24, 2025

cyliu0 commented Jan 24, 2025

diseku commented Jan 24, 2025

Ottovsky commented Jan 24, 2025

romilbhardwaj commented Jan 24, 2025 •

edited

Loading

bagerard commented Jan 27, 2025 •

edited

Loading

akhilputhiry commented Jan 27, 2025

cesarqdt commented Jan 27, 2025

rubix-git commented Jan 27, 2025

Authenticate issue with EKS cluster using latest version v32.0.0 #2333

Authenticate issue with EKS cluster using latest version v32.0.0 #2333

Comments

alyssa1303 commented Jan 24, 2025

66li commented Jan 24, 2025

cyliu0 commented Jan 24, 2025

diseku commented Jan 24, 2025

Ottovsky commented Jan 24, 2025

romilbhardwaj commented Jan 24, 2025 • edited Loading

bagerard commented Jan 27, 2025 • edited Loading

akhilputhiry commented Jan 27, 2025

cesarqdt commented Jan 27, 2025

rubix-git commented Jan 27, 2025

romilbhardwaj commented Jan 24, 2025 •

edited

Loading

bagerard commented Jan 27, 2025 •

edited

Loading