I would suggest that we add this section to the Katib Metrics Collector guide for now: https://www.kubeflow.org/docs/components/katib/user-guides/metrics-collector/
The getting started example, tell users to run Experiment in kubeflow namespace, which has the required label in default standalone installation.

I noticed something strange. When I previously ran the get started example with the kubeflow namespace, I encountered the "namespace lacks label" error. However, I just tried it again, and now it works fine with the kubeflow namespace.

On the other hand, when I use the API to run LLM hyperparameter optimization using kubeflow namespace too, the "namespace lacks label" error still occurs.

@andreyvelich Do you have any idea what might be causing this issue? I’ll continue investigating as well.

How did you install Katib and Training Operator when you run LLM hyperparameter optimization using kubeflow namespace?

I used the following commands from the installation page to install the latest changes of Katib control plane and Training Operator control plane:
kubectl apply -k "github.com/kubeflow/katib.git/manifests/v1beta1/installs/katib-standalone?ref=master"

kubectl apply --server-side -k "github.com/kubeflow/training-operator.git/manifests/overlays/standalone?ref=master"

So, when you deploy Training Operator after the Katib, is might override the namespace label, since it doesn't have it: https://github.com/kubeflow/training-operator/blob/master/manifests/overlays/standalone/namespace.yaml
You should install Training Operator before Katib ideally.
Also, to use PyTorchJob CRD within Katib Trial, it needs to be exist before Katib Controller starts: https://github.com/kubeflow/katib/blob/master/pkg/controller.v1beta1/trial/trial_controller.go#L121-L132

Oh I see! Problem fixed! Thank you so much

I would suggest that we add this section to the Katib Metrics Collector guide for now: https://www.kubeflow.org/docs/components/katib/user-guides/metrics-collector/
The getting started example, tell users to run Experiment in kubeflow namespace, which has the required label in default standalone installation.

SGTM.

@andreyvelich @Electronic-Waste should we add a pre-req section on https://www.kubeflow.org/docs/components/katib/user-guides/metrics-collector/ to include this step?

@andreyvelich @Electronic-Waste should we add a pre-req section on https://www.kubeflow.org/docs/components/katib/user-guides/metrics-collector/ to include this step?

SGTM. We could also declare that kubeflow namespace has already been configured with this label, that's why they can use tune function in kubeflow namespace without any prerequisite setup.

Such as "If you want to use pull-based metrics collector, you need to attach the label xxx to the target namespace (kubeflow namespace has built-in support for this label)"

Thanks for your review. I've updated the doc:

• move the detailed guidance to Metrics Collector
• add a Note in Getting Started to explain kubeflow namespace has built-in support. I also include a link to Metrics Collector for detailed instruments.

Please let me know if you have any suggestions.

@andreyvelich @Electronic-Waste should we add a pre-req section on https://www.kubeflow.org/docs/components/katib/user-guides/metrics-collector/ to include this step?

Since the user needs to update the namespace, no create a new one. I'd update the sentence by
"You can configure the namespace by adding the following label katib.kubeflow.org/metrics-collector-injection: enabled as is shown in the sample code."