Eval bug: b4547 doesn't see gpus

[anunknowperson]

Name and Version

C:\Users\Admin\Documents\llamacpp>llama-cli --version
version: 4547 (c5d9eff)
built with MSVC 19.29.30158.0 for

C:\Users\Admin\Documents\llamacpp>

Operating systems

Windows

GGML backends

CUDA

Hardware

Tesla P40 + 3090

Models

No response

Problem description & steps to reproduce

I tried both with cuda 11.7 and 12.4. I'm downloading
llama-b4547-bin-win-cuda-cu12.4-x64.zip and
cudart-llama-bin-win-cu12.4-x64.zip and putting them into the same folder.

C:\Users\Admin\Documents\llamacpp>llama-server --list-devices
Available devices:

C:\Users\Admin\Documents\llamacpp>

Llama server doesn't see any devices and doesn't offload layers to gpu with ngl.

It was working with previous versions. Latest koboldcpp also works (though it doesn't support R1 distilled models, so i'm trying to launch them with llamacpp)

CUDA_VISIBLE_DEVICES is not set. If I set it, still doesn't work.

First Bad Commit

No response

Relevant log output

C:\Users\Admin\Documents\llamacpp>nvidia-smi
Sat Jan 25 01:39:58 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.94                 Driver Version: 560.94         CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P40                    TCC   |   00000000:03:00.0 Off |                  Off |
| N/A   26C    P8              9W /  250W |       9MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090      WDDM  |   00000000:04:00.0 Off |                  N/A |
|  0%   48C    P8             30W /  350W |     635MiB /  24576MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

[anunknowperson]

C:\Users\Admin\Documents\llamacpp>llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Device 1: Tesla P40, compute capability 6.1, VMM: no
version: 4525 (3e3357f)
built with MSVC 19.29.30157.0 for

Everything works ok with cuda 12.4 on 4525

[slaren]

This is likely caused by the line breaks added to build.yml in #11392, the CUDA build is being built without CUDA enabled.

[lexasub]

@anunknowperson likely, you don't have cuda drivers, you install as nvidia-drivers without cuda) you may try run mining and see it

[anunknowperson]

@lexasub as I said, previous release (4525) works perfectly. I have Cuda, nvidia-smi works (though I don need it, I download cudart, llamacpp works without installed cuda-toolkit). Also the output of --version is strange. I think the problem is in builds, as @slaren said.

[ggerganov]

This is likely caused by the line breaks added to build.yml in #11392, the CUDA build is being built without CUDA enabled.

Yes, this is the problem:

https://github.com/ggerganov/llama.cpp/actions/runs/12956723171/job/36143642760#step:7:30

I think the line breaks on Windows have to be different than \. Looking into this now.

[ggerganov]

Should be fixed in the next builds. I didn't know that the line-breaks are different for the various shells:

• shell: \
• cmd: ^
• powershell: `