dockerTools.streamLayeredImage containers can't runt nix-friendly #!/usr/bin/env bash script

[jficz]

Describe the bug

Nix-friendly bash scripts which use #!/usr/bin/env bash shebang don't work in docker images built by buildLayeredImage (and, transitively, by streamLayeredImage)

This is caused by /usr/bin/env not being created and all packages' bins being linked to /bin instead.

Steps to reproduce

Steps to reproduce the behavior:

1. Build nix-based docker image:

...
      containers.nix = pkgs.dockerTools.buildLayeredImage {
       name = "import-tools";
       tag = "v1.0";
       contents = with pkgs; [
         bash
         which
         git
         jq
         yq
         coreutils
       ];

       extraCommands = ''
         mkdir -p repo
       '';

       config = {
         Cmd = ["./import.sh"];
         WorkingDir = "repo";
       };
     };
...

2. load image to podman and run image with a dir containing import.sh mounted to container's /repo:

podman run --mount type=bind,src=.,dst=/repo localhost/import-tools:v1.0 

3. {"msg":"exec container process (missing dynamic library?) '/repo/./import.sh': No such file or directory","level":"error","time":"2025-01-12T09:44:45.821363Z"}

Expected behavior

Import script being run.

Additional context

head -n1 import.sh
#!/usr/bin/env bash

I tried to understand where the /bin symlinking happens but so far I was unable to identify the place. I think it happens in the $stdenv/setup phase but the setup script is rather complicated and I have yet to analyze it properly.

It seems the linking happens in the symlinkJoin step but so far I don't see how to address the issue.

Therefore, I'm unsure who to tag.

I suggest a flag being added to streamLayeredImage which will make the build follow common distro practices and instead of putting links to /bin puts them in /usr/bin and symlinks /bin -> /usr/bin. This might incidentally also indirectly solve (or at least make easier to solve) issues like #129007 or #240919 .

Another solution would be to create the symlink ../bin/env usr/bin/env in the process of image creation but that would be a single-use-case only solution and would probably do more harm than good, especially considering that not all images necessarily contain coreutils (which provides env) - possibly put this behind a flag also? I don't think this is a good idea but it would work.

Metadata

 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.63, NixOS, 24.11 (Vicuna), 24.11.20241130.62c435d`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.10`
 - channels(user): `"home-manager"`
 - channels(root): `"nixos-hardware, stable-22.11"`
 - nixpkgs: `/nix/store/ljk3wvwqqzbmz0pys49ii18qmsi5aibc-source`

Notify maintainers

tbd.

---

Note for maintainers: Please tag this issue in your PR.

---

Add a 👍 reaction to issues you find important.

[jficz]

I'm aware I can work around this issue by doing /usr/bin -> /bin symlink in extraCommands but it is a workaround, not a solution imho. It also breaks when there is a fromImage based on, say, Debian, which has a symlink in the opposite direction.

[jficz]

OK, I see now this would be more complex than I initially thought. symlinkJoin links all paths' /bin and other dirs directly to $out which means we would need to override the function somehow for docker purposes to link to $out/usr instead. I think this is feasible as most of the directories should be linked to /usr anyway (like /share) if we want to follow FHS (which is I believe partially the point of doing the symlinks in images in the first place).

If this assumption is correct however, I don't understand what causes #129007, unless lndir itself nukes the existing /bin symlink.

[jficz]

Ok, this is more of a documentation issue. Took me some time to figure this out: include usrBinEnv in content:

... 
  content = [ dockerTools.usrBinEnv ... ];
...

(also dockerTools.caCertificates is rather useful, in case someone accidentally notices this :) )