Improve support for relative path inputs

[edolstra]

Motivation

We want it to be possible to refer to flake inputs that reside in the same repo via relative paths (e.g. inputs.foo.url = "path:./subdir"). Previously this didn't work very well because those inputs would be separately copied to the store, which is inefficient and makes references to parent directories tricky or impossible. Furthermore, they had their own NAR hash in the lock file, which is superfluous since the parent is already locked. In fact, if relative path inputs worked at all, it was by accident.

Now relative path inputs are accessed via the accessor of the calling flake. This avoids the unnecessary copy and makes it possible for relative path inputs to depend on flakes in a parent directory (so long as they're in the same tree).

Lock file nodes for relative path inputs now have a new parent field:

{
  "locked": {
    "path": "./subdir",
    "type": "path"
  },
  "original": {
    "path": "./subdir",
    "type": "path"
  },
  "parent": [
    "foo",
    "bar"
  ]
}

which denotes that ./subdir is to be interpreted relative to the directory of the bar input of the foo input of the root flake.

Extracted from the lazy-trees branch.

• Depends on Use SourcePath for reading flake.{nix,lock} #10088.
• Closes Allow flakes to refer to other flakes by relative path #3978

Context

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[dpulls]

🎉 All dependencies have been resolved !

[thufschmitt]

Discussed during the Nix maintainers meeting on 2024-03-04.

• Idea approved
• Document that directly accessing .. in subflakes is likely to break in the future
• Use path literals as the url for subflakes

• Great overall
• Right now, the PR allows import ../. in a subflake. Do we want that?
  • Directly accessing .. is convenient, but breaks the encapsulation property of flakes
    • Relates to nix flakes: flakes in subdirectories can access files in parent directories (breaking the implied seal of flake.nix) #4414
  • We can however access other parts of the tree with relative inputs (that can go up until the root)
    • inputs.something_else = ../foo
  • From an implementation perspective, hard to forbid without lazy trees
  • Decision:
    • Don't formally forbid direct access to .. since it's hard without lazy trees
    • Once lazy trees lands, make that conditional on a flag
    • Potentially make the default restrictive. Would be a breaking change, but
      • can be mitigated by only making that change effective after a flake update
      • small breakage as getting the old behavior back would only be a one-line change
• Syntax for relative flakes?
  • Currently reuses url = "path:./foo" that already kinda works on master
  • A more natural syntax would be url = ./foo
    • This is actually what most people seem to try first when they want subflakes

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-03-04-nix-team-meeting-minute-130/40830/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-11-27-nix-team-meeting-minutes-198/56691/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-12-18-nix-team-meeting-minutes-204/57602/1

[MagicRB]

Does this also mean that subflakes will auto-update? In other words, nix flake update subflake ought to be unnecessary with this right?

[roberth]

Reviewed together in meeting

• Release note: Generalizing the flake format can break tools that consume flake.nix files. For instance toJSON (import ./flake.nix).inputs will not work as expected anymore for flakes that use path values.

InputPath -> InputChain?

[roberth]

This should be in the expr / flake layer.
Open issue?

[roberth]

Off-topic: perhaps flake-ness should be "type-checked"

(existing code, moved)

[roberth]

Suggested change

	std::optional<InputPath> parentPath;
	std::optional<InputPath> parentInputChain;

as mentioned in the review header comment

[edolstra]

For consistency, let's postpone this to a future PR that resolves #12098.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-12-23-nix-team-meeting-minutes-205/57783/1

[tomberek]

(using: nix-2.26.0pre20241223_9223d64) An update to the input of a subflake does not get automatically picked up by the root when running a build.

1. Root and child are sync'd.
2. Parent uses path:./subdir to point to child and re-export its outputs.
3. Child has an input to a third, remote-flake.
4. Remote flake has a new commit pushed.
5. Child is updated with nix flake update in the subdir.
6. Run a build in the parent.

This will run the root with the "old" child and the mismatch/update is not detected. At this point the flake:root/subdir/src and flake:subdir/src are locked to different revisions.

7. Run nix flake update in the root.
8. Lock in the root now points to the same sources as the child.

Not sure if this is intended,,, but was a bit surprising. I'd guess that the root flake would detect the mismatch between its lock and that of the child - similar to how a significant change to an input is detected and locks updated when running things like nix build.

[roberth]

Those are really good observations. "Subflake" suggests some sort of directional relationship other than just an input. We should avoid that term in the release notes, and/or explain what's not implemented. The alternative term, relative path input would be ok to use for this feature. Maybe subflakes should be the name for the pattern or feature where all input declarations are moved to the root of the flake? Or is it merely a subdirectory relationship? Multi flake repositories might also be a useful name for the use case this pr enables.

[tomberek]

I still need to more thoroughly test it, but I'm afraid the unsync'd lockfiles is a problem. When evaluating the root flake with an outdated/updated/modified lockfile, it will still use the flake.nix from the current checkout via path:./subdir, but not the subdir/flake.lock, instead using the inputs from ./flake.lock. It would also be weird to have an old subdir/flake.nix be used - user keeps editing and doesn't see the changes.

Not sure if this is intended. Idea brainstorming:

• nix flake update --recursive
• detect de-sync and provide warning
• do not store the subflake's lock information, just a pointer to the other flake.lock
• document "multi-flake" vs "subflake".

note: I do love the new capability to avoid the "infinite updating lockfile"! Excited to keep playing with it.

[roberth]

At this point the flake:root/subdir/src and flake:subdir/src are locked to different revisions.

I think I'm reading pseudo-syntax that's a little ambiguous, but I would think that the locking of the local sources is fixed by this PR.

The locking of transitive dependencies between flakes in the same repo I think should be solved with #7730; otherwise we'll need a lot of new logic to deal with its symptoms.

[edolstra]

@tomberek

An update to the input of a subflake does not get automatically picked up by the root when running a build.

That's right, and probably not ideal, but it's consistent with how lock files currently work in general: lock files are independent, and the only time that we use the lock file of an input is if the referring flake doesn't have a lock for that input yet. So a change to the lock file of an input doesn't propagate to the referring flake.

This should indeed be fixed by having sparse lock files (#7730).

@roberth

"Subflake" suggests some sort of directional relationship other than just an input. We should avoid that term in the release notes, and/or explain what's not implemented.

Yeah we don't use the term "subflake" anywhere (except previously in the PR description, fixed).