[Experimental] Include both privileged and non-privileged binaries in Docker image

[nathancoleman]

Currently, the NET_BIND_SERVICE capability is added to the dataplane (and Envoy) entrypoints since we cannot know at build time whether a non-root user at runtime will attempt to bind to a privileged port. The reason this is the default behavior was to avoid a breaking change in ingress gateways, which had previously supported binding to privileged ports; however, there are still many use cases where it's preferable to avoid this capability when it isn't needed.

This change introduces a second set of entrypoints to the container that do not have the NET_BIND_SERVICE capability. In this way, we can support both use cases:

• Ingress gateways, which have historically supported binding to privileged ports
• All other use cases (sidecar and API, mesh, terminating gateways) which do not need to support binding to privileged ports

The container will default to the non-privileged entrypoint, and ingress gateway deployments that do want to support binding to privileged ports will need to override the entrypoint to privileged-consul-dataplane and provide a path to privileged-envoy. This way, no deployment needs the NET_BIND_SERVICE capability unless they are overriding the entrypoint in order to support binding to privileged ports.

See hashicorp/consul-k8s#4394 for an example of how this works in consul-k8s.

Warning

We still need to determine the impact – if any – of a change like this for VM and Nomad deployments of consul-dataplane

[jm96441n]

this shouldn't affect nomad/VM deployments as those don't use dataplane

[zalimeni]

This seems like a great idea! Excited to see it happen. Hoping/expecting the constraints around consul-dataplane are acceptable.