Cachepot distributed poc - JWT Mismatched address

[borjamunozf]

Hello everyone.

I'm preparing a PoC for my current job using Cachepot distributed build mode but I have encountered an issue when setting up the Scheduler & Worker on different machines.

I have some doubts with the X-Real-IP use and the suggestion to put a webworker in front of the scheduler, not sure what this meant, but let me explain better:

CURRENT SCENARIO

Scheduler and Worker

• W11 + WSL2 (Ubuntu 20.04)

Scheduler config

public_addr = "https://cachepot-scheduler:10600"

[client_auth]
type = "token"
token = "quefacemos"

[worker_auth]
type = "jwt_hs256"
secret_key = "blablablablaba"

Worker config

cache_dir = "/tmp/toolchains"

public_addr = "172.26.175.180:10501"
scheduler_url = "http://cachepot-scheduler:10600"

[builder]
type = "overlay"
build_dir = "/tmp/build"
bwrap_path = "/usr/bin/bwrap"

[scheduler_auth]
type = "jwt_token"
token ="blablalblab"

Port proxy on machines to WSL2. We have connectivity between two machines.
Cachepot succedeed to connect to the Windows IP port forwarded to the WSL2 IP -> 10501/10600

• On Scheduler machine

netsh interface portproxy add v4tov4 listenport=10600 listenaddress=0.0.0.0 connectport=10600 connectaddress=172.26.175.180

• On Worker machine

netsh interface portproxy add v4tov4 listenport=10501 listenaddress=0.0.0.0 connectport=10501 connectaddress=172.18.212.219

Generate the token for Worker:

cachepot-dist auth generate-jwt-hs256-worker-token --secret-key blablablabla --worker 172.26.175.180:10600

Result on Worker:

2022-10-24T15:34:42.372 DEBUG [PID 1214] dist worker response '401 Unauthorized' for http://cachepot-scheduler:10600/api/v1/scheduler/heartbeat_server
2022-10-24T15:34:42.372 ERROR [PID 1214] dist worker Failed to send heartbeat to server: didn't get a successful HTTP status, got `Error 401: {"description":"invalid_bearer_token_mismatched_address","cause":null}`

Explanation:

• The Scheduler's log shows that the Server Address & Request IP are not the same.
  The Request IP that arrives to the Scheduler is the gateway of the WSL2 network, not the source IP of my address.
  The last "hop", so of course the Worker address and the request ip arrived to the Scheduler does not match with the JWT.

Doubts

• Is this the expected behaviour? How could I solve it?
• What the doc means about setting up in front of the scheduler a Worker?
  HTTPS worker in front of the scheduler

Thanks. Hope to make myself clear.