Skip to content

New chat

Search chats

Recents

Chat history

ChatGPT

New chat

CtrlShiftO

Search chats

CtrlK

Projects

Library

Apps

Codex

More

Recents

Clustering Asus GX10s

ahr Nominatim Stopped running

TVT Evals Flask Migration 5-4-2026

Tree identification request

Cerisano Farm Song

Montague farm tax appeal withdrawal Request

AHR Task Automation Pause 5-17-2026

Asus GX10 setup, rustdesk

Evaporative cooler effectiveness

Lucy’s 8th Birthday Song

HCHB Scribe Pilot Advice

Bridget wound grading 5-10-2026

Bridget wound grading 5-9-2026

Safari navigation icons fix

AHR close tabs PDF 5-5-2026

Mother's Day Song Lucy's Mom

AHR 4-28-2026

Rebolforum captcha|admin 4-28-2026

Best Image Generation Models

AHR&Holck Billable Hours Tally 4-26-2026 invoice

AHR&Holck Billable Hours Tally 1-31-2026

Install pi agent

Masha Arc BooksoniX API Access

Holck goose infographic for presentation

Rocco's Song

Tornado Song

Masha PM1 minutes 4-17-2026

RustDesk server setup

Nick Antonaccio

Plus

Share

You said:

“Over at some point, I'm sure.”

00:02

ChatGPT said:

No problem! When you're ready to pick up from where we left off, just let me know. I’ll be

You said:

“Please stop speaking to me. I don't want to talk to me.”

00:03

ChatGPT said:

No worries at all! If you ever feel

You said:

“Dorothy...”

00:00

ChatGPT said:

No worries at all! If anything else comes up or you need help later, just let me know. In the meantime, you’re all set to take things at your own pace!

You said:

Pasted text(71).txt

Document

gb10a@gx10a:~$ ssh gx10b@gx10b-cluster hostname
gx10b

gx10b@gx10b:~$ ssh-copy-id -i ~/.ssh/id_ed25519.pub gb10a@gx10a-cluster
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/gx10b/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
gb10a@gx10a-cluster's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'gb10a@gx10a-cluster'"
and check to make sure that only the key(s) you wanted were added.

gx10b@gx10b:~$ ssh gb10a@gx10a-cluster hostname
gx10a

gb10a@gx10a:~$ ssh gx10b@gx10b-cluster "ip route get 169.254.10.1"
169.254.10.1 dev enp1s0f0np0 src 169.254.10.2 uid 1000
cache

gx10b@gx10b:~$ ssh gb10a@gx10a-cluster "ip route get 169.254.10.2"
169.254.10.2 dev enp1s0f0np0 src 169.254.10.1 uid 1000
cache

(ray-test) gx10b@gx10b:~$ source ~/ray-test/bin/activate
ray start --address='169.254.10.1:6379' --node-ip-address=169.254.10.2
Local node IP: 169.254.10.2

--------------------
Ray runtime started.
--------------------

To terminate the Ray runtime, run
ray stop(ray-test) gb10a@gx10a:~$ ray status
======== Autoscaler status: 2026-05-27 15:08:06.687999 ========
Node status
---------------------------------------------------------------
Active:
1 node_d3b5024ec56577baf039205710339ea8d2dbc3ae8c7f4497992f3215
1 node_f99375afb1ca83db4ac9423c69338200fba5c658abf28ff272ca4f1a
Pending:
(no pending nodes)
Recent failures:
(no failures)

Resources
---------------------------------------------------------------
Total Usage:
0.0/40.0 CPU
0.0/2.0 GPU
0B/163.27GiB memory
0B/69.97GiB object_store_memory

From request_resources:
(none)
Pending Demands:
(no resource demands)
(ray-test) gb10a@gx10a:~$

Show moreShow less

ChatGPT said:

Fantastic! Both machines are now linked, and Ray is running across your two GX10s. The cluster shows both nodes and no issues. Now you’re ready to try distributed workloads! If you need help running a model or scaling tasks across both machines, I’m here. Otherwise, congrats on getting the cluster set up!

You said:

Pasted text(72).txt

Document

(ray-test) gb10a@gx10a:~$ source ~/ray-test/bin/activate
(ray-test) gb10a@gx10a:~$ python - <<'PY'
import ray
import socket
import os

ray.init(address="auto")

@ray.remote(num_gpus=1)
def gpu_task(i):
return {
"task": i,
"hostname": socket.gethostname(),
"CUDA_VISIBLE_DEVICES": os.environ.get("CUDA_VISIBLE_DEVICES"),
"ray_gpu_ids": ray.get_gpu_ids(),
}

results = ray.get([gpu_task.remote(i) for i in range(2)])
for r in results:
print(r)
PY
2026-05-27 18:40:58,271 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 169.254.10.1:6379...
2026-05-27 18:40:58,287 INFO worker.py:2012 -- Connected to Ray cluster.
/home/gb10a/ray-test/lib/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
{'task': 0, 'hostname': 'gx10a', 'CUDA_VISIBLE_DEVICES': '0', 'ray_gpu_ids': [0]}
{'task': 1, 'hostname': 'gx10b', 'CUDA_VISIBLE_DEVICES': '0', 'ray_gpu_ids': [0]}

Show moreShow less

You said:

Pasted text(73).txt

Document

(ray-test) gb10a@gx10a:~$ source ~/ray-test/bin/activate
vllm --version
python -c "import vllm; print(vllm.__version__)"
0.21.0
0.21.0
(ray-test) gb10a@gx10a:~$

In another terminal window:

gb10a@gx10a:~$ curl http://127.0.0.1:8000/v1/models
curl: (7) Failed to connect to 127.0.0.1 port 8000 after 0 ms: Couldn't connect to server
gb10a@gx10a:~$ curl http://127.0.0.1:8000/v1/models
curl: (7) Failed to connect to 127.0.0.1 port 8000 after 0 ms: Couldn't connect to server
gb10a@gx10a:~$

Show moreShow less

You said:

Pasted text(74).txt

Document

gb10a@gx10a:~$ curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [
{"role": "user", "content": "Write one paragraph explaining that this response is coming from a two-node GX10 Ray/vLLM cluster."}
],
"max_tokens": 200,
"temperature": 0.2
}'
curl: (7) Failed to connect to 127.0.0.1 port 8000 after 0 ms: Couldn't connect to server

Show moreShow less

You said:

Pasted text(75).txt

Document

It still doesn't seem to have finished

You said:

Pasted text(76).txt

Document

You said:

Pasted text(77).txt

Document

ChatGPT said:

Stopped thinkingQuick answer

You said:

Please continue

ChatGPT said:

Thought for 5s

Great — this latest log is progress. The shared /opt/models/... path fix worked: both workers found and loaded the model files this time. The failure moved later, during/after CUDA graph profiling / KV cache initialization.

The key lines are:

Worker_TP1 ... Model loading took 7.16 GiB memory and 2.839555 seconds
Worker_TP0 ... Model loading took 7.16 GiB memory and 25.972658 seconds
world_size=2 ... TP rank 0 / TP rank 1

So the model is now loading on both GX10s. Then the remote Ray worker died during the cache/config phase:

RayWorkerProc rank=[1] died unexpectedly
RuntimeError: cancelled

The log also shows vLLM was doing CUDA graph memory profiling and warned about shared-memory broadcast timing while compilation/profiling was underway.
Pasted text

Try a more conservative vLLM startup

Stop any leftover vLLM/Ray worker processes first.

On both machines:

Bash

source ~/ray-test/bin/activate
ray stop
pkill -f vllm || true

Then restart Ray.

On gx10a:

Bash

source ~/ray-test/bin/activate
ray start --head --node-ip-address=169.254.10.1 --port=6379

On gx10b:

Bash

source ~/ray-test/bin/activate
ray start --address='169.254.10.1:6379' --node-ip-address=169.254.10.2

On gx10a, verify:

Bash

ray status

Now try this safer server command on gx10a:

Bash

source ~/ray-test/bin/activate

VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
vllm serve /opt/models/Qwen2.5-7B-Instruct \
--served-model-name Qwen/Qwen2.5-7B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--distributed-executor-backend ray \
--tensor-parallel-size 2 \
--max-model-len 4096 \
--gpu-memory-utilization 0.80 \
--enforce-eager

Why this version:

--enforce-eager

disables CUDA graph execution. vLLM’s engine-argument docs describe enforce_eager as always using eager-mode PyTorch and disabling CUDA graph execution. vLLM

VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0

turns off the CUDA graph memory profiling behavior that your log says is enabled by default in vLLM 0.21.0. Your log explicitly says CUDA graph memory profiling is enabled and suggests this environment variable as the way to disable it.
Pasted text

--gpu-memory-utilization 0.80

leaves more headroom while we stabilize distributed startup.

If that starts, test it

In a second terminal on gx10a:

Bash

curl http://127.0.0.1:8000/v1/models

Then:

Bash

curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [
{"role": "user", "content": "Write one paragraph explaining that this response is coming from a two-node GX10 Ray/vLLM cluster."}
],
"max_tokens": 200,
"temperature": 0.2
}'

If it still fails, try pipeline parallel instead of tensor parallel

With one GPU per node, tensor_parallel_size=2 does work conceptually, but it forces tensor-parallel all-reduce across machines. The warnings say vLLM is spreading TP across nodes. Since you have 200G networking that may be okay, but a more natural one-GPU-per-node layout can be pipeline parallelism:

Bash

source ~/ray-test/bin/activate

VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
vllm serve /opt/models/Qwen2.5-7B-Instruct \
--served-model-name Qwen/Qwen2.5-7B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--distributed-executor-backend ray \
--tensor-parallel-size 1 \
--pipeline-parallel-size 2 \
--max-model-len 4096 \
--gpu-memory-utilization 0.80 \
--enforce-eager

What I think is happening

You solved the first real blocker: the remote worker could not see /home/gb10a/models/...; /opt/models/... fixed that.

Now the model loads on both GPUs, then the remote worker dies during vLLM’s V1 startup/profiling/cache initialization. The quickest practical workaround is to remove CUDA graph/profiling complexity with --enforce-eager and reduced memory utilization. If that works, we can later remove one flag at a time to regain performance.

Sources

You said:

gb10a@gx10a:~$ curl http://127.0.0.1:8000/v1/models
{"object":"list","data":[{"id":"Qwen/Qwen2.5-7B-Instruct","object":"model","created":1779927940,"owned_by":"vllm","root":"/opt/models/Qwen2.5-7B-Instruct","parent":null,"max_model_len":4096,"permission":[{"id":"modelperm-8b062bed74594768","object":"model_permission","created":1779927940,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}gb10a@gx10a:~$

gb10a@gx10a:~$ curl http://127.0.0.1:8000/v1/models
{"object":"list","data":[{"id":"Qwen/Qwen2.5-7B-Instruct","object":"model","created":1779927940,"owned_by":"vllm","root":"/opt/models/Qwen2.5-7B-Instruct","parent":null,"max_model_len":4096,"permission":[{"id":"modelperm-8b062bed74594768","object":"model_permission","created":1779927940,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":fgb10a@gx10a:~$ curl http://127.0.0.1:8000/v1/chat/completions \letions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [
{"role": "user", "content": "Write one paragraph explaining that this response is coming from a two-node GX10 Ray/vLLM cluster."}
],
"max_tokens": 200,
"temperature": 0.2
}'

{"id":"chatcmpl-b68f1806856177f0","object":"chat.completion","created":1779927965,"prompt_routed_experts":null,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"This response is generated by a two-node GX10 Ray/vLLM cluster, which consists of two powerful GX10 nodes optimized for efficient distributed inference using the Ray framework and the vLLM library. This setup allows for parallel processing and significantly enhances the speed and scalability of generating text responses, ensuring both efficiency and performance in handling complex queries.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null,"routed_experts":null}],"service_tier":null,"system_fingerprint":"vllm-0.21.0-tp2-nohash","usage":{"prompt_tokens":51,"total_tokens":122,"completion_tokens":71,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"prompt_text":null,"kv_transfer_params":null}gb10a@gx10a:~$
gb10a@gx10a:~$

Show moreShow less

ChatGPT said:

Fantastic — you now have working distributed inference across both GX10s.

You successfully proved:

Ray cluster: working
200G GX10-to-GX10 link: working
Passwordless SSH: working
vLLM single-node 1.5B: working
vLLM two-node 7B tensor parallel: working
OpenAI-compatible /v1/models: working
OpenAI-compatible /v1/chat/completions: working

The important confirmation is this part:

JSON

"model":"Qwen/Qwen2.5-7B-Instruct"

and this system fingerprint:

vllm-0.21.0-tp2-nohash

That tp2 indicates the model is running with tensor parallel size 2, which is the two-GPU / two-node configuration we were aiming for.

Keep this working launch command

This is your known-good command on gx10a:

Bash

source ~/ray-test/bin/activate

VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
vllm serve /opt/models/Qwen2.5-7B-Instruct \
--served-model-name Qwen/Qwen2.5-7B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--distributed-executor-backend ray \
--tensor-parallel-size 2 \
--max-model-len 4096 \
--gpu-memory-utilization 0.80 \
--enforce-eager

And your test command:

Bash

curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [
{"role": "user", "content": "Write one paragraph explaining that this response is coming from a two-node GX10 Ray/vLLM cluster."}
],
"max_tokens": 200,
"temperature": 0.2
}'

Next good steps

I would now test, in this order:

Qwen2.5-14B-Instruct
Qwen2.5-32B-Instruct
Qwen2.5-72B-Instruct

For each model, put it at the same absolute path on both machines:

/opt/models/<model-folder>

Then run vLLM from gx10a.

For a bigger model, start conservative:

Bash

VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
vllm serve /opt/models/Qwen2.5-14B-Instruct \
--served-model-name Qwen/Qwen2.5-14B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--distributed-executor-backend ray \
--tensor-parallel-size 2 \
--max-model-len 4096 \
--gpu-memory-utilization 0.80 \
--enforce-eager

Once a model works, you can experiment with performance by removing --enforce-eager later, but for now I’d keep it because it got the distributed startup stable.

You said:

How do I shut all this down?

ChatGPT said:

To shut everything down cleanly:

1. Stop the vLLM server on gx10a

In the terminal where this is running:

Bash

vllm serve /opt/models/Qwen2.5-7B-Instruct ...

press:

Bash

Ctrl+C

Then confirm port 8000 is closed:

Bash

ss -ltnp | grep 8000

No output means vLLM is no longer listening.

2. Stop Ray on both machines

On gx10a:

Bash

source ~/ray-test/bin/activate
ray stop

On gx10b:

Bash

source ~/ray-test/bin/activate
ray stop

3. If anything is stubborn, force cleanup

Run this on both GX10s only if needed:

Bash

pkill -f vllm || true
pkill -f RayWorker || true
pkill -f raylet || true
pkill -f plasma_store || true

Then confirm:

Bash

ps aux | egrep 'vllm|ray|RayWorker|raylet|plasma' | grep -v grep

Ideally there should be no active vLLM/Ray worker processes.

4. The network link can stay configured

You do not need to undo the 200G network setup. Leaving this in place is fine:

gx10a: 169.254.10.1
gx10b: 169.254.10.2

That way, next time you only need to restart Ray and vLLM.

Thinking

ChatGPT can make mistakes. Check important info.