Below, we describe how to access log outputs and show a sample debugging flow. We are still currently working on making logs more accessible and improving the debugging flow.
There are two main ways to access logs.
Logs are automatically output onto the cluster, in the file
You can ssh into the cluster with
ssh cluster-name to access these logs.
Alternatively, to see logs on your local machine while running a remote function, you can add the
stream_logs=True argument to your function call.
For general debugging that doesn’t occur within remote function calls, you can add
import pdb; pdb.set_trace() whereever you want to set your debugging session.
If the code is being run locally at the point of the debugger, you’ll be able to access the session from your
local machine. If the code is being run remotely on a cluster, you will need to ssh into the cluster with
ssh cluster-name, and then run
screen -r inside the cluster. From there, you will see the RPC logs
being printed out, and can debug normally inside the
When debugging inside
screen, please use
Ctrl A+D to exit out of the screen. Do NOT use
which will terminate the RPC server.
If you accidentally terminate the RPC server, you can run
cluster.restart_server() to restart the
For debugging remote functions, which are launched using
ray, we can utilize Ray’s debugger. Add a
breakpoint() call inside the function where you want to start the debugging session, then ssh into the
ssh cluster-name, and call
ray debug to view select the breakpoint to enter. You can run
pdb commands within the debugging session, and can refer to
Ray Debugger for more information.