Below we document steps for manually setting up SSH tunnel and syncing files/packages, as well as some commonly encountered errors and solutions. Please create an issue on Github or message us on Discord for more support.
If you are having trouble forming successful SSH connections with the Runhouse pip package, want more control over SSH tunneling, or want to use latest unreleased features from main, you may perform the following manual setup steps.
Install Runhouse on your local box with:
SSH into your server with a tunnel. You can use a different local or remote port here as well; steps 5 and 6 contain additional info for setting up your port.
Install Runhouse on the remote box using the same command as in step 1:
Start Ray on the remote box:
If you receive an error
“WARNING: The scripts ray, rllib, serve and tune are installed
in '/home/user/.local/bin' which is not on PATH.”, you may need to add the destination of the installed packages
to the path, e.g.
Start the Runhouse server on the remote box:
If you previously already had a version of Runhouse installed and started, you will need to run:
To exit out of the process and run it in the background instead, you can do
Ctrl+C and then
runhouse start --screen.
Please first make sure regular
runhouse start does not error out prior to testing it out in a screen.
If you previously specified a custom port in step 2, add
--port <my_port> to the runhouse start command.
Initialize the Runhouse cluster in Python as follows. Make sure to use
localhost as the cluster address rather than the true IP,
as you manually created the tunnel above.
To specify another port, you can pass in
If your remote box already has a port open to http traffic, you can directly use it instead of forming a new tunnel.
host=["<my ip>:<my port>"].
If you are running into an rsync or module not found error, you can work around by manually syncing over local files or installing any packages on your remote box.
To sync over the local folder (e.g.
tutorials) to remote. Not the trailing slash
/ at the end of the local directory
to prevent an additional level of folder nesting on the remote cluster. Now is also a good time to install any packages on the remote box,
To ensure Runhouse doesn’t try to resync over your code again, pass in an empty env object,
when sending functions or modules to the cluster.
If you’re encountering a Module not found error, you may need to pass a Runhouse package object pointing to the package in the remote filesystem into your Env so the server can find the module properly. You can do that like so:
install_method can be
local (to skip install),
(pip installs requirements.txt)
Q: I’m running into the following error:
ImportError: Failed to import grpc on Apple Silicon.
pip uninstall grpcio; conda install grpcio, then rerun your code. This is due to the Ray/grpcio compatibility on Apple Silicon during setup.
Q: I’m running into an OpenSSL error.
pip install pip --upgrade; pip install pyopenssl --upgrade.
Q: I’m running into an rsync error.
Rsync is finicky and we are working to support more reliable file syncing. In the meantime, please refer to the Manual Package syncing instructions above. It is also worth noting that rsync will throw an error if
.bashrcoutputs anything to the terminal. See Issue.
Q: I’m running into a few (usually 5) consecutive errors of the following form:
Run the tests in
test_module.pyand see if the issue reproduces:
If you see the same error(s), SSH into
Check if you can resume the latest screen:
If the result is
There is no screen to be resumed., it means the Runhouse server is not up. Start it using:
Observe the output for any errors e.g.
ImportError: ...and fix them