Env

An Env is a Runhouse primitive that represents a compute environment.

Env Factory Methods

runhouse.env(reqs: List[str | Package] = [], conda_env: str | Dict = None, name: str | None = None, setup_cmds: List[str] = None, env_vars: Dict | str = {}, working_dir: str | Path | None = None, secrets: str | Secret | None = [], compute: Dict | None = {}, load_from_den: bool = True, dryrun: bool = False)[source]

Builds an instance of Env.

Parameters:
  • reqs (List[str]) – List of package names to install in this environment.

  • conda_env (Union[str, Dict], optional) – Dict representing conda env, Path to a conda env yaml file, or name of a local conda environment.

  • name (Optional[str], optional) – Name of the environment resource.

  • setup_cmds (Optional[List[str]]) – List of CLI commands to run for setup when the environment is being set up on a cluster.

  • env_vars (Dict or str) – Dictionary of environment variables, or relative path to .env file containing environment variables. (Default: {})

  • working_dir (str or Path) – Working directory of the environment, to be loaded onto the system. (Default: None)

  • compute (Dict) – Logical compute resources to be used by this environment, passed through to the cluster scheduler (generally Ray). Only use this if you know what you’re doing. Example: {"cpus": 1, "gpus": 1}. (Default: {}) More info: https://docs.ray.io/en/latest/ray-core/scheduling/resources.html

  • load_from_den (bool) – Whether to try loading the Env resource from Den. (Default: True)

  • dryrun (bool, optional) – Whether to run in dryrun mode. (Default: False)

Returns:

The resulting Env object.

Return type:

Env

Example

>>> # regular python env >>> env = rh.env(reqs=["torch", "pip"]) >>> env = rh.env(reqs=["reqs:./"], name="myenv") >>> >>> # conda env, see also rh.conda_env >>> conda_env_dict = >>> {"name": "new-conda-env", "channels": ["defaults"], "dependencies": "pip", {"pip": "diffusers"}) >>> conda_env = rh.env(conda_env=conda_env_dict) # from a dict >>> conda_env = rh.env(conda_env="conda_env.yaml") # from a yaml file >>> conda_env = rh.env(conda_env="local-conda-env-name") # from a existing local conda env >>> conda_env = rh.env(conda_env="conda_env.yaml", reqs=["pip:/accelerate"]) # with additional reqs
runhouse.conda_env(reqs: List[str | Package] = [], conda_env: str | Dict = None, name: str | None = None, setup_cmds: List[str] = None, env_vars: Dict | None = {}, working_dir: str | Path | None = None, secrets: List[str | Secret] = [], compute: Dict | None = {}, dryrun: bool = False)[source]

Builds an instance of CondaEnv.

Parameters:
  • reqs (List[str]) – List of package names to install in this environment.

  • conda_env (Union[str, Dict], optional) – Dict representing conda env, Path to a conda env yaml file, or name of a local conda environment.

  • name (Optional[str], optional) – Name of the environment resource.

  • setup_cmds (Optional[List[str]]) – List of CLI commands to run for setup when the environment is being set up on a cluster.

  • env_vars (Dict or str) – Dictionary of environment variables, or relative path to .env file containing environment variables. (Default: {})

  • working_dir (str or Path) – Working directory of the environment, to be loaded onto the system. (Default: None)

  • compute (Dict) – Logical compute resources to be used by this environment, passed through to the cluster scheduler (generally Ray). Only use this if you know what you’re doing. Example: {"cpus": 1, "gpus": 1}. (Default: {}) More info: https://docs.ray.io/en/latest/ray-core/scheduling/resources.html

  • dryrun (bool, optional) – Whether to run in dryrun mode. (Default: False)

Returns:

The resulting CondaEnv object.

Return type:

CondaEnv

Example

>>> rh.conda_env(reqs=["torch"]) >>> rh.conda_env(reqs=["torch"], name="resource_name") >>> rh.conda_env(reqs=["torch"], name="resource_name", conda_env={"name": "conda_env"})

Env Class

class runhouse.Env(name: str | None = None, reqs: List[str | Package] = [], setup_cmds: List[str] = None, env_vars: Dict | str = {}, working_dir: str | Path | None = None, secrets: str | Secret | None = [], compute: Dict | None = {}, dryrun: bool = True, **kwargs)[source]
__init__(name: str | None = None, reqs: List[str | Package] = [], setup_cmds: List[str] = None, env_vars: Dict | str = {}, working_dir: str | Path | None = None, secrets: str | Secret | None = [], compute: Dict | None = {}, dryrun: bool = True, **kwargs)[source]

Runhouse Env object.

Note

To create an Env, please use the factory method env().

add_env_var(key: str, value: str)[source]

Add an env var to the environment. Environment must be re-installed to propagate new environment variables if it already lives on a cluster.

static from_config(config: dict, dryrun: bool = False, _resolve_children: bool = True)[source]

Load or construct resource from config.

Parameters:
  • config (Dict) – Resource config.

  • dryrun (bool, optional) – Whether to construct resource or load as dryrun (Default: False)

install(force: bool = False, cluster: Cluster | None = None, node: str = 'all')[source]

Locally install packages and run setup commands.

Parameters:
  • force (bool, optional) – Whether to setup the installation again if the env already exists on the cluster. (Default: False)

  • cluster (Clsuter, optional) – Cluster to install the env on. If not provided, env is installed on the current cluster. (Default: None)

  • node (str, optional) – Node to install the env on. (Default: "all")

to(system: str | Cluster, node_idx: int | None = None, path: str | None = None, force_install: bool = False)[source]

Send environment to the system, and set it up if on a cluster.

Parameters:
  • system (str or Cluster) – Cluster or file system to send the env to.

  • node_idx (int, optional) – Node index of the cluster to send the env to. If not specified, uses the head node. (Default: None)

  • path (str, optional) – Path on the cluster to sync the env’s working dir to. Uses a default path if not specified. (Default: None)

  • force_install (bool, optional) – Whether to setup the installation again if the env already exists on the cluster. (Default: False)

Example

>>> env = rh.env(reqs=["numpy", "pip"]) >>> cluster_env = env.to(my_cluster) >>> s3_env = env.to("s3", path="s3_bucket/my_env")

Conda Env Class

class runhouse.CondaEnv(conda_yaml: str | Dict, name: str | None = None, reqs: List[str | Package] = [], setup_cmds: List[str] = None, env_vars: Dict | None = {}, working_dir: str | Path | None = './', secrets: List[str | Secret] = [], dryrun: bool = True, **kwargs)[source]
__init__(conda_yaml: str | Dict, name: str | None = None, reqs: List[str | Package] = [], setup_cmds: List[str] = None, env_vars: Dict | None = {}, working_dir: str | Path | None = './', secrets: List[str | Secret] = [], dryrun: bool = True, **kwargs)[source]

Runhouse CondaEnv object.

Note

To create a CondaEnv, please use the factory methods env() or conda_env().

install(force: bool = False, cluster: Cluster = None, node: str | None = None)[source]

Locally install packages and run setup commands.

Parameters:
  • force (bool, optional) – Whether to force re-install env if it has already been installed. (default: False)

  • cluster (bool, optional) – If None, installs env locally. Otherwise installs remotely on the cluster using SSH. (default: None)