Note
Go to the end to download the full example code.
Use a scheduling engine
from pathlib import Path
import time
from remote_run.run import (
ExecutionContext,
SlurmSchedulingEngine,
GuixRunner,
remote,
is_job_finished,
)
Define an execution context with a scheduling engine. For slurm you can pass slurm parameters directly here.
execution_context = ExecutionContext(
machine="shpc0003.ost.ch",
working_directory=Path("/cluster/raid/home/reza.housseini"),
runner=GuixRunner(channels=Path("channels.scm").read_text()),
scheduling_engine=SlurmSchedulingEngine(
job_name="test_remote_run",
mail_type="ALL",
mail_user="reza.housseini@ost.ch",
),
)
decorate your functions you want to run in the specified execution context
@remote(execution_context)
def add(a, b):
return a + b
this call will run on the remote machine specified in execution_context but due to the asynchronous nature of scheduling engines this will not return the result, instead you get the job id and a function to retrieve the result later.
job_id, result_func = add(1, 2)
now we wait for the remote execution to finish before retrieving the result
time.sleep(20)
we should check if the job id has finished before retrieving the result
if is_job_finished(execution_context, job_id):
assert result_func() == 3