var_resources

Simulation functions that use the MPIExecutor with dynamic resource assignment. six_hump_camel and helloworld python scripts are used as example applications, but these could be any MPI application.

Each simulation function uses the resources assigned to this worker to set CPU count and, in some functions, specify GPU usage.

GPUs are not used for the six_hump_camel function, but these tests check the assignment is correct. For an example that runs an actual GPU application, see the forces_gpu tutorial under libensemble/tests/scaling_tests/forces/forces_gpu.

See CUDA_variable_resources for an example where the sim function interrogates available resources and sets explicitly.

var_resources.gpu_variable_resources(H, persis_info, sim_specs, libE_info)

Launches an app and automatically assigns GPU resources.

The six_hump_camel app does not run on the GPU, but this test demonstrates how to automatically assign the GPUs given to this worker via the MPIExecutor.

The method used to assign GPUs will be determined by the MPI runner or user-provided configuration (e.g., by setting the platform or platform_specs options or the LIBE_PLATFORM environment variable).

var_resources.gpu_variable_resources_from_gen(H, persis_info, sim_specs, libE_info)

Input Fields: ['x']

Output Datatypes: [('f', <class 'float'>)]

Launches an app and assigns CPU and GPU resources as defined by the gen.

Otherwise similar to gpu_variable_resources.

var_resources.gpu_variable_resources_subenv(H, persis_info, sim_specs, libE_info)

Launches a chain of apps via bash scripts in different sub-processes.

Different MPI runners are specified for each submit. To run without dry_run these MPI runners need to be present. Dry_run is used by default.

Otherwise, this test is similar to gpu_variable_resources.

var_resources.multi_points_with_variable_resources(H, _, sim_specs, libE_info)

Evaluates either helloworld or six hump camel for a collection of points given in H["x"] via the MPI executor, supporting variable sized simulations/resources, as determined by the generator. The term rset refers to a resource set (the minimal set of resources that can be assigned to each worker). It can be anything from a partition of a node to multiple nodes.

Note that this is also an example that is capable of handling multiple points (sim ids) in each call.

See also

test_uniform_sampling_with_variable_resources.py

var_resources.CUDA_variable_resources(H, _, sim_specs, libE_info)

Launches an app setting GPU resources

The standard test apps do not run on GPU, but demonstrates accessing resource information to set CUDA_VISIBLE_DEVICES, and typical run configuration.

For an equivalent function that auto-assigns GPUs using platform detection, see GPU_variable_resources.

var_resources.py
  1"""
  2Simulation functions that use the MPIExecutor with dynamic resource assignment.
  3``six_hump_camel`` and ``helloworld`` python scripts are used as example
  4applications, but these could be any MPI application.
  5
  6Each simulation function uses the resources assigned to this worker to set CPU
  7count and, in some functions, specify GPU usage.
  8
  9GPUs are not used for the six_hump_camel function, but these tests check the
 10assignment is correct. For an example that runs an actual GPU application, see
 11the forces_gpu tutorial under libensemble/tests/scaling_tests/forces/forces_gpu.
 12
 13See CUDA_variable_resources for an example where the sim function
 14interrogates available resources and sets explicitly.
 15
 16"""
 17
 18__all__ = [
 19    "gpu_variable_resources",
 20    "gpu_variable_resources_from_gen",
 21    "gpu_variable_resources_subenv",
 22    "multi_points_with_variable_resources",
 23    "CUDA_variable_resources",
 24]
 25
 26import os
 27
 28import numpy as np
 29
 30from libensemble.message_numbers import TASK_FAILED, UNSET_TAG, WORKER_DONE
 31from libensemble.resources.resources import Resources
 32from libensemble.sim_funcs.six_hump_camel import six_hump_camel_func
 33from libensemble.specs import input_fields, output_data
 34from libensemble.tools.test_support import check_gpu_setting, check_mpi_runner
 35
 36
 37def gpu_variable_resources(H, persis_info, sim_specs, libE_info):
 38    """Launches an app and automatically assigns GPU resources.
 39
 40    The six_hump_camel app does not run on the GPU, but this test demonstrates
 41    how to automatically assign the GPUs given to this worker via the MPIExecutor.
 42
 43    The method used to assign GPUs will be determined by the MPI runner or
 44    user-provided configuration (e.g., by setting the ``platform`` or
 45    ``platform_specs`` options or the LIBE_PLATFORM environment variable).
 46
 47    """
 48    x = H["x"][0]
 49    H_o = np.zeros(1, dtype=sim_specs["out"])
 50    dry_run = sim_specs["user"].get("dry_run", False)  # logs run lines instead of running
 51    inpt = " ".join(map(str, x))  # Application input
 52
 53    exctr = libE_info["executor"]
 54
 55    # Launch application via system MPI runner, using assigned resources.
 56    task = exctr.submit(
 57        app_name="six_hump_camel",
 58        app_args=inpt,
 59        auto_assign_gpus=True,
 60        match_procs_to_gpus=True,
 61        stdout="out.txt",
 62        stderr="err.txt",
 63        dry_run=dry_run,
 64    )
 65
 66    if not dry_run:
 67        task.wait()  # Wait for run to complete
 68
 69        # Access app output
 70        with open("out.txt") as f:
 71            H_o["f"] = float(f.readline().strip())  # Read just first line
 72
 73    # Asserts GPU set correctly (for known MPI runners)
 74    check_gpu_setting(task, print_setting=True)
 75
 76    calc_status = WORKER_DONE if task.state == "FINISHED" else "FAILED"
 77    return H_o, persis_info, calc_status
 78
 79
 80@input_fields(["x"])
 81@output_data([("f", float)])
 82def gpu_variable_resources_from_gen(H, persis_info, sim_specs, libE_info):
 83    """
 84    Launches an app and assigns CPU and GPU resources as defined by the gen.
 85
 86    Otherwise similar to gpu_variable_resources.
 87    """
 88    x = H["x"][0]
 89    H_o = np.zeros(1, dtype=sim_specs["out"])
 90    dry_run = sim_specs["user"].get("dry_run", False)  # logs run lines instead of running
 91    inpt = " ".join(map(str, x))  # Application input
 92
 93    exctr = libE_info["executor"]  # Get Executor
 94
 95    # Launch application via system MPI runner, using assigned resources.
 96    task = exctr.submit(
 97        app_name="six_hump_camel",
 98        app_args=inpt,
 99        stdout="out.txt",
100        stderr="err.txt",
101        dry_run=dry_run,
102    )
103
104    if not dry_run:
105        task.wait()  # Wait for run to complete
106
107        # Access app output
108        with open("out.txt") as f:
109            H_o["f"] = float(f.readline().strip())  # Read just first line
110
111    # Asserts GPU set correctly (for known MPI runners)
112    check_gpu_setting(task, print_setting=True)
113
114    calc_status = WORKER_DONE if task.state == "FINISHED" else "FAILED"
115    return H_o, persis_info, calc_status
116
117
118def _launch_with_env_and_mpi(exctr, inpt, dry_run, env_script_path, mpi_runner):
119    """Used to launch each application in a chain"""
120
121    task = exctr.submit(
122        app_name="six_hump_camel",
123        app_args=inpt,
124        auto_assign_gpus=True,
125        match_procs_to_gpus=True,
126        dry_run=dry_run,
127        env_script=env_script_path,
128        mpi_runner_type=mpi_runner,
129    )
130
131    if isinstance(mpi_runner, dict):
132        mpi_runner = mpi_runner["runner_name"]
133
134    check_mpi_runner(task, mpi_runner, print_setting=True)
135    check_gpu_setting(task, print_setting=True)
136
137
138def gpu_variable_resources_subenv(H, persis_info, sim_specs, libE_info):
139    """Launches a chain of apps via bash scripts in different sub-processes.
140
141    Different MPI runners are specified for each submit. To run without dry_run
142    these MPI runners need to be present. Dry_run is used by default.
143
144    Otherwise, this test is similar to ``gpu_variable_resources``.
145
146    """
147    x = H["x"][0]
148    H_o = np.zeros(1, dtype=sim_specs["out"])
149    dry_run = sim_specs["user"].get("dry_run", False)  # logs run lines instead of running
150    env_script_path = sim_specs["user"]["env_script"]  # Script to run in subprocess
151    inpt = " ".join(map(str, x))  # Application input
152
153    exctr = libE_info["executor"]  # Get Executor
154
155    # Launch application via given MPI runner, using assigned resources.
156    _launch_with_env_and_mpi(exctr, inpt, dry_run, env_script_path, "openmpi")
157    _launch_with_env_and_mpi(exctr, inpt, dry_run, env_script_path, "srun")
158
159    mpi_runner_type = {"mpi_runner": "openmpi", "runner_name": "special_mpi"}
160    _launch_with_env_and_mpi(exctr, inpt, dry_run, env_script_path, mpi_runner_type)
161
162    # Now run in current environment.
163    task = exctr.submit(
164        app_name="six_hump_camel",
165        app_args=inpt,
166        auto_assign_gpus=True,
167        match_procs_to_gpus=True,
168        dry_run=dry_run,
169    )
170    check_mpi_runner(task, "mpich", print_setting=True)
171    check_gpu_setting(task, print_setting=True)
172
173    if not dry_run:
174        task.wait()  # Wait for run to complete
175
176        # Access app output
177        with open("out.txt") as f:
178            H_o["f"] = float(f.readline().strip())  # Read just first line
179
180    # Asserts GPU set correctly (for known MPI runners)
181    check_gpu_setting(task, print_setting=True)
182
183    calc_status = WORKER_DONE if task.state == "FINISHED" else "FAILED"
184    return H_o, persis_info, calc_status
185
186
187def multi_points_with_variable_resources(H, _, sim_specs, libE_info):
188    """
189    Evaluates either helloworld or six hump camel for a collection of points
190    given in ``H["x"]`` via the MPI executor, supporting variable sized
191    simulations/resources, as determined by the generator. The term `rset`
192    refers to a resource set (the minimal set of resources that can be assigned
193    to each worker). It can be anything from a partition of a node to multiple
194    nodes.
195
196    Note that this is also an example that is capable of handling multiple
197    points (sim ids) in each call.
198
199    .. seealso::
200        `test_uniform_sampling_with_variable_resources.py <https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/functionality_tests/test_uniform_sampling_with_variable_resources.py>`_ # noqa
201    """
202
203    batch = len(H["x"])
204    H_o = np.zeros(batch, dtype=sim_specs["out"])
205    app = sim_specs["user"].get("app", "helloworld")
206    dry_run = sim_specs["user"].get("dry_run", False)  # dry_run only prints run lines in ensemble.log
207    set_cores_by_rsets = True  # If True use rset count to set num procs, else use all available to this worker.
208    core_multiplier = 1  # Only used with set_cores_by_rsets as a multiplier.
209
210    exctr = libE_info["executor"]  # Get Executor
211    task_states = []
212    for i, x in enumerate(H["x"]):
213        nprocs = None  # Will be as if argument is not present
214        if set_cores_by_rsets:
215            resources = Resources.resources.worker_resources
216            nprocs = resources.num_rsets * core_multiplier
217
218        inpt = None  # Will be as if argument is not present
219        if app == "six_hump_camel":
220            inpt = " ".join(map(str, H["x"][i]))
221
222        task = exctr.submit(
223            app_name=app,
224            app_args=inpt,
225            num_procs=nprocs,
226            stdout="out.txt",
227            stderr="err.txt",
228            dry_run=dry_run,
229        )
230
231        if not dry_run:
232            task.wait()  # Wait for run to complete
233
234            # while(not task.finished):
235            #     time.sleep(0.1)
236            #     task.poll()
237
238        task_states.append(task.state)
239
240        if app == "six_hump_camel":
241            # H_o["f"][i] = float(task.read_stdout())  # Reads whole file
242            with open("out.txt") as f:
243                H_o["f"][i] = float(f.readline().strip())  # Read just first line
244        else:
245            # To return something in test
246            H_o["f"][i] = six_hump_camel_func(x)
247
248    calc_status = UNSET_TAG  # Returns to worker
249    if all(t == "FINISHED" for t in task_states):
250        calc_status = WORKER_DONE
251    elif any(t == "FAILED" for t in task_states):
252        calc_status = TASK_FAILED
253
254    return H_o, calc_status
255
256
257def CUDA_variable_resources(H, _, sim_specs, libE_info):
258    """Launches an app setting GPU resources
259
260    The standard test apps do not run on GPU, but demonstrates accessing resource
261    information to set ``CUDA_VISIBLE_DEVICES``, and typical run configuration.
262
263    For an equivalent function that auto-assigns GPUs using platform detection, see
264    GPU_variable_resources.
265    """
266    x = H["x"][0]
267    H_o = np.zeros(1, dtype=sim_specs["out"])
268    dry_run = sim_specs["user"].get("dry_run", False)  # dry_run only prints run lines in ensemble.log
269
270    # Interrogate resources available to this worker
271    resources = Resources.resources.worker_resources
272    slots = resources.slots
273
274    assert resources.matching_slots, f"Error: Cannot set CUDA_VISIBLE_DEVICES when unmatching slots on nodes {slots}"
275
276    num_nodes = resources.local_node_count
277
278    # Set to slots
279    resources.set_env_to_slots("CUDA_VISIBLE_DEVICES")
280    cores_per_node = resources.slot_count
281
282    # Set to detected GPUs
283    # gpus_per_slot = resources.gpus_per_rset_per_node
284    # resources.set_env_to_slots("CUDA_VISIBLE_DEVICES", multiplier=gpus_per_slot)
285    # cores_per_node = resources.slot_count * gpus_per_slot  # One CPU per GPU
286
287    print(
288        f"Worker {libE_info['workerID']}: CUDA_VISIBLE_DEVICES={os.environ['CUDA_VISIBLE_DEVICES']}"
289        f"\tnodes {num_nodes} ppn {cores_per_node}  slots {slots}"
290    )
291
292    # Create application input file
293    inpt = " ".join(map(str, x))
294    exctr = libE_info["executor"]  # Get Executor
295
296    # Launch application via system MPI runner, using assigned resources.
297    task = exctr.submit(
298        app_name="six_hump_camel",
299        app_args=inpt,
300        num_nodes=num_nodes,
301        procs_per_node=cores_per_node,
302        stdout="out.txt",
303        stderr="err.txt",
304        dry_run=dry_run,
305        # extra_args='--gpus-per-task=1'
306    )
307
308    if not dry_run:
309        task.wait()  # Wait for run to complete
310
311        # Access app output
312        with open("out.txt") as f:
313            H_o["f"] = float(f.readline().strip())  # Read just first line
314
315    calc_status = WORKER_DONE if task.state == "FINISHED" else "FAILED"
316    return H_o, calc_status