How to use the loky.wait function in loky

To help you get started, we’ve selected a few loky examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github michaelnowotny / cocos / cocos / multi_processing / device_pool.py View on Github external
# self._executor = ProcessPoolExecutor(max_workers=self._n_gpus,
        #                                      mp_context=ctx)

        if multiprocessing_pool_type == MultiprocessingPoolType.LOKY:
            from loky import get_reusable_executor, wait

            self._executor = get_reusable_executor(max_workers=self.number_of_devices,
                                                   timeout=None,
                                                   context='loky')

            futures = [self._executor.submit(_init_gpu_in_process,
                                             device_id=compute_device.id)
                       for compute_device
                       in self._compute_devices]

            wait(futures)

            [future.result() for future in futures]
        elif multiprocessing_pool_type == MultiprocessingPoolType.PATHOS:
            from pathos.pools import ProcessPool

            self._executor = ProcessPool(nodes=self.number_of_devices)
            futures = [self._executor.apipe(_init_gpu_in_process, device_id=compute_device.id)
                       for compute_device
                       in self._compute_devices]

            for future in futures:
                while not future.ready():
                    pass
        else:
            raise ValueError(f'Multiprocessing pool type {multiprocessing_pool_type} not supported')

loky

A robust implementation of concurrent.futures.ProcessPoolExecutor

BSD-3-Clause
Latest version published 1 year ago

Package Health Score

68 / 100
Full package analysis