chunkflow

To help you get started, we've collected the most common ways that chunkflow is being used within popular public projects.

seung-lab / igneous / igneous / tasks / tasks.py View on Github

def _inference(self):
        # prepare for inference
        from chunkflow.block_inference_engine import BlockInferenceEngine
        from chunkflow.frameworks.pznet_patch_inference_engine import PZNetPatchInferenceEngine
        patch_engine = PZNetPatchInferenceEngine(self.convnet_path)
        self.block_inference_engine = BlockInferenceEngine(
            patch_inference_engine=patch_engine,
            patch_size=self.patch_size,
            overlap=self.patch_overlap,
            output_key=self.output_key,
            output_channels=self.num_output_channels)


        # inference engine input is a OffsetArray rather than normal numpy array
        # it is actually a numpy array with global offset
        from chunkflow.offset_array import OffsetArray

        input_offset = tuple(s.start for s in self.input_slices)
        input_chunk = OffsetArray(self.image, global_offset=input_offset)
        self.output = self.block_inference_engine(input_chunk)

View more ways to use chunkflow

Problem

• Petabyte scale 3D image processing is slow and computationally demanding;
• Computation has to be distributed with linear scalability;
• Local cluster and public cloud computing are not fully used at the same time;
• Duplicated code across a variety of routine tasks is hard to maintain.

Features

• Composable operators. The chunk operators could be composed in a command line for flexible usage.
• Hybrid Cloud Distributed computation in both local and cloud computers. The task scheduling frontend and computationally heavy backend are decoupled using AWS Simple Queue Service. The backend could be any computer with an internet connection and cloud authentication. Benefit from the robust design, the cheap unstable instances (preemptable intance in Google Cloud, spot instance in AWS) could be used to reduce cost by about threefold!
• Petabyte scale. We have used chunkflow to output over eighteen-petabyte images and scaled up to 3600 nodes with NVIDIA GPUs across three regions in Google Cloud, and chunkflow is still reliable.
• Operators work with 3D image volumes.
• You can plug in your own code as an operator.

Check out the Documentation for installation and usage. Try it out by following the tutorial.

Image Segmentation Example

Perform Convolutional net inference to segment 3D image volume with one single command!

#!/bin/bash

chunkflow \
    load-tif --file-name path/of/image.tif -o image \
    inference --convnet-model path/of/model.py --convnet-weight-path path/of/weight.pt \
        --input-patch-size 20 256 256 --output-patch-overlap 4 64 64 --num-output-channels 3 \
        -f pytorch --batch-size 12 --mask-output-chunk -i image -o affs \
    plugin -f agglomerate --threshold 0.7 --aff-threshold-low 0.001 --aff-threshold-high 0.9999 -i affs -o seg \
    neuroglancer -i image,affs,seg -p 33333 -v 30 6 6

you can see your 3D image and segmentation directly in Neuroglancer!

Composable Operators

After installation, You can simply type chunkflow and it will list all the operators with help message. We keep adding new operators and will keep it update here. For the detailed usage, please checkout our Documentation.

Affiliation

This package is developed at Princeton University and Flatiron Institute.

Reference

We have a paper for this repo:

@article{wu_chunkflow_2021,
	title = {Chunkflow: hybrid cloud processing of large {3D} images by convolutional nets},
	issn = {1548-7105},
	shorttitle = {Chunkflow},
	url = {https://www.nature.com/articles/s41592-021-01088-5},
	doi = {10.1038/s41592-021-01088-5},
	journal = {Nature Methods},
	author = {Wu, Jingpeng and Silversmith, William M. and Lee, Kisuk and Seung, H. Sebastian},
	year = {2021},
	pages = {1--2}
}