Processing pipelines

While Nabu is primarily a library of tomography primitives, it is mostly used as a full-fledged processing pipeline for reconstructing synchrotron data.

Pipelines for plain reconstruction

To perform a slice or volume reconstruction, several options are available. Note that 'plain' reconstruction refers to a standard circular trajectory, but can be used with a parallel or cone-beam geometry.

Chunked pipeline

This pipeline reads several rows in all the projections, to reconstruct the corresponding slices. It's the most efficient one, but cannot always be used. It assumes that the GPU memory can hold the projections rows in memory, and the corresponding reconstructions.

The processing is first done in the "projections domain" (filtering, phase retrieval), then in the sinogram domain (normalization, de-ring, reconstruction).

Grouped pipeline

This is a slower, general-purpose processing mode.

In certain cases (eg. phase retrieval), many lines of each projection have to be read/stored at once, which might exceed the amount of memory. For example, a dataset with 24k projections of 16k pixels wide, with Paganin phase retrieval using 50 pixels margin, will need 155 GB of memory (each sinogram is 1.6 GB). When the chunk of data does not fit the RAM or GPU memory, a solution is to load only a part of the projections.

In this case, the processing is done in two stages: - Read a group of full projections, pre-process them, build a partial sinogram. Do the same for a second group of full projections, and so on. - In a second stage, reconstruct the sinograms that were written to disk

In python syntax: "if projections[:, :100, :] does not fit the memory, process projections[:500, :100, :], then projections[500:100, :100, :], etc".

This processing mode ("groups of radios") can handle very large datasets (in theory any size), at the expense of speed.

This mode will read all the data volume, so it's inefficient to use it for single-slice reconstruction. The worst-case scenario is single-slice CTF phase retrieval, since for CTF each full projection is needed (the CTF kernel spans the entire image).

Use nabu nabu.conf --pipeline grouped to enable this mode of processing.

Warning

Currently, nabu somehow duplicates the data with this mode. The sinogram is saved to disk in addition to the data. You can remove the sinogram (sinogram_{output_prefix}.hdf5 and folder sinogram_{output_prefix}) when you are satisfied with the reconstruction.

Accumulated pipeline

This processing mode aims at reconstructing data streams. It's roughly a "streaming mode" equivalent to the grouped pipeline above. It can be used for on-line data processing. For now, it's only available from the python API: nabu.pipeline.fullfield.accumulated.AccumulatedPipeline and nabu.pipeline.fullfield.accumulated_cuda.CudaAccumulatedPipeline.

Pipeline for helical reconstruction

Pipeline were created specially for helical reconstruction.

Sinogram pipeline

This extract sinogram(s) from the data volume and reconstruct the corresponding slices. It is suitable for reconstructing one or several slices. This method assumes that the CPU memory can hold (n_slices + detector_rows)/|dz| full projections.

Use nabu nabu.conf --pipeline sinogram to enable this mode of processing.

Accumulated

This is the counterpart of the "accumulated pipeline" for plain data.

All the data volume is read by angular sub-region, similarly to the 'grouped' pipeline above. The slice reconstruction are accumulated directly, no intermediate sinogram is built.

Note that this is fearfully inefficient if we only want to reconstruct a handful of slices. But it's well adapted for reconstructions that entail reading the whole data volume (full volume reconstruction or vertical slice reconstruction).

This accumulated pipeline itself has two versions: - A fast one, where accumulation takes place in a large GPU buffer. To use this one, the GPU memory has to be able to host at least 'n_z' reconstructed slices (n_z = detector number of rows) - A general-purpose one, where the accumulation takes place in a CPU buffer.

Use nabu nabu.conf --pipeline accumulated-gpu or nabu nabu.conf --pipeline accumulated-cpu