LiberTEM is an open source platform for high-throughput distributed processing of large-scale binary data sets using a simplified MapReduce programming model. The current focus is pixelated scanning transmission electron microscopy (STEM) [MNR+16, Oph19] and scanning electron beam diffraction data.
It is designed for high throughput and scalability on PCs, single server nodes, clusters and cloud services. On clusters it can use fast distributed local storage on high-performance SSDs. That way it achieves very high aggregate IO performance on a compact and cost-efficient system built from stock components. All CPU cores and CUDA devices in a system can be used in parallel.
LiberTEM is supported on Linux, Mac OS X and Windows. Other platforms that allow installation of Python 3.6+ and the required packages will likely work as well. The GUI is running in a web browser.