WP 7. Scalable parallel post-processing

To scale a post-processor for parallel computation and visualization of large amount of data, minimizing the I/O flux of data.

Existing commercial parallel, scalable post-processors (Paraview, VisIt) have been conceived with the assumption that one has a pipeline with very large capacity to connect to the parallel machine. This is rarely the case in production environments.

For this reason, within this task we will develop reduction techniques for communicating the postprocessing data on the parallel machine, and transmit to the (parallel) high-end desktop only the reduced data for visualization.

In this way, only the minimum amount of data required exits the parallel, exascale class machine, reducing I/O bandwidth, the main bottlenecks and wait times.

One of this postprocessor priorities will be the evaluation of which set of reduced data are needed to minimize the I/O bandwidth and storage without losing relevant information, equivalent to image and video compression techniques, i.e., by using adaptive streaming.

• Point sensors
• Cutting planes
• Isosurfaces

On the other hand, differently to the previous phases in the simulation pipeline, the visualization and, in general terms, postprocessing functionalities do not require the global knowledge of all the variables and unknowns to work with each piece of information, so it is probably the software module which can take best benefit of the parallel architecture. Furthermore, the postprocessors work naturally with graphics cards, so they are already prepared to exploit GPUs architectures.

Another possibility to minimize movement of data between the HPC and the end-user analyzing the data are smplification techniques. The goal of simplification post-processing is to adjust the outputs to be actually visualized to the characteristics and graphical resolution of local machine that will produce the final graphical otput. Simplification is performed on the server where data is produced, resulting in further reduction of the ammount of data trasferred from the data reduction methods.

Transfer of data can be reduced to a mínimum if the visualization of he data is produced using the graphic card of the server. In such case only the final produced image is trasferred to the local machine.

The management of the disk storage will be a changing paradigm, what still is a holistic problem because implies to save the whole results of the simulation. Our answer to this will reproduce other systems working with huge amount of data (for example, the maps navigation systems), including the in situ visualization, which means to integrate features of the postprocessing phase in the solvers. Therefore, only the results of the visualization will be saved to the disk (on the client side).

The implementation of new visualization algorithms in state-of-the-art graphics hardware available by the time of completion (e.g. new GPUs or similar hardware) will be a task in this WP.

Emphasis will be put in the identification of commonalities (as described in section B1.1.4 of the DoW) in parallel post-processing algorithms that can be of general interest and applicability for other HPC developments.

WP/task leader: CIMNE. Partners involved: CIMNE, LUH-IKM, NTUA, QUANTECH.

Starting date: month 21. Duration: 10 months

D7.1 Post-processing algorithms including description of commonality developments

Lead beneficiary: CIMNE