Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model

Transcription

1 Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model Rodrigo Virote Kassick 1 2, Francieli Zanon Boito 1 2, Philippe Navaux 1, Yves Denneulin² 1 GPPD II Federal University of Rio Grande do Sul (UFRGS), Brazil ² INRIA LIG Grenoble University, France June 13 th 2012

2 Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model Rodrigo Virote Kassick 1 2, Francieli Zanon Boito 1 2, Philippe Navaux 1, Yves Denneulin² 1 GPPD II Federal University of Rio Grande do Sul (UFRGS), Brazil ² INRIA LIG Grenoble University, France June 13 th 2012

3 Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model Rodrigo Virote Kassick 1 2, Francieli Zanon Boito 1 2, Philippe Navaux 1, Yves Denneulin² Weather/climate forecast model 1 Universidade Federal do Rio Grande do Sul (UFRGS), Brazil ² Laboratoire d'informatique de Grenoble (LIG), France June 13 th 2012

4 Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model Federal University of Rio Grande do Sul Rodrigo Virote Kassick 1 2, Francieli Zanon Boito 1 2, (UFRGS) Philippe Navaux 1, Yves Denneulin² Center for Weather Forecast and Climate Studies 1 Universidade Federal do Rio Grande do Sul (UFRGS), Brazil (CPTEC) ² Laboratoire d'informatique de Grenoble (LIG), France National Laboratory of Scientific Computing (LNCC) University of Miami (USA) June 13 th 2012

5 Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model Rodrigo Virote Kassick 1 2, Francieli Zanon Boito 1 2, Philippe Navaux 1, Yves Denneulin² 1 Universidade Federal do Rio Grande do Sul (UFRGS), Brazil ² Laboratoire d'informatique de Grenoble (LIG), France June 13 th 2012

6 Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model

7 Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model

8 Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model Damaris

9 Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model OLAM Damaris I/O Infrastructure

10 Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model OLAM Damaris I/O Infrastructure Initial results

11 Agenda OLAM and its Performance Problem OLAM + Damaris Performance Results Conclusions Future Work 11 / 58

12 Agenda OLAM and its Performance Problem OLAM + Damaris Performance Results Conclusions Future Work 12 / 58

13 Ocean-Land-Atmosphere Model Fortran 90 + MPI Developed at Duke University Global grid with local refinements 13 / 58

14 Ocean-Land-Atmosphere Model Initialization Timestep Finalization 14 / 58

15 Ocean-Land-Atmosphere Model Initialization Timestep Finalization Read input files (~600MB) All the processes 15 / 58

16 Ocean-Land-Atmosphere Model Initialization Timestep Finalization Every N timesteps, write output files (file per process) with HDF5 16 / 58

17 Ocean-Land-Atmosphere Model Initialization Timestep Finalization Each file: ~500KB Every N timesteps, write output files (file per process) with HDF5 17 / 58

18 Ocean-Land-Atmosphere Model Large number of small files access pattern Poor I/O performance Overhead of file creation + small requests Intra-node concurrency 18 / 58

19 Ocean-Land-Atmosphere Model Approach 1: OLAM MPI+OpenMP 19 / 58

20 Ocean-Land-Atmosphere Model Processing Node MPI Rank 0 MPI Rank 1 MPI Rank 2 MPI Rank 3 File System 0_1 0_2 1_1 1_2 2_1 2_2 3_1 3_2 20 / 58

21 Ocean-Land-Atmosphere Model Approach 1: OLAM MPI+OpenMP Processing Node MPI Rank 0 File System 0_1 0_2 21 / 58

22 Ocean-Land-Atmosphere Model Approach 1: OLAM MPI+OpenMP Processing Node MPI Rank 0 Previous Work: 8x less files -> File System 0_1 0_2 OLAM up to 20x faster 22 / 58

23 Ocean-Land-Atmosphere Model Approach 1: OLAM MPI+OpenMP Processing Node MPI Rank 0 Previous Work: 8x less files -> File System Not always... 0_1 0_2 OLAM up to 20x faster 23 / 58

24 Ocean-Land-Atmosphere Model Approach 2: OLAM + Damaris 24 / 58

25 Agenda OLAM and its Performance Problem OLAM + Damaris Performance Results Conclusions Future Work 25 / 58

26 OLAM + Damaris Damaris Library/infrastructure to help coordinate I/O Dedicate one core of each node to I/O Does I/O on behalf of the application 26 / 58

27 OLAM + Damaris Processing Node MPI Rank 0 MPI Rank 1 File System MPI Rank 2 MPI Rank 3 27 / 58

28 OLAM + Damaris Approach 2: OLAM + Damaris Processing Node MPI Rank 1 MPI Rank 2 MPI Rank 3 Damaris File System 28 / 58

29 OLAM + Damaris Output phase: 1. Create file 2. Write set of variables to file 1. Create HDF5 Dataset (H5Dcreate) 2. Write HDF5 Dataset (H5Dwrite) 3. Close file 29 / 58

30 OLAM + Damaris Output phase with Damaris: 1. Create file 2. Write set of variables to file 1. Create HDF5 Dataset (H5Dcreate) 2. Write HDF5 Dataset (H5Dwrite) 3. Close file 30 / 58

31 OLAM + Damaris Output phase with Damaris: 1. Create file 2. Write set of variables to file to Damaris 1. Create HDF5 Dataset df_chunk_set 2. Write HDF5 Dataset df_chunk_write 3. Close file 31 / 58

32 OLAM + Damaris Output phase with Damaris: 1. Create file 2. Write set of variables to file to Damaris 1. Create HDF5 Dataset df_chunk_set 2. Write HDF5 Dataset df_chunk_write 3. Close file 4. df_signal 32 / 58

33 OLAM + Damaris + Plugin to do the actual I/O When the application calls df_signal Create, write to HDF5 files and close + XML file List name and type of all variables 33 / 58

34 OLAM + Damaris Plugin 1: File-per-process Processing Node File System MPI Rank 1 MPI Rank 2 Damaris MPI Rank 3 34 / 58

35 OLAM + Damaris Plugin 1: File-per-process Processing Node File System MPI Rank 1 1_1 1_2 MPI Rank 2 MPI Rank 3 Damaris 2_1 2_2 3_1 3_2 35 / 58

36 OLAM + Damaris Plugin 2: Merger Processing Node File System MPI Rank 1 MPI Rank 2 MPI Rank 3 Damaris 1_1 1_2 36 / 58

37 Agenda OLAM and its Performance Problem OLAM + Damaris Performance Results Conclusions Future Work 37 / 58

38 Performance Evaluation 1. OLAM MPI 2. OLAM MPI + OpenMP 3. OLAM + Damaris File-per-process plugin 4. OLAM + Damaris Merger plugin 38 / 58

39 Performance Evaluation Rennes. Grid nodes from Parapluie (processing nodes) 12 cores 10 nodes from Parapide (FS nodes) OrangeFS (PVFS) 10 servers (meta and data servers) 39 / 58

40 Execution Time (seconds) Speedup Performance Evaluation OLAM-MPI #Processes #Threads / 58

41 Execution Time (seconds) Speedup Performance Evaluation OLAM-MPI OLAM-MPI+OpenMP #Processes #Threads / 58

42 Execution Time (seconds) Speedup Performance Evaluation OLAM-MPI OLAM-MPI+OpenMP ~27% #Processes #Threads / 58

43 Execution Time (seconds) Speedup Performance Evaluation OLAM-MPI OLAM-MPI+OpenMP ~27% #Processes #Threads Decrease in the number of files No intra-node concurrency / 58

44 Execution Time (seconds) Speedup Performance Evaluation OLAM-MPI OLAM-MPI+OpenMP OLAM+Damaris File-per-process #Processes #Threads / 58

45 Execution Time (seconds) Speedup Performance Evaluation OLAM-MPI OLAM-MPI+OpenMP ~26% OLAM+Damaris File-per-process #Processes #Threads / 58

46 Execution Time (seconds) Speedup Performance Evaluation OLAM-MPI OLAM-MPI+OpenMP ~26% #Processes #Threads OLAM+Damaris File-per-process Less processing nodes No intra-node concurrency / 58

47 Execution Time (seconds) Speedup Performance Evaluation OLAM-MPI OLAM+Damaris Merger OLAM-MPI+OpenMP #Processes #Threads / 58

48 Execution Time (seconds) Speedup Performance Evaluation OLAM-MPI OLAM+Damaris Merger OLAM-MPI+OpenMP ~32% #Processes #Threads / 58

49 Execution Time (seconds) Speedup Performance Evaluation OLAM-MPI OLAM+Damaris Merger OLAM-MPI+OpenMP ~32% Less processing nodes No intra-node concurrency #Processes #Threads 0 less files / 58

50 Execution Time (seconds) Speedup Performance Evaluation OLAM-MPI OLAM+Damaris File-per-process OLAM+Damaris Merger #Processes #Threads / 58

51 Agenda OLAM and its Performance Problem OLAM + Damaris Performance Results Conclusions Future Work 51 / 58

52 Conclusions Aplication (OLAM) Poor I/O performance 52 / 58

53 Conclusions OpenMP Aplication (OLAM) Poor I/O performance Damaris 53 / 58

54 Conclusions OpenMP ~27% Aplication (OLAM) Poor I/O performance Damaris ~26% ~32% 54 / 58

55 Conclusions OLAM+Damaris was simple Results better than the MPI+OpenMP version It s easy to test new I/O approaches Visualization tool 55 / 58

56 Agenda OLAM and its Performance Problem OLAM + Damaris Performance Results Conclusions Future Work 56 / 58

57 Future Work OLAM+Damaris Keep investigating! Can we use it to input too? Damaris How much information can we get about the applications access pattern? (Francieli) Can it help pipeline-style scientific workflow? (Rodrigo Kassick) 57 / 58

58 Investigating I/O approaches to improve performance and scalability of the Ocean-Land-Atmosphere Model Thank you! Rodrigo Virote Kassick 1 2, Francieli Zanon Boito 1 2, Philippe Navaux 1, Yves Denneulin² 1 GPPD II Federal University of Rio Grande do Sul (UFRGS), Brazil ² INRIA LIG Grenoble University, France June 13 th 2012