Converted to Confluence from: https://github.com/mrnorman/miniWeather/wiki/A-Practical-Introduction-to-GPU-Refactoring-in-Fortran-with-Directives-for-Climate authored by Matthew Norman
Introduction
This is intended to be an evolving document that covers the basics of what you should expect to do in a GPU refactoring and porting effort in a climate code. It will probably have some additions over time.
For climate codes, there are rarely large hot spots to be ported where the majority of the runtime is offloaded to an accelerator. Rather, when offloading a climate code, many different kernels must be created. This, coupled with the fact that the codes are traditionally in Fortran and that domain scientists are continually altering the code, means Fortran accelerator directives seem to be the most sensible approach. When approaching a GPU refactoring project, there are a few ideas that can guide the effort. The goal of this page is to overview those overarching goals / challenges for climate.
Helpful Links
- https://www.openacc.org
- https://www.openacc.org/sites/default/files/inline-files/OpenACC%20API%202.6%20Reference%20Guide.pdf
- https://www.openmp.org
- https://www.openmp.org/wp-content/uploads/OpenMP-4.5-1115-CPP-web.pdf
- https://devblogs.nvidia.com/getting-started-openacc/
Also checkout the many YouTube videos on OpenACC and OpenMP4.5 tutorials.
Quick-Running Testing Framework
The first thing you need is a quick-running testing framework, ideally a test or set of tests that can run on the order of 10 minutes. They need to be able to catch your most common bugs. One problem with testing the answers in climate is that it describes chaotic non-linear fluid dynamics where the answer changes rapidly from bit-level arithmetic changes (which you are very likely to introduce). My means of testing the answer is to run for a day at O0 optimization and then again at O3 optimization. This provides a baseline of the expected differences cause by bit-level changes after one day of simluation. Then, a three-way diff can be performed between the two baselines and the refactored simulation to determine if you are outside the expected envolope of answer changes. It's important to have these tests because finding bugs that could only have been caused by tens of lines of code changes is significantly easier and faster than finding bugs "in the wild" with a full-fledged debugger.