Reproducible results in computational mathematics

Author: David Silvester

All numerical results published in papers published in reputable scientific computing journals should be reproducible. (This is not the case at the present time.)  Authors can ensure this by (a) giving access to the source code as soon as the paper is accepted for publication, and (b) providing diary files that record the output of the code when the computational results reported in the associated research paper are generated.

A recently completed example of good practice is this GitHub repository.

Applying open research practices

The two parts of our technical article (Error estimation and adaptivity for stochastic collocation finite elements) were made available to anyone with access to the arXiv repository (https://arxiv.org) the same day as the manuscripts were submitted for publication in the SIAM journal of Scientific Computing. The source code was not made public at that time. This was to allay fears that the numerical experiments could be published by other people prior to acceptance of the paper in the target journal.

The reproducibility of the computational results was ensured by writing the source code in a high-level language (MATLAB, requires a licence) and then ensuring that the code also runs on the free alternative GNU Octave (https://octave.org). The code is readable and can be readily modified to explore other parameter choices and algorithmic refinements.

Overcoming challenges

The main challenge was that exposing the source code to everyone incurred significant effort on our part, to (a) ensure readability through provision of appropriate comments, and  (b) ensure that good coding practices, such as descriptions of function input/output arguments, were being adopted.  The help facilities that are built into MATLAB functionality considerably eased this burden. The code cleaning process also gave us (three different code writers) confidence that the algorithms that are the subject of the papers had actually been implemented correctly. 

Benefits of using these open research practices

There are two obvious benefits:

• The principle benefit is that the open access code base provides a “perfect” start point for future collaboration with others on the same topic and for future research student projects.
• A second benefit is that insight into the algorithmic performance is best gained by generating the results for oneself on one’s own computer.  When presenting timing comparisons between alternative strategies we can be confident that the results will stand the test of time in the sense that the results are really “reproducible”.

Top tip

GitHub is a wonderful resource.