Section 7.3
Implementation Tips

The following is a brief set of tips and standard practices that will help avoid algorithmic issues and bugs, or make it easier to debug when they inevitably occur. Perhaps the most important thing to remember when writing code of any sort is that software is written for people, not machines. That is, your software source code should be clearly structured, well commented, and easy to understand by other researchers or software developers (including a future version of yourself). Telling yourself that your code is being written for another person to understand, not for a machine to understand, will to along way to achieving this goal, and help enormously in avoiding or debugging problems when they do occur.

To catch bugs as early as possible it is always a good idea to check for NaN and Inf. These indicate that something has gone wrong in a numerical calculation (e.g., divide-by-zero), which render any subsequent calculations or model updates invalid. Similarly, always assert pre- and post-conditions of methods. For example, check that tensors match the size that you expect. When writing research code, it is better that your code crashes than produces invalid results.

When debugging code, try to write the simplest test case that will expose the bug. This not only rules out confounding factors and helps to rapidly narrow down the source of the error, but also gives you a free regression test for the future. It is also easier to share your test case with other developers and collaborators who may help you resolve the problem.

In a similar vain, use existing well-tested code where possible. The fact that other people have looked over and used the code means that it is more likely to be correct. We have already mentioned the many excellent deep learning frameworks—PyTorch, TensorFlow, and JAX. You would have to have a very good reason to write your own deep learning library. Often you’ll be extending someone else’s deep learning model, and hopefully they have shared code that implements their model. However, don’t blindly trust third-party code. Make sure you can reproduces published results before extending the code. If you can’t reproduce published results, the problem may be with the code or publication, not with you.

As with all software development, good software development practices should be applied. In particular, use software revision control (e.g., github), and commit your changes regularly. Do not store files that can be reproduced or downloaded from other sources. However, you should include scripts for performing downloads and other data pre-processing such as separating into training, validation and test sets. When running code, print out and save debugging information, but not within tight loops where is can slow down execution of your code. You should also save checkpoints of your models, which will allow you to resume training without having to start from scratch.¹

There are many tools that can be used to monitor your experiments as they run. Linux’s top, Window’s Task Manager, and Nvidia’s nvidia-smi are very simple tools to check for memory usage and potential leaks. And before embarking on a very large experiment that will consume many hours or days, make sure to run your code for a few iterations on a small example rather than your entire dataset.