Author and research engineer Joel Grus kicked off an important conversation about Jupyter Notebooks in his recent presentation at JupyterCon:
There’s no video yet available of Joel’s talk, but you can guess the theme of that opening slide, and walking through the slides conveys the message well, I think. Yuhui Xie, author and creator of the rmarkdown package, provides a detailed summary and response to Joel’s talk, where he lists Joel’s main critiques of Notebooks:
-
Hidden state and out-of-order execution
-
Notebooks are difficult for beginners
-
Notebooks encourage bad habits
-
Notebooks discourage modularity and testing
-
Jupyter’s autocomplete, linting, and way of looking up the help are awkward
-
Notebooks encourage bad processes
-
Notebooks hinder reproducible + extensible science
-
Notebooks make it hard to copy and paste into Slack/Github issues
-
Errors will always halt execution
-
Notebooks make it easy to teach poorly
-
Notebooks make it hard to teach well
Yihui suggests that many of these shortcomings of Notebooks could be addressed through literate programming systems, where the document you edit is plain-text (and so easy to edit, manage, and track), and computations are strictly processed from the beginning of the document to the end. I use the RMarkdown system myself, and find it a delightful way of combining code, output and graphics in a single document, which can in turn be rendered in a variety of formats including HTML, PDF, Word and even PowerPoint.
Yihui expands on these themes in greater detail in his excellent book (with JJ Allaire and Garrett Grolemund), R Markdown: The Definitive Guide, published by CRC Press. Incidentally, the book itself is a fine example of literate programming; you can find the R Markdown source here, and you can read the book in its entirety here. As Joel mentions in his talk, an automatically-generated document of that length and complexity simply wouldn’t be possible with Notebooks.
All that being said, RMarkdown is (for now) a strictly R-based system. Are there equivalent literate programming systems for Python? That’s a genuine question — I don’t know the Python ecosystem well enough to answer — but if you have suggestions please leave them in the comments.
Yihui Xie: The First Notebook War