Distill Update 2018

A little over a year ago, we formally launched Distill as an open-access scientific journal.Distill operated informally for several months before launching itself as a journal.

It’s been an exciting ride since then! To give some very concrete metrics, Distill has had over a million unique readers, and more than 2.9 million views. Distill papers have been cited 23 times on average. We measure this by Google Scholar citations. This would place Distill in the top 2% of academic journals indexed by Journal Citation Reports by this metric. That said, this is a somewhat unfair comparison given Distill’s smaller size; it is easier to publish mostly impactful papers when you only publish a small number of them. More importantly, we’ve published several new papers with a strong emphasis on clarity and reproducibility, which we think is helping to encourage a new style of scientific communication.

Despite this, there are a couple ways we think we’ve fallen short or could be doing better. To that end, we’ve been reflecting a lot on what we can improve. In particular, we plan to make the following changes:

Separate mentoring from evaluating articles.
Clarify and streamline the Distill review process, including a new reviewer worksheet.
Prioritize creating resources to help everyone over mentoring individuals, starting with the creation of a Slack workspace (join here).
Simplify our process for bringing in acting editors to resolve conflicts of interest.
Clarify our policy on dual submissions.
Grow the Distill editorial team and create a pathway to entering more fields. Arvind Satyanarayan is joining the editorial team to oversee an HCI + AI portfolio.

Things that Worked Well

Interfaces for Ideas

It’s tempting to think of explanations as a layer of polish on top of ideas. We believe that the best explanations are often something much deeper: they are interfaces to ideas, a way of thinking and interacting with a concept. Building on this, we’ve seen several Distill articles create visualizations that reify those ways of thinking and interacting with ideas.

One of the articles that best exemplifies this type of contribution is Gabriel Goh’s Why Momentum Really Works. Gabe, and other optimization researchers, have a perspective on this problem that may be unfamiliar to practitioners. It involves a mathematical formalism of the spectrum of eigenvalues of the optimization problem, as well as a more informal way of interpreting and thinking about them. In our opinion, traditional academic publications don’t emphasize articulating the intuition that accompanies technical ideas — the kind that teachers share with students at whiteboards.

In contrast, this diagram, taken from the article, not only conveys the formalism but also shares some of the author’s intuition. Bolstered by the interactivity, it invites readers to step into a way of thinking. Perhaps most interestingly, by reifying a mental model into a computationally-driven interface, the author discovered places where their thinking was incomplete — specifically, introducing momentum flattens the spectrum of eigenvalues in surprising ways.

This kind of interface design and this line of thinking isn’t new to Distill, but we’ve been able to help push it forward over the last year, and we’re excited to see where it will go.

Engagement as a Spectrum

One thing we’ve found particularly exciting is how articles can make engaging deeply with ideas an easier and smoother process. Normally, there’s a huge jump from reading a paper to testing and building on it. But we’re starting to see papers where engagement is a continuous spectrum:

Several recent articles have explored this idea. Not only are important concepts accompanied by interactive diagrams, but then also by a notebook that can reproduce that diagram. But including a notebook does more than allowing readers to test an idea — it lets them dive into new research without any setup. Interesting new experiments can sometimes be run in mere minutes.

As of today, over 6,000 readers have opened the notebooks. This represents around 3% of readers who viewed the articles with these notebook links.

Reproducibility has long been recognized as a critical component of maintaining scientific hygiene, and authors have increasingly taken to open sourcing their contributions and even putting intermediary artifacts (e.g., data analysis scripts) on GitHub. In-browser notebooks allow authors to go one step further, engaging in what we call “active reproducibility”: not only making it technically possible to reproduce their work, but making it convenient to do so. We hope to see more authors invest effort in this regard.

Software Engineering Best Practices for Scientific Publishing

Over the past year, we’ve also seen several advantages to using software engineering best practices to operate a scientific journal. Every Distill article is housed within a GitHub repository, and peer review is conducted through the issue tracker. We’ve found this setup gives readers greater transparency into the publication process. For instance, readers can see how Gabriel Goh’s momentum article has been updated from when it was first published, step through early development to see the genesis of the article’s ideas, and can read the back-and-forth of the review process. More excitingly, we’ve seen readers engage in on-going, post-publication peer review including sending pull requests to fix typos, making more thorough editing passes, and even sparked discussions with the author.

Authors also benefit from this setup as Distill provides continuous integration for scientific papers. Prior to publication, draft articles are automatically built and served from password-protected URLs. Authors are free to share these addresses to solicit initial feedback, and can be confident that their readers will always see the most up-to-date version of the draft.

Challenges & Improvements

The Distill Prize

We have, unfortunately, not yet awarded the 2018 Distill prize. We received 59 submissions, several of which were lecture series consisting of many hours of content. We did not anticipate content of this kind and did not (and do not) have a good process in place for evaluating it. Part of the issue has been about being a perfectionist in our evaluation of the content. In the future, we aim to better balance this with expeditious review.

We endeavor to award the Distill prize by Thanksgiving 2018, and to conduct the process in a more timely fashion moving forward.

Distill aims to cultivate a new style of scientific communication and build an ecosystem to support it. We hope to grow this community over time but, for the moment, we have a relatively small pool of potential authors and editors compared to more traditional academic venues.

One consequence of this has been that Distill has a low publication volume (12 papers as of present). We’ve grappled a lot with this: Should we change our standards? Is this something we should be concerned about?

At the end of the day, we believe that, as long as our content is outstanding, it’s fine to be a smaller “bespoke” journal for the foreseeable future. We believe Distill primarily serves the community by legitimizing non-traditional research artifacts, and providing an example of what is possible. That requires quality of publications, not quantity.

Another challenge of being in a small community is that there are often social ties between members of our community. This is great, but it means that we have to navigate a lot of potential conflicts of interest. These kind of challenges around conflicts of interest are typical of a young field, and we expect this issue to become less of a problem over the coming years as our editorial team grows. However, in the meantime, we often need to bring in independent acting editors to resolve conflicts of interest.

Our previous process for getting independent editors was to go through members of Distill’s steering committee. (We’re very grateful to Ian Goodfellow for his patient assistance with this.) However, since this is turning out to be a pretty common situation, we’d like a mechanism that doesn’t require us to bug our steering committee:

In the event of a conflict of interest, Distill editors will select a member of the research community to serve as a temporary “acting editor” for an article. The acting editor should be a member of the relevant research community, and at arm’s length to the authors. The use and identity of an acting editor will be noted in the review process log, and made public if the article is published.

Review Process

When we started Distill, we adopted a pretty radical review process. We knew that most researchers didn’t have the full skill set — especially the design skills — needed to write the kind of articles Distill aspires to. For that reason, we provided extensive mentorship and assistance to help autohrs improve articles.

Unfortunately, while this has led to some articles we’re really proud of, we’ve found ourselves struggling with it. Distill editors are volunteers who do their work on top of their normal role as researchers and the kind of mentorship we’ve been trying to do can take as much as 20-80 hours per article. As a result, editors have ended up severely over-capacity. We’ve also seen editors end up in paralyzing dual roles of simultaneously mentoring authors and needing to make editorial decisions about their work.

For some authors, the result has been a slow and indecisive review process.

Our authors and editors deserve better than that. To that end, Distill is implementing the following policy changes to our review process:

Distill’s review process will no longer involve mentorship. Instead, we’re creating alternative channels for supporting authors, discussed below.
Distill will only consider complete article submissions and will evaluate them as is.
We’re creating a public reviewer worksheet that more clearly defines our review criteria.

Our new policy is described in detail on the article submission page. Our old policy is archived here. By default, this policy change does not apply to articles submitted under our previous policy.

Part of Distill’s role is to enable authors to experiment and to provide a home for unusual types of academic artifacts. Formalizing our review process doesn’t change this; we are still open to submissions that our typical review process doesn’t anticipate, and will adapt our process if necessary. In particular, in the next year, we hope to increase the number of short articles focused on a narrow topic.

We’re also interested in finding ways to support communication of early-stage research results. We’re concerned that our current expectations may incentivize authors to not share results until they have reached a high level of maturity, and this may not be the best thing for the field. We’re not implementing any policy changes regarding this at the moment, but are actively considering options. Please reach out if you have ideas you’d like us to consider!

Other Changes

Dual Submission Policy

In order for Distill to be effective in legitimizing non-traditional publishing, it must be perceived as a primary academic publication. This means it’s important for Distill to follow typical “dual publication” norms. It’s also important for us to avoid the perception that Distill is an “accompanying blog post” for something like an arXiv paper.

The result is that Distill can only consider articles that are substantially different from those formally published elsewhere, and is cautious of articles informally published elsewhere. Below we provide guidance for particular cases:

No Prior Publication / Low-Profile Informal Publication: No concerns!
ArXiv Paper: We completely understand that researchers sometimes need to quickly get results out, and arXiv is a great vehicle for doing so. We’re happy to publish your paper as long as there’s a clear understanding that Distill is the formal publication. The Distill paper will almost certainly be more developed anyways. :)
Previous Workshop / Conference Papers: Distill is happy to publish more developed and polished “journal versions” of papers. (This is a normal pattern in scientific publishing, although less common in Machine Learning.) These must substantively advance on the previous publications, through some combination of improving exposition, better surfacing of underlying insights and ways of thinking, consolidating a sequence of papers, or expanding with better experiments.
High-Profile Informal Publication: We see this as being very similar to publication in a workshop or conference, and have the same expectations as above.

Supporting Authors

Over the last year, we’ve put a lot of energy into mentoring individuals on writing Distill articles. This has often involved editors volunteering tens of hours of mentorship, per submitted article. While this has been rewarding, it isn’t scalable.

In the next year, we plan to focus more on scalable ways of helping people by:

Continuing our work on the Distill Template, which provides many of the basic tools needed for writing beautiful web-first academic papers. We’ll be starting work on version 3 of the template, and plan to make it even easier to use by making more opinionated decisions about author workflow.
Writing a Distill Style Guide describing the best practices we’ve discovered. We see many of the same issues in the articles we edit, and hope that by consolidating the solutions in a style guide we can help all authors avoid those pitfalls.
Sharing our Distill Reviewer Worksheet so that authors can use it to self-evaluate their article and look for areas to improve.
Starting a Distill Community Slack workspace where people can seek advice, mentorship, and co-authors.

Growing Distill’s Team

We believe that growing Distill’s editorial team is one of the most important ingredients for its long-term success. Growing the circle of people involved in Distill’s day-to-day operations makes it more robust, better able to scale, and reduces conflict of interest issues.

As important as it is to expand our editors, it’s equally important to make sure we pick the right editors. This means building up a team deeply aligned with Distill’s unusual values and mission. To achieve this, we plan to use the following evaluation process for potential editors:

Write an outstanding Distill paper, demonstrating to us that they deeply understand Distill’s mission and have the technical skills needed to evaluate others’ work.
Interviews with existing editors discussing Distill’s mission and the role of editors.

Being a Distill editor means taking on ownership and responsibility for the success of Distill and for publication decisions within your subject matter portfolio. Distill editors are volunteer positions with no compensation — except playing a critical role in advancing a new kind of scientific publishing.

As a first step in this direction, we’re pleased that Arvind Satyanarayan has joined the Distill editorial team. Arvind comes from the data visualization and human-computer interaction (HCI) communities and will initially focus on articles at the intersection of these fields with machine learning. If Distill finds the right additional editor, we would be happy to expand our coverage of HCI more broadly.

Growing Distill’s Scope

In the long-run, we believe Distill should be open to expanding to other disciplines, with new editors taking on different topic portfolios.

We had previously believed that, in exploring a new kind of publishing, Distill would be best served by focusing on a single “vertical” (machine learning) where it had editorial expertise. We still believe that Distill should only operate in areas where it has expert editors, but we also think that isn’t the full story. Focusing only on machine learning has exacerbated our small community issues, by restricting us to the intersection of machine learning and this style of communication.

We now believe that the right strategy for Distill is to expand to other disciplines, slowly and cautiously, if and when we find the right editors. These new topics would become part of a single cross-disciplinary Distill journal, like PLoS or Nature. We do not plan to subdivide or franchise at this point.

In considering editors for new topics, Distill will have the same expectations we have for all editors (described above) with two modifications:

Although Distill does not normally review papers outside its existing topic portfolio, we will make an exception to review papers from potential editorial candidates. The existing editorial team would evaluate exposition while soliciting a third party editor to help us evaluate scientific merit, following Distill’s regular review process. Because this type of review is especially difficult and expensive, we will only move forward if the submission plausibly appears to be a very strong article.
There needs to be a second editor who can share responsibility for the new topic. This can either be an existing editor expanding to another topic, or someone applying along with the new editor. Having a second editor is important so that editors have someone to talk over difficult cases with, and so that there isn’t a single point of failure.

Conclusion

Distill is a young journal exploring a new style of scientific communication. We have learned a lot of valuable lessons in our first year, but we still have a lot of room to grow. We hope that you will join us in pushing the boundaries of what a scientific paper can be!

Distill is grateful to all the members of the research community who have supported it to date — our authors, reviewers, editors, members of the steering committee, every one providing feedback on GitHub, and, of course, our readers. We’re glad to have you with us!