[REVIEW]: Pigeons.jl: Distributed sampling from intractable distributions

[whedon]

Submitting author: @nikola-sur (Nikola Surjanovic)
Repository: https://github.com/Julia-Tempering/Pigeons-Paper
Branch with paper.md (empty if default branch):
Version:
Editor: @pitsianis
Reviewers: @nsailor, @georgebisbas
Archive: Pending

Status

Status badge code:

HTML: <a href="https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d"><img src="https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d/status.svg"></a>
Markdown: [![status](https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d/status.svg)](https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@nsailor & @georgebisbas, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @pitsianis know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @nsailor

Conflict of interest

• As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Authorship: Has the submitting author (@nikola-sur) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

• Authors: Does the paper.tex file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
• Page limit: Is the page limit for full papers respected by the submitted document?

Content

• Context: is the scientific context motivating the work correctly presented?
• Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
• Results: are the results presented and compared to approaches with similar goals?

Review checklist for @georgebisbas

Conflict of interest

• As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Authorship: Has the submitting author (@nikola-sur) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

• Authors: Does the paper.tex file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
• Page limit: Is the page limit for full papers respected by the submitted document?

Content

• Context: is the scientific context motivating the work correctly presented?
• Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
• Results: are the results presented and compared to approaches with similar goals?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @nsailor, @georgebisbas it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/JuliaCon/proceedings-review) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/JuliaCon/proceedings-review:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Failed to discover a Statement of need section in paper

[whedon]

Wordcount for paper.tex is 308

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.10 s (503.0 files/s, 350305.9 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
TeX                                     37           1011           1624          30073
Windows Module Definition                1              0              0           1585
Julia                                   10             59             50            330
Ruby                                     1              8              4             45
YAML                                     1              0              0             30
---------------------------------------------------------------------------------------
SUM:                                    50           1078           1678          32063
---------------------------------------------------------------------------------------


Statistical information for the repository '301fc6098cae505466f5f2a7' was
gathered on 2023/11/18.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Nikola Surjanovic                1            57              0          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Nikola Surjanovic            57          100.0          0.0                7.02

[whedon]

Failed to discover a valid open source license.

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1137/141000671 is OK
- 10.1007/s11222-012-9328-6 is OK
- 10.2307/2348941 is OK
- 10.1093/sysbio/syt022 is OK
- 10.1016/j.cma.2021.114264 is OK
- 10.1615/int.j.uncertaintyquantification.2022038478 is OK
- 10.1038/s41467-022-31830-2 is OK
- 10.1145/3314221.3314642 is OK
- 10.2307/2684568 is OK
- 10.21105/joss.02741 is OK
- 10.1186/s12918-017-0433-1 is OK
- 10.1103/physrevlett.57.2607 is OK
- 10.1111/rssb.12464 is OK
- 10.1093/bioinformatics/17.8.754 is OK
- 10.1093/sysbio/sys029 is OK
- 10.1109/wisp.2007.4447579 is OK
- 10.1007/s11222-008-9110-y is OK
- 10.3390/galaxies11010006 is OK
- 10.3847/1538-4357/ac09ee is OK
- 10.21105/joss.04457 is OK
- 10.1198/106186006x100579 is OK
- 10.1198/jcgs.2010.10039 is OK
- 10.1198/jcgs.2011.10167 is OK
- 10.1073/pnas.1408184111 is OK
- 10.4324/9781003289173-2 is OK
- 10.1111/rssb.12336 is OK
- 10.1145/62959.62969 is OK
- 10.1145/2714064.2660195 is OK
- 10.1080/01621459.1990.10474918 is OK
- 10.1093/sysbio/syq085 is OK
- 10.21105/joss.01816 is OK
- 10.1145/3551624.3555300 is OK

MISSING DOIs

- 10.1111/rssb.12464 may be a valid DOI for title: Non-reversible parallel tempering: a scalable highly parallel MCMC scheme

INVALID DOIs

- None

[whedon]

👋 @georgebisbas, please update us on how your review is going (this is an automated reminder).

[whedon]

👋 @nsailor, please update us on how your review is going (this is an automated reminder).

[nsailor]

This work is an implementation of the Parallel Tempering method in Julia, an improvement over MCMC for obtaining samples form otherwise difficult to sample probability distributions.
The package supports parallelization using both Julia's built-in multi-threading features, as well as MPI.jl, a Julia wrapper for MPI.

I greatly appreciate the attention given to what the authors term "strong parallelism invariance" (SPI), roughly meaning that the result should not depend on the degree of parallelism (especially considering the complexities of finite-precision floating-point arithmetic). This package also builds upon a Julia translation of a Java library called Splittable Random previously published by the authors, allowing deterministic random number generation across multiple threads.

Overall, I think this work is a great contribution to the Julia ecosystem, notwithstanding the following minor points:

• Although definitely not a requirement, I think it would be very interesting to see the scaling characteristics of this implementation, especially with respect to the other algorithm parameters, for instance, the number of chains.
  Additionally, the paper mentions Pigeons being able to run on "thousands of MPI-communicating machines". It may be worth clarifying if this is something that has been tested or a possibility given the package's design. Having said that, I was glad to find this issue so it seems this is something on the authors' radar.

• Unfortuantely, I was not able to get a speedup with the toy MVN example using multiple threads (see this issue in the project repository). In general, it would be very helpful to have some hints in the package's documentation for tuning the degree of parallelism for a given problem, especially if the speedup from parallelism is not linear.

• It would be great to have additional examples, as described in this issue

In writing the above, it is possible that I have misunderstood some part of this work, in which case, please feel free to correct me.

[nikola-sur]

Thank you for your comments! We will work on updating the manuscript, package, and documentation to account for your feedback.

[miguelbiron]

Dear @pitsianis -- we were wondering what the expected timeline for the reviews is? Should we perhaps just respond to @nsailor and not wait for @georgebisbas, or should we wait until we have both sets of comments? Thank you.

[pitsianis]

Please answer any outstanding issues; you don't need to do this sequentially. This exchange will also remind @georgebisbas to wrap up his review.

[miguelbiron]

Thanks for the clarification.

[georgebisbas]

Dear @miguelbiron and @pitsianis, my apologies for the delay here.
Feel free to respond to the outstanding issues; I aim to prioritize this review in my task list.

[miguelbiron]

No worries @georgebisbas, thank you for prioritizing

[georgebisbas]

Hi again, apologies for the delay, I have partly written a draft, aiming to complete my review in the next days.

[georgebisbas]

First, I would like to thank the authors for their patience over the last few months.

Pigeons.jl offers a high-level API to leverage shared- and distributed-memory parallelism via Julia's built-in multi-threading features and MPI.jl, a Julia wrapper for MPI.
Overall, this is a great work suitable for JuliaCon proceedings.
My review below lists a few weaknesses/questions that could help clarify my understanding of the work in a few places or act as constructive feedback to improve this work.

Strengths :

• Automated deployment to MPI DMP without necessary user experience
• Strong parallelism invariance is guaranteed.
• Applications are discussed in the Introduction.
• Smooth transition for the reader, problem formulation, and running examples help in that.
• Compatibility with other targets is a plus
• Section 2.2.1 helps to describe Figure 2
• I very much like how code is accompanied by text describing the symbolic math process

Weaknesses/Questions:
I understand and like that it "works out of the box." It feels that authors focus more on correctness rather than performance. A few questions I have, are:

• What are the memory requirements of a "typical large enough" problem to be tackled?
• What are the advantages of DMP versus SMP only?
• Why are any strong scaling graphs not included?
• After reading the paper, I feel that MPI works correctly, but with no clue on why it is needed and what performance benefits it brings to the table

Minor:

• Have any of the additional targets been implemented in the meantime? Could the paper be updated there?

[nikola-sur]

Thank you to both reviewers for their comments! Now that all reviews are in, we will get back to you shortly with updates and responses to the questions raised.

[nikola-sur]

We thank the reviewers for their insightful comments! Our responses to each reviewer are presented below.

Reviewer 1 (Jasan Barmparesos @nsailor)
It would be very interesting to see the scaling characteristics of this implementation, especially with respect to the other algorithm parameters, for instance, the number of chains.

• Thank you for this comment. We have added weak scaling plots in the paper in the new section on weak scaling of PT (Section 3.4). In this context, the problem size is the number of PT chains. As the problem size increases, we also increase the number of processors for the weak scaling plots. For strong scaling plots, where the problem size (number of chains) is fixed, we propose a future addition to the Pigeons software package: “parallel parallel tempering”. Here, more than one copy of PT is run while keeping the total number of chains across all copies fixed. This parallel PT approach has been previously studied in the PT literature (Syed et al., 2022), but is not yet available in the current version of Pigeons.

Additionally, the paper mentions Pigeons being able to run on "thousands of MPI-communicating machines". It may be worth clarifying if this is something that has been tested or a possibility given the package's design.

• This has been tested (with up to 1000 MPI-communicating machines) on the computer clusters available at our institution. We have clarified this in the updated version of the paper.

I was not able to get a speedup with the toy MVN example using multiple threads (see this issue in the project repository). In general, it would be very helpful to have some hints in the package's documentation for tuning the degree of parallelism for a given problem, especially if the speedup from parallelism is not linear.

• Thank you for raising this point. This issue has been closed as completed in the meantime. The reason for this is that the toy_mvn_target() has a special explorer performing iid sampling. We use this when we need very fast running tests (e.g. in CI) but in terms of performance profile this is atypical. Since the i.i.d. multivariate normal explorer is so quick the overhead of multithreading is not worth it. Typical explorers use algorithms such as slice sampling or gradient informed samplers, and for those algorithms we see improvements once the problem is large enough (and for small problems with fast explorers, absolute time spent will be small anyway, so relative speedup is less important). We have included new plots in the paper (added here for reference) comparing the speedup obtained with Pigeons and multi-threading in realistic scenarios.
  

It would be great to have additional examples, as described in this issue

• We agree that this is a great addition. In fact, we have added a link in the paper to several real-life example use cases of Pigeons with these targets. In the meantime Pigeons has been applied to problems in physics, economics, astrostatistics, psychology, and more. Some examples are made publicly available on InferHub (https://julia-tempering.github.io/InferHub/), our new website dedicated to showcasing examples of applications of Pigeons. We plan to continue to grow this repository of examples over time.

Reviewer 2 (George Bisbas @georgebisbas)

What are the memory requirements of a "typical large enough" problem to be tackled?

• We emphasize that the memory requirements of a “difficult” sampling task are problem dependent. However, Pigeons has facilities to support cases requiring large memory. For instance, we allow for the computation of online statistics (https://pigeons.run/stable/output-online/), as well as off-memory sample storage (https://pigeons.run/stable/output-off-memory/). These have been commented on in the updated version of the manuscript.
• As a second remark, we note that Pigeons is intended for use with “complex” sampling problems. This does not necessarily mean that the memory requirements are large. For instance, a 100-dimensional real-valued distribution may have a very complicated structure, but only has a memory requirement of storing a floating point vector with 100 entries (800 bytes). Nevertheless, due to the complex nature of the problem, it may be necessary to use parallel tempering and distribute computation on up to several machines. In practice, the memory requirement is usually O(d), where d is the dimension of the target distribution. In the Bayesian setting, we may need to store n data points in order to evaluate the likelihood, in which case the memory requirement is O(d+n). Therefore, the memory requirements of a “difficult” sampling task are very much problem dependent.

What are the advantages of DMP versus SMP only?

• Thank you for this question. We note that Pigeons supports both shared memory parallelism (SMP) and distributed memory parallelism (DMP). In certain cases, it might be useful to run parallel tempering with very many chains (e.g., 100) or to run several instances of parallel tempering. In these cases, it is not reasonable to expect all users to have access to a shared memory resource capable of executing 100 processes in parallel. Our implementation in Pigeons ensures that the communication cost in the DMP setup is minimal. In fact, in most cases, the communication cost is negligible compared to other computations performed on each process. We have clarified these points in the paper. Additionally, DMP can be useful when the user's code is not thread-safe. In such cases, concurrency can still be achieved via DMP.

Why are any strong scaling graphs not included?

• Thank you for this comment. We have added weak scaling plots in the paper in the new section on weak scaling of PT (Section 3.4). In this context, the problem size is the number of PT chains. As the problem size increases, we also increase the number of processors for the weak scaling plots. For strong scaling plots, where the problem size (number of chains) is fixed, we propose a future addition to the Pigeons software package: “parallel parallel tempering”. Here, more than one copy of PT is run while keeping the total number of chains across all copies fixed. This parallel PT approach has been previously studied in the PT literature (Syed et al., 2022), but is not yet available in the current version of Pigeons. We are planning to implement this feature in the near future (Can I run k copies of PT easily? Julia-Tempering/Pigeons.jl#188).

After reading the paper, I feel that MPI works correctly, but with no clue on why it is needed and what performance benefits it brings to the table.

• Thank you for raising this point. We have added to the paper comments in a new section (Section 3.5) that emphasize that Pigeons allows for both DMP and SMP. In cases where the sampling problem is particularly complex, it can be advantageous to request many processes, in which case a distributed memory setup (e.g., on a cluster) is the most logical when it cannot be expected from the user to have access to a machine with, e.g., 100-1000 processors. MPI is the go-to interface for DMP settings that require communication between workers. Another notable example of the benefit of MPI is for the multi-GPU use case. For instance, in some situations the likelihood evaluation of a single chain requires its own GPU (e.g., for high-resolution black hole imaging, which relies on running a Fast Fourier Transform on a GPU).

Have any of the additional targets been implemented in the meantime? Could the paper be updated there?

• Yes, there are new targets, which are made publicly available on InferHub (https://julia-tempering.github.io/InferHub/), our new website dedicated to showcasing examples of applications of Pigeons. We plan to continue to grow this repository of examples over time. We have added a link in the paper to InferHub. In the meantime, Pigeons has also been applied to problems in physics, economics, astrostatistics, psychology, and more.

References
Syed, S., Bouchard-Côté, A., Deligiannidis, G., & Doucet, A. (2022). Non-reversible parallel tempering: a scalable highly parallel MCMC scheme. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(2), 321-350.

[editorialbot]

My name is now @editorialbot

[nikola-sur]

@pitsianis Do we need to use the editorial bot for our rebuttal? Just checking that the reviewers (@nsailor, @georgebisbas) have been notified of our responses. Thanks!

[pitsianis]

@nsailor & @georgebisbas, please look at the author's answer to your comments. If appropriate, check the remaining boxes in your checklists and/or follow up with any remaining questions and challenges so the review will move ahead.

[georgebisbas]

Hi @pitsianis , @nikola-sur ,

I would expect that the editorial bot will create some new pdf. Is that right?
Is there any other way to check the updated manuscript?

[nikola-sur]

@editorialbot generate pdf

[nikola-sur]

@whedon generate pdf

[editorialbot]

My name is now @editorialbot

[nikola-sur]

@editorialbot commands

[editorialbot]

Hello @nikola-sur, here are the things you can ask me to do:

# List all available commands
@editorialbot commands

# Get a list of all editors's GitHub handles
@editorialbot list editors

# Adds a checklist for the reviewer using this command
@editorialbot generate my checklist

# Set a value for branch
@editorialbot set juliacon-paper as branch

# Run checks and provide information on the repository and the paper file
@editorialbot check repository

# Check the references of the paper for missing DOIs
@editorialbot check references

# Generates the pdf paper
@editorialbot generate pdf

# Generates a LaTeX preprint file
@editorialbot generate preprint

# Get a link to the complete list of reviewers
@editorialbot list reviewers

[nikola-sur]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[nikola-sur]

@georgebisbas The new PDF can be found above.

[georgebisbas]

Hi @nikola-sur , thanks for updating the manuscript.

I have updated my checklist above accordingly. I am more than happy with the response and the edits.

I think my only remaining checkboxes are:

Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

You may need to have a look here: https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/setting-guidelines-for-repository-contributors

References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

I did not see DOIs in the software as far as I recall.

I think we are nearly there.
Best,
George

[nsailor]

Hi @nikola-sur , thank you for responding to the review points, I think they have been addressed sufficiently.

I don't have any other comment except for the two checklist items mentioned by @georgebisbas above.

Best wishes,
Jason

[miguelbiron]

Thank you both @nsailor @georgebisbas . Just to be clear on the contribution guidelines issue: this is referring to the repo, correct? Like having a proper CONTRIBUTING.md, for example. Not the paper itself, right?

[georgebisbas]

@miguelbiron , yes.
To the best of my understanding, this refers to the repo, and a CONTRIBUTING.md should be enough.

[miguelbiron]

Thank you @georgebisbas . Then Julia-Tempering/Pigeons.jl#289 should cover it.

And we have added dois to the reference to the best of our abilities. There are 2 conference papers without known dois. I'm generating a new version now.

[miguelbiron]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[miguelbiron]

Ah no wait a second. I forgot to commit stuff...

[miguelbiron]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[georgebisbas]

Thank you for that @miguelbiron .

@pitsianis , the checklist is now complete from my side.
All the points have been addressed.

[nikola-sur]

@pitsianis It seems that both reviewers are now satisfied with the changes. Is there anything else that we should do? Please let us know!

[pitsianis]

Indeed very nice work and paper. Unrelated, it would be nice to have slurm commands to submit to a job queue. There exist some very nice examples where an otherwise normal .jl script can also act as a job submission script. That coupled with DrWatson.jl can build a powerful environment to run and rerun heavy computations.

[pitsianis]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1093/sysbio/syac033 is OK
- 10.3847/2041-8213/abe71d is OK
- 10.3847/2041-8213/ac6736 is OK
- 10.1007/s11222-008-9110-y is OK
- 10.1186/s12918-017-0433-1 is OK
- 10.1007/s11222-012-9328-6 is OK
- 10.1137/141000671 is OK
- 10.18637/jss.v103.i11 is OK
- 10.1198/106186006X100579 is OK
- 10.1017/S0956796800000320 is OK
- 10.21105/jcon.00068 is OK
- 10.1073/pnas.1408184111 is OK
- 10.18637/jss.v076.i01 is OK
- 10.3847/1538-4357/ac09ee is OK
- 10.1145/3314221.3314642 is OK
- 10.21105/joss.01816 is OK
- 10.1038/s41467-022-31830-2 is OK
- 10.17863/CAM.42246 is OK
- 10.1198/016214504000001132 is OK
- 10.1088/1741-4326/ac2521 is OK
- 10.1198/jcgs.2011.10167 is OK
- 10.1111/rssb.12336 is OK
- 10.1093/sysbio/syt022 is OK
- 10.1145/62959.62969 is OK
- 10.1198/jcgs.2010.10039 is OK
- 10.1023/A:1008929526011 is OK
- 10.1002/sim.3680 is OK
- 10.1201/b13613 is OK
- 10.1016/j.cma.2021.114264 is OK
- 10.1287/opre.38.5.911 is OK
- 10.1080/01621459.1990.10474918 is OK
- 10.1093/sysbio/sys029 is OK
- 10.1038/s41586-021-03648-3 is OK
- 10.7717/peerj-cs.55 is OK
- 10.1080/17509653.2016.1142191 is OK
- 10.1145/2660193.2660195 is OK
- 10.1103/PhysRevLett.57.2607 is OK
- 10.1111/rssb.12464 is OK
- 10.1111/rssb.12464 is OK
- 10.21105/joss.04457 is OK
- 10.48550/arXiv.1506.03164 is OK
- 10.48550/arXiv.1706.04780 is OK
- 10.1093/sysbio/syq085 is OK
- 10.1145/3551624.3555300 is OK
- 10.1093/nsr/nwx044 is OK
- 10.48550/arXiv.2102.07720 is OK
- 10.48550/arXiv.2310.16782 is OK
- 10.48550/arXiv.2405.11384 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Understanding the Metropolis-Hastings algorithm
- No DOI given, and none found for title: Gibbs sampling
- No DOI given, and none found for title: Church: a Language for Generative Models
- No DOI given, and none found for title: MultiBUGS: a parallel implementation of the BUGS m...
- No DOI given, and none found for title: How many data clusters are in the Galaxy data set?...
- No DOI given, and none found for title: The No-U-Turn sampler: adaptively setting path len...
- No DOI given, and none found for title: JAGS: a program for analysis of Bayesian graphical...
- No DOI given, and none found for title: High-fidelity Bayesian inference of transient FRC ...
- No DOI given, and none found for title: The Bayesian Choice: From Decision-Theoretic Found...
- No DOI given, and none found for title: Parallel Tempering With a Variational Reference
- No DOI given, and none found for title: Parallel tempering on optimized paths
- No DOI given, and none found for title: A New Approach to Probabilistic Programming Infere...

❌ MISSING DOIs

- 10.2307/2348941 may be a valid DOI for title: A language and program for complex Bayesian modell...
- 10.1093/bioinformatics/17.8.754 may be a valid DOI for title: MRBAYES: Bayesian inference of phylogenetic trees
- 10.1615/int.j.uncertaintyquantification.2022038478 may be a valid DOI for title: Hamiltonian Monte Carlo in inverse problems; Ill-c...
- 10.3390/galaxies11010006 may be a valid DOI for title: The ngEHT’s Role in Measuring Supermassive Black H...
- 10.21105/joss.02741 may be a valid DOI for title: ParaMonte: A high-performance serial/parallel Mont...
- 10.1109/wisp.2007.4447579 may be a valid DOI for title: Large scale Bayesian data analysis for nuclear fus...

❌ INVALID DOIs

- 10.5555/3122009.3153003 is INVALID - 10.5555 is a known broken prefix, replace with https://dl.acm.org/doi/{doi} in the {url} field
- 10.5555/3600270.3600311 is INVALID - 10.5555 is a known broken prefix, replace with https://dl.acm.org/doi/{doi} in the {url} field

[pitsianis]

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

• Double check authors and affiliations (including ORCIDs)
• Make a release of the software with the latest changes from the review and post the version number here. This is the version that will be used in the JuliaCon Proceedings paper.
• Archive the release on Zenodo/figshare/etc and post the DOI here.
• Make sure that the title and author list (including ORCIDs) in the archive match those in the JuliaCon Proceedings paper.
• Make sure that the license listed for the archive is the same as the software license.

Editor Tasks Prior to Acceptance

• Read the text of the paper and offer comments/corrections (as either a list or a pull request)
• Check that the archive title, author list, version tag, and the license are correct
• Set archive DOI with @editorialbot set <DOI here> as archive
• Set version with @editorialbot set <version here> as version
• Double check rendering of paper with @editorialbot generate pdf
• Specifically check the references with @editorialbot check references and ask author(s) to update as needed
• Recommend acceptance with @editorialbot recommend-accept

[pitsianis]

@nikola-sur please have another look at the DOIs of your references, see the report above.
Also, see the post-review checklist above, and post here your answers for every bullet point, so that I can check the remaining of the checkboxes as well.