Summary

  1. Models must be transparent.
  2. Models should adopt organizational and interface recommendations for community and interoperability.


Introduction

Open-source modeling ensures that model results are reproducible, that experts around the world can collaborate to make models better, and that users can explore the parameter space for themselves. These outcomes systematically improve public policy decisionmaking.

We have developed the criteria in this document to facilitate the growth of an open source public policy modeling ecosystem.


Mandatory Criteria for Transparency and Quality

Models must meet the following requirements for acceptance into the catalog:

  1. Release under an OSI-approved open source license or the Creative Commons Public Domain Dedication (CC0).
  2. Make data publicly available, unless release is restricted by a third party.
  3. For any data that should not be disclosed, provide:
  4. A complete descriptive list of all data variables;
  5. Descriptive statistics for all data variables for such data (including averages, standard deviations, number of observations, and correlations to other variables), to the extent that the descriptive statistics do not violate the rule against disclosure;
  6. Contact information for the individual or entity who has unrestricted access to the data.
  7. Include unit tests and ensure that they sources code passes the tests.
  8. For at least one test, generate key outputs from source materials; the test must be run with every new version, and the outputs of the test must be checked into the repository.
  9. Report names and contact information for at least one maintainer.
  10. Have a suggested citation.
  11. Have a project overview.
  12. Have installation instructions.
  13. Version control in GitHub.
  14. Adopt a consistent versioning scheme.
  15. Include a PSL_catalog.json configuration file in the project's repository for cataloging these criteria. Specific instructions for creating this file can be found in the Catalog-Builder Documentation.


Recommended Community Criteria

We also encourage, but do not require, projects to adopt the following practices:

  1. Report code coverage.
  2. Use semantic versioning.
  3. Display a public roadmap.
  4. Display contributor documentation and guidelines.
  5. Hold regular office hours, webinars, or standing meetings.
  6. List technical contributors.
  7. List funders.
  8. List user citations and case studies.
  9. Include subject matter repository topics.
  10. Include a disclaimer.
  11. Track issues publicly (e.g. on GitHub).
  12. Show a changelog.
  13. Be written in an open source language.
  14. Add a "psl-cataloged" tag to the About section of the project's GitHub repository.
  15. Follow a standard contributor workflow, such as GitHub flow.
  16. Include continuous integration tests to ensure that tests are passing for proposed changes in pull requests and to document publicly that all unit tests are passing.
  17. Adopt a consistent code formatting scheme, perhaps through an automated code formatter like Black.


Optional Additional Practices

Projects may also consider the following:

  1. Have a Stack Overflow channel.
  2. Include a "News" translation of the changelog for users.
  3. Include criteria for participating in cross-model PSL initiatives.
  4. Include a link to a webapp version.
  5. Include a list of consultants.