Appendix. SSI Evidence of originality, significance and rigour for software outputs

The extent to which an output makes an important and innovative contribution to understanding and knowledge in the field. For software this might include, but is not limited to: 

• Implementation of a novel algorithm
• Development of innovative research methods
• A solution to a problem that has not been solved before in practice
• Interpretation of or engagement with novel forms or scales of data
• Improvement in the scale, resolution or accuracy at which research can be performed

The extent to which the work has influenced, or has the capacity to influence, knowledge and scholarly thought, or the development and understanding of policy and/or practice. For software this might include, but is not limited to: 

• Enabling or accelerating significant research advances
• Enabling research or thinking that was not previously possible
• Influencing policy, practice or users/audiences
• Being used by a large number of researchers, or within a diverse range of disciplines

Rigour is the extent to which the work demonstrates intellectual coherence and integrity, and adopts robust and appropriate concepts, analyses, sources, theories and/or methodologies. For software this might include, but is not limited to: 

• Being used to produce peer-reviewed research outputs
• Clear documentation that explains: 
  • The software purpose and intended users, including the research context
  • How the software works, including the research process it enables
  • Architecture and design rationale, to provide confidence in the appropriateness of the approach
• Practices to improve reproducibility of results, e.g. automation, recording of provenance, use of standard data formats
• A clear strategy for internal software quality assurance, e.g. a unit test suite to provide confidence in research results, and consistent coding standards to ensure readability of code
• Formal verification
• Adoption of community-accepted standards
• Certification
• Code and documentation held in version control
• Regular releases
• Structured end-user evaluation
• Quality-based peer review (e.g. publishing a software paper in a specialist software journal, or receiving an ACM artefact badge)