Rigour is the extent to which the work demonstrates intellectual coherence and integrity, and adopts robust and appropriate concepts, analyses, sources, theories and/or methodologies. For software this might include, but is not limited to:
- Being used to produce peer-reviewed research outputs
- Clear documentation that explains:
- The software purpose and intended users, including the research context
- How the software works, including the research process it enables
- Architecture and design rationale, to provide confidence in the appropriateness of the approach
- Practices to improve reproducibility of results, e.g. automation, recording of provenance, use of standard data formats
- A clear strategy for internal software quality assurance, e.g. a unit test suite to provide confidence in research results, and consistent coding standards to ensure readability of code
- Formal verification
- Adoption of community-accepted standards
- Certification
- Code and documentation held in version control
- Regular releases
- Structured end-user evaluation
- Quality-based peer review (e.g. publishing a software paper in a specialist software journal, or receiving an ACM artefact badge)