Digging Deeper into the State of the Practice for Domain Specific Research Software

To improve the productivity of research software developers we need to first understand their development practices. Previous studies on this topic have collected data by surveying as many developers as possible, across a broad range of application domains. We propose to dig deeper into the state of the practice by instead looking at what developers in specific domains create, as evidenced by the contents of their repositories. Our methodology prescribes the following steps: i) Identify the domain; ii) Identify a list of candidate software; iii) Filter the list to a length of about 30 packages; iv) Collect repository related data on each package, like number of stars, number of open issues, number of lines of code; v) Fill in the measurement template (the template consists of 108 questions to assess 9 qualities (including the qualities of installability, usability and visibility)); vi) Rank the software using the Analytic Hierarchy Process (AHP); vii) Interview developers (the interview consists of 20 questions and takes about an hour); and, viii) Conduct a domain analysis. The collected data is analyzed by: i) comparing the ranking by best practices against the ranking by popularity; ii) comparing artifacts, tools and processes to current research software development guidelines; and, iii) exploring pain points. We estimate the time to complete an assessment at 173 person hours. The method is illustrated via the example of Lattice Boltzmann Solvers, where we find that the top packages engaged in most of recommended best practices, but still show room for improvement with respect to providing API documentation, a roadmap, a code of conduct, programming style guide and continuous integration.