Methodology for Assessing the State of the Practice for Domain X
Journal Articles
Overview
Research
View All
Overview
abstract
To improve software development methods and tools for research software, we
first need to understand the current state of the practice. Therefore, we have
developed a methodology for assessing the state of the software development
practices for a given research software domain. For each domain we wish to
answer questions such as: i) What artifacts (documents, code, test cases, etc.)
are present? ii) What tools are used? iii) What principles, process and
methodologies are used? iv) What are the pain points for developers? v) What
actions are used to improve qualities like maintainability and reproducibility?
To answer these questions, our methodology prescribes the following steps: i)
Identify the domain; ii) Identify a list of candidate software packages; iii)
Filter the list to a length of about 30 packages; iv) Gather source code and
documentation for each package; v) Collect repository related data on each
software package, like number of stars, number of open issues, number of lines
of code; vi) Fill in the measurement template (the template consists of 108
questions to assess 9 qualities (including the qualities of installability,
usability and visibility)); vii) Interview developers (the interview consists
of 20 questions and takes about an hour); viii) Rank the software using the
Analytic Hierarchy Process (AHP); and, ix) Analyze the data to answer the
questions posed above. A domain expert should be engaged throughout the
process, to ensure that implicit information about the domain is properly
represented and to assist with conducting an analysis of the commonalities and
variabilities between the 30 selected packages. Using our methodology,
spreadsheet templates and AHP tool, we estimate (based on our experience with
using the process) the time to complete an assessment for a given domain at 173
person hours.
