skDER and CiDDER: two scalable approaches for microbial genome dereplication

An abundance of microbial genomes have been sequenced in the past two decades. For fundamental comparative genomic investigations, where the goal is to determine the major gain and loss events shaping the pangenome of a species or broader taxon, it is often unnecessary and computationally onerous to include all available genomes in studies. In addition, the over-representation of specific lineages due to sampling and sequencing bias can have undesired effects on evolutionary analyses. To assist users with genomic dereplication, we developed skDER and CiDDER (https://github.com/raufs/skDER) to select a subset of representative genomes for downstream comparative genomic investigations. skDER is a nucleotide-based genomic dereplication tool that can dereplicate thousands of microbial genomes leveraging recent advances in average nucleotide identity (ANI) inference. CiDDER dereplicates microbial genomes based on saturation assessment of distinct protein-coding genes. To support usability, auxiliary functionalities are incorporated for testing the number of representative genomes resulting from applying various clustering parameters, automated downloading of genomes belonging to a bacterial species or genus, clustering non-representative genomes to their closest representative genomes and filtering plasmids and phages prior to dereplication. From benchmarking against other ANI-based dereplication tools, skDER, when run in the default mode, was efficient and achieved comparable pangenome coverage and strictly adhered to user-defined cutoffs for both ANI and aligned fraction (AF). Further, we showcase that CiDDER is a convenient alternative to ANI-based dereplication that allows users to more directly optimize the selection of representative genomes to cover a large breadth of a taxon's pangenome.