| Reviewer name and names of any other individual's who aided in reviewer | Trevor Pesout |
| Do you understand and agree to our policy of having open and named reviews, and having your review included with the published manuscript. (If no, please inform the editor that you cannot review this manuscript.) | Yes |
| Is the language of sufficient quality? | Yes |
| Please add additional comments on language quality to clarify if needed | |
| Is there a clear statement of need explaining what problems the software is designed to solve and who the target audience is? | Yes |
| Additional Comments | |
| Is the source code available, and has an appropriate Open Source Initiative license <a href="https://opensource.org/licenses" target="_blank">(https://opensource.org/licenses)</a> been assigned to the code? | No |
| Additional Comments | There is no license in the github repository. |
| As Open Source Software are there guidelines on how to contribute, report issues or seek support on the code? | Yes |
| Additional Comments | Github provides issue tracking, which can be used to report issues or seek support. |
| Is the code executable? | Yes |
| Additional Comments | I needed to refer to their issues to find the modifications to the makefile necessary to compile the code. |
| Is installation/deployment sufficiently outlined in the paper and documentation, and does it proceed as outlined? | Yes |
| Additional Comments | |
| Is the documentation provided clear and user friendly? | Yes |
| Additional Comments | Sparse but sufficient. |
| Is there a clearly-stated list of dependencies, and is the core functionality of the software documented to a satisfactory level? | Yes |
| Additional Comments | |
| Have any claims of performance been sufficiently tested and compared to other commonly-used packages? | No |
| Additional Comments | In their evaluation of the fruit fly, they do not describe how misassembled_contigs and total_aligned_length_mb were determined. Total misassembly count would be a better metric of structural integrity in the assemblies than misassembled contig count. This is especially so as total number of contigs generated by the assemblers vary significantly, and the ratio of misassembled contigs to total contigs is worst for smartdenovo. There is also not any description of base-level accuracy, which should be included especially as they have a reference sequence. The runtime statistics should be presented for this sample. In their evaluation of the wild tomato (Table 2), they include hours per CPU as a metric. Total CPU hours (num_cpu x total_runtime) is the correct metric; this is likely just mislabeled. No assembly quality metrics (such as were present in the fruit fly evaluation) were presented for this sample; I don't find the size and runtime statistics alone strong enough evidence of a high-quality assembly using ONT reads. |
| Are there (ideally real world) examples demonstrating use of the software? | Yes |
| Additional Comments | |
| Is automated testing used or are there manual steps described so that the functionality of the software can be verified? | Yes |
| Additional Comments | |
| Any Additional Overall Comments to the Author | In the Introduction at L55 the authors do not cite Shasta as a current high-quality SMS assembler; I believe it should be included, especially as they refer to it in the methods at L78 and in the discussion at L211. In the Discussion at L207, the authors claim to have introduced novel algorithms which "have been useful for improving other programs." I think this claim should be supported by more examples, especially as they don't describe any algorithms shared between themselves and MECAT or Flye. In Table 2, "peek MEM/Gb" should be "peak". |
| Recommendation | Minor Revisions |