Single-cell RNA-sequencing (scRNA-seq) is a powerful technique for profiling the transcriptome at the single cell resolution and has gained tremendous popularity since its emergence in 2009. In recent years, there has been an increasing number of simulation tools designed specifically for simulating scRNA-seq data. For simulation data to be useful to aid in the development of analytical algorithms, simulation methods must generate a faithful and realistic representation of the scRNA-seq data. Using a systematic framework, the aim of our study is to evaluate each method at capturing the underlying biological structure of scRNA-seq datasets. In our evaluation framework, we evaluate a total of 12 simulation methods across 35 diverse datasets with a variety of tissue types, biological conditions and sequencing platforms. We discover that some measures are harder to capture by current methods than others and identified areas that could benefit from further methodological development.