Satellite observations show that the Arctic sea ice melt season is getting longer. This lengthening has important implications for the Arctic Ocean's radiation budget, marine ecology and accessibility. Here we assess how passive microwave satellite observations of the melt season can be used for climate model evaluation. By using the Community Earth System Model Large Ensemble (CESM LE), we evaluate the effect of multiple possible definitions of melt onset, freeze onset and melt season length on comparisons with passive microwave satellite data, while taking into account the impacts of internal variability. We find that within the CESM LE, melt onset shows a higher sensitivity to definition choices than freeze onset, while freeze onset is more greatly impacted by internal variability. The CESM LE accurately simulates that the trend in freeze onset largely drives the observed pan-Arctic trend in melt season length. Under RCP8.5 forcing, the CESM LE projects that freeze onset dates will continue to shift later, leading to a pan-Arctic average melt season length of 7-9 months by the end of the 21st century. However, none of the available model definitions produce trends in the pan-Arctic melt season length as large as seen in passive microwave observations. This suggests a model bias, which might be a factor in the generally underestimated response of sea ice loss to global warming in the CESM LE. Overall, our results show that the choice of model melt season definition is highly dependent on the question posed, and none of the definitions exactly match the physics underlying the passive microwave observations.