This article presents an instrument for evaluating the quality of news aggregators. For its creation, it takes as a reference the set formed by the seven Chinese news aggregators with the largest number of active users. These platforms have been studied applying heuristic evaluation and expert analysis techniques. In this way, the characteristics that give quality to these applications have been identified, formalized and operationalized through the creation of a new evaluation instrument. This instrument is presented as a checklist, characterized by the high number of quality factors that it analyses. In total, the checklist is made up of 8 parameters and 73 indicators, all of them grouped around three quality dimensions: user experience, content and business model. It is concluded that these three dimensions of quality are key to identify and understand the characteristics that, from the point of view of journalistic sustainability, make these news aggregators successful.