Decarbonization processes around the world are pressing system operation significantly, particularly in terms of ensuring appropriate reliability and stability levels when integrating massive amounts of converter-based generation technologies (CGTs). In this context, this work proposes an optimization-simulation model to optimize system operation with nadir considerations via simulation of detailed system dynamic responses. We propose a Gauss–Seidel iterative method to incorporate the precise governors’ dynamic responses determined by the simulations within the sequential optimization problems. Through several case studies, we demonstrate the benefits of our approach and analyze the impacts of incorporating a detailed representation of units’ dynamic responses. We also compare our frequency nadir-constrained approach against an alternative inertia-constrained approach that ensures minimum amounts of inertia levels. Particularly, we show that ensuring minimum amounts of inertia may lead to inefficient economic outputs in terms of costs and integration of renewable generation without security gains in terms of frequency stability compared with constraining frequency nadir.