Timber construction experiences a growing trend in different countries due to its inherent environmental benefits and proven lateral load performance. However, most of the previous studies on structural and seismic performance have focused on undamaged structures without any signs of deterioration. This paper focuses on the analysis of the effects of the initial damage state on the seismic response and fragility of a five-story CLT building designed under a force-based approach. A detailed 3D finite element model was developed and validated through experimental data in order to perform incremental dynamic analyses that considered different arbitrarily imposed initial damage states. The residual response and the fragility functions are analyzed to characterize the impact of the initial state on seismic behavior. The results of this work highlight the need to properly consider the effect of previous load actions for the seismic performance evaluation during the operating life of CLT structures. Findings suggest that the initial state can significantly modify the probability of reaching a given limit state. Moreover, it was found that if the initial damage is defined as severe, the collapse margin ratio is reduced by 58.8% compared to the case in which the initial state is undamaged.