In this paper, we present the cosmological perturbation formalism for theories within the framework of affine gravity. These theories are distinguished by their connection, devoid of any metric. Our approach involves segregating perturbations into symmetric and antisymmetric components (related to torsion), each further decomposed into irreducible elements, namely scalars, pseudoscalars, vectors, pseudovectors, 2-tensors, and 3-tensors. Finally, we have fully addressed the gauge freedom in this context.