Aspergillus flavus contamination in peanut kernels poses significant health risks and economic losses, hence requiring accurate and fast detection methods to ensure postharvest safety and quality. This study investigated the detection of Aspergillus flavus contamination in peanut kernels using visible-near infrared (VNIR) hyperspectral imaging and hyperspectral microscopic imaging (HMI). The research explored the structural damage to peanut kernel cells and tissue caused by contamination, as revealed through both electron microscopy and hyperspectral imaging. Generalized two-dimensional correlation spectroscopy analysis was applied to determine the sequence of molecular changes, providing insights into fungal metabolism. The spatial-spectral features of the peanut kernels and peanut kernel sections were extracted, and a hybrid convolutional transformer-feature fusion network (HCT-FFN) was employed for features integration and classification. The model demonstrated superior accuracy compared to classic deep learning models, with test accuracy of 100.00 % for both VNIR hyperspectral imaging and HMI. Using smaller regions of interest in peanut kernel sections maintained high accuracy and improved the efficiency of the model. The study concluded that Aspergillus flavus contamination significantly altered peanut kernel structure and spectral properties. The HCT-FFN model proved highly effective for detecting and classifying contamination with minimal computational costs, highlighting its potential as a valuable tool for ensuring the safety and quality of postharvest nuts.