Soil microbial community composition is sensitive to environmental changes and can be used as a predictor of grassland degradation. However, the response of microbial communities to alpine grassland degradation has not been well elucidated. Illumina MiSeq high-throughput sequencing technology was used to analyze the composition and diversity of soil microbial communities in three degraded grassland types, namely, non-degradation, moderate degradation, and severe degradation, in the alpine grasslands of Qilian Mountains. With the deterioration of alpine grasslands, soil pH increased significantly (P < 0.01). Soil water content (SWC), soil organic matter (SOC), total nitrogen (TN), total phosphorus (TP), available nitrogen (AN), and available phosphorus (AP) showed a significant decreasing trend (P < 0.01). Soil microbial biomass carbon (MBC), microbial biomass nitrogen (MBN), and microbial biomass phosphorus (MBP) significantly decreased (P < 0.01). Alpine grassland degradation significantly changed the community composition, and beta-diversity of soil bacteria and fungi changed significantly, but their alpha-diversity did not change significantly. Soil pH, SWC, AN, and MBN were the key factors limiting the changes in soil bacterial and fungal community composition. Alpine grassland degradation has changed soil microbial community composition, which should be the future focus of alpine grassland degradation restoration management.