We have selected 337 intermediate- and high-mass young stellar objects (YSOs; 1.5-20 M ⊙) well-characterized with spectroscopy. By means of the clustering algorithm HDBSCAN, we study their clustering and association properties in the Gaia DR3 catalog as a function of stellar mass. We find that the lower-mass YSOs (1.5-4 M ⊙) have clustering rates of 55%-60% in Gaia astrometric space, a percentage similar to that found in the T Tauri regime. However, intermediate-mass YSOs in the range 4-10 M ⊙ show a decreasing clustering rate with stellar mass, down to 27%. We find tentative evidence suggesting that massive YSOs (>10 M ⊙) often (yet not always) appear clustered. We put forward the idea that most massive YSOs form via a mechanism that demands many low-mass stars around them. However, intermediate-mass YSOs form in a classical core-collapse T Tauri way, yet they do not appear often in the clusters around massive YSOs. We also find that intermediate- and high-mass YSOs become less clustered with decreasing disk emission and accretion rate. This points toward an evolution with time. For those sources that appear clustered, no major correlation is found between their stellar properties and the cluster sizes, number of cluster members, cluster densities, or distance to cluster centers. In doing this analysis, we report the identification of 55 new clusters. We tabulated all of the derived cluster parameters for the considered intermediate- and high-mass YSOs.