Stars form with a complex and highly structured distribution. For a smooth star cluster to form from these initial conditions, the star cluster must erase this substructure. We study how substructure is removed using N-body simulations that realistically handle two-body relaxation. In contrast to previous studies, we find that hierarchical cluster formation occurs chiefly as a result of scattering of stars out of clumps, and not through clump merging. Two-body relaxation, in particular within the body of a clump, can significantly increase the rate at which substructure is erased beyond that of clump merging alone. Hence the relaxation time of individual clumps is a key parameter controlling the rate at which smooth, spherical star clusters can form. The initial virial ratio of the clumps is an additional key parameter controlling the formation rate of a cluster. Reducing the initial virial ratio causes a star cluster to lose its substructure more rapidly.