Optical nonlinearities are fundamental in several types of optical information processing protocols. However, the high laser intensities needed for implementing phase nonlinearities using conventional optical materials represent a challenge for nonlinear optics in the few-photon regime. We introduce an infrared cavity quantum electrodynamics (QED) approach for imprinting nonlinear phase shifts on individual THz pulses in reflection setups, conditional on the input power. Power-dependent phase shifts on the order of 0.1 pi can be achieved with femtosecond pulses of only a few mu W input power. The proposed scheme involves a small number of intersubband quantum well transition dipoles evanescently coupled to the near field of an infrared resonator. The field evolution is nonlinear due to the dynamical transfer of spectral anharmonicity from material dipoles to the infrared vacuum, through an effective dipolar chirping mechanism that transiently detunes the quantum well transitions from the vacuum field, leading to photon blockade. We develop analytical theory that describes the dependence of the imprinted nonlinear phase shift on relevant physical parameters. For a pair of quantum well dipoles, the phase control scheme is shown to be robust with respect to inhomogeneities in the dipole transition frequencies and relaxation rates. Numerical results based on the Lindblad quantum master equation validate the theory in the regime where the material dipoles are populated up to the second excitation manifold. In contrast with conventional QED schemes for phase control that require strong light-matter interaction, the proposed phase nonlinearity works best in weak coupling, increasing the prospects for its experimental realization using current nanophotonic technology.