The azimuthal asymmetry of direct photons originating from primary hard scatterings between partons is calculated. This can be accounted for by the inclusion of the color dipole orientation, which is sensitive to the rapid variation of the nuclear profile. To this end we introduce the dipole orientation within the saturation model of Golec-Biernat and Wusthoff, while preserving all its features at the cross-section level. We show that the direct photon elliptic anisotropy v2 coming from this mechanism changes sign and becomes negative for peripheral collisions, albeit it is quite small for nuclear collisions at the RHIC energy.