Lepton flavor violating Higgs decays can arise in flavor symmetry models where the Higgs sector is responsible for both the electroweak and the flavor symmetry breaking. Here we advocate an S-4 three-Higgs-doublet model where tightly constrained flavor changing neutral currents are suppressed by a remnant Z(3) symmetry. A small breaking of this Z(3) symmetry can explain the 2.4 sigma excess of Higgs decay final states with a mu tau topology reported recently by CMS if the new neutral scalars are light. The model also predicts sizable rates for lepton flavor violating Higgs decays in the e tau and e mu channels because of the unifying S-4 flavor symmetry.