We construct an extension of f(T) gravity with the inclusion of a nonminimal torsion-matter coupling in the action. The resulting theory is a novel gravitational modification, since it is different from both f(T) gravity, as well as from the nonminimal curvature-matter-coupled theory. The cosmological application of this new theory proves to be very interesting. In particular, we obtain an effective dark energy sector whose equation-of-state parameter can be quintessence- or phantom-like, or exhibit the phantom-divide crossing, while for a large range of the model parameters the universe enters in a de Sitter, dark-energy-dominated, accelerating phase. Additionally, we can obtain early-time inflationary solutions too, and thus provide a unified description of the cosmological history.