Measurements of exclusive J/ψ, ψ(2s), and electron-positron (e+e-) pair photoproduction in Au+Au ultraperipheral collisions are reported by the STAR experiment at sNN=200GeV. We report several first measurements at the BNL Relativistic Heavy Ion Collider, which are (i) J/ψ photoproduction with large momentum transfer up to 2.2(GeV/c)2, (ii) coherent J/ψ photoproduction associated with neutron emissions from nuclear breakup, (iii) the rapidity dependence of incoherent J/ψ photoproduction, (iv) the ψ(2s) photoproduction cross section at midrapidity, and (v) e+e- pair photoproduction up to high invariant mass of 6GeV/c2. For measurement (ii), the coherent J/ψ total cross section of γ+Au→J/ψ+Au as a function of the center-of-mass energy WγN has been obtained without photon energy ambiguities. The data are quantitatively compared with the Monte Carlo models STARlight, Sartre, BeAGLE, and theoretical calculations of gluon saturation with color glass condensate, nuclear shadowing with leading twist approximation, quantum electrodynamics, and the next-to-leading-order perturbative QCD. At the photon-nucleon center-of-mass energy of 25.0 GeV, the coherent and incoherent J/ψ cross sections of Au nuclei are found to be 71%±10% and 36%±7%, respectively, of that of free protons. These data provide an important experimental constraint for nuclear parton distribution functions and a unique opportunity to advance the understanding of the nuclear modification effect at the top RHIC energy.