We present an analysis of the spatial distribution and dynamics of neutral hydrogen gas around galaxies using new X-Shooter observations of z similar to 2.5-4 quasars. Adding the X-Shooter data to our existing data set of high-resolution quasar spectroscopy, we use a total sample of 29 quasars alongside similar to 1700 Lyman Break Galaxies (LBGs) in the redshift range 2 less than or similar to z less than or similar to 3.5. We measure the Ly alpha forest auto-correlation function, finding a clustering length of s(0) = 0.081 +/- 0.006 h(-1) Mpc, and the cross-correlation function with LBGs, finding a cross-clustering length of s0 = 0.27 +/- 0.14 h(-1) Mpc and power-law slope gamma = 1.1 +/- 0.2. Our results highlight the weakly clustered nature of neutral hydrogren systems in the Ly alpha forest. Building on this, we make a first analysis of the dependence of the clustering on absorber strength, finding a clear preference for stronger Ly alpha forest absorption features to be more strongly clustered around the galaxy population, suggesting that they trace on average higher mass haloes. Using the projected and 2-D cross-correlation functions, we constrain the dynamics of Ly alpha forest clouds around z similar to 3 galaxies. We find a significant detection of large-scale infall of neutral hydrogen, with a constraint on the Ly alpha forest infall parameter of beta(F) = 1.02 +/- 0.22.