In this paper we consider general port-Hamiltonian formulations of multi-dimensional Maxwell's viscoelastic fluids. Two different cases are considered to describe the energy fluxes in isentropic compressible and incompressible fluids. In the compressible case, the viscoelastic effects of shear and dilatational strains on the stress tensor are described individually through the corresponding constitutive equations. In the incompressible case, an approach based on the bulk modulus definition is proposed in order to obtain an appropriate characterization, from the port-Hamiltonian point of view, of the pressure and nonlinear terms in the momentum equation, associated with both dynamic pressure and vorticity of the flow. Copyright (C) 2021 The Authors.