One particular strength of the boundary element method is that it allows for a high-order pointwise approximation of the solution of the related partial differential equation via the representation formula. However, the high-order convergence and hence accuracy usually suffers from singularities of the Cauchy data. We propose two adaptive mesh-refining algorithms and prove their quasi-optimal convergence behavior with respect to an a posteriori computable bound for the point error in the representation formula. Numerical examples for the weakly-singular integral equations for the 2D and 3D Laplacian underline our theoretical findings.