The expectation value of Wilson loop operators in three-dimensional SO(N) Chern-Simons gauge theory gives a known knot invariant: the Kauffman polynomial. Here this result is derived, at the first order, via a simple variational method. With the same procedure the skein relation for Sp(N) are also obtained. Jones polynomial arises as special cases: Sp(2), SO(-2), and SL(2, R). These results are confirmed and extended up to the second order, by means of perturbation theory, which moreover let us establish a duality relation between SO(+/- N) and Sp(-/+ N) invariants. A correspondence between the first orders in perturbation theory of SO(-2), Sp(2) or SU(2) Chern-Simons quantum holonomy's traces and the partition function of the Q = 4 Potts model is built.