In the wireless communication, deep reinforcement learning (DRL) techniques promise performance optimizations at a low cost. Considering the time-varying property of the wireless downlink channels, this paper proposes a deep deterministic policy gradient (DDPG) approach and a hierarchical DDPG (h-DDPG) approach to optimize the sum-rate at the user equipment (UE) side, by jointly designing the power control and the beam-forming at the base station (BS). Our results demonstrate that the proposed DDPG enables continuous data representation through the deterministic policy functions, while the proposed h-DDPG is able to mitigate the sparse reward problem. Both of the two DRL algorithms are superior to the conventional deep Q-learning (DQN) algorithm, in terms of improving the communication performance over the time-varying wireless downlink channels.