Considering the fact that it is very difficult to fully model an autonomous underwater vehicle (AUV) in the complex water environment, this paper presents a model-free tracking control strategy for an AUV in the presence of unknown disturbances. We first formulate an optimized control problem by defining a tracking Hamilton–Jacobi–Isaac (HJI) equation. Then, we present a reinforcement learning (RL) algorithm to compute an optimized solution by learning from the HJI equation online. It is noted that during the learning period, no information about the AUV's dynamics is needed. In order to demonstrate the efficiency of the proposed strategy, numerical simulation is considered, results are validated and discussed.