The paper proposes an optimized leader-follower formation control for the multi-agent systems with unknown nonlinear dynamics. Usually, optimal control is designed based on the solution of the Hamilton-Jacobi-Bellman equation, but it is very difficult to solve the equation because of the unknown dynamic and inherent nonlinearity. Specifically, to multi-agent systems, it will become more complicated owing to the state coupling problem in control design. In order to achieve the optimized control, the reinforcement learning algorithm of the identifier-actor-critic architecture is implemented based on fuzzy logic system (FLS) approximators. The identifier is designed for estimating the unknown multi-agent dynamics; the actor and critic FLSs are constructed for executing control behavior and evaluating control performance, respectively. According to Lyapunov stability theory, it is proven that the desired optimizing performance can be arrived. Finally, a simulation example is carried out to further demonstrate the effectiveness of the proposed control approach.