This paper investigates the optimal consensus problem for heterogeneous discrete-time(DT) linear multi-agent systems. The optimal consensus problem is formulated as finding a global Nash equilibrium solution subjected to the defined local performance index. A reinforcement learning(RL) value iteration(VI) algorithm is introduced to obtain the optimal policies in the sense of Nash equilibrium. To ensure the effectiveness of the VI algorithm, the admissibility of the iterative control policies for multi-agent systems is considered. With theoretical analysis, a new termination criterion is established to guarantee the admissibility of the iterative control policies. Furthermore, an online learning framework is designed with an actor-critic neural network(NN) to implement the VI algorithm. Finally, two simulation examples are presented respectively for leader–follower and leaderless multi-agent systems to verify the effectiveness of the proposed method.