This paper studies the joint fleet sizing and charging system planning problem for a company operating a fleet of autonomous electric vehicles (AEVs) for passenger and goods transportation. Most of the relevant published papers focus on intracity scenarios and adopt heuristic approaches, e.g., agent based simulation, which do not guarantee optimality. In contrast, we propose a mixed integer linear programming model for intercity scenarios. This model incorporates comprehensive considerations of 1) limited AEV driving range; 2) optimal AEV routing and relocating operations; 3) time-varying origin-destination transport demands; and 4) differentiated operation cost structure of passenger and goods transportation. The proposed model can be computational expensive when the scale of the transportation network is large. We then exploit the structure of this program to expedite its solution. Numerical experiments are conducted to validate the proposed method. Our experimental results show that AEVs in passenger and goods transportation have remarkable planning and operation differences. We also demonstrate that intelligent routing and relocating operations, charging system and vehicle parameters, e.g., charging power, battery capacity, driving speed etc., can significantly affect the economic efficiency and the planning results of an AEV fleet.