Aggregation of graphene oxide (GO) in cement matrix greatly reduces its reinforcing efficiency to the mechanical properties of cement-based materials. However, current technology is too difficult to distinguish the aggregated GO from the complex components in hardened cement matrix, so how does the GO aggregation influence the mechanical properties of cement-based materials is still not clear. In this study, the aggregation size effect of GO on its reinforcing efficiency to cement-based materials was investigated by pre-aggregating GO in aqueous solutions by a novel extrusion method before mixing with cement particles. It was found that with the increasing aggregation size of GO from nanometers, micrometers to millimeters, its reinforcing efficiency to the compressive strength of cement paste was gradually reduced, because the intrinsic advantages of excellent mechanical properties and high specific surface area of GO faded away with the increasing size of GO. More importantly, the mechanism behind that the addition of GO always showed a stronger mechanical reinforcement to cement mortar than paste was revealed by measuring the particle size variation of aggregated GO in a simulated alkaline cementitious solution with and without sand under mixing process. The sand shearing effect on de-aggregation of GO by reducing their particle size, for the first time, was proposed. Compared with the aggregated GO with a larger size, the de-aggregated GO with a smaller size and larger specific surface area due to the sand shearing effect should be responsible for the stronger mechanical reinforcement to cement mortar. In conclusion, this study provides a new perspective to the GO reinforcing efficiency to the mechanical properties of cement-based materials in terms of aggregation size of GO and sand shearing effect.