In recent years, global garbage production has increased dramatically, and the garbage has not been treated effectively. To address such problems as the size of the current garbage classification detection model is too large, processing speed is slow, and it is not suitable for deployment to embedded terminals, this paper proposes a YOLOv4 based on lightweight feature fusion (YOLOv4-LFF). The model uses two lightweight neural networks, MobileNetV3 and GhostNet, to execute feature fusion, which is used instead of the CSPDarknet53. It serves to extract preliminary feature information from the images based on the lightweight model. To further reduce the model's size, we replace the standard convolution in PANet with the depthwise separable convolution in the model, which is used for the enhanced feature information extraction work. The final experimental results show that YOLOv4-LFF achieves 93.2% accuracy on the homemade dataset and reduces the number of model parameters to 26.5% of YOLOv4, which significantly reduces the model parameters and memory consumption. Therefore, the YOLOv4-LFF garbage classification detection model meets the requirements of edge computing devices and has theoretical research significance and practicality.