With the advances in massive computing ability and big data science, deep neural network (DNN) has been developing rapidly for different applications. However, due to its extensive computation and memory usage requirements, it calls for the design of an energy-efficient DNN accelerator with a high potential for implementation on low-end devices. Moreover, a depthwise separable convolution (DSC) layer can reduce the network complexity while sustaining classification accuracy. In this work, we propose an energy-efficient, digital signal processor (DSP)-less DSC accelerator. We design a dataflow to process the three sub-layers of the DSC layer with an end-to-end evaluation to reduce by 80.5% the repeated memory accesses from the layer-by-layer dataflow. The proposal accelerator achieves a throughput of 413.2 GOPs and an energy efficiency of 65.18 GOPs/W for the MobileNetV2. Furthermore, we can reconfigure the accelerator to evaluate a custom DSC network for the CIFAR-10 dataset with similar energy efficiency.