Over recent years, there has been an increase in research interest regarding depth estimation using multiple-spectrum images from Visible-Light (VIS) and Thermal-Infrared (TIR) cameras. Depth estimation with multi-spectrum images has the potential for more reliable depth estimation under varying illumination conditions. However, most of the existing multi-spectrum depth estimation methods rely on the supervision of ground-truth depth information or other additional complex labels or equipment, resulting in limited convenience for practical application. To address these challenges, we propose a new unsupervised all-day depth estimation framework with multi-spectrum stereo images, specifically, a thermal image (acting as the right view) and a visible-light image (acting as the left view), and they are denoted as a single frame of multi-spectrum stereo sequences. The network uses multiple frames of multi-spectrum stereo images as input during training and a single frame of multi-spectrum stereo images as input for testing. We improve the depth estimation accuracy by combining the intra-spectrum temporal consistency and the cross-spectrum spatial consistency. A unique self-teacher structure is integrated to augment the quality of nighttime depth estimation using daytime image data. Simultaneously, we release a large multi-spectrum stereo dataset, which includes TIR-VIS stereo images collected during the day and night. Experimental results reveal that our method achieves a significant improvement in all-day depth estimation by effectively utilizing multi-spectrum stereo images.