Rare cancer cells, such as circulating tumor cells (CTCs) and cancer stem cells (CSCs), are small cell population found in cancer patients. CTCs have been recognized as tumor avatars for real-time cancer monitoring, while CSCs are the most malignant tumor cells that play a dominant role in drug resistance and metastasis. Interestingly, these two types of cells share the same surface markers, such as EpCAM, CD44, and CD133. While capturing these rare cells is available, the expansion of these cells is still challenging due to the limited cell number. These cells are susceptible to the microenvironment and lose the capability to grow in vitro, especially after an intense capturing process. A technology called patient-derived tumor organoids (PDOs) or tumoroids is a rising start in cancer modeling but the applicability is still questionable. Recently, assembloids containing multiple tumor-related cells have been developed which is one step closer to the real tumor. In this review, strategies for in vitro expansion of tumoroids are summarized implying that artificial tumor niche composed of optimized biophysical and biological cues is vital in the tumoroid generation. Tumoroids containing rare cancer cells is believed to be beneficial in the diagnosis, therapeutic regimen, and drug discovery for personalized therapy.