Building energy demand prediction (BEDP) concerns sensing the environment using the Internet of Things (IoT), making seamless decisions and responding and controlling certain devices automatically, intelligently, and quickly. Typically, the BEDP application can be empowered by fog computing where the sensed data are processed at the edge nodes rather than in a central cloud. The challenge is that in this decentralized IoT environment, the machine learning algorithm implemented at the fog node must learn a model from the incoming data accurately and fast. Which type of incremental learning algorithms, combined with traditional or swarm types of stochastic feature selection methods, are more suitable for BEDP? In this article, this topic is investigated in detail by introducing a new incremental learning model, the swarm decision table (SDT) in comparison with the classical decision tree. The simulation experiments using an empirical energy consumption data set that represent a typical IoT-connected BEDP scenario are tested, and the SDT shows superior results in terms of accuracy and time, demonstrating it as a suitable machine learning candidate in a fog computing environment.