Various techniques have been proposed to improve the level of automation in stages of architectural design. We present an interactive interface, along with optimization algorithms focused on designing early stage symbolic architecture. The architect specifies some simple shape requirements by inputting a few binary images that resemble the shape of the building from different viewpoints. Our optimizer uses these to generate a conceptual 3D design that is guided by various aesthetic and structural requirements. We also develop an approach for planning the inner space of the given architecture model. The architect specifies idealized requirements on the key functional rooms by using instances of simple 3D primitive shapes. The pose and location of each instance are parameters of the optimization. By coupling this process together with the shape inspired exterior design process, we can construct good initial conceptual designs. We develop modified cuckoo search algorithms to efficiently obtain feasible solutions via offline optimization. Several examples are presented to illustrate the methodology and results. User studies based on a proposed dataset and interviews with domain experts have been carried out to demonstrate the usability and effectiveness of our system and algorithms.