Based on Takenaka–Malmquist (TM) system, a new nonparametric estimator for probability density function is proposed. The TM estimation method is completely different from the existent density estimation methods in that the estimator depends on an approximate system with poles in a complex plane. Compared with the classic Fourier estimator, the TM estimator will offer more flexibility and adaptivity for real data due to the poles and nonlinearity of the phase of TM system. We compare the TM estimator with kernel, wavelet, and spline estimators by simulations. It shows that the introduced TM estimator is a more promising method than the existing and commonly used methods. Copyright © 2016 John Wiley & Sons, Ltd.