This is a longitudinal study of the change in the academic achievement gap between African American and European American students from elementary to high schools with large administrative data from a school district in the United States. Analysis of variance between eight tracks of students defined by the school environment of isolated schools or diverse schools indicated that middle school is a critical period for closing the achievement gap and that students who stayed in diverse schools from elementary to high schools benefited the most in both reading and mathematics standardized test scores. Multilevel linear growth models show that staying in isolated elementary and middle schools has a negative impact on the students’ reading achievement and their annual growth rate in mathematics for all students regardless of race.