Children’s early language exposure impacts their later linguistic skills, cognitive abilities, and academic achievement, and large disparities in language exposure are associated with family socioeconomic status (SES). However, there is little evidence about the neural mechanisms underlying the relation between language experience and linguistic and cognitive development. Here, language experience was measured from home audio recordings of 36 SES-diverse 4- to 6-year-old children. During a story-listening functional MRI task, children who had experienced more conversational turns with adults—independently of SES, IQ, and adult-child utterances alone—exhibited greater left inferior frontal (Broca’s area) activation, which significantly explained the relation between children’s language exposure and verbal skill. This is the first evidence directly relating children’s language environments with neural language processing, specifying both an environmental and a neural mechanism underlying SES disparities in children’s language skills. Furthermore, results suggest that conversational experience impacts neural language processing over and above SES or the sheer quantity of words heard.