Semantic control allows us to shape our conceptual retrieval to suit the circumstances in a flexible way. Tasks requiring semantic control activate a large-scale network including left inferior prefrontal gyrus (IFG) and posterior middle temporal gyrus (pMTG) - this network responds when retrieval is focussed on weak as opposed to dominant associations. However, little is known about the biological basis of individual differences in this cognitive capacity: regions that are commonly activated in task-based fMRI may not relate to variation in controlled retrieval. The current study combined analyses of MRI-based cortical thickness with resting-state fMRI connectivity to identify structural markers of individual differences in semantic control. We found that participants who performed relatively well on tests of controlled semantic retrieval showed increased structural covariance between left pMTG and left anterior middle frontal gyrus (aMFG). This pattern of structural covariance was specific to semantic control and did not predict performance when harder non-semantic judgements were contrasted with easier semantic judgements. The intrinsic functional connectivity of these two regions forming a structural covariance network overlapped with previously-described semantic control regions, including bilateral IFG and intraparietal sulcus, and left posterior temporal cortex. These results add to our knowledge of the neural basis of semantic control in three ways: (i) Semantic control performance was predicted by the structural covariance network of left pMTG, a site that is less consistently activated than left IFG across studies. (ii) Our results provide further evidence that semantic control is at least partially separable from domain-general executive control. (iii) More flexible patterns of memory retrieval occurred when pMTG co-varied with distant regions in aMFG, as opposed to nearby visual, temporal or parietal lobe regions, providing further evidence that left prefrontal and posterior temporal areas form a distributed network for semantic control.