Patients with epilepsy are often able to predict seizure occurrence subsequent to an acute stress experience. However, neuroimaging investigations into the neural basis of this relationship or the potential influence of perceived life stress are limited. The current study assessed the relationship between perceived stress and the neurobehavioral response to stress in patients with left temporal lobe epilepsy (LTLE) and healthy controls (HCs) using heart rate, salivary cortisol level, and functional magnetic resonance imaging and compared these effects between HCs and LTLE. Matched on perceived stress levels, groups of 36 patients with LTLE and 36 HCs completed the Montreal Imaging Stress Task, with control and stress math task conditions. Among LTLEs, 27 reported that prior (acute) stress affected their seizures (LTLES+), while nine did not (LTLES−). The results revealed that increased perceived stress was associated with seizure frequency in LTLE. Further, cortisol secretion was greater in LTLE, but did not vary with perceived stress as observed in HCs. A linear mixed-effects analysis revealed that as perceived stress increased, activation in the hippocampal complex (parahippocampal gyrus and hippocampus) decreased during stressful math in the LTLES+, increased in HCs, but did not vary in the LTLES−. Task-based functional connectivity analyses revealed LTLE differences in hippocampal functional connectivity with sensory cortex specific to stressor modalities. We argue that the current study demonstrates an inhibitory hippocampal mechanism underlying differences in resilience to stress between HCs and LTLE, as well as LTLE patients who report stress as a precipitant of seizures.