Previous neuroimaging studies have extensively evaluated the structural and functional connectivity of the Ventral Attention Network (VAN) and its role in reorienting attention in the presence of a salient (pop-out) stimulus. However, a detailed understanding of the “directed” functional connectivity within the VAN during the process of reorientation remains elusive. Functional magnetic resonance imaging (fMRI) studies have not adequately addressed this issue due to a lack of appropriate temporal resolution required to capture this dynamic process. The present study investigates the neural changes associated with processing salient distractors operating at a slow and a fast time scale using custom-designed experiment involving visual search on static images and dynamic motion tracking, respectively. We recorded high-density scalp electroencephalography (EEG) from healthy human volunteers, obtained saliency-specific behavioral and spectral changes during the tasks, localized the sources underlying the spectral power modulations with individual-specific structural MRI scans, reconstructed the waveforms of the sources and finally, investigated the causal relationships between the sources using spectral Granger-Geweke Causality (GGC). We found that salient stimuli processing, across tasks with varying spatio-temporal complexities, involves a characteristic modulation in the alpha frequency band which is executed primarily by the nodes of the VAN constituting the temporo-parietal junction (TPJ), the insula and the lateral prefrontal cortex (lPFC). The directed functional connectivity results further revealed the presence of bidirectional interactions among prominent nodes of right-lateralized VAN, corresponding only to the trials with saliency. Thus, our study elucidates the invariant network mechanisms for processing saliency in visual attention tasks across diverse time-scales. © 2021 The Author(s)