Understanding the nature and dynamics of material defects in superconducting circuits is of paramount importance for improving qubit coherence and parameter stability and much needed for implementing large-scale quantum computing. Here we present measurements on individual highly coherent environmental two-level systems (TLS). We trace the spectral diffusion of specific TLS and demonstrate that it originates from the TLS coupling to a small number of low energy incoherent fluctuators. From the analysis of these fluctuations, we access the relevant parameters of low energy fluctuators: dipole moments, switching energies, and, more importantly, interaction energies. Our approach opens up the possibility of deducing the macroscopic observables in amorphous glassy media from direct measurements of local fluctuator dynamics at the microscopic level—a route towards substantiating commonly accepted, but so far phenomenological, models for the decohering environment.