We have studied the optical conductivity of several intermetallic compounds of the Al-Cr-Fe system in a broad range of frequencies. Depending on composition, the crystallographic structure of these compounds evolves from that of a close approximant of the decagonal phase to a γ-brass structure, a superstructure of the β-cubic phases. For comparison, pure icosahedral samples were studied as well in the same experimental conditions. Characteristically, as for this kind of aperiodic crystal, no Drude peak was observed at low frequencies in the Al-Cr-Fe approximant samples. By contrast, such a peak was recovered in the γ-brass sample, which furthermore showed evidence of anisotropic conductivity. Using our crystallographic data, the conductivity measurements and the corresponding optical indices were interpreted on the basis of tunneling transitions. We found good agreement between these results and a theoretical approach developed for aperiodic intermetallics assuming an anomalous diffusion of the electron wavepacket.