Silicone rubber (SR) composites are candidate radiation-shielding materials, but their irradiation and annealing responses remain incompletely understood. Three predefined formulations containing boron carbide (B₄C), lead (Pb)/B₄C, or Pb were exposed to 100 kGy gamma irradiation in air and then annealed under nitrogen at 60–120 °C for 30–240 min. Attenuated total reflectance Fourier-transform infrared (ATR-FTIR) spectra retained the same characteristic band positions after irradiation, although their normalized band areas changed. Shore A hardness increased after irradiation in all formulations and remained above the corresponding pre-irradiation values after annealing. Electron paramagnetic resonance (EPR) spectra of the SR-B₄C formulation varied qualitatively with the annealing conditions, but absolute radical quantification and assignment of a unique reaction pathway were not possible. Because filler identity, filler loading, and silicone content varied simultaneously, differences among the formulations cannot be independently attributed to Pb, B₄C, or filler–matrix interactions. Based on one independently prepared parent specimen per formulation, these results describe irradiation and annealing responses in composites with micrometer-sized fillers. They may inform future comparisons with silicone-based shielding nanocomposites under matched compositions and treatment conditions.