Thermally conductive and flame-retardant polyolefin composites are facing great challenges in meeting the increasing demands for fire safety and thermal management. Aiming at simultaneously enhancing thermal conductivity and flame retardancy, hexagonal boron nitride (hBN) and magnesium hydroxide (MH) were adopted in ethylene–vinyl acetate copolymer/polyolefin elastomer (EVA/POE) blends to design composites with selective filler distributions and co-continuous networks via different processing schemes. The thermal conductivity and flame retardancy show strong dependence on the distributed structure of hBN and MH. The composites with hBN-rich centers and MH-rich edges in the filled POE phase show a thermal conductivity of 0.70 W/(m·K) and an LOI of 27.7%, which are very close to the thermal conductivity of EVA/POE/hBN and the LOI of EVA/POE/MH at the same total filler content. The composites with MH-rich centers and hBN-rich edges show pHRR, THR and TSP values of 169 kW/m², 49.8 MJ/m² and 1.8 m², which are decreased by 40%, 33% and 62% in comparison with EVA/POE/MH, respectively. Modulating the filler structure distribution provides a strategy to co-enhance thermal conductivity and flame retardancy.