The structure of human cortical bone evolves over multiple length scales from its basic constituents of collagen and hydroxyapatite at the nanoscale to osteonal structures at near-millimeter dimensions, which all provide the basis for its mechanical properties. To resist fracture, bone’s toughness is derived intrinsically through plasticity (e.g., fibrillar sliding) at structural scales typically below a micrometer and extrinsically (i.e., during crack growth) through mechanisms (e.g., crack deflection/bridging) generated at larger structural scales. Biological factors such as aging lead to a markedly increased fracture risk, which is often associated with an age-related loss in bone mass (bone quantity). However, we find that age-related structural changes can significantly degrade the fracture resistance (bone quality) over multiple length scales. Using in situ small-angle X-ray scattering and wide-angle X-ray diffraction to characterize submicrometer structural changes and synchrotron X-ray computed tomography and in situ fracture-toughness measurements in the scanning electron microscope to characterize effects at micrometer scales, we show how these age-related structural changes at differing size scales degrade both the intrinsic and extrinsic toughness of bone. Specifically, we attribute the loss in toughness to increased nonenzymatic collagen cross-linking, which suppresses plasticity at nanoscale dimensions, and to an increased osteonal density, which limits the potency of crack-bridging mechanisms at micrometer scales. The link between these processes is that the increased stiffness of the cross-linked collagen requires energy to be absorbed by “plastic” deformation at higher structural levels, which occurs by the process of microcracking.