The point of attachment of spindle microtubules to metaphase chromosomes is known as the centromere. Plant and animal centromeres are epigenetically specified by a centromere-specific variant of Histone H3, CENH3 (a.k.a. CENP-A). Unlike canonical histones that are invariant, CENH3 proteins are accumulating substitutions at an accelerated rate. This diversification of CENH3 is a conundrum since its role as the key determinant of centromere identity remains a constant across species. Here, we ask whether naturally occurring divergence in CENH3 has functional consequences. We performed functional complementation assays on cenh3-1, a null mutation in Arabidopsis thaliana, using untagged CENH3s from increasingly distant relatives. Contrary to previous results using GFP-tagged CENH3, we find that the essential functions of CENH3 are conserved across a broad evolutionary landscape. CENH3 from a species as distant as the monocot Zea mays can functionally replace A. thaliana CENH3. Plants expressing variant CENH3s that are fertile when selfed show dramatic segregation errors when crossed to a wild-type individual. The progeny of this cross include hybrid diploids, aneuploids with novel genetic rearrangements and haploids that inherit only the genome of the wild-type parent. Importantly, it is always chromosomes from the plant expressing the divergent CENH3 that missegregate. Using chimeras, we show that it is divergence in the fast-evolving N-terminal tail of CENH3 that is causing segregation errors and genome elimination. Furthermore, we analyzed N-terminal tail sequences from plant CENH3s and discovered a modular pattern of sequence conservation. From this we hypothesize that while the essential functions of CENH3 are largely conserved, the N-terminal tail is evolving to adapt to lineage-specific centromeric constraints. Our results demonstrate that this lineage-specific evolution of CENH3 causes inviability and sterility of progeny in crosses, at the same time producing karyotypic variation. Thus, CENH3 evolution can contribute to postzygotic reproductive barriers. As populations evolve into new species they acquire mutations that are compatible with their own genetic background, but often lead to defects when crossed to others. Here, we show that naturally evolved differences in the centromere-specific histone H3 (CENH3) can contribute to this process. Unlike canonical histones, CENH3 differentiates rapidly even between closely related species. To better understand the functional role of natural CENH3 variation, we complemented a null allele of Arabidopsis with progressively more distant orthologs. Contrary to previous findings, we discovered that all tested variants, even the highly diverged maize CENH3, could restore normal growth and reproduction in selfing individuals. However, when crossed to the wild type, hybrid progeny suffered from extensive mis-segregation. Genotypes include simple aneuploids, novel genetic rearrangements, and in extreme cases haploids where all the chromosomes from one parent are lost. This indicates that while wide variation in CENH3 is compatible with its essential function, epigenetically different centromeres do not function well when brought together in a hybrid embryo. A better understanding of haploid generation would have profound effects on plant breeding and our results suggest that the natural variation of CENH3 could offer a cache of testable variation.