Cholesterol and its oxidized forms, oxysterols, are ingested from foods and are synthesized de novo. Cholesterol and oxysterols influence molecular and cellular events and subsequent biological responses of immune cells. The amount of dietary cholesterol influence on the levels of LDL cholesterol and blood oxysterols plays a significant role in the induction of pro-inflammatory state in immune cells, leading to inflammatory disorders, including cardiovascular disease. Cholesterol and oxysterols synthesized de novo in immune cells and stroma cells are involved in immune homeostasis, which may also be influenced by an excess intake of dietary cholesterol. Dietary compounds such as β-glucan, plant sterols/stanols, omega-3 lipids, polyphenols, and soy proteins, could lower blood cholesterol levels by interfering with cholesterol absorption and metabolism. Such dietary compounds also have potential to exert immune modulation through diverse mechanisms. This review addresses current knowledge about the impact of dietary-derived and de novo synthesized cholesterol and oxysterols on the immune system. Possible immunomodulatory mechanisms elicited by cholesterol-lowering dietary compounds are also discussed.