Tumors evolve in close interaction with their microenvironment, which encompasses a continual tension between the developing tumor and the host immune system. Clinical trials have shown that appropriate enhancement of a tumor immune response can lead to long-lasting clinical responses and patient benefit. Understanding the contribution of the immune contexture, in addition to the molecular subtype across different tumor indications, is a significant knowledge gap with limited sagacity to drive rational immunotherapy combinations. To better inform clinical studies, we must first strive to understand the multifaceted elements of the tumor-immune interaction, the spatiotemporal interplay of numerous different immune cell types, in conjunction with an understanding of the oncogenic drivers and mutations that may lead to presentation of neoepitopes and could drive changes within the tumor microenvironment. In this review, we discuss the Immunoscore and its probable universal characteristic. The overlay of immune quantification with the molecular segments of disease and how this may benefit identification of patients at high risk of tumor recurrence will be discussed. The Immunoscore may translate to provide a tumor agnostic method to define immune fitness of a given tumor and predict and stratify patients who will benefit from certain therapies (in particular immune therapies) and, ultimately, help save the lives of patients with cancer.