Owing to their tremendous diversity and plasticity, immune cells exert multifaceted functions in tumor-bearing hosts, ranging from anti-tumor to pro-tumor activities. Tumor immune landscapes differ greatly between and within cancer types. Emerging evidence suggests that genetic aberrations in cancer cells dictate the immune contexture of tumors. Here, we review the current understanding of the mechanisms whereby common drivers of tumorigenesis modulate the tumor immune milieu. We discuss these findings in the context of clinical observations and examine how cancer-cell-intrinsic properties can be exploited to maximize the benefit of immunomodulatory therapies. Understanding the relationship between cancer cell-intrinsic genetic events and the immune response may enable personalized immune intervention strategies for cancer patients.