Despite the remarkable progress of large language models (LLMs) in understanding and generating unstructured text, their application in structured data domains and their multi-role capabilities remain underexplored. In particular, utilizing LLMs to perform complex reasoning tasks on knowledge graphs (KGs) is still an emerging area with limited research. To address this gap, we propose KG-EGV, a versatile framework leveraging LLMs to perform KG-based tasks. KG-EGV consists of four core steps: sentence segmentation, graph retrieval, EGV, and backward updating, each designed to segment sentences, retrieve relevant KG components, and derive logical conclusions. EGV, a novel integrated framework for LLM inference, enables comprehensive reasoning beyond retrieval by synthesizing diverse evidence, which is often unattainable via retrieval alone due to noise or hallucinations. The framework incorporates six key stages: generation expansion, expansion evaluation, document re-ranking, re-ranking evaluation, answer generation, and answer verification. Within this framework, LLMs take on various roles, such as generator, re-ranker, evaluator, and verifier, collaboratively enhancing answer precision and logical coherence. By combining the strengths of retrieval-based and generation-based evidence, KG-EGV achieves greater flexibility and accuracy in evidence gathering and answer formulation. Extensive experiments on widely used benchmarks, including FactKG, MetaQA, NQ, WebQ, and TriviaQA, demonstrate that KG-EGV achieves state-of-the-art performance in answer accuracy and evidence quality, showcasing its potential to advance QA research and applications.