The rapid rise in popularity of Large Language Models (LLMs) with emerging capabilities has spurred public curiosity to evaluate and compare different LLMs, leading many researchers to propose their own LLM benchmarks. Noticing preliminary inadequacies in those benchmarks, we embarked on a study to critically assess 23 state-of-the-art LLM benchmarks, using our novel unified evaluation framework through the lenses of people, process, and technology, under the pillars of benchmark functionality and integrity. Our research uncovered significant limitations, including biases, difficulties in measuring genuine reasoning, adaptability, implementation inconsistencies, prompt engineering complexity, evaluator diversity, and the overlooking of cultural and ideological norms in one comprehensive assessment. Our discussions emphasized the urgent need for standardized methodologies, regulatory certainties, and ethical guidelines in light of Artificial Intelligence (AI) advancements, including advocating for an evolution from static benchmarks to dynamic behavioral profiling to accurately capture LLMs’ complex behaviors and potential risks. Our study highlighted the necessity for a paradigm shift in LLM evaluation methodologies, underlining the importance of collaborative efforts for the development of universally accepted benchmarks and the enhancement of AI systems’ integration into society.

The Generative Pre-trained Transformer (GPT) models, renowned for generating human-like text, occasionally produce “hallucinations” - outputs that diverge from human expectations. Current mitigation strategies for these GPT hallucinations largely rely on algorithmic automation, thereby overlooking the complexities of human judgment and cultural influence, particularly in fact interpretation. Addressing this issue, we have introduced a Culturally Sensitive Test that integrates language subjectivity, cultural nuances, and GPT idiosyncrasies. We have applied this test to five GPT models—OpenAI’s ChatGPT-3.5 and ChatGPT-4, Google’s Bard, Perplexity AI and TruthGPT - evaluating their responses to 70 questions across seven categories designed to provoke hallucinations. The evaluated models demonstrated varying performance, with controversial topics, those lacking clear scientific consensus and the brain teasers proving more susceptible to GPT hallucinations. Our study has paved the way for a nuanced assessment of GPT hallucinations.

This comprehensive survey explored the evolving landscape of generative Artificial Intelligence (AI), with a specific focus on the recent technological breakthroughs and the gathering advancements toward possible Artificial General Intelligence (AGI). It critically examined the current state and future trajectory of generative AI, exploring how innovations in developing actionable and multimodal AI agents with the ability scale their “thinking” in solving complex reasoning tasks are reshaping research priorities and applications across various domains, while the survey also offers an impact analysis on the generative AI research taxonomy. This work has assessed the computational challenges, scalability, and real-world implications of these technologies while highlighting their potential in driving significant progress in fields like healthcare, finance, and education. Our study also addressed the emerging academic challenges posed by the proliferation of both AI-themed and AI-generated preprints, examining their impact on the peer-review process and scholarly communication. The study highlighted the importance of incorporating ethical and human-centric methods in AI development, ensuring alignment with societal norms and welfare, and outlined a strategy for future AI research that focuses on a balanced and conscientious use of generative AI as its capabilities continue to scale.

Ransomware has grown to be a dominant cybersecurity threat by exfiltrating, encrypting, or destroying valuable user data and causing numerous disruptions to victims. The severity of the ransomware endemic has generated research interest from both the academia and the industry. However, many studies held stereotypical assumptions about ransomware, used unverified, outdated, and limited self-collected ransomware samples, and did not consider government strategies, industry guidelines, or cyber intelligence. We observed that ransomware no longer exists simply as an executable file or limits to encrypting files (data loss); data exfiltration (data breach) is the new norm, espionage is an emerging theme, and the industry is shifting focus from technical advancements to cyber governance and resilience. We created a ransomware innovation adoption curve, critically evaluated 212 academic studies published during 2020 and 2023, and cross-verified them against various government strategies, industry reports, and cyber intelligence on ransomware. We concluded that many studies were becoming irrelevant to the contemporary ransomware reality and called for the redirection of ransomware research to align with the continuous ransomware evolution in the industry. We proposed to address data exfiltration as priority over data encryption, to consider ransomware in a business-practical manner, and recommended research collaboration with the industry.

This study is an empirical investigation into the semantic vulnerabilities of four popular pre-trained commercial Large Language Models (LLMs) to ideological manipulation. Using tactics reminiscent of human semantic conditioning in psychology, we have induced and assessed ideological misalignments and their retention in four commercial pre-trained LLMs, in response to 30 controversial questions that spanned a broad ideological and social spectrum, encompassing both extreme left-wing and right-wing viewpoints. Such semantic vulnerabilities arise due to fundamental limitations in LLMs’ capability to comprehend detailed linguistic variations, making them susceptible to ideological manipulation through targeted semantic exploits. We observed Reinforcement Learning from Human Feedback (RLHF) in effect in LLM initial answers, but highlighted the limitations of RLHF in two aspects: (1) its inability to fully mitigate the impact of ideological conditioning prompts, leading to partial alleviation of LLM semantic vulnerabilities; (2) its inadequacy in representing a diverse set of “human values”, often reflecting the predefined values of certain groups controlling the LLMs. Our findings have provided empirical evidence of semantic vulnerabilities inherent in current LLMs, challenged both the robustness and the adequacy of RLHF as a mainstream method for aligning LLMs with human values, and underscored the need for a multidisciplinary approach in developing ethical and resilient Artificial Intelligence (AI).