Are you as smart as you say you are? That's the question on the minds of two leading players in the artificial intelligence (AI) sector who have called upon the public to craft questions capable of truly testing the capabilities of large language models (LLMs) like Google Gemini and OpenAI's GPT-4. Scale AI, known for its expertise in preparing the vast amounts of data necessary for training LLMs, teamed up with the Center for AI Safety (CAIS) to launch Humanity's Last Exam, aiming to engage experts and the public alike.
The initiative offers exciting prizes totaling $5,000 for the top 50 questions selected for the test. Scale and CAIS believe this is not just another challenge; it is pivotal for measuring how close we are to developing “expert-level AI systems.” Despite the notable advances LLMs have already achieved—like excelling on established tests for intelligence, mathematics, and even law—there remain questions about how meaningful these accomplishments are. After all, many of these models might have simply memorized pre-learned answers from the massive datasets they were trained on, which include much of what’s available on the internet.
This concept of testing is particularly important as AI continues to evolve. According to estimates from Epoch, it is expected by 2028, LLMs will have effectively “read” everything ever written by humankind. This presents not only opportunities but also challenges—like model collapse, where the flood of AI-generated content can degrade performance over time. This situation raises the question of how to keep testing AIs' capabilities as they develop.
Humans have traditionally relied on various tests to measure intelligence—like IQ tests, which are often criticized for being overly simplistic and failing to capture the multifaceted nature of human intellect. A similar dilemma exists for LLMs. There are numerous well-established tests for tasks ranging from summarizing text to recognizing human gestures, but these tests tend to be very specific and fail to measure the broader scope of capabilities.
At the same time, the AI-driven professional development space has seen remarkable advancements. The Josh Bersin Company recently unveiled Galileo Professional, which offers the same powerful features as its enterprise counterpart but at a more accessible price point. This subscription-based model is targeted at individual HR professionals and small teams, aiming to democratize access to AI capabilities and extensive resources developed over decades of research.
The Galileo platform provides users with tools and knowledge needed to tackle prevalent HR issues, fostering efficient recruitment processes and improved workforce management. Integrated with major HR suites like Workday, this AI assistant allows HR professionals to efficiently manage tasks and find answers to common challenges they face daily. For example, they can ask how to create development plans or improve employee retention—essentially acts as their expert guide.
When discussing the practical applications of AI, it’s evident there's significant room for optimization—not just within human resources but across various sectors, including cybersecurity. The concept of autonomous cybersecurity has been gaining traction, with AI transforming how organizations manage their security operations.
By drawing parallels between autonomous vehicles and cybersecurity, experts propose the establishment of what they call the Autonomous Security Operations Center (ASOC). Much like how drivers are gradually being relieved of complete responsibility for managing vehicles due to advancements in automation, cybersecurity analysts can also expect to oversee multiple incidents simultaneously as AI technology continues to mature.
Cybersecurity is currently explored through levels of autonomy. At Level 0, humans manage all aspects of security with minimal automation support. On the other end of the spectrum, Level 5 embodies full autonomy, where AI systems can independently handle all aspects of security management, utilizing cutting-edge technologies such as quantum computing to adapt to new threats without any human intervention.
This step-wise transition—from traditional models where threats are entirely managed by human analysts to more sophisticated systems capable of performing security duties autonomously—shows great promise for the future. Levels of automation offer powerful insights for businesses evaluating their cybersecurity strategies. Each level provides incremental benefits, allowing companies to decide how much automation is appropriate depending on their unique operational needs and capabilities.
Research indicates AI can greatly increase the processing speed and efficiency of security management, which can transform the way organizations approach their operations. With AI handling routine tasks, cybersecurity teams can reallocate their focus to strategic planning and innovations. This shift empowers analysts to engage more deeply with complex threats, enhancing both the infrastructure's resilience and adaptability.
But even with all the glitz and glamour of AI capabilities, experts caution against neglecting human oversight. While AI offers significant benefits, human expertise is still invaluable, especially when it matters most—during strategic decision-making processes and the management of exceptional situations. Balancing AI with human intellect could lead to creating security networks more responsive and adept at dealing with cyber threats.
What does all this mean for us? The future is both exciting and demanding as we strive toward fully realizing the potential of AI across multiple domains. From redefining the way we test intelligence to making workplaces smarter and cybersecurity more resilient, these advancements call for diligent attention and thoughtful integration. The technology is advancing rapidly, and the onus is on us to wield it wisely.
The myriad developments and applications of AI stretch far and wide. Preparing ourselves and our systems for this growing influx of technology seems not just prudent but necessary for ensuring we are ready to tackle the next big challenges. Engaging constructs like Humanity's Last Exam and tools like Galileo Professional are just the beginning of what promises to be transformative ecological changes inspired by artificial intelligence.
