Navigating Security: Opportunities and Challenges in the AI Era

The new generation of AI poses both huge opportunities and risks. While AI can open up a world of new capabilities, it also presents new security concerns, that require our focus at three levels:

1. LLMs Reliability
2. Security risks posed by GenAI
3. AI-powered security solutions

In this article, we will explore the new reality with AI in each of these areas.

LLMs Reliability

LLMs have demonstrated remarkable capabilities in natural language processing, but their reliability remains a concern. In 2023, researchers from Stanford University discovered that GPT-4 could generate highly persuasive disinformation articles that were difficult to distinguish from real news, highlighting ongoing reliability challenges with state-of-the-art language models. We see a growing number of companies addressing issues like biased outputs, hallucinations, and the potential for generating harmful content, through improved AI infrastructure like RAG and mechanisms to test and validate LLMs.

There are a few different methods enabling to test and promise the reliability of the LLMs in our usage, among them:

1. Red teaming: Actively trying to find ways to make the model produce undesirable outputs, to identify weaknesses. Companies like Anthropic, Halcyon, and Adept AI are using red teaming in their AI development processes. Startups like Haize Labs, Robust Intelligence, and Scale AI have products helping provide solutions to handle Red Teaming.

2. Oversight sampling: Regularly sampling outputs and having them reviewed by human raters for quality and safety issues. Startups like Fiddler AI provide solutions with humans in the loop to check for quality issues

3. Runtime monitoring: Analyzing model inputs and outputs in real-time to detect potential reliability issues. Guardrails AI, Galileo and TrueEra are building infrastructure for runtime monitoring of LLMs in production.

Security risks posed by GenAI

Generative AI introduces new security challenges. For example, deepfakes can produce highly realistic fake content, potentially leading to misinformation and fraud, and cybercriminals are leveraging tools like Midjourney and Stable Diffusion to generate synthetic media for social engineering attacks. Additionally, GenAI systems are especially vulnerable to unique threats:

• Prompt injection attacks attempt to craft inputs that cause the model to ignore instructions and do something else, like disclosing sensitive data. In 2023, prompt injection was used to get GPT-4 to reveal training data.
• Jailbreaking aiming to bypassing safeguards and performing unintended actions, like creating harmful outputs or giving illegal instructions.
• Model integrity erosion happening when an AI system’s performance deteriorates over time due to adversarial or unforeseen inputs, corrupting the effectiveness of of AI driven security measurements.

Companies like Flow Security (now CrowdStrike), Sentra, Protect.ai and HiddenLayers are developing solutions to protect data and models from unauthorized access and malicious activity. Cohere, Anthropic, OpenAI, Adept and others are exploring new AI architectures that are more resistant to prompt attacks and jailbreaking attempts.

AI powered security solutions

Alongside these risks, AI offers an outstanding opportunity to address security challenges like never before. AI-driven tools can enable high-quality observability, accurate detection, clear prioritization, and accelerated mitigation. Overall, AI can transform the way we handle and mitigate security risks today. Here are a few areas with significant potential for improvement in the new era of AI:

1. Anomaly and Threat Detection: LLMs are designed to analyze large amounts of data and identify anomalies more efficiently than humans. This enables the creation of better alert systems that detect fraud and security threats effectively and in real-time. For example, Noname uses AI to identify data leakage, suspicious behavior, and API security attacks, as they happen. Redcoat AI and Abnormal Security identify phishing attempts and malicious email activity.

2. Penetration Testing: AI-powered tools can be used not only to test the reliability of LLMs, as demonstrated by companies like Adept and Haize Labs, but also to perform intensive and sophisticated penetration testing on systems to identify vulnerabilities, as offered by XBOW. AI-driven simulations of cyber-attacks on networks and systems can test their resilience and train cybersecurity professionals in incident handling, regularly improving security layers.

3. Code as language: While GenAI-generated code can raise concerns among tech leaders due to potential vulnerabilities and logical flaws, LLMs can read code as if it were natural language, enabling the identification of problematic code blocks and configurations that may lead to security breaches. AI-powered tools and security-oriented LLMs like Snyk DeepCode and Codacy embody the ‘shift left’ philosophy, focusing on identifying and resolving security issues early in the development lifecycle rather than addressing them post-deployment.

4. Vulnerability Management and prioritization: AI can be highly effective in assisting engineers with intelligent security vulnerability management and prioritization. By creating a unified source of truth for existing security vulnerabilities and analyzing factors such as severity and potential impact, platforms like Wiz and Balbix offer advanced vulnerability management and prioritization, resulting in decreased engineers confusion and response time.

5. Incident Response and auto mitigation: AI can significantly enhance incident response and automated mitigation, like applying security patches and updates to vulnerable software components in real-time, reducing the time required to contain and resolve security breaches. Solutions like Palo Alto’s Cortex XSOAR, also leverage AI to speed up incident investigation, automate and expedite tedious, manual SOC work, towards the vision of mitigating risks with minimal human intervention.

While the breakthroughs in AI present exciting opportunities, it is crucial to address the risks related to AI models and security. By focusing on the reliability of LLMs, understanding the new threats posed by GenAI, and leveraging AI to enhance security measures, we can navigate this new era of technology safely. Are you building in this space? Let’s talk.

Acknowledgements: I would like to thank Pear AI Fellow Libby Meshorer for significant contributions to this post, as well as Avika Patel and Pear team members Lucy Lee Duckworth, Arash Afrakhteh, and Jill Puente for contributing.