4 Critical LLM Security Risks in RAG Systems: Protecting Your AI’s Knowledge Base

In the race to implement AI, we’re often dazzled by seemingly endless possibilities.

But a recent security audit of Ask Astro, an open-source chatbot based on a16z’s reference LLM architecture, serves as a sobering reminder: our AI systems may be more vulnerable than we think.

Ask Astro isn’t just any chatbot. It’s a showcase of modern AI techniques, particularly Retrieval Augmented Generation (RAG), which allows AI to pull from a curated knowledge base.

It’s the kind of system many companies aspire to build. But as the audit revealed, it’s also a Pandora’s box of potential security issues.

The audit, conducted by cybersecurity firm Trail of Bits, uncovered four critical vulnerabilities:

Imagine a malicious actor posting false information, waiting for the system to ingest it, then deleting the original post.

The false data remains in the system, invisible to moderators.

This high-severity issue highlights the need for robust content verification.

A vulnerability that could potentially leak sensitive information if public and private databases share infrastructure.

This issue stems from a bug in the Weaviate Python client library’s string sanitization function. The problematic code uses a regular expression that fails to properly escape certain input patterns:

This regex incorrectly handles cases where multiple backslashes precede a quotation mark. For instance, the input \\" isn't transformed because the look-behind assertion fails.

As a result, an attacker could craft a query that prematurely terminates a string literal, potentially allowing them to inject arbitrary GraphQL syntax and retrieve data from unintended collections.

A lower-severity but insidious technique, where attackers could forge entire conversation threads, making false information appear to come from authoritative sources.

This arises from the fact that the Ask Astro implementation downloads GitHub issues during document ingestion routines and concatenates them using a markdown template. This can allow attackers to inject false information that appears to come from official or trusted sources, potentially misleading users or contaminating the AI’s knowledge base.

While rated low-severity, this could lead to resource exhaustion and potential denial of service.

This vulnerability occurs in the question expansion step, where GPT-3.5 Turbo is used to generate alternative phrasings of user questions.

An attacker could craft a malicious prompt that causes the model to generate an excessive number of questions or produce arbitrarily large outputs. This could potentially exhaust computational resources or, in a worst-case scenario, trigger a denial of service, disrupting the entire system’s operation.

These aren’t just Ask Astro’s problems. They’re symptomatic of wider issues in the AI industry, particularly in RAG systems.

The audit shines a light on the complex challenge of maintaining data integrity in AI systems that ingest information from various, often untrusted, sources.

So, what can we learn from this?

The Ask Astro audit is a wake-up call. As the boundaries of AI capabilities continue to expand, an equally innovative approach to AI security becomes crucial.

The landscape of cybersecurity is evolving beyond traditional threats. A new era is dawning where the integrity of an AI’s knowledge base is paramount to its security and reliability.

As AI continues to permeate every aspect of our digital lives, from customer service to critical decision-making systems, the stakes couldn’t be higher.

The vulnerabilities exposed in Ask Astro stand as a stark reminder: in the rush to embrace AI’s potential, security cannot be an afterthought. It must be an integral part of AI development and deployment from the very beginning.

The road ahead is challenging, but not insurmountable.

By learning from audits like this, implementing best practices, and fostering a culture of security-first AI development, we can build AI systems that are not just powerful, but trustworthy and resilient.

In the end, the lesson from Ask Astro is clear: in the world of AI, security isn’t just a feature. It’s a fundamental necessity.

Thank you for reading! If you enjoyed this post and would like to stay up to date, then please consider subscribing.

NOTE: This article was originally written and published in 2024 on my previous personal blog and on Medium.