A Deep Dive into OWASP Top 10 for LLM Applications

Artificial Intelligence

July 29, 2024 . 4 Min Read

In the ever-evolving technological sphere, the incorporation of Large Language Models (LLMs) and Generative AI (GAI) into applications is experiencing a notable surge in prevalence. A prime illustration in this arena is evident in OpenAI's GPT (Generative Pre-trained Transformer) series. GPT models undergo initial pre-training on diverse datasets, enabling them to absorb the nuances of language, grammar, context, and even factual information. Subsequently, these models can be fine-tuned to address specific tasks or applications. While the utilization of such models holds the promise of remarkable advantages in terms of automation and efficiency, it concurrently introduces unique security challenges. The widespread adoption of Large Language Models (LLMs) is gaining traction across diverse industries and sectors. Prominent instances include OpenAI's GPT-3 and analogous models, renowned for their proficiency in comprehending and generating human-like text across various contexts. This pervasive adoption underscores the expanding influence of LLMs while recognizing the associated intricacies and security considerations within the swiftly evolving technological landscape.
The most recent development, from OWASP is the Top 10 List for Large Language Models, serves as a strategic blueprint crafted to fortify the defenses of architects, testers, developers, designers, and managers operating within the domain of language models.

OWASPs Top 10 for Large Language Models are as follows:

When malicious actors exploit the functions of a reliable large language model through the manipulation of meticulously crafted input prompts, whether done directly or indirectly through various channels, prompt injection vulnerabilities arise. This subversion of the LLM often goes undetected due to the inherent trust placed in its output. Consequences stemming from these particular LLM vulnerabilities may involve the exposure of sensitive information and the execution of unauthorized actions, all transpiring without triggering alerts in the user security system.

LLM01: Prompt Injection

When malicious actors exploit the functions of a reliable large language model through the manipulation of meticulously crafted input prompts, whether done directly or indirectly through various channels, prompt injection vulnerabilities arise. This subversion of the LLM often goes undetected due to the inherent trust placed in its output.Consequences stemming from these particular LLM vulnerabilities may involve the exposure of sensitive information and the execution of unauthorized actions, all transpiring without triggering alerts in the user security system.

Mitigating Prompt Injection:

Constrain the privileges of an LLM to the bare minimum essential for its designated functionality.
Forge dependable relationships among the LLM, external sources, and its intended functionality.
Strengthen input validation by employing methodologies that curtail potential unauthorized prompt inputs originating from unfamiliar sources.

LLM02: Insecure Output Handling

Insecure output handling occurs when an application indiscriminately accepts output from a Large Language Model (LLM) without proper scrutiny, enabling it to directly reach the backend systems. Since content generated by Large Language Models can be manipulated through prompt input, this action is akin to granting users indirect access to additional functionality. In certain scenarios, this leads to vulnerabilities such as XSS, CSRF, SSRF, and remote code execution on backend systems.

Remedial Measures for Insecure Output Handling:

Treat the output from the model with the same caution as any other untrusted user content and implement appropriate input validation on responses.
Encode the output from the model before presenting it to users to mitigate the risk of undesired code interpretations.

LLM03: Training Data Poisoning

The occurrence of training data poisoning arises when data manipulation is deliberately generated to introduce vulnerabilities and backdoors within the Large Language Model (LLM).This nefarious practice could severely compromise the security, functionality, and ethical conduct of the ML model, leading to the exposure of users to false results or misinformation.

Mitigation Strategies for Training Data Poisoning:

Scrutinize the supply chain of the training data and verify the legitimacy of both external and internal data sources.
Incorporate LLM vulnerability scans into the testing phases of the LLM's lifecycle to identify and address potential threats.
Ensure the availability of sufficient sandboxing to prevent the model from unintentionally accessing and incorporating data from unapproved sources.

LLM04: Denial of Service

Denial of Service occurs when assailants strategically engage with an LLM, leading to substantial resource consumption and consequent service degradation or an escalation in costs. The heightened usage of LLMs across diverse applications, coupled with the intensive resource requirements, amplifies the severity of Denial of Service incidents.

Countermeasures for Denial of Service:

Restrict the overall number of actions in a system responding to LLM outputs.
Set a cap on resource utilization per request to mitigate the impact of potential Denial of Service attacks.
Impose limits on the number of queued actions.

LLM05: Supply Chain

Vulnerabilities within the supply chain of large language models can jeopardize the integrity of training data, machine learning models, and deployment platforms, culminating in compromised outcomes, system failures, and potential security breaches. Traditionally, supply chain vulnerabilities have been predominantly associated with third-party software components, but now, they have extended into the realm of artificial intelligence (AI).

Mitigating Strategies for Supply Chain:

Integrate adversarial robustness training to identify and address extraction queries.
Conduct thorough auditing processes.
Inspect and validate sources and suppliers to enhance supply chain security.

LLM06: Permission Issues

In scenarios where authorization is not meticulously monitored within the plugins and treats every Large Language Model (LLM) content as if it were entirely user-created, there is a risk of initiating commands without proper authorization. This situation could potentially result in privilege escalation, compromise of confidentiality, and reliance on accessible plugins.

Addressing Permission Issues:

Safeguard against the invocation of sensitive plugins following any other plugins.
Introduce manual authorization for any actions executed by sensitive plugins.
Restrict the calling of more than one plugin with each user input to mitigate potential permission-related concerns.

LLM07: Data Leakage

Data Leakage or unintentional disclosure of sensitive information happens when Large Language Models (LLMs) inadvertently reveal confidential data and proprietary algorithms in their responses. This can result in privacy invasion and security breaches, emphasizing the importance for users of LLM applications to understand how to interact and identify potential risks associated with LLMs.

Mitigation Strategies for Data Leakage:

Sustain continuous supply chain risk mitigation through methodologies like Static Application Security Testing (SAST) and Software Bill of Materials (SBOM).
Conduct regular vulnerability scanning for LLMs.
Implement robust input validation and sanitization methods to enhance data security and privacy.
Incorporate effective data sanitization and scrubbing techniques.

LLM08: Excessive Agency

Excessive Agency vulnerability, as implied by its name, arises due to an abundance of authorizations and functionalities. Without adequate restrictions, any unwarranted operation of Large Language Models (LLMs) can lead to unintended actions.

Mitigating Strategies for Excessive Agency:

Limit permissions to the minimum necessary for LLMs.
Implement rate-limiting mechanisms to minimize the occurrence of unnecessary actions.
Introduce human approval for significant actions.

LLM09: Overreliance

Overreliance occurs when a system excessively relies on Large Language Models (LLMs) for decision-making and content generation without appropriate oversight and validation mechanisms. While LLMs can produce informative content, they may contain factual errors, potentially resulting in misinformation and misguidance.

Addressing Overreliance:

Implement automatic validation mechanisms to enhance accuracy.
Regularly monitor and review the outputs generated by LLMs.
Verify the information produced by LLMs before incorporating it into the decision-making process.

LLM10: Insecure Plugins

Plugins are crafted to link Large Language Models with external resources, enabling the utilization of unstructured text as input rather than structured and validated inputs. This opens up the possibility for cyber attackers to formulate a malicious request to the plugin, potentially resulting in undesirable consequences.

Addressing Insecure Plugins:

Whenever feasible, plugin calls should adhere to strict parameterization.
Design the plugin with a least-privileged perspective.

While extensive language models embody sophisticated and advantageous technology, it is imperative to stay alert to the risks associated with their application. The swift evolution of technology, expanding adoption, and the introduction of new tools heighten the potential for novel vulnerabilities. While the OWASP Top 10 list for LLM streamlines threat modeling for LLM-related applications, it is not exhaustive. Sustained vigilance remains crucial for detecting and mitigating emerging vulnerabilities promptly.