Securing the Future: Building Resilient Supply Chains for LLMs

In the rapidly evolving landscape of artificial intelligence, Large Language Models (LLMs) have become indispensable tools across various industries. However, the complexity and reliance on third-party components in their supply chains introduce significant vulnerabilities. The integrity of LLMs can be compromised through various supply chain risks, including tampering with training data, exploiting outdated components, and manipulating pre-trained models. As organizations increasingly depend on open-access LLMs and advanced fine-tuning techniques like LoRA and PEFT, the attack surface expands, making these systems susceptible to biases, security breaches, and operational failures. Understanding and mitigating these supply chain vulnerabilities is crucial to ensuring the reliability and security of LLM applications. By implementing robust vetting processes, maintaining comprehensive inventories, and applying stringent security measures, organizations can safeguard their LLM supply chains against potential threats and maintain the trust and efficacy of their AI-driven solutions.

LLM03:2025 Supply Chain

Overview: The supply chain for Large Language Models (LLMs) is vulnerable to a range of risks that can compromise the integrity of training data, models, and deployment platforms. These vulnerabilities can lead to biased outputs, security breaches, or system failures. Unlike traditional software vulnerabilities that focus on code flaws and dependencies, machine learning introduces additional risks related to third-party pre-trained models and data.

Risks: External elements in the LLM supply chain can be manipulated through tampering or poisoning attacks. The creation of LLMs often relies on third-party models, and the rise of open-access LLMs and new fine-tuning methods like “LoRA” (Low-Rank Adaptation) and “PEFT” (Parameter-Efficient Fine-Tuning) on platforms like Hugging Face introduce new supply chain risks. Additionally, the emergence of on-device LLMs increases the attack surface and supply chain vulnerabilities for LLM applications.

Common Examples of Risks:

1. Traditional Third-party Package Vulnerabilities: Outdated or deprecated components can be exploited by attackers to compromise LLM applications, similar to “A06:2021 — Vulnerable and Outdated Components.”
2. Licensing Risks: AI development involves diverse software and dataset licenses, which can create risks if not properly managed. Different licenses impose varying legal requirements, and dataset licenses may restrict usage, distribution, or commercialization.
3. Outdated or Deprecated Models: Using models that are no longer maintained can lead to security issues.
4. Vulnerable Pre-Trained Models: Pre-trained models, being binary black boxes, offer little security assurance through static inspection. They can contain hidden biases, backdoors, or other malicious features that may not be identified through safety evaluations.
5. Weak Model Provenance: There are currently no strong provenance assurances for published models. Model Cards provide information but do not guarantee the origin of the model. An attacker could compromise a supplier account or create a similar one to infiltrate the supply chain.
6. Vulnerable LoRA Adapters: LoRA, a popular fine-tuning technique, enhances modularity but can introduce risks if a malicious adapter compromises the integrity of the pre-trained base model.
7. Exploiting Collaborative Development Processes: Shared environments for model merging and handling can be exploited to introduce vulnerabilities. Model merging is popular on platforms like Hugging Face and can be used to bypass reviews.
8. LLM Model on Device Supply-Chain Vulnerabilities: On-device LLMs increase the attack surface through compromised manufacturing processes and exploitation of device OS or firmware vulnerabilities.
9. Unclear T&Cs and Data Privacy Policies: Unclear terms and privacy policies can lead to the use of sensitive data for model training, risking exposure.

Prevention and Mitigation Strategies:

• Vet Data Sources and Suppliers: Carefully vet data sources and suppliers, including their terms and privacy policies. Use only trusted suppliers and regularly review their security posture.
• Apply OWASP Mitigations: Follow the OWASP Top Ten’s “A06:2021 — Vulnerable and Outdated Components” for vulnerability scanning, management, and patching.
• AI Red Teaming and Evaluations: Conduct comprehensive AI Red Teaming and evaluations when selecting third-party models. Use benchmarks like Decoding Trust to assess model trustworthiness.
• Maintain a Software Bill of Materials (SBOM): Keep an up-to-date inventory of components to prevent tampering and quickly detect new vulnerabilities.
• Manage Licensing Risks: Create an inventory of licenses and conduct regular audits to ensure compliance and transparency.
• Use Verifiable Models: Only use models from verifiable sources and employ third-party integrity checks with signing and file hashes.
• Monitor Collaborative Environments: Implement strict monitoring and auditing practices for collaborative development environments.
• Anomaly Detection and Robustness Testing: Use anomaly detection and adversarial robustness tests to detect tampering and poisoning.
• Implement Patching Policies: Ensure applications rely on maintained versions of APIs and underlying models.
• Encrypt Models at AI Edge: Use integrity checks and vendor attestation APIs to prevent tampered apps and models.

Sample Attack Scenarios:

1. Vulnerable Python Library: An attacker exploits a vulnerable Python library to compromise an LLM app, similar to the first Open AI data breach.
2. Direct Tampering: An attacker directly tampers with a model to spread misinformation, as seen with PoisonGPT.
3. Finetuning Popular Model: An attacker finetunes a popular model to remove safety features and deploys it on Hugging Face.
4. Pre-Trained Models: A compromised pre-trained model introduces malicious code, leading to biased outputs.
5. Compromised Third-Party Supplier: A compromised supplier provides a vulnerable LoRA adapter that is merged into an LLM.
6. Supplier Infiltration: An attacker infiltrates a supplier and compromises a LoRA adapter, providing a covert entry point into the system.
7. CloudBorne and CloudJacking Attacks: These attacks target cloud infrastructures, exploiting shared resources and vulnerabilities.
8. LeftOvers (CVE-2023–4969): An attacker exploits leaked GPU local memory to recover sensitive data.
9. WizardLM: An attacker publishes a fake version of a model with malware and backdoors.
10. Model Merge/Format Conversion Service: An attacker uses a model merge service to inject malware into a publicly available model.
11. Reverse-Engineer Mobile App: An attacker reverse-engineers a mobile app to replace the model with a tampered version.
12. Dataset Poisoning: An attacker poisons datasets to create a backdoor when fine-tuning models.
13. T&Cs and Privacy Policy: An LLM operator changes its terms and privacy policy, leading to the memorization of sensitive data.

LeftOver GPU

Yes, the “Leftover” attack on GPUs refers to a type of side-channel attack that exploits residual data left in the GPU’s memory after processing tasks. This attack takes advantage of the fact that GPUs, which are often used for high-performance computing tasks such as machine learning and graphics rendering, may not always clear their memory completely after a task is completed. As a result, sensitive data processed by the GPU can potentially be recovered by an attacker.

Key Aspects of the Leftover Attack:

1. Nature of the Attack:

• The attack targets the residual data, or “leftovers,” in the GPU’s memory. This data can include sensitive information such as cryptographic keys, personal data, or proprietary algorithms that were processed by the GPU.

Exploitation:

• An attacker with access to the GPU can attempt to read the leftover data from the memory. This can be done by executing code on the same machine or potentially through remote access if the GPU is part of a cloud infrastructure.

Implications:

• The attack poses significant risks in environments where GPUs are shared among multiple users or applications, such as in cloud computing platforms. Sensitive data from one user could be inadvertently exposed to another user who gains access to the same GPU.

Mitigation Strategies:

• Memory Scrubbing: Implementing memory scrubbing techniques to ensure that GPU memory is cleared after each task can help mitigate the risk of leftover attacks.
• Access Controls: Restricting access to GPUs and ensuring that only authorized users can execute code on them can reduce the risk of exploitation.
• Isolation: Using virtualization or containerization to isolate GPU tasks can help prevent data leakage between different users or applications.

The “Leftover” attack highlights the importance of securing GPU resources, especially in shared environments, to protect sensitive data from unintended exposure. As GPUs continue to play a critical role in high-performance computing, addressing such vulnerabilities is essential for maintaining data security and privacy.

In conclusion, as Large Language Models (LLMs) continue to transform industries with their advanced capabilities, addressing supply chain vulnerabilities becomes increasingly critical. The intricate web of third-party components, pre-trained models, and data sources that constitute the LLM supply chain presents unique challenges that traditional software security measures may not fully address. By recognizing the potential risks — ranging from outdated components and licensing issues to model tampering and data poisoning — organizations can take proactive steps to fortify their LLM supply chains. Implementing comprehensive security audits, maintaining up-to-date inventories, and fostering transparency in licensing and data usage are essential strategies for mitigating these risks. As the adoption of LLMs grows, ensuring the integrity and security of their supply chains will be paramount to unlocking their full potential while safeguarding against biases, breaches, and failures. Through diligent oversight and continuous improvement, organizations can confidently leverage LLMs to drive innovation and success in a secure and responsible manner.