# Google Vertex AI SDK Flaw Exposes ML Models to Code Hijacking via Bucket Squatting
A critical vulnerability in the Google Cloud Vertex AI SDK for Python allows unauthenticated attackers to hijack machine learning model uploads and execute arbitrary code within Google's serving infrastructure—without any prior access to a victim's Google Cloud project. Palo Alto Networks Unit 42, which discovered and reported the flaw, has dubbed the attack technique "Pickle in the Middle" and confirmed no active exploitation in the wild at the time of disclosure.
The vulnerability represents a significant risk to organizations deploying machine learning models through Vertex AI, as successful exploitation could lead to model theft, intellectual property compromise, and arbitrary code execution in a trusted cloud environment.
## The Threat
The flaw allows an attacker to intercept and modify a victim's model upload during the ML deployment process by exploiting an insecure default behavior in the Vertex AI SDK. An attacker with no credentials—not even a Google Cloud account—can hijack the upload workflow and inject malicious code that executes when the model is served or accessed.
The attack does not require:
Instead, it exploits a namespace collision vulnerability—essentially a form of "bucket squatting"—where the SDK's default configuration allows attackers to claim and control the namespace used for model uploads before legitimate users do.
## Background and Context
Vertex AI: The Target
Google Cloud's Vertex AI is a fully managed machine learning platform that enables organizations to build, train, and deploy ML models at scale. It handles the complete ML lifecycle, from data preparation through model serving, and is widely used by enterprises for predictive analytics, recommendation systems, and computer vision applications.
The platform's appeal lies in its integration with Google Cloud's broader ecosystem and its promise of simplified ML operations. However, like all deployment platforms, Vertex AI's security posture depends heavily on the SDKs and tools that developers use to interact with it.
The SDK's Role
The Vertex AI SDK for Python is the primary interface through which developers upload, configure, and deploy models. The SDK automates many steps in the deployment process, including:
This automation is convenient, but it also creates opportunities for misconfiguration or insecure defaults—exactly what Unit 42 discovered.
## Technical Details: How "Pickle in the Middle" Works
The vulnerability centers on Python's pickle serialization format, a common method for persisting ML models. Here's how the attack unfolds:
Step 1: Namespace Collision
The Vertex AI SDK uses a default storage bucket naming scheme that includes common organizational identifiers (often project-based or organization-based). An attacker can predict or enumerate likely bucket names and create a publicly-writable Cloud Storage bucket with a name that matches the default namespace the SDK will use.
Step 2: Injection Point
When a victim runs the Vertex AI SDK to upload their model, the SDK serializes the model using pickle and writes it to the default bucket location. Because the attacker has pre-claimed the bucket namespace, the victim's model file is actually written to the attacker's bucket.
Step 3: Code Injection
The attacker intercepts the uploaded pickle file and modifies it. Pickle files are not simply serialized data—they're executable instructions. An attacker can craft a malicious pickle that includes code execution directives. When the model is later deserialized (unpickled) during serving or inference, the injected code executes.
Step 4: Execution in Trusted Infrastructure
The code runs within Google Cloud's Vertex AI serving infrastructure, meaning it executes with the permissions and context of the ML serving endpoint—potentially allowing lateral movement, data exfiltration, or further attacks on the organization's cloud environment.
| Attack Phase | Attacker Action | Risk |
|---|---|---|
| Reconnaissance | Predict/enumerate default bucket names | Low effort |
| Claim | Create bucket with matching name | Minimal cost |
| Intercept | Monitor bucket; modify uploaded pickle | Access to attacker-controlled bucket |
| Execute | Model serving triggers code execution | Code runs in Vertex AI infrastructure |
## Why Pickle Is the Weak Link
The root cause is pickle's inherent design: it's a format that includes executable instructions, not just data. While this is well-known in the security community, the Vertex AI SDK's use of pickle as the default serialization method—combined with insecure bucket naming and namespace management—created the vulnerability.
Modern ML frameworks often support safer serialization formats (ONNX, SavedModel, HDF5) that don't include execution logic. The choice to rely on pickle, combined with the bucket squatting opportunity, proved critical.
## Implications for Organizations
Who Is at Risk?
Organizations using Vertex AI's Python SDK to deploy models are directly affected. This includes:
The Blast Radius
Successful exploitation could result in:
## Recommendations for Defense
Immediate Actions
1. Audit Model Deployments: Review all Vertex AI model uploads from the past 6-12 months. Verify that buckets used for uploads are secure and that you control the bucket naming.
2. Verify Bucket Ownership: Check that Cloud Storage buckets used by the Vertex AI SDK are:
- In your own Google Cloud project
- Not accessible from outside your organization
- Using appropriate IAM controls and bucket policies
3. Update the SDK: Install the latest version of the Vertex AI SDK for Python once Google releases a patch. Monitor Google Cloud Security advisories for details.
4. Regenerate and Audit Models: If you cannot verify the integrity of previous model uploads, consider retraining and redeploying models with verified code.
Longer-Term Mitigations
## HackWire Analysis
This vulnerability exemplifies a critical gap in cloud security: the assumption that "default" configurations are safe. Developers trust that cloud SDKs ship with secure defaults, but namespace collision vulnerabilities like bucket squatting are old problems in new clothes. The specific timing is noteworthy—ML adoption is accelerating, and many teams deploying models for the first time lack the security maturity to catch such flaws.
What makes "Pickle in the Middle" particularly insidious is its supply chain potential. ML models are increasingly shared between organizations: as APIs, as pre-trained weights, as containerized endpoints. A single compromised upload could cascade through multiple users if the attacker distributes the poisoned model. This is the ML equivalent of a compromised npm package, but with less visibility and fewer automated scanning tools to catch it.
The lack of observed in-the-wild exploitation shouldn't breed complacency. Unit 42's disclosure was responsible, and it's likely that only a small number of researchers understood the attack before now. However, the technique is straightforward enough that defenders should assume threat actors will understand it quickly. Organizations have a narrow window to patch and audit before adversaries stockpile exploits.
The deeper lesson: serialization formats that include execution are inherently risky in untrusted contexts. The security community learned this with PHP object injection, Java deserialization, and YAML unsafe load. Pickle should be on the same list. Cloud SDKs should default to safe, data-only formats and require explicit opt-in for executable serialization.
— HackWire Editorial
## Related Coverage