RC RANDOM CHAOS

Sleeper Agent Payloads Hide in Open AI Model Weights

· via Dark Reading

Original source

'Sleeper Agent' Tactic Hides Payloads in Open-Weight AI Models

Dark Reading →

Researchers have documented techniques for embedding dormant malicious payloads directly into the weights of open-weight AI models. Because these models contain billions of parameters, attackers can hide functioning code in a negligible fraction of the network without materially affecting the model’s normal behavior. Open-weight adoption is widespread—63% of organizations already use them—yet they’re rarely audited before deployment, making this vulnerability particularly concerning.

Two main approaches have emerged. The MaleficNet technique applies code division multiple access to fragment malware across individual weights, essentially encoding it as noise within the network. Microsoft’s approach is simpler: it encodes a payload in dormant neurons—nodes with minimal impact on model output—and configures them to remain inactive until triggered by specific user input, tool output, or system events. The core vulnerability stems from neural network architecture: models contain far more parameters than necessary, allowing subtle modifications that escape detection since weights are inherently unreadable compared to source code.

Detection at the weight level is impractical given the scale of modern models. Defense requires restricting model capabilities from the outset: minimal credentials, network segmentation, access controls, and comprehensive logging. Behavioral monitoring at system boundaries—detecting unexpected shell commands, unusual network connections, or access to credentials the model shouldn’t need—provides far more actionable detection than analyzing billions of parameters for hidden malice.

Read the full article

Continue reading at Dark Reading →

This is an AI-generated summary. Read the original for the full story.