A recently identified security flaw in FFmpeg, named ‘PixelSmash,’ could allow attackers to run malicious code on Jellyfin servers in specific scenarios. It can also cause crashes or service disruptions in widely used applications such as Kodi, Emby, Nextcloud, PhotoPrism, and OBS Studio.
Tracked as CVE-2026-8461, this high-severity vulnerability (rated 8.8) involves a heap-based buffer overflow in the MagicYUV video decoder. Attackers can exploit it by crafting a malicious video file in common formats like AVI, MKV, or MOV.
Any software relying on libavcodec—the core FFmpeg library responsible for encoding and decoding video—is potentially at risk.
Remote code execution (RCE) becomes feasible only if the system’s Address Space Layout Randomization (ASLR) protection is turned off, or if an attacker combines this flaw with another vulnerability to circumvent ASLR.
Underlying issue and potential consequences
According to researchers at JFrog, a firm specializing in software supply-chain security, PixelSmash arises from how MagicYUV handles video slices—distinct sections of a frame that can be decoded independently.
“This flaw is a one-row heap buffer overflow occurring during slice processing in the MagicYUV decoder. It results from a mismatch between how the frame memory allocator and the decoder calculate the height of chroma planes,” JFrog clarifies.
PixelSmash can be activated simply by opening a malicious AVI, MKV, or MOV file, browsing a folder containing such a file (which may trigger thumbnail previews), or running any automated media processing pipeline.
JFrog confirmed that several well-known applications—including Kodi, OBS Studio, PhotoPrism, and thumbnail generators in GNOME, KDE, and XFCE desktop environments—use FFmpeg with the MagicYUV decoder enabled, leaving them exposed to this attack.
Messaging platforms like Slack, Discord, Telegram, and WhatsApp might also be affected since they rely on FFmpeg for server-side video preview generation, though these were not directly tested.
JFrog’s lead researcher, Yuval Moravchick, demonstrated that PixelSmash can lead to full remote code execution on Jellyfin and Nextcloud (when movie previews are enabled).
“To illustrate real-world impact, we successfully executed arbitrary code on a Jellyfin 10.11.9 media server—the second most widely adopted self-hosted media server after Plex—using its standard media library scanning process,” JFrog explains.
“Attack flow: a specially crafted MagicYUV AVI file is placed into the media library → Jellyfin automatically runs ffprobe to extract metadata → the out-of-bounds write occurs → AVBuffer.free is redirected to system() → attacker-defined commands run under the jellyfin service account.”
However, Moravchick emphasized that achieving RCE requires ASLR to be disabled; CVE-2026-8461 by itself cannot bypass this security mechanism.
In theory, a separate information-leak vulnerability in FFmpeg’s FlashSV decoder could be combined with PixelSmash to defeat ASLR.
Another exploitation method involves torrent downloads and requires no user action. An attacker could distribute a malicious video file targeting Jellyfin users who configure their download directory as part of the media library.
“Jellyfin’s real-time file watcher detects the new file and immediately initiates an ffprobe metadata scan. The exploit activates during this scan—AVBuffer.free is hijacked to system(), and the attacker’s reverse shell command executes under the jellyfin service account.”
Even when full code execution isn’t possible, CVE-2026-8461 is reliable enough to consistently cause a denial-of-service (DoS) condition on affected systems.
The researchers noted that Plex, the leading media server, avoids this risk by using a customized FFmpeg build where most decoders are disabled and only a minimal set of approved formats are allowed.
Beyond FFmpeg releasing version 8.1.2 to fix the issue, Jellyfin has updated its bundled FFmpeg, and PhotoPrism is implementing a blocklist for risky file formats to reduce exposure.
Nextcloud’s development team acknowledged the report via HackerOne but chose not to patch it, stating the vulnerability lies outside their codebase.
JFrog discovered PixelSmash (CVE-2026-8461) and responsibly disclosed it to the FFmpeg security team on May 13. The fix was included in FFmpeg version 8.1.2, released on June 17.
Researchers caution that PixelSmash poses a broad threat because the MagicYUV decoder is embedded in hundreds of projects that “rely on FFmpeg to safely process untrusted media,” effectively turning this into a supply-chain security concern.
Security teams detect 54% of successful attacks but only generate alerts for 14%. The remaining threats go unnoticed within your infrastructure.
The Picus whitepaper explains how breach and attack simulation validates your SIEM and EDR detection rules to ensure threats no longer evade monitoring.
Get the whitepaper
