Skip to content
How Jupyter Notebook Diffs Opened a Backdoor to GitLab Servers
Article

How Jupyter Notebook Diffs Opened a Backdoor to GitLab Servers

The problem

We tend to treat git repositories as passive storage boxes. You push some text, the server indexes it, and you move on. Sure, there’s always the risk of a rogue developer pushing a malicious script, but we assume that unless someone executes that script, or unless a CI/CD runner picks it up, the server itself is safe.

A newly disclosed remote code execution (RCE) vulnerability in self-managed GitLab CE and EE instances breaks that assumption completely. If an attacker has permission to push a commit to a repository, even a private one, they can compromise the underlying host.

They don’t need administrative permissions, CI runners, or special access to anyone else’s projects. All they have to do is upload a crafted Jupyter Notebook file. The moment you open the GitLab web interface to view the commit diff, the server’s web processes execute arbitrary shell commands. It’s a zero-interaction trigger for anyone looking at code changes.

What happened

Jupyter Notebooks are widely used by data scientists and developers to mix code, narrative text, and visual outputs. When stored in a Git repository, these notebooks are just large JSON files. Because reading raw JSON diffs is painful, GitLab tries to be helpful by rendering a visual, side-by-side comparison of the notebook cells.

To build this comparison, GitLab relies on an internal Ruby gem called ipynbdiff. The gem parses the notebook files and reconstructs them into readable text blocks. However, parsing large files in standard Ruby can be slow. To speed things up, ipynbdiff hands the parsing job to a native C extension for Ruby called Oj (Optimized JSON).

Security firm depthfirst audited Oj and discovered multiple memory-safety bugs in the C code that had gone unnoticed for almost five years. By chain-linking two of these bugs, researchers showed that an attacker can trick the server into running shell commands as the system git user. The vulnerability affects GitLab versions spanning 15.2.0 all the way to 19.0.1.

How the exploit worked

To pull off the exploit, the attacker has to chain together two different bugs in the Oj parser: a memory leak to find where things are, and a write primitive to redirect execution.

Modern operating systems use Address Space Layout Randomization (ASLR) to randomize where program code sits in memory. To bypass this, the attacker first commits a notebook containing a JSON key that is exceptionally long (specifically, 65,565 bytes). A bug in how the parser converts number sizes (an integer narrowing flaw) causes it to read from the wrong layout view. Instead of reading the key text, the parser reads a raw memory address from the heap. GitLab then prints this leaked address directly into the HTML diff page, letting the attacker calculate the server’s memory layout.

Next, the attacker corrupts the parser’s internal state. The parser tracks how deeply nested a JSON structure is using a small 1,024-byte stack. If the attacker commits a notebook nested deeper than that, the parser writes right past the end of the stack without checking limits. By carefully aligning the memory allocations, the attacker uses this stack overflow to overwrite an internal parser function pointer with the address of system().

Finally, the attacker triggers the command execution. Because GitLab’s web worker process (Puma) is long-lived and handles multiple file parses in a single request, the armed parser is reused to parse the next part of the exploit payload. The parser runs the overwritten function pointer, executing the attacker’s shell commands as the system git user.

The awkward part

The awkward part of this story is that GitLab is written in Ruby, a language chosen specifically because it is memory-safe. Ruby developers are not supposed to worry about buffer overflows, integer narrowing, or manual memory management.

But developers love speed. To make JSON parsing fast, libraries like Oj implement their performance-critical code as native C extensions. The moment you pull in a C extension, you inherit all the classic, dangerous memory corruption problems of C. A memory-safe web application is suddenly vulnerable to low-level heap corruption because of a dependency.

Making matters worse, these bugs survived in the wild for nearly five years before the audit surfaced them. Since the visual notebook diff feature was introduced in GitLab in 2022, self-managed instances have been exposing this native C parser to user-supplied files. Because exploitation happens automatically when the diff is rendered, an attacker just needs to push a commit and wait for a developer, or a webhook, to trigger the diff view.

What GitLab is doing

GitLab moved quickly once the report came in. The company resolved the vulnerability on June 10 by releasing a set of security patches that updated their bundled dependency to Oj 3.17.3.

For self-managed administrators, the fix is packaged in GitLab releases 18.10.8, 18.11.5, and 19.0.2. If you run any version between 15.2.0 and 19.0.1, your installation is vulnerable and should be updated immediately.

According to GitLab’s security advisory, GitLab.com and Dedicated instances have already been patched. If you rely on their hosted services, you are safe from this specific attack chain. The action item lies entirely with teams maintaining their own installations.

What to do about it

If you are self-hosting a GitLab instance, there are a few practical steps you should take right now:

  • Upgrade your instance immediately. This is the only real fix. Get your server onto GitLab 18.10.8, 18.11.5, or 19.0.2 depending on which release line you track.
  • Turn off public registrations. If your GitLab server is exposed to the web and allows anyone to sign up and create a repository, they can target your server. Disable public sign-ups and restrict repository creation.
  • Isolate your server container. The shell commands run under the system git user, which handles the Puma worker processes. Ensure your container environment restricts what this user can do. Block Puma workers from talking to internal network databases, cloud metadata endpoints, or other internal APIs that they have no business visiting.
  • Audit other native dependencies. This is a reminder that memory-safety in Ruby or Python only goes as deep as your dependencies. When choosing packages for your stack, treat any library shipping with native C extensions with extra scrutiny.

Sources

Related