Nocom

From 2b2t Wiki
Jump to navigation Jump to search
Nocom heatmap of the overworld, from -245k to +245k, from early 2020 to mid-2021

Nocom was a coordinate exploit used by Nerds Inc on 2b2t from July 2018 to July 2021.

The flaw, summarized very briefly

You could punch any block coordinate in the entire map, and 2b2t would either:

  • Tell you what block is placed there, if the chunk is currently loaded
  • Or ignore you, if the chunk is not currently loaded.

2b2t would immediately reveal whether or not any chunk coordinate that you ask about is currently loaded. If it is currently loaded, it tells you which specific block is there at the coordinate you asked about.

The flaw

Nocom was a flaw/oversight in the 196th patch to PaperMC for Minecraft 1.12.2, called Fix block break desync (ghost blocks). It only exists in servers using the PaperMC server software; it does not exist in vanilla Minecraft.

The patch changes the behavior of Paper servers when a player says they are digging a block that appears to be too far for them to reach. In vanilla Minecraft, a CPacketPlayerDigging packet is simply ignored entirely when it is over six blocks away from the player. However, due to lag, sometimes the server and client disagree on where the player is. It is therefore possible for some good-faith block digging packets to be discarded by this policy. Since the server simply ignores the client stating "I have mined this block", this caused desync, a ghost block. The client thinks it is air while the server thinks it is still a block that you cannot walk through. This is because the vanilla client is "optimistic" in a sense: if it mines a block, it assumes it is mined unless the server tells it otherwise, which did not happen in this case because the server ignored the client and never told it "put that back", resulting in a ghost block.

Initial ghost block patch and lag exploit

The initial version of the patch, added in the initial port of Paper to 1.12.2, in September 2017, simply changed the behavior so that instead of ignoring the block dig, the server instead replies saying essentially "No, I think you can't mine that. Put (X:1, Y:2, Z:3) back as obsidian". In code terms, that's simply the only line added by the patch: this.sendPacket(new PacketPlayOutBlockChange(worldserver, blockposition)); // Paper - Fix block break desync

While this does patch ghost blocks (the patch is aptly named "Fix block break desync"), it also opens up a massive vulnerability. This applies to every CPacketPlayerDigging packet about a coordinate that is more than six blocks away from the player. There is no upper bound, just a lower bound of six. Simply by spamming "I am digging this block" on any coordinate across the map, the server would be forced to generate that chunk on the spot just to reply with what block should be there, to "fix" the desync that the server is concerned has just happened (while actually the client is sending the packets in bad faith).

Nocom used as a queue skip by Babbaj, April 2018

This causes a virtually unlimited amount of server lag, handling these packets and generating/loading those chunks. This was used as a queue skip exploit as early as April 2018. This is because at that time Hause had no packet ratelimiter, at all. Hause would add such a ratelimit in 2019 as a response to the off-hand switch sound lag exploit.

Coordinate exploit

The crash log that Hause saw and reported to Paper (recreation, as hastebin sadly expired the actual crash log he linked to)

In July 2018, 0x22 and Babbaj created and executed a plan to turn this lag/crash exploit into a coordinate exploit. They realized that this patch was practically inches away from being a coord exploit: the only difference between the patch as it stood, and a coord exploit, was the fact that the server would generate/load a chunk on the spot in reply to a block punch. If that behavior were removed, it would be a coord exploit, because it would reveal if the block you punched was in a loaded chunk or not, no matter how far away it was. That quirk of behavior, that the server would respond regardless of if the chunk was loaded, is exactly the behavior that would be removed in "the obvious patch" to the lag exploit.

So, instead of simply causing survivable amounts of lag (such as to skip queue), they began intentionally, repeatedly, and blatantly pushing it to the point where it would cause a server crash. That crash log included the exact line of that patch, and it was clear that spamming the packet was causing this. Hause[1] reported the issue to paper here, and electricboy applied the obvious patch in this commit. It was so trivial that there was no pull request or comments on the PR, it was simply applied directly to the development branch.

That obvious patch makes the "fix block break desync" only reply if the block is in a loaded chunk. Specifically, it just adds if (worldserver.isChunkLoaded(blockposition.getX() >> 4, blockposition.getZ() >> 4, true)) // Paper - Fix block break desync - Don't send for unloaded chunks beforehand, making the entire patch two lines of code.

What does the server do when you punch a block? (in other words, when you send CPacketPlayerDigging for a specific coordinate)
Vanilla Minecraft server Original version of the Paper patch, only exploitable to crash or lag the server Updated version that added nocom
Less than 6 blocks away The normal thing happens
More than 6 blocks away and in a currently loaded chunk No reply, server ignores you (causes desync/ghost block) The server tells you which block is at that coordinate (telling you to put it back; you were not permitted to break it)
More than 6 blocks away and NOT in a currently loaded chunk The server generates from seed or loads the chunk, and tells you which block is at that coordinate (telling you to put it back; you were not permitted to break it) No reply, server ignores you

As you can see from the above table, prior to the patch that added nocom, there was no way to tell if a chunk was loaded. The server would respond either way. Spamming this would cause it to generate or load chunks though, which needed to be patched.

After that patch, that added nocom, the server now reveals if any chunk you're curious about is currently loaded. You simply need to say "I click the block at (X,Y,Z)" (aka CPacketPlayerDigging). The server will respond with "No. Put (X,Y,Z) back as SOME_BLOCK" (aka SPacketBlockChange) if the chunk containing that block is currently loaded by a nearby player. The server will not send any reply if the chunk containing that block is not currently loaded. As such, the server reveals the loaded or unloaded status of any chunk you ask about, by if it replies to you, or ignores you claiming to have punched a block in that chunk.

Using common sense, the Paper developers most certainly intended for this patch to reply to the player only if the chunks were loaded by your player, as that would make logical sense (that's all the blocks you could reasonably be digging in good faith). Applying an upper bound of "but only up to your render distance" is a perfectly reasonable fix, as previously there was only the lower bound of "more than 6 blocks away from you". The problem is that the way the code was written, the server will reply to you if the chunk is loaded by any player on the server, which is clearly an unintended side effect.

The reason why this has flown under the radar for so long is that there is no actual "exploit" or "backdoor" in the sense that you might think. In other words, the server doesn't "misbehave" or do anything suspicious. It's fully expected and intended behavior, the code doesn't do anything sneaky or surreptitious, it's actually perfectly simple. The fact that you can click a block and get its contents (due to this patch) is a well-known fact. Many mods have used this over the years. For example, Orebfuscator is a server plugin to combat X-Ray. It doesn't send where ores actually are until you get close to them. Sadly, you can simply abuse this patch and click every block in every chunk in render distance, and the server will dutifully tell you which blocks are truly stone and which are truly ore. This anti-orebfuscator module has been in many mods. Constantly punching blocks near to you to fix ghost blocks or desync is also a feature in many clients. It's just that not many people thought of intentionally going outside your render distance to extract information. The realization is that you can click absolutely any block, anywhere on the server, even a million blocks away, and learn if it's currently a loaded chunk, by if the server responds to you or remains silent.

Exploiting the flaw

Spiral

The most natural and obvious way to exploit coordinates using this: just click one block per render distance, going out in an expanding spiral from spawn.

If the spiral search hits a loaded chunk, 2b2t will send back a reply, which would presumably be logged to chat, or to a log file.

This was done starting on the same day that the patch was added to 2b2t, July 13 2018, and continued until late 2019 when the packet ratelimit was added.

Tracking

With the packet ratelimit that Hause implemented in 2019, it became no longer practical to happen upon people at their base through a brute force spiral search. The packet budget was far too low to continue with that approach, for anything but very close spawn stashes. It could never work into the mid-hundred-Ks.

The new approach was a shift in mindset: instead of finding people at their base, the exploit would now follow them as they traveled to the location in the first place.

In broad strokes, players were found in the first place by running straight-line checks on every axis and diagonal highway in the nether. 3.75 million blocks to the OW border, divided by one render distance (144 blocks), means that one highway requires running 26,041 checks (punching that many blocks). The practical ratelimit was about 150 packets per second for much of 2020 and 2021, meaning this would take 173 seconds, just under three minutes, to scan a highway. There are eight such highways (four axis, four diagonal), and the time is the same for each (even for diagonal because square render distances are spaced the same even when aligned corner-to-corner[2]). In total, 23 minutes to scan the entire highway system.

After being located, a Monte Carlo particle filter was used to keep up with the player. This is an adaptive system that learns over time (after just a few seconds, really) where the player probably is, and their movement speed. The utility of doing this is that players very commonly move in straight lines, such as elytraflying in the overworld, or traveling through nether tunnels, or, the most obvious of all, traveling on a nether highway. A particle filter is able to understand that a track in motion is likely to stay in motion, and be more efficient with its checks, not being surprised each time "oh, they moved! which direction??". For particularly consistent movement such as straight line afk travel, it relaxed all the way down to 1 to 2 checks per second. For more erratic target movement, it could go up to a dozen checks per second if it becomes unsure what is happening. If it lost the target for 5 seconds, it triggered the "hail mary" check, which is a 11 by 11 grid of checks (1232 by 1232 blocks, centered on the last confirmed hit), as a last resort to make sure that it really wouldn't lose anyone. As there were rarely any exploits to allow traveling 1323/2=616 blocks in under 5 seconds, this caused the tracking to be quite sticky and resilient, no matter what creative maneuvers were performed by the target. Taking everything into account, all told, the particle system on average used just over two checks per second per player. If a player was seen to be perfectly stationary, the speed would relax even further by special code for a "stationary filter", which could go all the way down to 1 check every 8 seconds. All in all, the tracking system would commonly adapt its chunk checking budget by 1000x in about 5 seconds, ranging from 0.125 checks per second to 121 checks per second, based on a direct feedback of what the player was doing at that moment and how predictably they were moving.

Simple rules were added to follow people. If someone disappears from the nether, multiply the coords by eight and ask the overworld bot to check there, and vice versa. If someone logs in, check the last time they logged out, and run through all tracks that went cold within a minute of that time, and recheck the last hit on all of those. Etc.

Various automated analyses were performed on the resulting tracking data. The primary one was tallying up which areas are hotspots of activity, likely indicating a base or stash. The threshold was set at 90 minutes of time spent in a given area for it to be called a base. At that point, the system would start to track player logins and logouts at that location and start to build up statistical correlations of "who could be who". It would also at that point start to download the base block-by-block by clicking it, to enable remote viewing. Analyses were also performed on that data, such as the commonly used query to print out all bases from the last week, sorted by number of chests.

Statistics on collected data

  • 3,250,000 player sessions. (this is player, login time, logout time. so, timestamps of 3 million "X joined the game"s and the 3 million matching "X left the game"s)
  • 300,000 unique players seen in the tab menu
  • 15,000 bases (defined as an overworld location with over 90 minutes of someone spending time there, but excluding within 25k of spawn, and excluding within 2k of a highway/diagonal)
  • 400,000 association events. These were only tracked for logout that happened within one of those 15000 bases. Each event is simply "player X logged out at the same time that it lost a track at base Y". As it counted up many events for that player and that base, it became more and more certain that it's them at that base.
  • Of those bases, two thirds (10,000) have a world download. This WDL is pristine for most large/active bases, and has the full block-by-block timeline of the base's history at 30 minute intervals. This was only done for bases above 100k from spawn.
  • 140,000,000 chunk positions where it started to track online time, to perhaps see if there is a base there. Bases within 512 blocks of each other were merged into one, so while there are 15,000 bases, there are actually 1,150,000 chunks that make up those bases, chunks where it had observed significant time being spent occupied by a player.
  • The table of blocks, which is just coordinate, timestamp, and block, has 10,000,000,000 rows (ten billion). It takes up over a terabyte just for this table and its associated indexes. It's all the data behind the remote viewing system.
  • The table of hits, which is every time it got a "chunk is loaded" response back from 2b2t, has 3,000,000,000 rows (three billion). This is essentially high resolution real-time tracking data for about 70% of the server's online players, going back years. It's everyone's location at 1 second update intervals.
  • The table of tracks, which is a grouping of hits into which ones were collected as a part of one continuous track. This has 10,000,000 rows (ten million). So, the average track therefore has about three hundred hits. This helped the system do things like resume tracks after someone leaves and rejoins the server.

Bases griefed via Nocom

External links

Notes

  1. The "ghost" GitHub username isn't a real username, it's just a placeholder indicating that they have deleted their GitHub account
  2. Thanks Casparov for pointing this out