6-year Anniversary Event

This year’s anniversary event was a huge success, thank you to everyone who logged on and participated - we hope you enjoyed gambling your time for block game rewards. This event was unique due to the gameplay-changing effect of relics dropping from cobblestone mining. Although this was a significant modification, it allowed us to run an event which could accomodate everybody who wanted to join rather than the lucky few who happened to be online and at spawn at the right time to receive relics from a drop. Thanks to the success and positive feedback from players, there may be more custom events like this in future, although not for a long time.

Folia Update Status

Public testing of the new Folia plugins is now live on /server folia with exclusive cosmetic rewards for any successful bug reports filed on our Discord server (provided that the bugs relate to the Folia server plugins or configuration specifically). The initial testing has not raised any major issues and so the update to Folia for the main server should go ahead as planned.

The current timeline for the update is:

1. Temporary (fresh) map on main server running Folia on Sunday 13th April
2. Update main map to Folia on Tuesday 15th April

The temporary map will replace the main map for 2 days to allow for testing of the new plugins and Folia in general on a larger scale than the testing server.

At this point, the significant parts of the project are almost complete:

• New custom plugins (~80%)
• Replacment plugin configuration (~90%)
• Anticheat (Grim) configuration and fixes (~70%)
• Fixing existing plugins (50%)

Folia Server Configuration

As well as the development and configuration of new plugins, the server itself will be reconfigured for Folia. This will include resetting many Bukkit, Spigot and Paper server settings to their default values which should bring entities closer to their vanilla behaviour. In future, a comprehensive list of server configurations will be published and updated regularly, but as there are many sensitive options such as terrain and feature seeds, the entire files themselves can’t be made public.

Currently, the following configuration options are set on the test server and will be set on the main server once we update. All other options will be left at their default values. See the full list of configuration options and their defaults here.

Please remember this list of configurations is subject to change and may be invalid if you are reading this post in the future.

server.properties

• difficulty=hard
• enforce-secure-profile=false
• simulation-distance=5 (Likely to be reduced)
• view-distance=8 (May be reduced)

bukkit.yml

• spawn-limits.monsters: 20 (Likely to change)
  This value was reduced greatly from the default of 70 due to the low mob spawning range of 3 chunks. This may still result in an excessive number of mobs and so may be changed further. Note that this is not the global limit for monster spawns like in CraftBukkit, but uses Paper’s per-player mob spawning limits.

spigot.yml

• mob-spawn-range: 3 (Current value preserved to keep spawn perimeter size constant)
• ticks-per.hopper-check: 4 (Current value preserved initially, likely to be reset to vanilla after update)

paper-global.yml

• entities.armour-stands.do-collision-entity-lookups: false
• entities.armour-stands.tick: false
• entities.despawn-ranges.<ALL>.hard: 48
• entities.despawn-ranges.<ALL>.soft: 32
• environment.nether-ceiling-void-damage-height: 127
• environment.optimize-explosions: true
• misc.legacy-ender-pearl-behavior: true
• chunks.entity-per-chunk-save-limit.arrow: 32
• chunks.entity-per-chunk-save-limit.ender_pearl: 64
• chunks.entity-per-chunk-save-limit.fireball: 32
• chunks.entity-per-chunk-save-limit.snowball: 16
• chunks.entity-per-chunk-save-limit.villager: 1024 (High placeholder value, likely to be reduced)
• chunks.entity-per-chunk-save-limit.wither_skull: 16
• chunks.entity-per-chunk-save-limit.wither: 4 (Likely to change)
• chunks.entity-per-chunk-save-limit.tnt: 16
• collisions.fix-climbing-bypassing-cramming-rule: true
• behaviour.disable-chest-cat-detection: true
• spawning.count-all-mobs-for-spawning: true
• maps.item-frame-cursor-limit: 12
• maps.item-frame-cursor-update-interval: 60
• misc.redstone-implementation: EIGENCRAFT
• misc.update-pathfinding-on-block-update: false

Tasks in Progress

The update to Folia is all that is being worked on at the moment. Updates about ongoing issues with the main and side servers will resume after Folia is live.

New Updates from Staff

To help increase player engagement in the community, we want to be more transparent about the work that goes on behind the scenes on Purity. This starts with reviving this blog site for more detailed updates on the server than will appear on Discord, including new semi-regular summary posts from me (penguin). These posts will let you know what is currently being worked on and what is planned for the future. I’ll still announce any significant changes on Discord, with a link to any associated posts here for more information. I’m also aiming to talk with players and staff more regularly about current issues to stay on top of what needs done.

Folia Update Status

The big news is about our update to Folia, which has obviously been delayed significantly since first being announced. We’re still in a similar situation to our previous post where plugin compatibility is the main barrier, however a few plugins that claimed to support Folia have been shown to have broken features in our testing. This means more time is needed to either fix or find replacements for these plugins. We’ve scaled back our custom plugin project so that any development time is focused on what is essential for server functionality on Folia.

Due to projects outside of Purity, I’ve had much less time than I expected to work on the update. At this time, the server is expected to update to Folia on or before April 15th.

The most significant parts of the project are:

• New custom plugins (~30% completed)
• Replacement plugin configuration (~50% completed)
• Anticheat (Grim) configuration and fixes (~10%)
• Fixing existing plugins (0%)

Player Suggestions

We’ve started collecting and officially accepting player requests from players. Once a suggestion is accepted, it’s added to an internal list to which is then ordered based on priority and ease of implementation. While work on these suggestions may not begin immediately, we’ll keep track of them until I have the time to implement them. In the case of large changes, their progress will be published in these updates.

See current suggestions in this Discord channel.
Before submitting your own suggestion, search both the suggestions channel and this channel to make sure it hasn’t been answered yet.

Tasks in Progress

Folia

• Custom plugin development
• Configure and test Folia (e.g. new view distance, mob caps etc)
• Configure Grim (may be combined with some checks from NoCheatPlus if necessary)

Purity PVP

• Modify Arena plugin to reset arenas more frequently than server restarts.

Todo List

Folia

• Fix and repladce broken plugins (ChatManager, Grim, Vanish, PlaceholderAPI)
• Update Folia server for public testing

Discord Server

• Create media channel for Purity content creators to post new uploads etc
• Add dedicated FAQ channel for server configuration info (likely automated with simple bot)

Purity PVP

• Update to 1.21
• Add elytra and rockets
• Expand FFA arenas below current bedrock
• Finish drop map below spawn

Purity Creative

• Museum of historic bases
• Build battles
• New donor perks

What is Folia?

If you are active on the Discord or have been hanging around on the server recently then you may have heard me (penguin) promising to resolve all server perfromance issues by changing the server software to Folia. Folia is a fork of the popular Minecraft server software Paper, currently the most widely used CraftBukkit-based server.

Purity’s Server Software

For the last couple years, we have run Purpur on our server, a seperate fork of Paper. We used this fork in particular as it has a large degree of customisability which we used to limit villager AI to improve performance. Purpur also was the first Paper fork to introduce configurable projectile saving limitations (preventing the server from crashing due to snowballs or wither skulls) as well as seperate feature seeds for dungeons and end spikes, which helps prevent terrain seed cracking. These important features have since been included in Paper which now generates a completely unique seed for every feature in a particular world.

However, when running Purpur, the server had intermittent performance issues that could not be identified as entity or chunk related from Timings/Spark reports. This was eventually attributed to the large featureset of Purpur resulting in some unnecessary logic being checked, even though the vast majority of the features were disabled in the config. While this is not a particularly satisfying or complete explanation, it was proven to be largely true when we switched to a custom fork of Purpur. This fork included a hand-picked selection of Purpur’s patches, only the ones needed to help with performance and seed cracking on Purity, and when we first switched to it for 1.16 there was a marked improvement in server stability and high playercounts (70-80).

In general, a major limitation for larger single-instance Minecraft servers like Purity Vanilla is there is a limit to the number of entities, blocks and players that can be updated by a single server each tick, when this limit is exceeded the server runs slower than it should with less than 20 ticks per second (TPS). This is why networks such as Hypixel use thousands of smaller instance servers to be able to accomodate tens of thousands of concurrent players - they obviously could not achieve this using just one server. You can get more information about the reasons for server lag from this (fairly outdated) post.

What does Folia do?

Folia allows one Minecraft server (a single instance) to hold many more players than previously possible by breaking up each world into many groups of adjacent chunks which can each be ticked in parallel. See the Folia docs for some details about the implementation. This allows our primary server hardware, which only runs the main Purity Vanilla server and a Velocity proxy, to be able to handle more players by distributing the load more evenly across its many CPU threads. This is untested for Purity as of yet, although we have been running a small scale test of Folia for many months to test stability, we are yet to test the server with the same load from all of the players, farms and entities we see on the main map.

Changes to the API when compared to Paper mean that most plugins need some changes to work on Folia, while some may need to be significantly reworked. This is the case for most of our plugins, although our custom plugins could be easily used, we are reliant on the developers of larger plugins like EssentialsX updating their own plugins. Fortunately, many plugins already support Folia with more actively developing for it. A convient list for reference of popular plugins that support Folia can be found here.

Plugin Support

As we already have a small test server running the latest version of Folia, the only thing preventing us from switching the main server to it is the lack of plugin support. This, combined with some of our plugins like ChatManager being poorly supported at the moment, has prompted the development of a new set of custom plugins for Purity Vanilla which will replace the majority of the plugins. While features will mostly be the same for players, this means it will be easier to customise the server further in future and will help with implementing new cosmetic perks for example.

These plugins are very early on in development, so no precise ETA can be given. Hopefully most of the functionality of the main server will be re-implemented within a couple of months (please consider Purity Time™️). Further development updates will be posted to this blog, along with the GitHub repositories for any open source elements which will allow for our players to see the progress and potentially contribute to the server.

When will we update?

With the update to Folia dependant entirely on me developing these new plugins, the timeline for the update may change depending on the time I have available. That being said, I’m intending on having the plugins running on the test server as soon as possible, with the update for the main server coming before December. I will post any major changes to this timeline on Discord and give more detail on this site.

Thanks for your patience with any server stability or TPS issues in the meantime.

With the update to 1.16, text colours have been overhauled. This means instead of being stuck with the old colour format codes, full RGB can be used in chat, for player nicknames and on signs.

As a reminder, you can colour code and format your message by using the chat code of the colour you want before your message. If you want to use font and colour formatting, the colour format code must come before the font format code. For example, if I wanted to type a message in gold with bold font, I would type &6&lHello World!.

This quick and easy formatting is all you need in most cases as typing out RGB hex codes for every chat message can be time-consuming. However, if you want more customisability for your nickname or placed signs. To start, choose a colour your like using an RGB colour picker. The formatting is very similar to the old system, just replace the single number or letter with your hex code prefaced with ‘#’. For example, if I wanted to colour my nickname with the colour #FF0061, I would type /nick &#FF0061penguin

Of course, multiple colour codes and formatting codes can be used together in the same message or nickname, it can just take a while to type in. I recommend using a simple text editor to draft your nickname command so you can copy and paste it into chat and make adjustments as needed without needing to fully re-enter the command. For example, If I wanted to make the first part of my nickname red, the second part green and the third part blue, I would type /nick &#FF0000pen&#00FF00gu&#0000FFin

If you want to use an RGB gradient in your nickname, here is a useful tool for getting hex codes in as many steps as you want.

Despite looking complicated and hard to understand, hex codes for RGB are very simple. They are written in hexadecimal (base 16), which means there are 16 symbols for numbers rather than the regular 10 arabic numerals we are used to. These symbols are 0-9 and A-F. The first two characters of a hex code is for the red channel. 00 meaning no red and FF meaning as much red as possible. The same is true for the remaining green and blue channels. In RGB, these channels can range from 0 to 255, the range of values that can be represented with two digits of hexadecimal.

Overview

One of the most frequently asked questions we get is, in one form or another, “why is this server so laggy?”. While we do our best to keep TPS as high as possible, lag is still an issue. This is for a variety of reasons, some out of our control.

In this post I will highlight the major factors and explain their causes and trade-offs. The most important thing to remember is running a server is a balancing act between giving players the features and the gameplay they want, having enough capacity to handle all the players that want to be on the server and ensuring the performance is acceptable. No optimisation-related decision can be made without impacting one of these factors, if it could, all servers would run with 20TPS and unlimited player slots.

Entities

The primary and most obvious cause of lag is a great number of entities. This includes all dropped items, monsters, animals and NPCs such as villagers. Each of these kinds of entities has its own source of lag. For example, pathfinding was found to be very intensive not only for villagers, but for monsters too. This is why, at the time of writing, zombie aggressiveness towards villagers is disabled on Purity Vanilla. From our testing, it was found that that process alone was using up to 15% of each tick. Entity collisions are another source of lag although they can be reduced with configuration. This is why our server has a per-chunk limit for each mob type in addition to bukkit’s spawning limits, as our players build huge mob farms and grinders. Our goal is to allow these farms to function as efficiently as possible without causing too much lag while not deviating from vanilla spawning too much outside of farms. However, these limits often result in fewer naturally spawning mobs which can be irritating for players. Paper’s per-player mob spawning is useful, however, still leads to trying to balance farm efficiency and vanilla gameplay.

To evidence the extent of the impact of huge mob farms on server lag, we took sampler reports during one of our temporary maps, for which the world had only been running for a few hours. At 80 players online we only had 33ms ticks on average (20TPS means < 50mspt). Compared to our normal map, which usually runs at 50-60mspt at 80 players, this is a huge improvement. Considering chunks are likely being loaded at the same rate, if not more as people are more likely to explore a fresh map, this shows the impact mob farms have and the benefits gained from nerfing them.

Chunks

Another contributor to overall server lag is chunk loading and generation. Paper actually implements asynchronous loading/saving of chunks which is a huge performance boost. However, not all chunk loads are asynchronous. “Sync loads” can be triggered by a variety of events. A chunk loading synchronously essentially means the server is unable to process anything else until the chunk has loaded, which can cause lag. A very common cause of sync loads in survival servers is player teleporting such as /tpa or /home, although more recent plugins should use asynchronous teleporting. However, even though servers like Purity have no player teleporting, sync loads can still be triggered by maps, cartographers or certain EntityPlayer methods when called by the server. Brief but severe lag spikes can sometimes be attributed to these synchronous map loads.

Just because the majority of chunk loading in Paper is asynchronous does not mean it doesn’t cause any lag. Players using elytra with a trident can easily cause lag from loading chunks too quickly on less powerful hardware or in a shared server environment. This is why pregenerating chunks is recommended for survival servers with a reasonable world border. However, for anarchy servers or just survival servers with large world borders (over 50k blocks) pregenerating is unfeasible due to the file size of a fully generated world - Purity’s overworld is currently 1.5TB at only 8% generated.

Without pregenerating, the server is constantly generating new chunks while players are online, which puts more strain on the CPU.

Lack of Threading

The majority of the Minecraft Server process is run on a single thread. This means, for a single-instance server such as Purity Vanilla, no matter how many threads our server’s CPU has, the Minecraft server can’t take advantage of them all. There are plenty of reasons why Minecraft isn’t properly threaded to take advantage of more CPU threads and going back and optimising the server’s code in this way is a huge task for even Mojang to undertake. As it stands, our server runs on one of the most powerful CPUs in terms of single-threaded performance, the 3950X. We have used a 9900k in the past but we have found Ryzen 3 has given us better performance. This is why, no matter how many donations the server receives, we cannot just “buy a better server”. The fact we are using a 3950X dedicated server for a single Minecraft server is already overkill.

As for why minigame servers like Hypixel can have thousands of simultaneous players, these servers don’t just run a single Minecraft server and fill it with players. They use a proxy like bungeecord and a load-balancer to distribute their players among any number of backend servers. If you think about it, even without any entities or chunk loading, Hypixel’s 100+ player lobbies can still lag. This means each minigame lobby has its own server instance and will only handle 20-40 players.

This kind of load-balancing is impossible for a survival server like Purity since all of our players are playing in the same world at all times.

Hardware Limitations

As I mentioned above, how many players that can play on your Minecraft server is heavily bound by your CPU and its single-threaded performance. Many shared hosts will advertise their use of “enterprise” hardware for the utmost performance, however this invariably means Intel Xeon CPUs, not intended for single-threaded applications. Even hosts which use Intel i9s or high-end Ryzen CPUs will often run more servers on each node than is optimal. This is known as overallocation and should be avoided if at all possible.

Regarding Purity, we run on the most powerful available hardware because it is within our budget. At the lower end, you need to be much more conscious of what hardware you are paying for. We are instead limited by the limitations of the Minecraft server itself.

Garbage Collection

In Java, objects in memory that are no longer used do not need to be explicitly marked to be deleted by the programmer. Instead, The JVM preforms “garbage collection”, during which the program’s heap is checked for unused objects that can be deleted to free up memory. On recent Java versions, the default garbage collector is G1GC. This process can cause lag spikes if misconfigured. If you are using G1GC, make sure to use Aikar’s Flags. On Purity, we use ZGC, a garbage collector designed for the lowest possible pause times. (Your mileage will vary with ZGC, I may write a post about our setup and how it works for us at some point.)

Either way, garbage collection involves additional work to be done by the CPU which can cause lag spikes while it is happening. This could be as a result of GC pauses from G1 or higher CPU utilisation with ZGC. Appropriate memory allocation and heap size is important, don’t just allocate as much memory as possible.

Summary

There are a large number of reasons why servers can seem laggy to a player, in addition to those highlighted above. This post was written to explain the main sources of lag on Purity Vanilla, although they are likely applicable to other survival servers. Each properly configured server will have its own limitations and changes to reduce lag which you will need to adapt to in order to play on.

A lazily configured or misconfigured server will lag no matter what. Read pre-existing optimisation guides for a good baseline, but constant tweaking is necessary to meet your own server’s needs. Purity’s optimisation has been an ongoing process for the entire lifetime of the server and will likely never be completed.