Analyse
FeaturesA/B tests

Features

A/B tests

Test changes on real players: two spawn layouts, two starter kits, two join messages. Analyse picks the winner for you.

A/B testing is how you figure out if a change actually helps. You show half your players version A, the other half version B, and Analyse tells you which one wins.

It's the most underused feature on most servers. If you've ever argued in Discord about whether a new spawn or a new starter kit is "better", you should run an A/B test instead.

Plan

A/B testing is available on current plans.

How it works on Analyse

  • You create a test in the Analyse dashboard with at least two variants.
  • Each variant is a list of actions Analyse runs for players in that bucket (for example: send a chat message, run a console command).
  • A trigger decides when variants are assigned and run. The most common trigger is a player's first join.
  • Each player is permanently bucketed into one variant using their UUID.
  • A goal defines what counts as a conversion: retention, first purchase, minutes of playtime, or a custom event.
  • Analyse tracks exposures and conversions per variant and shows you which one is leading.

What you can test

Anything you can express as a chat message, a command, or (via the SDK) a branch in your plugin code:

  • Two welcome messages on first join
  • Two starter-kit commands (/kit starter vs /kit starter_v2)
  • Two store prompt strings
  • Two join teleport commands pointing at two different spawn points

Start small

Pick one thing. Run the test. Look at the result. Ship the winner. Then pick the next thing. Running five tests at once in your first month is how you end up ignoring all of them.

Creating a test

The create flow walks you through a few short steps.

  1. 1

    Basics

    Give the test a unique key (like new-spawn), a readable name, and a description. Set the target percent of players who should be eligible (100% by default).

  2. 2

    Trigger

    Pick when variants get assigned. "First join" is the usual choice, which locks each new player into a variant the first time they connect.

  3. 3

    Variants

    Add two or more variants. Each has a weight (split by percent, must total 100) and one or more actions (send a chat message, run a console command, and so on). A/B tests start with a control and one variant; add more if you want A/B/C.

  4. 4

    Goal

    Pick what you're optimizing for: retention, purchase, playtime, or a custom event you already track. For a custom event goal, pick the event name from the list.

  5. 5

    Save as draft

    The test is created in DRAFT status. Nothing runs yet.

Starting the test

After creating the test, open it and click Start test. From that moment on, every new player that hits the trigger is bucketed into a variant and the variant's actions run. Before you click start, the test is a saved plan that isn't affecting anyone.

You can pause the test at any time from the same page.

The SDK path (advanced)

The default flow is server-run actions (messages, commands) configured in the UI. If your plugin needs to branch its own code based on the variant, you can read the variant directly via the SDK:

java
String variant = Analyse.getVariant(player.getUniqueId(), "new-spawn");
if ("variant-a".equals(variant)) {
  // show the new spawn
} else {
  // show the old spawn (or null, if the test isn't live)
}

Full code walkthrough: SDK: A/B tests.

Reading the results

The test's detail page shows, per variant:

  • Exposed: how many unique players landed in this variant
  • Converted: how many hit the goal
  • Conversion rate: converted / exposed
  • A Leading badge on the variant with the highest conversion rate

With low traffic, one or two lucky conversions can make a variant look like a runaway winner. Let the test run long enough that both variants have comfortably over 100 exposures before you call it. When in doubt, give it another week.

Common A/B tests to run first

  • Spawn vs new spawn via two different teleport commands, goal = D1 retention.
  • Join message v1 vs v2, goal = first purchase.
  • Free starter kit vs cheap $3 starter rank for newbies, goal = 30-day purchase.

Related