Beating annoying minigames with Java☕ - Or: How to create a smart auto-clicker 🤖🎮

Last week was slow. After a rather busy holiday season, I felt like I needed to relax a bit (pro tip: do that occasionally; it does help!) and booted up my PS4 for the first time in months. My game of choice was one that I already played ad nauseam when I was a kid: Final Fantasy 10; only I was playing the HD remaster.

Wait, is this a story, or are you coding something today?

There will be code, promise! But to understand what the code is supposed to do, we first need to understand the problem, which is a rather specific one. Bear with me!

As with most RPGs by Square Enix, and with most Final Fantasy parts in general, the end game, right before the final story boss, is where most of the gameplay is hidden. I got there some time ago and started collecting the ultimate gear for all party members, levelling them up, etc., to face a hidden final super boss called "Penance". This can easily take up dozens of hours and includes a lot of repetitive gameplay, but alas. I like numbers going up.

The player must beat a series of minigames to acquire the ultimate gear for all party members. Some of them can be done in half an hour, and others require days of work and perfect button input to beat them, a single wrong input resetting the progress of the minigame so far.

One of these is called "Dodge the Lightning." In a specific region of the game world, called the "Thunder Plains", there are occasional lightning strikes to the player character that stun them for a couple of seconds.

This image of the Thunder Plains is taken from the Final Fantasy Wiki, all rights to their respective owners.)

Lightning is announced by a screen flash of around 400ms (believe it or not, I measured), followed by a roughly 1-second period. The character dodges the lightning if the player hits the cross button within that timeframe.

Now, to beat that minigame, the player needs to dodge 200 (two hundred) lightning strikes. In succession. Without a single mistake. In one go.

Ugh.

Right? The most I've managed was 47. The frustration was real. I was already spending hours waiting for a flash of light blue on the screen and pressing the button at the right moment. If a single car outside drew my attention a little too much, I missed the time window and had to start over.

This wasn't fun anymore. Or relaxing. At all.

I got up to get a coffee (haha, in a Java post, get it? Just kidding, I actually did get a coffee...) and started thinking: If this challenge is so dull and not fun, how could I alter it to make it more fun and fit my skill set better? I'm not used to precision work for hours on end.

So, if I can't do it, perhaps a machine can?

Opening up the borders

First challenge: How was I supposed to execute anything on the PS4? Running custom software on a console is tricky, and I did not want to tamper with my beloved PS4 too much. I didn't want to lose any game progress or, worse, brick it entirely. So, running software on the console was out.

Luckily, Sony has this nifty thing called "Remote Play" that lets users connect to their console using a computer (as if the console isn't a computer). The screen output is then visible in a window on the computer and can be interacted with.

For example, streamers on Twitch and YouTube do that a lot. Installing streaming software on a PS is more tedious than using an existing setup, grabbing the screen and streaming that.

However, there is a little issue: Remote Play no longer allows cross-button input in the latest build.

So I used Chiaki instead.

Chiaki is a Remote Play protocol client and allows any button input. Even with custom key mapping. Perfect!

Letting the machine do the job

Once I played PS on my PC (this sentence alone was already an adventure), I got to coding. I needed to detect the screen flash and get the timing just right to press the cross button, which I mapped to the "Enter" key.

I decided on Java since Java's AWT has a built-in class called "Robot" that does exactly what I wanted.

Step one was to figure out where to take a sample from. I decided to use the centre of the screen since I would put the PS4 Chiaki window roughly there anyway:

(Note: I'm leaving out any import statements since this is bog-standard Java stuff that doesn't require any additional dependency.)

public class Main
{
  public static void main(String[] args) {
    try {
      Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
      double width = screenSize.getWidth();
      double height = screenSize.getHeight();
      int sampleX = (int) width / 2;
      int sampleY = (int) height / 2;

      //More code later

    } catch (Exception e) {
      System.out.println("Oh noes: " + e.getMessage());
    }
  }
}

Now, I needed the Robot to sample the screen and repeatedly wait for a specific brightness. Through trial and error (quite literally), I figured that a brightness value of 75% was ideal. The screen would get bright quickly for about a frame or two, every other frame is either build-up (brightness increasing) or fade-out (brightness decreasing).

public class Main
{
  public static void main(String[] args) {
    try {
      Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
      double width = screenSize.getWidth();
      double height = screenSize.getHeight();
      int sampleX = (int) width / 2;
      int sampleY = (int) height / 2;

      Robot robot = new Robot();

      while (true) {
        Color c = robot.getPixelColor(sampleX, sampleY);
        float[] hsbColor = Color.RGBtoHSB(c.getRed(), c.getGreen(), c.getBlue(), null);

        if (hsbColor[2] > 0.75) {
          // Screen flash detected, time to spam-click!
        }

        Thread.sleep(16);
      }
    } catch (Exception e) {
      System.out.println("Oh noes: " + e.getMessage());
    }
  }
}

Perfect. This was already working quite reliably.

We're using the HSB colour space (which stands for "Hue, Saturation, Brightness") to determine the brightness value.

I've explained the HSB (or HLS in some cases) in my post on how to create a rainbow from scratch 🌈📐.

In Java, a Color object is always in the RGB (Red, Green, Blue) colour space, whereas the static function RGBtoHSB returns a float array that contains the hue of the colour at index 0, the saturation at index 1 and the brightness at index 2. The brightness value is between 0 and 1, indicating a percentage.

I also added a sleep of 16ms to the loop to prevent wasting CPU cycles on checking frames that didn't change. Chiaki allows setting the frames per second (FPS) of the mirrored screen, which was either 30 or 60. To get the timing right, I needed 60 FPS. A second has 1000 milliseconds. At 60 FPS, every frame is up for roughly 16-17ms.

This code detected all of the screen flashes, and through the magic of debugging, I was confident that I wouldn't miss any of the lightning strikes.

Last but not least, I needed to add the actual button press. Luckily, the Robot class can do that too:

public class Main
{
  public static void main(String[] args) {
    try {
      Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
      double width = screenSize.getWidth();
      double height = screenSize.getHeight();
      int sampleX = (int) width / 2;
      int sampleY = (int) height / 2;

      Robot robot = new Robot();

      int dodged = 0;
      while (true) {
        Color c = robot.getPixelColor(sampleX, sampleY);
        float[] hsbColor = Color.RGBtoHSB(c.getRed(), c.getGreen(), c.getBlue(), null);

        if (hsbColor[2] > 0.75) {
          Thread.sleep(120); // Simulates reaction time of a really _really_ fast human.
          for (int i = 0; i < 14; i++) {
            robot.keyPress(KeyEvent.VK_ENTER);
            Thread.sleep(50); // More or less the lower bound of the average length of a key press.
            robot.keyRelease(KeyEvent.VK_ENTER);
            Thread.sleep(50); // A little waiting time between releasing the button and pressing it again.
          }

          dodged++;
          System.out.println("Dodged: " + dodged);
        }

        Thread.sleep(16);
      }
    } catch (Exception e) {
      System.out.println("Oh noes: " + e.getMessage());
    }
  }
}

So, what happens here precisely? Once a lightning strike is detected, we wait for around 120ms, which simulates a ridiculously fast human, and then we spam-click the "Enter" button. Since the Robot class method keyPress is actually a keyDown, we also need to call keyRelease because it otherwise registers as a continuous button press.

This, however, lets us steer the duration of the key press. Some sources suggest that the average key press is between 50 and 300ms for humans, so I figured if the Robot is already reacting at twice the average human reaction speed, why not give it super-human button-pressing abilities as well?

Last but not least, the spam-clicking is essential here. If the network connection or Chiaki has a bit of delay, the spam-clicking guarantees that at least one button press gets through to the game.

A word of caution: Beware of auto-clickers

During development, I once made the mistake of opening a different window with a white background. White means a brightness of 100%, which is larger than 75%, which means spam-clicking Enter.

Needless to say, this almost wreaked havoc by constantly pressing Enter. It started all sorts of other apps, basically behaving like a fork bomb after mere seconds. My CPU almost melted. And worst of all, I missed a lightning strike and had to start over.

So, did the auto-clicker work in the end?

Yes. And just for the flex, I dodged 222 lightning bolts.

Take that, game!

I hope you enjoyed reading this article as much as I enjoyed writing it! If so, leave a ❤️! I write tech articles in my free time and like to drink coffee every once in a while.

If you want to support my efforts, you can offer me a coffee ☕! You can also support me directly via Paypal!