Image created by me out of the following free resources: Coordinates, Radio signal, Minecraft screenshot, Screen, Phone

This project has two key components: streaming the sensor data from the mobile device/sensor to the computer, and processing the incoming data to generate movement in the game.

Streaming data

Most mobile phones have an orientation sensor or a gyroscope which can be easily accessed. This real-time data then needs to be transmitted to the computer with -ideally- low latency. This can be done with a lightweight UDP connection. Luckily for Android users, there is a free app, Sensorstream IMU+GPS, which does this out of the box. Just select the desired sensors to stream, and specify the receiving IP address. Most likely, similar apps exist for iOS and different platforms.

Receiving and processing

The data stream can be received in the computer using a UDP socket listening for incoming data. This can easily be achieved in Python using the socket library. By default, if using the aforementioned Android app, the incoming port is 5555. Once the data stream is being received, the mobile device orientation angles can be parsed and the appropriate movement keys pressed by software. In more detail, if the player tilts the phone forward, it is interpreted as walking forward. Analogously, when tilting the phone backward. When the phone is turned left or right, the player looks around in that direction. These rotations are controlled by mouse movements, and are more challenging to implement by software. While plenty of libraries allow to control mouse movements from Python code, they do not seem to work with Minecraft, at least in Mac OS and Ubuntu. There is some evidence online talking about this working in Windows systems. In order to bypass this difficulty, raw player movement and direction can be accessed by employing the MCPI Python library released by Mojang a few years ago. This not only solves the problem, but it also increases responsiveness.

Two scripts can be found in the repository. Firstly, the Minecraft specific script which employs MCPI and works only with Minecraft. Secondly, a general purpose script which uses pyautogui to control the mouse. It does not work with Minecraft in Mac OS and Ubuntu, but it may work in Windows. Moreover, it does work fine in other 3D games I tried it with.

Real-time demo

Creative Commons License
MinecraftController by Alvaro Perez-Diaz is licensed under a Creative Commons Attribution 4.0 International License.