AI - Fitness Game Using Body Pose Detection (Difficulty: 3)

CreatiCode

Key Topics Covered

Introduction

In this tutorial, you’ll create a fitness game using AI-powered body pose detection. The game will display a target pose on the screen, and the player must quickly match that pose using their body.

Step 1 - Start a New Project

Create a new project in the CreatiCode Playground.
Delete the sprite named “Sprite1” and rename the “Empty1” sprite to “Main”.

Step 2 - Enable Body Part Detection

Add the body part detection block. Set debugging to “yes” to highlight the key body points on the video feed.

A new table named “i” will automatically appear under the Variables category. Check its box to display it on stage.

Once you run the program, you’ll see the body part markers on screen, and the table data will update in real time:

Note: To capture your entire body in the camera frame, you’ll need to stand farther away—but this makes it difficult to read the table data on the screen. A good solution is to work with a classmate: one person performs the poses in front of the camera, while the other monitors how the data in the table changes.

Step 3 - Add the Squat Costume

Now, uncheck the “i” table and add a costume representing a squat pose. Use the costume named “Dorian-d”, and rename it “Squat”.

Set its size to 130 and position it at the bottom right of the stage. This will serve as the first target pose.

Step 4 - Understand the Squat Pose

We know where each body part is, but we don’t yet know how to consistently recognize the squat pose. To do that, focus on the X-positions of these 4 parts:

Left hip
Right hip
Left knee
Right knee

In a squat pose, the knees spread out wider than the hips. So:

Knee Distance = Right Knee X - Left Knee X
Hip Distance = Right Hip X - Left Hip X

After we calculate these 2 distances, we can use them to determine when the player is in a squat pose.

Step 5 - Read Data from the Table

Create 4 variables to represent the X positions of these 4 body parts: “left hip x”, “left knee x”, “right hip x”, “right knee x”.

Display the “i” table and note each part’s row number. For example, the left hip X position is in row 12, under the column x:

Therefore, we can read this value into the “left hip x” variable like this:

Repeat this process for all 4 variables:

Step 6 - Calculate Distances

Create 2 more variables:

hip distance
knee distance

Now calculate the distances:

hip distance = right hip x - left hip x
knee distance = right knee x - left knee x

Step 7 - Determine Pose from Distances

Create a variable called pose. Use a simple rule like the one below to classify the pose:

If knee distance > 2 times of hip distance, then it’s a Squat
Otherwise, it’s Unknown

Step 8 - Organize with a Custom Block

Group the logic for pose detection into a new block called “calculate pose”, and move the related code into it.

Step 9 - Handle Invalid Data

Sometimes body parts are not detected and return a value of -10000 in the table.

To avoid errors, only calculate pose when all 4 X values are > -250.

Step 10 - Continuously Calculate Pose

Since the table updates continuously as the player moves, wrap the pose calculation inside a forever loop:

Step 11 - Detect Success

Now check if pose = “Squat”. If so, show a success message:

Result:

Step 12 - Set the Target Pose

Now we have the basic game mechanics worked out. The next step is to make it more fun. We will give the player more than one target pose, so the player has to try to complete these poses to earn points.

As a first step, we should create a new variable “target pose”, which will represent the next target pose the player needs to make.

We can rewrite our code using this new variable:

Step 13 - Switch Between Squat and Stand

When the user has completed the target pose, we should change the target to a new pose. To keep it simple, let’s make the target toggle between 2 poses: “Squat” and “Stand”.

We can define a new block “change target pose” for this purpose. If the value of “target pose” is “Squat”, then the next target should be “Stand”. Otherwise, if the value of “target pose” is not “Squat”, it must be “Stand”.

Step 14 - Create the Stand Pose Costume

To show that the target pose has changed, we should change the costume of the sprite as well.

Please find the “Dorian-c” costume and add it to your sprite. Rename it as “Stand”.

We need to make some small adjustments to the costume. The leg on the left is reaching out sideways. We can select the whole leg and rotate it to an upright position:

In addition, the head is looking sideways, which may confuse the player. We can remove this head, and copy the head from the “Squat” costume:

Step 15 - Switch Costume with Target Pose

When changing the target pose, also update the costume accordingly:

Step 16 - Add Visual/Sound Effects

To give the player clear feedback that the target pose has been completed, we should show some visual effects. For example, we can make the sprite brighter for a short time before changing to the next costume. Also, we can play a sound for collecting a coin as well:

Step 17 - Detect the Stand Pose

In the “calculate pose” block’s definition, currently we only check for the “Squat” pose. We also need to check if the player is in “Stand” pose. To keep it simple, we can still look at the distance between the knees and the distance between the hips. If the player is standing up, the knee distance should just be slightly more than the hip distance. For example, we can require that the knee distance is less than 1.2 times the hip distance:

Now our game is ready. Whenever the player hits a target pose, we switch to the next target:

Additional Challenges

Here are some fun ways for you to build upon this project:

Game Rules: You can design different rules for playing the game. For example, you can give the player a time limit (e.g. 1 minute), and whoever earns more coins wins. You can also require the player to complete 60 target poses, and whoever finishes in the shortest time wins.
Improve the Pose Detection: Currently we only look at knees and hips. You can also add additional checks on the arms. For example, you can require the arms to be held horizontally. You can compare the Y position of the wrists with the shoulders.
Different Target Poses: You can try to use some other poses. For example, make the player do an “X” pose like below. You will need to use the X and Y positions of the wrists and ankles to detect such a pose. Note that you can also generate more positions using the AI image generation tool (first generate an image for a standing person, then generate a variation).

2-Player Mode: When 2 players are standing in front of the camera, both can be detected by the body pose detection engine. This allows you to create 2-player games. For example, whenever player A makes a pose, we set that to be the target for player B. This way, player B has to copy player A’s poses.

CreatiCode Scratch Forum