AI for Hand Detection

CreatiCode

Description

In many AI applications, it’s useful for the program to “see” the user’s hand gestures and respond to them. For example, hand waves could control a character, or finger poses could trigger actions in a game.

In the CreatiCode Playground, you can use a special AI block to automatically detect hands using your computer’s front-facing camera:

This block does all the hard work for you — it recognizes the hands, tracks their position, and gives you detailed data to use in your program.

️ Block Inputs Explained

This block takes 3 input parameters:

Output Table
This is where the hand data will be stored.

When you add this block to your code, a table will automatically be created for you.
You can rename the table if needed.
The data inside this table updates in real-time as the camera detects hand movements.

Debug Mode
Set this input to “yes” to enable visual debugging:

The hand will be drawn over the camera feed using 21 landmark points: 1 for the wrist and 4 for each of the 5 fingers.
Red dots show the left hand, and blue dots show the right hand.

3. Show Video: If this input is “yes”, then the video from the camera will be displayed on the stage as a background. If this input is “no”, then the video will not be displayed, which will make the program run faster. If your program simply uses the data in the output table, then you can choose not to show the video.

The Output Table

When no hand is detected in the camera video, the output table will be empty. It contains these 7 columns.

When one hand is detected, the table will be populated with 47 rows of data. If 2 hands are detected, another 47 rows will be added below, making it a total of 94 rows.

For each hand, there are 3 types of data, and each type uses a different subset of the columns.

Finger Pose (Most Commonly Used)

For each hand, the first 5 rows contain the curl angle and direction of each finger. These 5 rows contain sufficient information for most simple use cases.

When a hand is detected, only the first 4 columns are populated with data for these 5 rows:

Hand: This column tells you whether this is the left hand or the right hand.
Part: This column tells you which finger this row is describing.
Curl: This is the curl angle of that finger, or how much the finger is bending. It should be a number between 0 and 180. For example, for the index finger, when it is straight, the curling angle is 180 degrees. As you curl your finger, this angle drops down to as low as about 40 degrees.
Dir: This is the pointing direction of this finger. It uses the same direction system as the “point in direction” motion block. For example, when the finger is pointing up, the direction is about 0 degrees. When the finger is pointing to the right, the direction is about 90 degrees. For the left direction, it is about -90 degrees. For the down direction, it is about 180 or -180 degrees. Of course, the actual direction value will not be exact, so your program needs to check if the value is in a range. For example, if the direction is more than -30 or less than 30, you can treat it as pointing up.

2D Finger Position On Stage (Advanced)

Below the 5 rows of finger pose data, there are 21 rows of data describing the X and Y positions of the 21 landmark points. 4 columns are populated with data:

Hand: This column tells you whether this is the left hand or the right hand.
Part: This column tells you which of the 21 points this row is describing. The first row represents the “wrist” point, followed by four rows for each of the five fingers. For example, for the thumb, the 4 points are “thumb 1”, “thumb 2”, “thumb 3” and “thumb 4”, from bottom up.
X and Y: These values use the same position system as the sprites, with the center of the stage at X = 0 and Y = 0. For example, in the picture below, the wrist is at the bottom center of the stage window, so its X position is around 0, and its Y position is around -180.

3D Finger Position On Stage (Advanced)

Below the 21 rows of 2D finger position data, there are another 21 rows of data describing the 3D positions of the same 21 landmark points. 5 columns are populated with data:

Hand: This column tells you whether this is the left hand or the right hand.
Part: This column tells you which of the 21 points this row is describing.
X/Y/Z: These values use a 3D position system. If you use a small box to represent each point, then when you hold your hand forward, these boxes will form a 3D hand facing forward (same direction as your real hand):

You can open this link to play with this demo:

play.creaticode.com/projects/ef0d19a26944613ac1805db7