fxgl-intelligence

name	fxgl-intelligence
description	Integrate ML-powered intelligence features into an FXGL game via the fxgl-intelligence module — face detection from webcam, hand gesture recognition, hand landmark tracking, speech recognition for voice commands, and platform text-to-speech. Use this skill when building accessibility features, gesture-controlled games, voice-commanded gameplay, facial-expression-driven mechanics, or spoken NPC dialogue.
triggers	["face detection","gesture recognition","speech recognition","voice command","hand tracking","text to speech","TTS","webcam","ML","fxgl-intelligence","facial expression","hand gesture"]
compatibility	Java 17+, FXGL 21.x, fxgl-intelligence module. Requires Python ML backend running locally (see references/ml-backend-setup.md). TTS requires platform speech engine (SAPI/NSS/espeak).
category	fxgl/intelligence
tags	["fxgl","java","javafx","intelligence"]
metadata	{"author":"fxgl-skills","version":"1.0","fxgl-version":"21.1"}
allowed-tools	["Read","Write","Edit","Bash"]

FXGL Intelligence Module

Dependency Setup

Add to pom.xml (replace VERSION with current FXGL version):

<dependency>
    <groupId>com.github.almasb</groupId>
    <artifactId>fxgl-intelligence</artifactId>
    <version>VERSION</version>
</dependency>

The intelligence module communicates with a local Python ML service via WebSocket on configurable ports. See references/ml-backend-setup.md for how to install and run the Python service.

---

Face Detection

Detects faces via webcam and provides bounding box + landmark positions.

// Register
settings.addEngineService(FaceDetectService.class);

// Use in initGame
FaceDetectService faceDetect = getService(FaceDetectService.class);

faceDetect.setOnFaceDetected(faceData -> {
    // faceData contains bounding box and 68 facial landmarks
    double faceX = faceData.getBounds().getMinX();
    double faceY = faceData.getBounds().getMinY();

    // Example: move camera to face position
    getGameScene().getViewport().setX(faceX * getAppWidth());
});

faceDetect.setOnFaceLost(() -> {
    System.out.println("No face detected — pause game?");
    getGameController().pauseEngine();
});

faceDetect.start(0);  // webcam index 0 = default camera

Gesture Recognition

Recognises predefined hand gestures from webcam frames.

settings.addEngineService(GestureRecognitionService.class);

   getService(GestureRecognitionService.class);

gestures.setOnGestureRecognised(gesture -> {
     (gesture) {
         THUMBS_UP   -> jump();
         FIST        -> punch();
         OPEN_HAND   -> shield();
         PEACE       -> specialMove();
         POINTING    -> aimAtPoint(getInput().getMousePositionWorld());
         THUMBS_DOWN -> decreaseVolume();
    }
});

gestures.start();