Posture monitoring desktop application
About this project
A desktop application using live data from the user's camera and Google MediaPipe to detect slouching and notifiy the user. It also tracks posture over time to monitor progress
Key Features
- Fully local: The application is fully offline: images processed locally to determine the posture of the user, then are discarded. No images are ever stored. Historical data (i.e. how long the user spends slouching) is stored locally in a SQLite database
- Detection of poor posture: slouching back, leaning in, head tilts...: Google MediaPipe provides coordinates for different body parts based on images from the machine's live camera, which are then used to determine the user's posture
- Notifications when bad posture is detected: The program notifies the user of bad posture. Notifications are automatically dismissed when the user is sitting straight again
- Track progress over days, weeks and months of usage: The application stores timestamps of changes in posture, allowing it to track how long the user has had good or bad posture on a specific day, week or month.
Challenges
The main challenges of this project was to determine the posture of the user based on coordinates and to communicate between a Python server and a Rust client. The server uses opencv and mediapipe to capture images from the camera, and get coordinates for specific points of the upper body (ears, eyes, shoulders). This data is sent over a socket to a Rust application, which uses it to determine the user's posture, trigger notifications, and save changes to the database
Learnings
This project was an occasion to experiment with TCP socket communication, as well as a chance to experiment with the Iced Rust framework to build desktop applications