Hi 👋 I’m looking to use eye tracking on Linux to control the desktop mouse cursor position, with mouse clicks triggered via keyboard keybinds (no dwell clicking). Eye tracking would be used only for pointing, not clicking.
I’m considering Pupil Labs hardware/software and wanted to ask: is Pupil Labs a good fit for this kind of gaze-to-cursor control use case?
I’m particularly interested in how difficult it is to integrate Pupil Labs on Linux as a system mouse input (i.e. mapping real-time gaze data to cursor movement with filtering/smoothing).
Thanks in advance!
Hi @user-efc105 👋 ! Over the years, there have been several projects that implemented similar approaches using Pupil Core, many of which you can find in the community-contributed repository: https://github.com/pupil-labs/pupil-community?tab=readme-ov-file
In particular, these may be useful references:
- https://github.com/trishume/PolyMouse
- https://github.com/emendir/PupilCore-CursorControl
If you’re looking for a more modern and easier-to-use setup, I would recommend Neon, for which you may also want to check this practical guide on gaze-contingent and assistive applications: https://docs.pupil-labs.com/alpha-lab/gaze-contingency-assistive/#a-practical-guide-to-implementing-gaze-contingency-for-assistive-technology
In essence, both approaches shown here rely on using fiducial markers to remap gaze from the scene camera to screen coordinates. From there, it’s up to you whether you trigger interaction via dwell time, blinks, or an external input (e.g. keyboard). FYI The Neon tutorial is open source, so you can adapt it freely.
If fiducial markers (e.g. AprilTags) are too intrusive for your use case, you may also want to look at: https://github.com/pupil-labs/ir_plane_tracker
Hi!
I’m using Pupil Core with 3D gaze mapping (binocular, gaze.3d.01). Even when fixating ~2m away, gaze_point_3d.z seems to saturate around ~600–800mm. Also, it doesn't look stable. I’ve tried Screen Marker and Single Marker, but the depth behavior didn’t change much.
Is this expected due to binocular disparity limits at far distances? Is application-side depth calibration (e.g. mapping observed z to real distance using multiple known depths like 0.5m / 1m / 2m) the recommended approach, or are there other calibration methods that could improve depth accuracy?
Thanks!
Hi @user-2c3e08 👋 The 3D gaze point depth component provided by Core can be inaccurate, especially at viewing distances over 1 metre. But it also depends a lot on the physiological characteristics of the wearer. I've written about this before (a quick search for 'depth' will yield lots of messages), here's one example you'll want to check out: https://discord.com/channels/285728493612957698/285728493612957698/1140575743609409586 In order to measure depth more accurately, you might want to look at the head pose tracker plugin.
hi. I have some difficulties using a mouse, so I would like to use your project to control the cursor using a smartphone camera. Is this possible? Or it works with your devices only? Sorry for my English.
Hi @user-8d04f2 , do you mean that you want to use a smartphone camera to measure your eye movements and gaze?
I want to use on my pc (WINDOWS)
Thanks for the clarification. We do have tools for Pupil Core and Neon that let you use your eyes to control the mouse and click. However, please note that they are more intended as examples. If you are looking to discuss a more full-featured solution, feel free to reach out to us at [email removed] if that works for you. Otherwise, I can ask my colleague to reach out to you here on Discord.
by the way link from DIY is broken https://i.materialise.com/en/shop/designer/pupil-labs it redirected to main page. ty for support. i will check it all
Hi @user-8d04f2 , we are in the process of transferring that to a new provider, since the iMaterialise service discontinued. I will update you when it is online again. And, you are welcome.