Hey Rob, I was able to give this a try,
Hi guys. I wonder whether you can help me please? I'm interested in integrating Core headset eye tracking functionality into a Unity app and wondering what the best method is. I've read there's a hmd-eyes plugin and am aware that there are are Plugin APIs and Network APIs also, but note sure which method is best. We are interested in real-time performance, so efficiency and latency is important. I assume that I'd need to run the Core software independently of my Unity application and have some form of communication in between? Thanks for any pointers, Dave.
Hi @user-be1dcc , are you making a VR app or a desktop app?
Hi Rob. The output of the Unity app will be shown on custom AR hardware. We would run Unity on a PC/desktop and connect to the AR device. So, I think the answer to your question is that it's a desktop app.
Ok, so for that, you will want to look at hmd-eyes. Generally, how it works:
The hmd-eyes tool already has routines for all of this. It might already be sufficient for your use case and if not, you could certainly adapt it to your custom setup. Depending on the Unity version you are using, you might need to update portions of the tool, but it can certainly be used as a reference.
And, yes, the Unity app and Pupil Capture will run in parallel and communicate to each other over the Network API, which is based on ZMQ.
Thanks Rob. That's really useful, Dave.
Rob, another question on calibration. Is it possible to come up with a calibration mechanism that doesn't use the forward looking camera? Or does the Pupil Capure calibration not use that anyway?
The Pupil Capture calibration does use the forward looking camera. The hmd-eyes calibration, however, does not use the forward looking camera and follows the process described here (https://discord.com/channels/285728493612957698/285728493612957698/1469260595219009587). Using and/or referencing the hmd-eyes code would be a good way forward.
Hi @user-12944a (cc: @user-da7106 , @user-8d04f2 , @user-3f4a1b ), yes, sorry about that. We made the DIY geometry, including the frame, open-source. You can download it here and use any 3D printing service you prefer. Thank you for the reminder and we will update the Documentation accordingly.
Make sure to also take note of the printing recommendations:
We recommend printing using a SLS or HPSLS printer for best durabilty and fit. Services like shapeways and similar work well for this.
Excellent! Thank you so much! I've had my modified cameras ready to go on my own custom frames and was running into many problems that hopefully the official frames resolve.
Hi, I am using the Core to run a user study with some eye-tracking tasks. The system report "No camera intrinsics available for camera (disconnected) at resolution (192, 192)! eye1 - [WARNING] camera_models: Loading dummy intrinsics, which might decrease accuracy!" And there is nothing displayed in the Core. Could anyone please help me fix that? Much Appreciated! I have tried to reconnected and also in other computer, but same error report.
Hi @user-3a2026 , can you try the Restart with default settings button in the General Settings tab?
Hi Rob, thanks for your help! I have tried with the restart button, it works in my friend’s laptop but not working in my one. I also tried to reinstall the Core software but still not working. How could I fix it?
Hey, I'm a researcher using the pupil labs core in a flight sim, and I am having trouble with pupil detection in one of my night flight scenarios. Once the participant's eyes have adjusted to the darkness, I completely lose any confidence and the detection is all over the place. So far, I have adjusted up the max pupil size, adjusted intensity, and turned on auto exposure to try to fix this, yet the same issue prevails. Any assistance would be greatly appreciated
Hi @user-f7e8f6 , would you be able to share an example recording with [email removed] Then, we can provide more direct feedback.
Greetings from the Netherlands... I'm trying to setup a DIY frame with two eye camera's VID:0x0EDC and PID:0x2076 and a world camera: VID:0x1E45 and PID: 0x3035. Downloaded Pupil Labs 3.5.1 for my Windows 10 machine. Camera's work fine in the native Windows Camera app, but Pupil Player doesn't detect them. Think it has to do with libusbK, but can not find any references on how to fix this. Could anyone give me a hint where to look for documentation?
Hi @user-bcec99 , while we cannot provide dedicated support for DIY Pupil Core, other members of the community here might be able to chime in about their experiences. At the least, you can try the instructions here.
If that does not do it, make sure that the cameras are UVC compatible and support these features:
Hi! I’m looking to visualize a cumulative scan path , connect gaze samples (or fixation centers) in time order and show the path accumulating over time, optionally aggregated across multiple participants, overlaid on the world camera video. what should I do?
Hi @user-74b1c6 , may I first confirm that you are using Pupil Core? Or are you using a different eyetracker?
Yes. I am using Pupil Core and Pupil Capture V3.5.1.
Ok, one approach is:
If the recording is long, then extracting all frames, as per that tutorial, could take up significant harddrive space. In that case, you would be better off using pyav to go frame by frame, and look up the corresponding fixation for that video frame by correlating the world_timestamps.csv and fixations.csv files. The lookup technique remains similar to the linked Tutorial above. You can again draw a fixation point and save out an animation using OpenCV or Matplotlib.
We also have an Alpha Lab showing how we did it for Neon on a single Reference Image. Although the data formats and final output are different, many of the principles and even good portions of the code, remain the same.
Thanks! To clarify, I meant a group-level aggregated scanpath across participants. I have ~10 subjects viewing the same stimulus. What’s the recommended way to compute or visualize a single aggregate scanpath from multiple subjects’ fixations?
Hi @user-74b1c6 , as far as I know, a single aggregate scanpath is not conventional and not usually studied. May I ask what insight you are seeking with such a visualization?
My goal is mainly illustrative: to communicate whether there is a common viewing sequence across 10 participants (same stimulus/trial), and to compare two designs at a high level.
Ok. Since people will make different numbers of fixations, with different durations and positions, even when they share similar viewing behaviour, I am not immediately sure what to recommend for a single aggregate scanpath.
Beeswarm plots might be a more conventional choice here.
Alternatively, if you have specific Areas of Interest (AOI) that you want to study, then you can mark those as Surfaces in Pupil Player. Then, you could make a visualization that highlights which Surface received the majority of fixation 1 across participants, which received the majority of fixation 2, etc.
This is not a standard method, just an initial idea. Some care would be needed, because it would be misleading when only 2 or 3 out of 10 participants look at one AOI, while everyone else equally distributes their gaze around the scene.
Thanks, Rub, and one more question. With only 10 participants, is it acceptable to produce a heatmap as an illustrative visualization? Some references suggest that comparing heatmaps between conditions may require ≥40 participants for reliability.
This depends on the type of research and the question that is being asked. I am unable to provide dedicated support on this, although more data is better, but do check our Publications List to see how others have used Pupil Core. And you are welcome.
Hi Rob, thanks for your help! I have
Hi there, I am a research student investigating blink completeness (e.g., complete and incomplete blinking, defined as less than 75% of the eye covered). We were given the Pupil Core wearable eye tracker as our equipment. However, when my colleagues and I were trying to figure out how we could assess whether a participant had a complete or incomplete blink, we could not determine a method. Is it possible for the Pupil Core to assess blink completeness?
Hi @user-bde59a. Pupil Core does have a blink detector. However, it provides a binary classification for blink onset and offset based on pupil detection, rather than a continuous eye opening signal. As such, it's not optimised to compute percentage of the eye that's covered. You can read more about how the blink detector works in this section of the docs.
For eye openness, you would need to look at something like eyelid segmentation, similar to how our Neon eye tracker works. Or alternatively, manually digitise the eyelid opening aperture using the eye videos.
Hi Team! I have a hareware problem.. The wires on the JST connector broke off (shown in picture). What is the easiest way to fix this? Thanks!
Hi @user-fbbf44 👋. Sorry to hear about that! This system will need returning to our Berlin site for recabling. Could you please reach out to [email removed] mentioning this chat and include your original order ID? Someone from the ops team will look after you from there.
Hello, I've read quite a bit about using the pupil core with prescription glasses. However I haven't seen anything about the IR LEDs. My question is: I managed to get the prescription glasses to sit on top of the eye cameras such that the cameras do not capture through the lens but from below the prescription glasses. However, the prescription glasses end up blocking the IR LEDs. Would this have a large impact on the performance?
I have roughly tested it and it seems to be capturing well and estimating the pupil quite accurately (except for the extremes), albeit that the image is now a bit darker but i increased the brightness and contrast in post processing.
So I was wondering if anyone else has any recommendations or experiences to share?
Hi @user-4c4fec! As you've found, using Core with third-party/user spectacles is not an ideal condition, but can work for some. The goal is to capture the eyes from underneath the lenses, i.e not through the lenses, which it sounds like you've achieved. Before post-processing, you might also want to adjust the eye camera exposure settings. But that said, if pupil detection is working fine, a slightly darker image isn't something to worry about.
Hi Neil, thanks for the feedback. I've found that the most problems are when I have participants with the large glasses, those that are more square-ish than rectangluar. Rectangular prescription glasses do not pose as difficult of a problem, although positioning is still very limited and not ideal.
About tweaking exposure settings: I have tried changing the exposure timing, however this reduces the framerate of the eye cameras at high settings. I assume I should be aiming for the brightest picture I can achieve while keeping the fps at 120 (i think the model I have available only has 120fps eye cameras)?
Secondly, while adjusting post-processing parameters, should I be aiming for a darker pupil with lighter surroundings? However, I find that it does not increase the reliability of detection sometimes. At times, I see that the pupil detection is delayed as well, especially after a blink, it takes close to maybe half a second before the red and orange/yellow ring to appear on the pupil. Could this be due to the large amount of post-processing that has to be done if tweaked too much?
I also have a question regarding calibration. For context, I'm using the eye-tracker to conduct a subjective video quality assessment experiment. My setup is located in a fairly dimly lit environment. I have tried using the built-in screen marker and single marker calibration methods, however due to the white background, the screen is blinding and the user feels uncomfortable due to having to stare intensely at a bright white screen. Thus, I have settled for a single-marker calibration process, where I have edited the calibration marker to have a gray background.
This solves two problems: 1. I have to leave the world camera exposure to manual, as auto exposure will dim the camera image and this results in the apriltags around the monitor not being recognized. 2. The calibration process is not as blinding and uncomfortable for participants. The marker is also larger and easier to be detected.
The questions I have are: 1. Would changing the background color of the calibration marker be detrimental to the system? I have not noticed any problems and the bigger black donut circle seems to be detected properly (green circle appears in pupil capture when calibration is conducted)
Does the size of the calibration marker play a role? I found that the size of the calibration marker when using the built-in calibration methods look a bit small, even at a setting of 2.0. Thus I have enlarged the marker I am using as a replacement.
I only get a small calibration area that does not fill the whole monitor screen. Is this normal and expected? I am using the 3D pipeline.
If there is more information required, do ask and I will see if I can provide better descriptions. I apologize if i'm asking too much or unclear, but I would be very grateful if my questions can be answered.
Hi @user-4c4fec! Thanks for the detailed context.
On exposure/FPS: Core can run eye cameras up to 200 Hz depending on the eye camera resolution selected in the eye camera window settings. Rather than aiming for maximum brightness, the goal is to keep pupil detection confidence high (typically > ~0.7).
On post-processing: you’re aiming for good contrast between pupil and iris. Pupil detection should recover very quickly after a blink. ~0.5 s delay doesn’t sound normal, and it’s unlikely you should need extreme post-processing to get stable detection.
On calibration: changing the marker background colour shouldn’t affect calibration in principle, but with the default calibration choreography there’s typically no need to modify it. If the screen is uncomfortably bright, it’s better to reduce monitor brightness.
Marker size: the default is recommended and should be reliably detected; enlarging is feasible but usually shouldn’t be necessary.
Could you share a full raw recording including the calibration sequence so we can see what’s happening concretely (especially blink recovery and what you mean by the “small calibration area” in the 3D pipeline)? Please send it to [email removed] referencing this discussion 🙂