thanks @rob

After repeating the experiments several times, I found that the accuracy loss caused by transferring a calibration between recordings is actually acceptable. The more significant issue is the gaze estimation error introduced by the mismatch between the calibration distance and the actual viewing distance, which can vary considerably during use.

To clarify my workflow, the custom images I mentioned are simply images similar to the example shown below, with the markers placed at different positions. I do not perform calibration during recording. Instead, after recording, I use the marker detection feature in Pupil Player together with Post-hoc Calibration to generate the calibration.

Regarding your second point, I have not yet come up with a suitable approach for multi-distance calibration. My intuition is that allowing the participant to move during the calibration process would likely introduce a substantial loss in calibration accuracy, although I have not tested this systematically yet.

Hi @user-7c9706 , would it be possible to share one of the recordings with [email removed] Even a test session where you run it on yourself would be helpful, so that we can see exactly how it is being done and what the issue might be.

Hi @rob Thank you for your guidance. I have sent the three recordings to data@pupil-labs.com for your review.

Please let me know if any additional information or clarification is needed.

Hi @user-7c9706 , thanks for sending some example recordings.

We have taken a look and it will be useful to clarify a few points about Pupil Core:

• First, it seems you changed to the narrow-angle lens, which will produce spurious results without recalculating the intrinsics. We would recommend switching back to the default lens.

• Next, you do not need to present calibration targets of different sizes or at different distances.This is not standard and could actually make calibration worse.

• Then, the green calibration rectangle that is shown in Pupil Capture does not need to be as large as possible. The default 3D calibration pipeline can extrapolate outside of this boundary. Trying to make it as large as possible can actually make calibration worse.

• After that, when running the calibration, it is important to fit the 3D eye model first and only then do a calibration. In your recordings, it looks like the 3D eye model has not been fit.

Try implementing these recommendations and let us know how it goes.

Thank you for your help.@rob

1.Regarding your first point, I did not switch to a narrow-angle lens—I am using the default lens. However, when using a Raspberry Pi as a relay, the video stream in Pupil Capture appears to be distorted, so camera intrinsics estimation seems necessary. The issue is that grayscale images cannot be used for intrinsics calibration, and using grayscale leads to the error shown in the screenshot. At the moment, neither the transmission speed nor the Raspberry Pi’s computing power can fully support color video at my target settings. Therefore, I am currently trying to perform intrinsics calibration using low-frame-rate color video first, and then switch back to grayscale for the actual experiment. If you have a better solution, I would really appreciate your advice.

2.Regarding your comments on calibration targets and the calibration rectangle, should I understand correctly that even in a scenario where the gaze target distance varies, the standard 3D Screen Marker Calibration Choreography is still the recommended approach?

3.Regarding the 3D eye model fitting, I will try this after resolving the camera intrinsics estimation issue.

Thank you!

Hi @user-7c9706 , I see. Here are my responses:

• You may want to check with the creator of that video backend plugin to see what a recommend solution is. We are unable to provide support for their package.
• Yes.
• Understood. Once you are ready to evaluate calibration accuracy again, it is important to fit the 3D eye model first.

Hi @rob , I see. Thanks a lot!

Hi Rob. I wondered whether you could give me some pointers please? I'm using Pupil Capture/Service together with the Unity hmd-eyes plugin with two Core cameras and the associated USB hub (no world camera) attached to a custom AR device. My issue is with the quality and stability of the gaze output vector which seems to be very noisey/jumpy and suffers from occasional jumps of angle. The view of the eye from the Core cameras is good, but I note looking at the behaviour of the 3D model in Pupil Capture that there is signficant jitter and some instability. It's not clear to me whther this is behaving as it should. I've had a good look at the camera res (400x400) and update rate (120Hz), and changed various options to try to improve the stability of the output without luck. Do you have any recommendations or pointers that might reduce jitter and improve stability please?

Would you be able to share an example recording with us at [email removed] Then, we can provide better feedback.

Yes, probably. I'll see if I can record some.

Just a standard Pupil Capture recording will be sufficient.

Just sent.

Hi @user-be1dcc , thanks, but the eye camera videos are missing. Is it possible to supply those, too?

Yes, probably although they make the data large obviously.

If you can do a brief recording with yourself demonstrating the issue, that is sufficient. The eye videos will be necessary to assess what has gone wrong.

Thanks for sending the new recording.

So, the eye cameras are well positioned, the images are clear, and the 3D eye models look decently fit. However, it seems the wearer's eyelashes are a bit darker, which seems to be intermittently confusing the Pupil Core's dark pupil detection algorithm.

Or, it might be the darker regions at the top and bottom of the iris, where it meets the eyelid. Do you have additional IR illumination in your setup?

You could try using the ROI method to force the algorithm to just focus on the region of the image that usually contains the pupil.

You can also consider the tips here.

Thanks Rob. I made a slight discovery. If I freeze the model in the Pye3D detector, then the stability of the virtual eye ball in the Debug Visualiser window is much improved. However, when I then try to calibrate using the hmd-eyes method, there is no data output immediate following calibration. Any ideas why this might be?

Is it possible to disable the Pye3D detector?

The 2D and 3D detection pipelines are always running in parallel and you can choose which is used for calibration & gaze mapping. May I ask why you want to fully disable it?

Hi @user-be1dcc , yes, that will improve the stability of the 3D eye model, but you only want to freeze it after it is well fit. Aside from that, freezing the eye model will have no influence on the jittery pupil detection that you reported and which was seen in the example recording you sent.

The foundation of the Pupil Core pipeline is a robust and stable pupil detection, so you will want to improve that before going to the 3D eye model.

With respect to the hmd-eyes method, could you clarify how you are doing it? Did you write your own calibration script in Python or are you now running a Unity program that uses the hmd-eyes plugin?

Hi Rob. We are using Unity and the example hmd-eyes scripts, Game Controller, Visualiser, etc... embedded in to our Unity application. We are calibrating using the in built mechanism that this provides. With respect to the Pye3D influence, freezing the model significantly improves the stability of the virtual eye and it still seems to track fine. However, using the Unity calibration method, it doesn't receive any data post-calibration.

Is it possible to get a 2d output through the Unity hmd-eyes method I wonder?

When you say 2D output, do you mean x/y coordinates in Unity pixel space? Or do you mean using the 2D pupil detection pipeline of Pupil Core?

I see, so your AR application is potentially using passthrough in a VR headset?

Actually, now I've looked a bit closer, I think it's the reflection of the illuminating LEDs causing the instability in particularly when the glint coincides with the edge of the pupil. Is this something you've seen before Rob?

Hello, I'm having trouble with the core headset seemingly not able to consistently estimate where my gaze is - the pupil ID confidence remains high, but when calibrating, the centre of targets flicker green and red, and in player the gaze location looks like it is flickering and drifting. I've tried resetting all of the settings, rebooting, etc. but nothing changed. Any insight would be appreciated, thanks!

Hi @user-6c482a 👋 ! Could you please share a recording that includes the calibration routine with [email removed] This will allow us to review it in context, and provide more tailored feedback on what might be causing the flickering and drift you're observing.

If possible, please mention this Discord post when sending the recording so we can correlate the information more easily.

Hi @user-6c482a 👋 ! I had a look at the recording you shared. Overall, it doesn't look too bad, but there are a few things you can do to improve the gaze estimation.

First, it appears that you're only using a single eye camera. In general, using both eye cameras is more robust, especially in situations involving blinks, partial occlusions, eccentric gaze angles, or corneal reflections that may prevent one camera from properly detecting the pupil.

The main thing I noticed is that the eye model was likely not fully established before calibration. Ideally, before calibrating, you should spend some time rotating your eyes and looking around so that Pye3D can build a good 3D eyeball model. Have a look at our eye model fitting video, which demonstrates this process.

Since the recording has already been collected, the best option now is to reprocess it post hoc. In Pupil Player, open the Pupil Positions plugin and run Post-Hoc Pupil Detection. This will redetect the pupils throughout the entire recording and rebuild the 3D eye model. You can tell when the model is fitting properly because you'll see a blue circle overlay that closely matches the shape of the eye.

This step is particularly important if you're using 3D gaze mapping. You can read more about the available gaze mapping methods here. In many cases, 2D gaze mapping can actually provide better accuracy than 3D mapping, so it may be worth trying both and comparing the results.

If you recorded the calibration data, you can also rerun the gaze mapping post hoc after rebuilding the eye model. That will generally give you the best accuracy you can achieve from the existing recording.

Hey! Hoping someone can sanity check a plan before we order. I want to add eye tracking for my thesis (real-time monitoring) on a budget, so instead of a full Core headset we're thinking of buying the 200Hz Eye Camera accessory and 3D printing our own frame using the published mount CAD.

A few things I'd like to confirm:

1. Does the released camera-mount CAD cover the current 200Hz accessory Eye Camera (Right/Left)? i.e. will a frame printed to that interface actually accept it?
2. Which headset/frame revision should our printed mount match for the current 200Hz camera? (I saw the mounts carry the revision number of the headset they were designed for, so want to print the right one.)
3. The triangle-mount geometry in the repo looks to be STL, but our printer workflow only takes STEP would converting this cause any problems? And any tips on print process and tolerances for a reliable slide-fit on the mount rail?
4. How is the camera powered, and how does data transfer to the laptop; wired USB and bus-powered over the same cable, or is there a separate power/ wireless link?
5. Once connected, is it plug-and-play in Pupil Capture or does it need extra config?
6. Can the camera be used for real time analysis i.e. live gaze/pupil data streamed to our own application as it happens (low latency), rather than only recorded for post-hoc processing?

For context we'd likely run one camera for monocular tracking. Thanks so much!

Hi @user-29ff85 , first, please note that you will also need a World Camera, in addition to an eye camera. Otherwise, you will not be able to calibrate (nor know what they are looking at) and thus, you would not have gaze data.

Considering that the DIY World Cameras are harder to find these days and sometimes users report trouble with newer cameras that are not fully UVC compliant, it could be worth it to still consider a standard Pupil Core.

Also, keep in mind that eye tracking monocularly has inherently less accuracy than tracking both eyes, but depending on your research goals, that might not be a problem. Considering that, to provide better support, may I ask what you research goals are?

Here are responses to your questions:

1. Yes
2. Simply use the latest in the Github repository
3. You should be able to convert to STEP without problems. Tips on 3D printing are here.
4. Wired USB 2.0; same cable. If it helps, you can order a decased Pupil Core, so just the cable tree.
5. The eye camera will be plug-and-play. You will just need to install the drivers, which Pupil Capture should do automatically when you start it. The world camera should also be plug-and-play, depending on which you choose.
6. Yes, Pupil Capture fully supports real-time streaming. You can find more details in the Network documentation.

If your application and Pupil Capture run on the same computer, then you will have extremely low latency. Otherwise, it depends on the quality of your network.

So my recommendation would be:

1. Run Post-Hoc Pupil Detection.
2. Verify that a good 3D eye model is obtained.
3. Recompute the gaze mapping.
4. Compare both the 2D and 3D gaze mapping methods.

That should noticeably improve the quality of the gaze estimates. For future recordings, ensure the model is properly built by rolling the eyes (you can also check them in the debug option of the eye windows) , use both eye cameras if available.

Thanks, this is really helpful! On the research goals; at a high level it's a human factors study using pupillometry to estimate cognitive workload and attention in real time while participants interact with a system. So we need two things: reliable pupillometry (pupil diameter as a physiological index) and gaze/fixation behaviour, streamed live to our own application running on a different computer.

Given that, a few questions:

1. The DIY guide recommends the Logitech C615 as the world camera would that specific camera still pose the calibration / UVC-compliance issue you mentioned, or is it known to work reliably for calibration and gaze mapping?
2. Where can I find a quote for the decased pupil core? And when you say just the cable tree, do you mean that that's a component I can purchase separately?
3. If we go with the full DIY set instead, can it also run over a single USB 2.0 cable for both power and data straight to the laptop, the same as the decased Core; or does the DIY build need any intermediate hardware between the cameras and the laptop (e.g. a microcontroller like a Raspberry Pi Pico) rather than connecting directly?

Thanks again for all the detail!

Hi @user-29ff85 , you are welcome.

Here are my responses to your questions:

• Yes, our DIY guide recommends the Logitech C615 since it is UVC-compliant and works with the Pupil Capture software.
• Please send an email to info@pupil-labs.com to receive a quote for a decased Pupil Core. And yes, the cable tree can be purchased separately. They can also provide you with an image of it.
• The full DIY set does not need any intermediate hardware. It connects directly. The DIY guide links to the full Bill of Materials and the instructions there are complete.

However, please note that we are unable to provide dedicated support for DIY projects, so if you run into issues, we will unfortunately not be able to assist in that case.

Hi team — I have a Pupil Core (academic) and need help with the left eye camera (eye1 / Pupil Cam2 ID1). The ball joint on the left arm separated while adjusting it and the orange flex is now exposed. Since then eye1 won't produce video. What I've confirmed:

eye1 still enumerates — Pupil Capture shows "Found device. Pupil Cam2 ID1 @ Local USB," and Windows chimes when I unplug/replug it, so power/detect lines are intact. But it's stuck at 0 FPS, blank image, cycling "Camera disconnected. Reconnecting…" — same at 120/60/30 fps, after refresh and "Restart with default settings." eye0 works perfectly on the same PC/drivers/USB, so it's isolated to the left arm. Tested direct on USB 3.0, drivers installed as admin.

Looks like the flex data lines broke in the separation. Could you advise on a replacement left eye-camera arm or repair? Happy to share photos/video. Thanks!

Hi @user-9fce12 👋 ! Sy to hear that your eye camera is not working. Since you've also reached out via email, I've replied there so we can keep the communication streamlined and avoid duplicating troubleshooting steps across different channels.

Let's continue the investigation over email, and we'll be happy to help you get this resolved.

Hi. I have a question about the hmd-eyes Unity plugin. Does anyone know what the typical update rate is for the data output received at the Unity end? Is it independent of the camera capture rate (30,60,120 Hz, etc..) or is it related? Thanks for any pointers.

Hi @user-be1dcc , the upper limit will be the sampling rate of Pupil Capture on your system, but the rate in practice will depend on how you obtain the data within your Unity application. For instance, if you obtain it within Update on a GameObject, then the refresh rate of your Unity app will be the upper bound. You could alleviate this by receiving data in a separate thread. The Demos in hmd-eyes are also a good place to see some approaches.

Thanks Rob. I believe the examples use a 'OnReceive' callback which presumably is called independently of the Unity Update() function? I suppose what I'm trying to understand is whether the Pupil Service just spits out data at the camera rate or whether there is a 2 way communication between the hmd-eyes plugin and Pupil Service whereby the plugin sends data requests and PupilService responds. So, for example, if my Unity app runs at 60Hz but my cameras are running at 30Hz then presumably I'm losing out. However, if I run the cameras at 120Hz, will the OnReceive function get called at 120Hz or is it limited somehow?

@user-be1dcc , I went to the code to refresh myself. It works as follows:

• The SubscriptionsController creates Subscription instances for each data stream you subscribe to (e.g., gaze, blinks, etc).
• Within its Update function, it calls the UpdateSubscriptionSockets function, that loops through all subscriptions and polls each for new data.
• That data is then parsed and passed to the associated OnReceiveData callback.

So, to answer your question:

• Pupil Capture/Service send data over the network as fast as they can (this is dependent on the power of the system running them).
• Any subscriber can receive that data, but if they want all transmitted data, then they need to consume it as fast as possible, since the PUB-SUB arrangement of ZeroMQ is being used (relevant hmd-eyes code here).
• So, if your cameras run at 120Hz, this does not mean that OnReceive is called at 120Hz, even when the data are transmitted at 120Hz over the network. The rate at which OnReceive is called is dependent on how you integrate it into your Unity app and the power of the computer running that app.

To get an idea of the effective sample rate in your setup, you can save the incoming data with Unity timestamps and then compute the sample rate from those.

Hello! I need some help with the surface tracker plugin. When I recorded with pupil capture, my apriltag markers were not detected. Though, I thought of trying with pupil player afterwards. While doing so, no surface were detected and I cannot add a new surface without the detection of the markers. Do you have any solution for this issue?

Hi @user-69c9af , you can share the recording with us [email removed] but if the markers were not detected during the recording, then this does lower the chances of detecting them afterwards. However, once we see the recording, then we can say more.

Hi @user-69c9af , thanks for your email. I have taken a look. The AprilTags look primarily a bit too small and a bit overexposed/blurry. A good rule of thumb is if it is the tags are not clearly visible to you in the image, then they will not be easily detected by the algorithm.

If you want, you can print out the tags and place them around the monitor. You do not need tags around each individual AOI.

The quality was good when we recorded, and I've done it before with no issues. We think it might be because the screen is not flat. We are done with the recordings so apply what you said, but is there a way we can generate ourselve, for the whole recording, the surfaces?

Ah, it is probably not related to the screen not being flat, at least not based on what I see in the video. When you say the quality was good when you recorded, do you mean in the Pupil Capture preview? I only ask, because in your original message, it was said that in Pupil Capture, the tags were not detected.

Would you be able to send the original recorded data from Pupil Capture to [email removed] rather than a video made with a phone?

Yes the tags were not detected but the quality was better than after recording. I never got the issue before with other screens, and it detected when we did the pre-test, but when we started the experiment, it stopped detecting the qr codes. We thought we could manage to create our own surface in PupilPlayer, but apparently it's not possible or very complicated.

The video has confidential info in it, that is why you received a video made with a phone.

Thanks for the clarification. The primary reason why AprilTag detection usually fails is because of the tags being too small and/or overexposed, which seems to be the case in the example video. If you want, you could try using an AI enhancement tool on the video. We have had a report from one user that running such a tool in Adobe Premiere on their video did actually improve AprilTag detection when the tags were overexposed/blurry.

We'll try to crop the video and send it to you

Ok, if possible, that would be helpful. Thanks.

Thank you, we will try if we can to use AI enhancement.

@rob Hi there. I was wondering whether you made the CAD models for the Core camera module available ? the ones in the docs are for webcams. Thanks!

Hi @user-9a85ca , do you mean for the world camera or the eye cameras?

world camera!

Ok, may I ask why the world camera mount geometry and the world camera reference mount in the full DIY STL are not sufficient? Those are for the world camera that we supply, not for webcams.

I'll transfer that to my colleague working on the project. Thanks.

we'd need the geometry of the camera case itself to 3D print a brace that would clip on it (to stick mocap markers) hence the need for the file. From what i've seen (through github viewer), the geometry of the world camera module is a primitive cylinder ?

I see. We have not released the geometry of the world camera itself. Rather, a world camera can be obtained through us. If you would like a quote, please send an email to info@pupil-labs.com

I have one already. We will make the measurement and cad file ourselves. Thank you very much for the quick replies !

Understood and you are welcome!