Hi @user-65c5e1 ! The RTSP packet doesn’t include Unix timestamps. Currently, NeonXR is a lightweight package that only leverages RTSP through WebSockets and doesn’t implement the additional functions available in the real-time Python client, such as timestamp handling.

There’s ongoing work on a client that can handle multiple streams and provide timestamps. A minimal example can be found here, but we can’t provide an ETA for it.

So, if you need multiple streams or Unix timestamps, you’d need to implement the RTCP SR and RTP logic as described here

Hi, @user-4ad7d3 - that's normal - the IMU requires calibration in order to track yaw properly

Well it's not "raw" so to speak, but pre-calibration is not possible. The IMU needs to be calibrated every time you use it

Hi @user-ffc425 , does it do this if you plug the Raspberry Pi into mains again? Or, if the eye camera is plugged into a standard Pupil Core headset and PC? Also, what do you get if you instead use cam.get_frame(0.05), as used in Pupil Capture?

Hi Rob, I should have been clearer earlier. #1. The mgpack.loads line takes 20 ms. We did this by adding GetSecs before and after this line and checking the time difference. Isn't 20ms itself too long given eye data updates every 5ms. # 2. Is there another way to access the norm_pos in gaze_positions which is randomly sized cell dict everytime the loads is run, other than using the for loop which might be time efficient? #3. The recv_string only takes 5 to 6 ms. Which I think is still ok. I did use tic n toc as well as GetSecs before and after each line to see the difference. #4. My purpose is to get real time values abd check if eye reached target. Currently I am getting a lot of delayed information. So by the time my target updates, the actual eye has already left the target and I never get a succesful trial.

Actually, that package is something of an adaptation of Pupil Core's Surface Tracker

Hi @user-da1455 👋

It looks like you're using the normalized gaze coordinates from the egocentric scene camera space — in other words, gaze position relative to the scene camera, not the screen.

That’s likely why the cursor control isn’t looking as expected. To control the cursor on a screen, you’ll need to:
1. Define a surface on the monitor using a surface marker.
2. Translate gaze coordinates into the surface coordinate system.
3. Use those transformed coordinates to control the cursor.

You can achieve this by using the Surface Tracker and subscribing to the surface topic in the network api, which gives you gaze data already mapped to the surface.

This has come up before — so you might find this past discussion helpful:
https://discord.com/channels/285728493612957698/285728493612957698/1199003211881775154

Hi @user-da1455 , following up on what my colleague, @user-d407c1 , wrote, you might also find this Python example helpful.

Hi @user-05ba05 👋 ! If you're looking for real-time blinks, fixations, or eyelid openness data, check out the Play Store for updates. 😉

Hi @user-da1455 , to clarify, the green rectangle and the pink rectangle are distinct:

• The green rectangle shows the region of the scene camera image within which the calibration is valid.
• The pink rectangle demarcates the boundary of your Surface, as defined by the AprilTags.

They do not necessarily need to coincide. So long as your gaze remains within the green & pink rectangle, then your gaze can be reliably mapped to the screen, even if the rectangles are shifted relative to each other.

It seems in your case that the calibration could be improved. That spiral of orange lines towards the middle (see attached zoomed image with red arrow) suggests that the person may have been looking off center when the calibration started and then they moved their head to comfortably look at the target.

It can help when doing a calibration to already have your head in position before starting and then, try and minimize head motion during the calibration. In other words, try to only move your eyes until the calibration has completed. Once calibration is complete, you can comfortable move your head again. At the end of calibration, you will see an accuracy display. 2.5 degrees or less should be sufficient for the mouse control script.

Hi @user-d1f142 , serial number refers to the serial number for Pupil Invisible devices, our previous eyetracker. With Neon, you want to check the module_serial field. Thanks for reporting. That part of the example will be updated.

I see you are using Python 3.13. That is the newest Python which is not fully supported throughout the Python ecosystem yet. Can you try again with a fresh virtual environment and Python 3.11 or 3.12?

Ah, Python 3.8 is not supported by that library. Python 3.11 would probably be the best choice right now.

Hi @user-e83888 , Pupil Core can sample data at up to 200Hz, depending on the power of the recording computer. We recommend Intel i7 and newer, or if you are on a Mac, then the M1 chip and newer.

To determine the average sampling rate in the case of your data, you could do the following:

duration = timestamps_seconds[-1] - timestamps_seconds[0]
sample_rate = len(timestamps_seconds) / duration

Sure. Our Python tutorials for working with Pupil Core data are hosted here. You may be interested in this one on working with the pupil diameter data.

You may also want to take a look at this third party library, PyPlr.

We don't really have any tutorial snippets that would be able to identify which of your trials contained pupil size measurements only within a given range, which I think is what you're asking me. But with that said, it is likely, as you suggest, that the confidence signal would be much lower during those trials in which the pupil size is measured as close to zero.

Hi, @user-f0ed79 - historically the async API was more performant, but they should be comparable now thanks to a recent update. Under-the-hood though, the simple API is just a wrapper for the async API, so it will always have at least a little more overhead

Thank you for clarifying. I wanted to make sure I fully understand your goal and implementation so I can provide accurate advice.

I’ve looked at the code you shared. Conceptually, it isn’t quite right—you’re adding an extra layer on top of our real-time API in the form of LSL, which is likely to increase latency rather than reduce it.

Instead, I’d recommend using our dedicated LSL plugin to stream gaze or pupil data: https://github.com/labstreaminglayer/App-PupilLabs/tree/master/pupil_capture

My suggested approach would be as follows: 1. Send your Psychtoolbox event corresponding to stimulus onset via LSL. 2. Stream eye movement data from Pupil Core via LSL.

This way, you can compare the two data streams post hoc and directly measure the latency between stimulus onset and gaze shift.

It’s important to note that this approach should isolate the latency measurement from any gaze-contingent or processing latency. That is, to obtain a screen-mapped gaze datum, we must acquire the eye image, run 2D pupil detection, generate a gaze datum, detect the AprilTags, map gaze to the screen, and finally, serve the result over the network. As you can imagine, each of these steps can add additional—and potentially variable—latency to your estimation. In my view, it’s better to isolate your core metric from as many parts of this processing chain as possible.

From what I can see, the gaze-contingent part of your experiment is mostly used to update the stimulus to a smiley face when it's gazed at. So for this part, processing latency isn't so critical.

Great. So the critical metric part is solved! As for the stimulus update, you'll have to live with the current constraints of the surface tracker in this case. As @user-f43a29 had stated in his message, a workaround would be to set the scene cam to 120 Hz. If you aren't able to track the markers well with this, then feel free to share a screen-capture showing why and we can try to provide some feedback on that!

Hi @user-e962d7 , I'm not in a position to provide explicit support for DIY Pupil Core projects, but the IR filter is found in the lens housing of the camera. What you have marked with the red circle is the camera sensor on the PCB. The video here demonstrates what you should be removing. I've attached a snapshot of one of the key points.

Hi @user-e962d7 , I am unable to provide explicit support to that extent on Pupil Core DIY projects. The main steps are contained in the associated Documentation. Otherwise, members of the community who have experience with this step should feel free to chime in 🙂

how do we access the raw data at 200Hz and the previous img to surface transformation values and calculate surface values ourseleves

This is already done for you by our software, but if you really want to do the mapping yourself you need three things * Gaze data * Camera calibration * A transformation matrix

So, * To do this in real-time, you'd want to subscribe to gaze to receive gaze points and surfaces.YOUR_SURFACE_NAME to receive the img_to_surf_trans transformation matrix. * To do this post-hoc you should make a recording with Pupil Capture, then load it in Pupil Player. Enable the surface tracker in Player and export data. This will give you gaze data in one CSV and the img_to_surf_trans transformation matrices in another CSV. * If you haven't calibrated the camera yourself, the default calibration data is in the source code

To convert your gaze points from scene camera space to surface points, you need to first undistort the gaze points using the camera calibration and then apply the transformation matrix.

If you want to calculate the transformation matrix yourself from a world frame, I recommend starting with OpenCV's homography tutorial

Hi @user-3bcb3f. Can you please share the full code you have used?

Hi, @user-3bcb3f - if you don't mind me jumping in here, I just wanted to add a couple of notes.

Core's surface tracker buffers gaze data and processes it in chunks when each world camera frame is captured. In other words, you should receive as much data as the gaze sampling rate implies - as long as a surface is always detected - but that data will come in chunks. You'll probably see 6 or 7 gaze samples in each chunk (assuming 200 Hz gaze / 30 Hz world)

That particular piece of code is receiving a group of samples, but only examining the last one (gaze_positions{end}), so it's a bit misleading (although I'm not really sure what the original purpose was for it). Another point about that snippet is that it's also measuring network latency. This will be small when using localhost/127.0.0.1, but not zero.

There are some other factors that will affect your observed surface-gaze-sampling rate. * Insufficient resources - if you don't empty the ZMQ buffer quickly enough, data will be lost. Not a problem for that snippet, but something to keep in mind * Missing surface detections - if no surface is detected in a world frame, none of the gaze data associated with that frame will be mapped and sent. People often use just 4 tags on the corners of their display, which is usually "ok", but I'd suggest 10-12 markers framing the display all the way around. Besides improving detection, it also improves accuracy.

Could you make and share a recording?

Yes, please

I think we will need you to share a full recording as asked by @user-cdcab0. This will be necessary to provide concrete feedback

Hi @nmt and @user-cdcab0 . I am sharing here in this folder https://1drv.ms/f/c/FB1DFB83EAA92E53/EstwP0SdCJVChBNMH90V4M8BFF8yl_MiNHKucwjK0gaDKA?e=iRTA9F I have added two recording: one at 120Hz ( 640x480 resolution) of scene camera and second one at 30Hz. While the recording was going on, I also ran the matlab code which uses zmq msgpack through matlab python client to access real time data. The matlab workspace variables norm_pos120Hz and norm_pos30Hz has the positions saved while the ts120Hz and ts30Hz has the time taken for the 300 runs of realtime surface gaze data access. Hope this will help you let me know what can I do to optimize realtime data access.

Additionally, I am also including a subfolder 'python_realtime' which has the python code I tried use to create a custom lsl stream that I am able to access in matlab (this is alos in the subfolder). This method seems to be able to get data at 30-35 ms delay in case of 30 Hz refresh rate of scene camer and around 10-15 ms delay in case of 120Hz refresh rate of scen camera. Previously, you mentioned that this method only will increase delay, but my observations are not the same. Could you suggest if this would be a feasible solution in my case?

Thank you for the detailed reply. 1. I guess that could be the reason for the delays I face with matlab? I am not using manual exposure mode but it was on aperature priority mode. For some reason I am not able to select auto mode. How do I choose the right exposure time? Can I set it at the bare minimum to be able to see the screen and surface tags clearly?

1. But the MATLAB file, along with this Python script, can access the latest surface gaze data with lower delays (upto 10 ms in case of 120Hz refresh of scene camera) while the purely MATLAB py.zmq. msgpack code takes ~ 45 ms read the latest sample ? In this case, do you suggest I replace my code with the file "pupil_capture/pupil_capture_lsl_relay/gaze_surface.py" you have created? If so, could you guide me on how to use it to obtain surface gaze datum?

2. I will add more markers around the screen of bigger size and see if the surface stabilizes. Also, can we put a higher resolution camera separately for the scene camera and overcome this issue?

Thanks for your answer. I’ll try and comment the result here