Hi Neil, thank you for the information. We are using the Neon eye tracker to monitor fixation on a central fixation point presented on a screen while participants respond to visual stimuli in the periphery. As we review the recordings, we want a better understanding of the differences in exported data between Pupil Cloud and Neon Player.
After going through the full breakdown linked in your message, it seems that, aside from the advanced capabilities listed, the two may also differ in the reports they generate. For example, a few questions I have are listed below:
Thank you very much and Happy New Year!
Thanks for providing further context. Which Companion app version was used to make those recordings, and on which model of phone?
can 'is this thing on' model record and analyse pupil diameter? how can this measurement be extracted?
Our most recent recordings were made on app version 2.9.26-prod, while our oldest recordings date back to 2.8.31-prod. The phone model number is NE2213EEA, which I believe corresponds to the OnePlus 10 Pro 5G.
Thanks for confirming!
Things can get a little convoluted because your dataset spans multiple Companion app versions. For example, we added real-time fixation detection in version 2.9.0, but whether those fixations were computed in real-time depends on a few variables, such as your specific app settings and phone performance. Before that app version, there would be no real-time fixations. Thus, you would actually need different versions of Neon Player to open the older recordings compared to the newer ones if you wanted to see fixations locally (more here).
Pupil Cloud, on the other hand, provides fixation results regardless of your phone's performance, the app version (v2.8 vs v2.9), or which settings were selected. In that sense, you don't have to worry about whether the phone kept up, or which settings were selected. Every recording in your dataset will have fixations that are ready to use.
Hi- why is the recommended method to align the sensor data from IMUs and gaze with the start of the video stream? Via the neon_recording python api. Is recording.start_time relative to the video stream or just when the actual recording was started? Thanks!
Hi @user-f89b02 , just to clarify, did you mean "what is the recommended way" or "why is the recommended way"?
And, recording.start_time is relative to when the recording started, so the moment that the white button is pushed (or the moment that the device.recording_start() command is received by the device). The sensors always need to warm up and start a bit later, after recording start.
Hi Rob - I meant what is the recommended way sorry. Ok so recording start time would be the actual start time and the sensors start a little after (as I have seen based on the data I have worked with). So is time point 0 in the video stream recording.stat_time or is that also later? And from your experience, what is the offset typically roughly?
@user-f89b02 and to follow up on @user-d407c1 's helpful snippet:
recording.start_time and is almost never equal to recording.start_time. It usually takes ~1-2 seconds for the scene camera to warm up and start recording video.Note that this is relevant if you are making many short recordings to track the trials of an experiment. For that purpose, we rather recommend making one long recording and tracking trials using Events. If relevant, see this guide.
Since all of the datastreams from Neon are timestamped by the same high-precision clock, you can just follow @user-d407c1 's snippet to sample the various streams at the same timestamps. That is, you can just directly compare the timestamps to each other.
Hi @user-f89b02 👋 ! Please allow me to step in for @user-f43a29 , are you looking to how to get IMU, gaze and scene camera frames using pl-neon-recording ?
If so, this might be helpful for you.
import pupil_labs.neon_recording as nr
import tqdm from tqdm
recording = nr.open("REC_PATH")
timestamps = recording.scene.time
# Could be also gaze.time for gaze timestamps or imu.time for imu.
combined_data = zip(
timestamps,
recording.scene.sample(timestamps),
recording.imu.sample(timestamps),
recording.gaze.sample(timestamps),
strict=False,
)
# combine with data sample to those timestamps
for timestamp, scene_frame, imu_datum, gaze_datum in tqdm(combined_data, total=len(timestamps)):
# iterate over
frame = scene_frame.bgr
gaze_xy = gaze_datum.point
rotation = imu_datum.rotation
...# whatever you want to do
Thankyou, i had some other doubt. 1. the frame of scene video output is more wider is there any option to record the video itself little zoomer. 2. The scene video output recording from eyetracker, I want to use to check where they are exactly looking while driving but the quality of video is not good, i had an instax 360 cam with me so while recording with eyetracker if i record parallelly with instax 360, then with that video output can i able to get gaze plus fix integrated on new intax video output. if yes how to do,
It's not possible to change the zoom. The scene camera indeed covers a wide field of view that captures extreme gaze angles / stimuli that appear in the wearer's periphery. You mention issues with video during driving. Perhaps you mean scene camera exposure? E.g. a sharp change in contrast, such bright light coming in through the windscreen contrasted against a dark car interior, can be challenging. So in the first instance, I'd recommend experimenting with Neon's exposure settings. You can read more about that here: Exposure Settings.
To answer your final point, yes! It's indeed possible to integrate Neon's gaze measurements with third-party video, such as your instax360. You can follow this Alpha Lab guide: Egocentric Video Mapper
Hello, I just bought the "Just act natural" frames, but as my Neon module stopped working, I would like to purchase another module while awaiting repairs of this one. Is it possible to buy just the module and companion device without the frames or to return the unopened frames I just bought?
Hi @user-451948. Yes, it's possible to buy just a module and Companion device, and also to return the frame if that's what you want. With that said, I assume the faulty module you're referring to is the one we discussed in the troubleshooting ticket? Because we haven't yet diagnosed whether the fault lies with the module or the bare metal nest. If it's the nest, e.g. a loose connection, then you might not need a new module.
Since you mention you have a new frame, it would be worth putting your module into it. If it everything works as expected, then it's just your bare metal nest which is broken. If the problem follows the module, then we know the module will need fixing.
Hi! I have a recording where no fixations were mapped from the raw gaze data. I was wondering why this is the case and if there might be any solutions to obtain the fixation data?
Hi @user-13d297! Could you expand a bit on what you mean by no fixations being mapped? For example, is this about mapping fixations to AOIs via an enrichment, such as Reference Image Mapper? Or something else?
Hi team. may I know if there are suggested ways to extract fixations from the gaze data? I am processing the raw recordings using the python libraries pl-neon-recording. My current data structure has around 200 gaze points, and I want to extract the fixations from these gaze data points
Hi @user-9a1aed , pl-neon-recording has a fixations stream that already contains the data you are looking for. You can simply sample the fixations data at the same timestamps as those 200 gaze points, similar to here.
thx!
I am currently using pupil-labs-neon-recording==1.0.3 and it seems that it does not have module 'pupil_labs.neon_recording' has no attribute 'match_ts'. May I know if there is another attribute I could use to match the events? thx! matches = nr.match_ts(target_time, events.start_time, method="backward")
May I ask if there is a reason to not upgrade to the latest version of pl-neon-recording?
I have a running script that is not yet updated since I encountered quite a few errors after updating it
Ok, in that case, it depends on how exactly you want to match the timestamps. You could try using np.searchsorted from numpy and then check if the matched timestamps are close enough for your purposes. You could also just find the timestamp with smallest difference from the target timestamp with something like np.min(np.abs(events.start_time - target_time)).
Alternatively, you can adapt the match_ts function for version 1.0.3.
Oh, sorry for the confusion! Gaze was sampled for the entirety of the recording, but none of these samples were parsed into fixations. The fixation data stream in Pupil Cloud is empty as well as the fixation file from the Timeseries download.
Hi @user-13d297 , could you open a Support Ticket in 🛟 troubleshooting about this? Thanks.
Hi Neil, great, thank you. How could we purchase a module and Companion device without the frames, or initiate a frame return? And yes, I did try the module with the two different frames ("ready set go!" and "bare metal"), and both do not connect tot he companion app. My "just act natural" frames are at my lab in another location and have been unopened. We will plan to send this module back for inspection/repair but wish to have another working module during this time. Thank you!
Hi @user-451948 , as mentioned in the Support Ticket, please send an email to info@pupil-labs.com and the Operations team can answer all of these questions and coordinate the process with you.
We prioritize repairs, so you will probably save time and cost by organizing a module return, rather than ordering a new one.
Hello! I am using the neon glasses in combination with another, highspeed, egocentric camera and I am trying to use the egocentric video mapper code from github. I am not able to use the cloud for IRB reasons, so I am exporting the files and copying them from the android phone. I then used neon player to generate a folder of files. It looks like the egocentric video mapper is looking for a different set of files though (.csv instead of .npy) and it was looking for a world camera video with 0 in the name, which was not output by neon player. Is there another step I am missing to generate the propper file format for the egocentric video mapper? Thanks!
Hi and healthy new year to everyone! Is there a possibility to share just some recordings in the workspace and not the whole workspace? I know it was never possible, but maybe there is some way...or at least some of the recordings to be shared as an Editor and others as a Viewer.
Hi @user-3c26e4 , thanks; wishing the same to you.
While you can share select recordings from a Workspace (right click and choose Share recording), it is not possible to do either of the following:
If you would like to see either of these features added, be sure to add them to the 💡 features-requests list.
Hi @user-e544ee , after loading the recording into Neon Player, you then need to export it, by either pressing the E key or clicking the button that looks like a download symbol. Make sure also that the World Video Exporter is enabled. To clarify, the .npy files are an intermediate format that are not meant for typical usage. If that does not resolve it, then just let us know.
Hi Rob, I am now running into an issue where the egocentric video mapper expects column names like 'recording id", but the only column in the csv output from Neon Player is "timestamps [ns]". Can you advise a solution to this issue?
Thank you for getting back to me! It looks like I did not let the world video exporter complete the first time.
I see. Thanks! So even if we download the native recording data from Pupil Cloud, that folder also only contains contents from real-time recording. I guess my last question is: for the latest version of the Neon Companion app, would the fixation results be the same between Pupil Cloud and Neon Player? Which I suppose in this case specifically means whether Pupil Cloud is doing anything additional to the real-time detected fixations.
Hi @user-743e57 , if you have Compute fixations enabled in the Neon Companion app settings, then the fixations are computed in real-time and the same data are used in Neon Player and Pupil Cloud. If you have this setting disabled, then Pupil Cloud runs the fixation detector across the whole recording upon upload. It does not do anything additional, aside from re-computing the fundamental gaze data from all recordings at a full 200Hz.
Hi there. I would like to connect Neon to my own computer to receive the data stream (including gaze estimation from NeonNet, eye pose, etc.). I think the best way is to the c++ client real-time API (https://github.com/pupil-labs/pl-realtime-cpp-client). I am wondering how stable this API is, since I don't see much information on its user feedback either on the website or here in discord. Are there examples of using this c++ client real-time API? Thanks.
Hi @user-3d4b81 👋. We generally recommend our Python client as a starting point because it has the most extensive documentation and examples. It is our most widely used entry point for developers. That said, the C++ client is a valid option, but less widely used and discussed. Architecturally, both the Python and C++ clients abstract the API protocols (see under-the-hood). So you can choose the client that best fits your existing stack.
Hi. I have noticed that for one of my participant the enrichment marker mapper has not run properly and I would like to re-run it just for that participant. I am trying to create a new enrichment but I'm not able to i) select the recordingID that I want and ii) locate the 'run enrichment' command. See screenshot attached. Can you please help? Thank you
Hi @user-7413e1 , it seems that this Enrichment was potentially already started? Could you click the red Cancel button and then do the following:
Then, either try to re-start that Enrichment or create a new one. Let us know how it goes.
Can I stay in the same project or do I need to create a new project to run a new enrichment for this participant? Is there a way to simply overwrite my old enrichment?
You can stay in the same Project. To "overwrite" the old Enrichment, you can right click it in the Enrichments tab and choose to delete it and then make a new one. Also, depending on your experiment, you don't necessarily need to make a separate Enrichment for every participant. You could potentially save significant time by using Pupil Cloud's default batch processing.
If I delete my old enrichment, will it delete my whole sample (including 'good' participants)? I just want to re-run one participant and overwrite its file ideally. After refreshing as you suggested, it seems to be giving me more options (thank you), but I am still unsure about how to correct the misplacement of the markers (see screenshot) and to run the enrichment only on this recording.
Then, you can instead create a single new Enrichment with just that participant's recording.
Regarding the marker positions, would you be willing to temporarily invite us as Collaborators on your Workspace [email removed] Then, we can take a closer look, but please note that then we will be able to see all data in that Workspace while you provide us access.
Thank you - the same participant seem to have issue also for the face mapper enrichment, where a face seems to be dected but heavily misplaced.
Hi @user-7413e1 , we see that these recs were made with very outdated app versions, 2.8.2-prod and 2.8.25-prod. We can reprocess the affected recordings for you, but it is highly recommended to upgrade to the latest Neon Companion app version, 2.9.26-prod, as we have made many stability & feature updates since then.
Would you be able to send, via DM or email [email removed] a list of the recordings that are exhibiting displaced marker & face detections?
Thanks. We received your invite and will take a look.
@user-4a6a05 Hi, surface tracking in the new Neon Player is not functioning. It resets the surface as soon as I move the sliders or uncheck Edit. When I add another surface, the old one extends to the old boundaries. There is also no heatmap.
Hi @user-3c26e4 , thanks. My colleague, @user-cdcab0 , is responsible for handling this feedback. Perhaps sharing a screen recording could help determine what is going on?
Thank you Neil for information. Just to confirm, this real-time API (both python and c++) should work in different operating systems (Linux, Windown, MacOS, etc.), right? In what operating system has the API been tested?
Hi @user-3d4b81 , the API is Operating System agnostic, programming language agnostic, and network agnostic. The API itself runs in the Neon Companion app on the phone, making it independent of the PC's OS. We even have an example of a small Android app, written in Kotlin, that uses the API, as well as a C# implementation for Unity and a MATLAB wrapper.
It has been tested on several releases of MacOS, Windows 10 & 11, and a few flavors of Linux, and the Python package has been tested with a variety of Python versions.
Hello! I had purchased the imotions and neon glasses with our grant, but was recently notified that there is now a subscription for the pupil labs cloud to access our data. Is there a discount for universities or a package for those that adopted before this format rolled out?
Hi @user-72e42a , if you want to work within the free 2 hour recording quota, then you do not need an Unlimited Plan. If you simply want to access the data that is currently there, you can analyze & download the 2 hours worth of data, then delete those, and work with the next 2 hour chunk. Just note that deleting recordings on Pupil Cloud will permanently remove them from Pupil Cloud, so be sure to make local backups if you go with this approach.
If you rather want to obtain an Unlimited Plan, it is not exactly a subscription. You obtain a Plan in yearly units and decide in advance, when it should start & end.
And, yes, academic discount is available for Unlimited Plans.
Thank you for your response! We collect 30 min of data twice a week for seven weeks from patients (approximately 7 hours of recording per person) and are anticipating a total of 80 patients, so I think keeping that in the cloud would be ideal? (Sometimes we have found the file gets corrupted when we download/analyze so have to redownload later on )
Could I get the pricing for a university license ?
Whichever method is most ideal for your situation.
Could you clarify what you mean though by "the files are corrupted when we analyze"?
If you use the Unlimited Plan calculator, it will show you the academic discount at the bottom left, as it depends on how many devices and years you wish to include in your Plan.
Hello. I am using my PupilLabs Neon and streaming data through LSL. When connected to "Neon Companion_Neon Gaze" I receive a sample of data with 22 entries:
[799.9196166992188, 471.5899658203125, 4.100955009460449, -27.5625, 14.5, -43.21875, 0.20794349908828735, 0.3420678675174713, 0.9163782596588135, 4.54191780090332, 34.78125, 12.34375, -49.40625, -0.08832237869501114, 0.3171752393245697, 0.9442452192306519, -0.63671875, -0.72705078125, 1.0433200597763062, -0.28125, -1.3251953125, 12.412347793579102]
Could you provide me information about what each value corresponds to? I have read the documentation (https://github.com/sccn/xdf/wiki/Gaze-Meta-Data) but it doesn't specify what values have been used, as I only receive 22 in the data stream and more than 22 are reported in the documentation.
Thank you!
Hi @user-13078d , may I first briefly ask why you are using LSL for real-time streaming? That is not really its principal design focus.
Hello Rob, thank you for your quick reply! I'm using LSL to manage and synchronize data threads coming from other biosensors. I have also used the pupillabs-api in real time and this works fine. However, I would also like to implement the LSL approach to extract gaze and eye data. Thank you!
Hi, I am experiencing some issues that seem quite concerning. I opened up some events.csv files that I downloaded last July and had not touched since, and I realised they contain different events from the ones I can see on pupil cloud. I also checked that the events on pupil cloud have been inserted programatically via API (and not manually) confirming that the discrepancy does not result from a post-doc edits on pupil cloud. By looking at other back-up files of the session + the video on pupilcloud, the events on my july events.csv file are correct, while the events on pupilcloud seem to be messed up. Can you help me understand where this discrepancy come from? So far I have noticed this in two files but there may be more and will check more.
Hi @user-7413e1 , is it the naming of the events or the timestamps that are discrepant? Do I understand correctly that you also confirmed the discrepancy against the original code that sent the Events?
Both. And I just checked this applies to my whole sample. It looks like the same event structure (naming AND timestamps) has been applied to all my recordings (wrongly). But I am 100% this is a later error (and quite recent) because the data I downloaded last summer looks different (and seems correct).
Thanks for the clarification. Could you open a Support Ticket in 🛟 troubleshooting about this?
will do, thanks.
You are welcome, @user-13078d . I see. Is there any reason to avoid LSL's standard LabRecorder tool, which saves the automatically synchronized data to an XDF file? Do you need to do real-time processing of the data?
I ask, because the channels are labelled in the XDF file, making it a bit easier to keep track of your data. The streamed channels should have the same format as the channels that are saved in the XDF file, which are not actually in the same exact order as the Gaze Meta Data documentation. I will confirm for you.
Exactly, I'm doing real time processing of the data. I'm trying to avoid secondary tools and APIs to reduce any possible delay.
I will be waiting for your confirmation. Thank you!
Hi @user-e544ee , you can either add columns with the respective values to your data file (e.g., insert the recording ID in the info.json file into a “recording id” column) or modify the code to ignore those columns.
Thank you!
@user-39ee45 Sorry for the delay getting back to you. Could you send us an email to info@pupil-labs.com about this, so we can send you the files?
No worries! Just sent the email. Thank you so much for your help!
Hello @user-f43a29 : Happy new year. Sorry, but has this code been released or is there a pre-release version we can work with? Thanks
Hi @user-ebd8d5 , not yet. It will be released as an Alpha Lab, so keep an eye on the 📯 announcements for its release. And, happy new year also!
@user-f43a29 Thank you. Sorry but do you know by when would it be released approximately?
Hi @user-ebd8d5 , there is not a set deadline. The tool is being made more robust and simpler.
Hi all, I have been working with the open source local version of the egocentric video mapper and just had a few issues with it. 1. I was wondering what these 3 colored circles are in the output? 2. I noticed that the world seems to be the wrong color in the neon output video. For instance, the table captured is brownish-red in real life. 3. The red circle, which is also mapped onto my alternative camera, seems to be advanced in time in both the neon video and the alternative video compared to what I know is true. The two videos themselves look well synced in global time, but this red dot seems equally advanced in both of them. Any thoughts on this?
Hi @user-ffc425 👋 ! Could you clarify a bit more what you’re running exactly? Which code are you using, and how are you opening or rendering the video?
A few things that might explain what you’re seeing:
Color change (red ↔ blue):
If you’re using OpenCV, note that it works in BGR color order by default, not RGB. So a red overlay drawn in RGB can appear blue if the image isn’t converted first or your brown table look blueish.
You’d typically want to convert with something like cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) before visualising.
Red vs numbered circles:
The red circle is usually the gaze point, while the numbered circles typically indicate the fixation scanpath, which usually appears when you’re using video renderer (exported in Cloud) rather than the raw scene camera video.
Timing / alignment issues:
If you downloaded a rendered video from Cloud, it may not have the same length as the raw scene video (see gray frames explanation here https://discord.com/channels/285728493612957698/1047111711230009405/1364251433612087416) . That could explain the misalignment you’re seeing.
If you can share whether you’re using the raw scene video, a Cloud-rendered video, or your own rendering code, we can narrow this down further.
Hello PupilLabs team. I have encountered a problem when implementing surface tracking using the Neon eyetracker and the companion device (Motorola). I have followed the README instructions (https://github.com/pupil-labs/real-time-screen-gaze/blob/main/README.rst) to create the QR markers and i have successfully displayed them on an interface at each corner of my screen. I have added the surface but when I use this function "gaze_mapper.process_frame(frame, gaze)", I receive no data. However, the rest of the functions seem to be working fine, I am receiving outputs of other functions such as: frame, gaze = device.receive_matched_scene_video_frame_and_gaze().
What could be the cause of this problem?
Thank you!
Hi, @user-13078d - it's most likely that your AprilTag markers aren't being detected, which usually happens from the markers being too small, not having large enough margins, and/or not being well-lit. Special care has to be taken with markers rendered on a digital display. Specifically, you may need to adjust the contrast settings within the companion app to suit your display and environment.
The first thing you should do though is to visually examine one of your video frames. It's, generally, quite easy to tell for youself when a marker may not be well detected. Of course, you should feel free to share the frame grab here if you have any doubts!
Hello. Problem solved! Adjusting the contrast settings worked, as well as rescaling the AprilTags. Thank you!
Hi everyone, I’m working on vergence variation and I need to project each eye’s gaze coordinates separately onto the scene camera image. Pupil Labs provides: * gaze coordinates in pixels (but only as the average of both eyes), * gaze coordinates in degrees (azimuth & elevation), but not per eye. So I started by trying to compute the azimuth and elevation for each eye from the optical axes. Here is what I currently do:
df["azimuth_left_deg"] = np.rad2deg( np.arctan2(df["optical axis left x"], df["optical axis left z"]) ) df["azimuth_right_deg"] = np.rad2deg( np.arctan2(df["optical axis right x"], df["optical axis right z"]) )
a = np.arcsin( df["optical axis right y"] / np.sqrt( df["optical axis right x"]2 + df["optical axis right y"]2 + df["optical axis right z"]**2 ) )
b = np.arcsin( df["optical axis left y"] / np.sqrt( df["optical axis left x"]2 + df["optical axis left y"]2 + df["optical axis left z"]**2 ) )
df["elevation_left_deg"] = -np.rad2deg(b) df["elevation_right_deg"] = -np.rad2deg(a)
However, I’m not even sure this angle computation is correct, and I don’t know how to convert these per-eye angles (deg) into pixel coordinates in order to project them onto the scene camera video. Does anyone have an idea whether this approach makes sense, or how to correctly go from optical axis → per-eye gaze angles → scene camera pixels? Thank you !!
Hi @user-4a6a05 , do you intend to implement dynamic AOIs in PupilCloud soon, as this would be of high importance for dynamic situations? I know that there is a tool in the browser, but whatever I tried, it didn't function.
Hi @user-3c26e4 , rather than tagging Marc, you can simply write here. We will all see it.
Could you clarify what did not work? Do you mean the Dynamic AOI Tracking With Neon and SAM2 Segmentation guide?
Yes, exactly this.
Thanks. Could you clarify what did not work?
Please excuse me for asking again, but do you intend to implement dynamic AOIs in Pupil Cloud?
Unfortunately I can't explain it thoroughly, it just seemed to be calculating and nothing happened. That's why I asked whether you're going to implement it into pupil cloud.
@user-3c26e4 , if the tool did not complete, then you either need to:
Thanks for the detailed response! In short, I am using this GitHub repo from pupil labs (https://github.com/pupil-labs/egocentric_video_mapper) in addition to the Neon Player and Neon Player Beta to output the required csv and videos to pass into this. I've used the jupyter notebook in this file titled "PL_EgocentricVideoMapper.ipynb" for this analysis. Other than that, I've made no direct modification of how the neon video is read in (regarding the open CV part, which I could change in the repo, but I am more than likely passing in some parameter wrong or something). In regards to the timing issue, which is the biggest issue, is there any supported way to get the offset of where the gaze point should actually be in respect to the videos? If there is some constant calculatable factor, I can simply modify the timestamps of the output gaze points to fix this, as I am more interested in the actual points aligned correctly to global time than the videos themselves. The videos are primarily for a sanity check the gaze points are correct in time and space
Hi @user-ffc425 ! The egocentric video mapper code expects the output from Pupil Cloud (Timeseries + Video). Could you kindly try using that format first?
Hi @user-3c26e4! This is something we're looking into, but it's not on the immediate roadmap.
Hi @user-4c21e5 I hope that this feature will be implemented soon. Looking forward for it.
Hi @user-2b5d07 , since it seems you are only interested in vergence angle, may I ask why you are attempting to project the optical axes into the scene camera image?
Hi, you’re right that for vergence computation alone, I don’t need to project monocular positions into the scene camera. However, I’m also working on cognitive load, and I need to visualize the monocular behavior of each eye during specific task phases, to see how the subject looks at the object over time. I’m not fully confident that my current conversion from optical axis to azimuth/elevation is correct. Are these formulas valid, or should the angles be computed using the eyeball center ? Finally, is there any recommended way to project per-eye gaze directions onto the scene camera image, given that Pupil Labs only provides binocular gaze in pixel space? Thanks a lot for your help.
Hello, I am currently working on a study using PsychoPy for a user interface, run an EEG recorded through LSL, as well as eye tracking with a Neon eye tracker. Right now, I'm recording the eye tracking data in 3 different ways: (1) export the raw data from the companion device and process it with Neon Player to get the gaze mapped onto the screen, (2) use the plugin for PsychoPy to directly get gaze mapped onto screen, and (3) stream raw gaze to LSL for synchronization with EEG. In the attached image, you can see a plot I created to check if all these sources give me synchronous data. As you can this, this unfortunately is not the case: the data streamed to LSL somehow drifts compared to the other 2 sources. This does not happen for all of my trials - for some they perfectly align - but so far I haven't be able to find out what causes this. One thing to point out is that the raw gaze data values I get from PsychoPy (together with the gaze mapped onto the screen) and from LSL actually match exactly (at least for the few datapoints I checked), but the timing is off a little bit, so they drift over time. For example, in the run shown in the figure, the data recorded to LSL has a pretty constant frame rate of 198 FPS, while the frame rate from Neon App is exactly 200 and that of PsychoPy around 198.5 with a bit more variability (but PsychoPy and Neon App data stay perfectly aligned over time, so I guess there is some interpolation between these going on) I noticed that the XDF file I get from LabRecorder contains some clock_offset values, but it seems like these stay constant over time. Have you ever seen anything like this? I've looked at this for some time now, but am a bit clueless on what's going on here. Please let me know if I can provide more information or if any of what I tried to explain is unclear.
Hi @user-a6c56f , to clarify one point first:
clock_offset values are a central component of LSL and is what enables it to do its automatic time synchronization. More details are contained in their documentation.Can I also confirm something -> Are you taking these different datastreams, and plotting them directly with their individual timestamps? I ask because it is not really possible to compare them so directly, since they all have different clocks.
With respect to the LSL drift, how are you loading the XDF file?
Hi, I am trying to run an experiment with PsychoPy + Pupil Labs and I would like to confirm I am understanding everything correctly.
The time (s) of my stimulus onset times in the PsychoPy behavior output should correspond to the column "time" (not logged_time) in my eye-tracking PsychoPy output and both should be synchronized. Is this correct? Is there anything I should specifically check for to double-check my eye-tracking is properly synchronized to behavior?
Hi @user-b71b46 , adding onto @user-480f4c 's message, also, please note, that depending on your needs, very precise synchronization of stimulus presentation involves a few components:
These have the most influence. Other factors that have an influence are whether you have other programs running in the background, the task priority that is assigned to your experiment, and the resource demands and overall structure of your experiment. It could also be worth it to consider a high-quality DisplayPort cable that supports the latest standards.
Hi @user-b71b46! With our dedicated PsychoPy-Neon plugin, the synchronization is handled automatically for you. If you'd like to verify the sync, you can send a "Neon Event" at the start of the stimulus and check if the timestamp of a "stimulus_start" event in your eye-tracking data matches the onset time in your behavioral log. Does this make sense?
Hi @user-f43a29, no, I first synchronize the different data streams: the stream from Neon App is a timestamp in ns, PsychoPy is in seconds since the experiment start, and LSL is in seconds since recording start (which is different from PsychoPy start). From PsychoPy, I send events for synchronization to LSL and the PsychoPy log. These are two different events that are sent right after another, so they're not perfectly in sync. For the visualization I showed before, I then manually adjusted the alignment by around 1 second, and as you can see from the green and orange plot, in the beginning they are perfectly in sync.
I'm loading the XDF file into Python with pyxdf.
Hi @user-a6c56f , do you use our PsychoPy plugin's Neon Events component in Builder? When you load the XDF file with pyxdf, could you try disabling jitter correction. We have seen this resolve problems when synchronizing with EEG before.
Question about the face mapper enrichment: I know it has to be added in Pupil Cloud and can't be done in Neon Player, and also that videos can't be reuploaded once they're deleted from Pupil Cloud.
Is there anything that can be done if we accidentally delete a video before adding the enrichment?
My supervisor is trying to figure out if there are any contingency options in case of accidental deletion, and I can't see any.
Hi @user-fa126e ! Once a recording is deleted from Pupil Cloud, it remains in the trash before being permanently removed. If you need to restore a trashed recording for further analysis, you can simply click on the three-dot icon under the workspace name, then select "Show trashed". From there, locate the recording you wish to restore, right-click on it, and choose "Restore recording".
Thanks Nadia!
I meant in the case of it being permanently removed, is there anything that could be done to add an enrichment to that video?
Obviously we're hoping it won't happen but I just wanted to check if there are ways to do it just in case
@user-fa126e There are two steps required for a recording to be permanently deleted from Pupil Cloud. When you first delete a recording, it is moved to the Trash. It will remain there until it is manually and permanently deleted. This two-step process acts as a safeguard against accidental data loss.
Thank you. In the case where a video is permanently deleted, that means there's no longer a way to add an enrichment to that video, is that correct?
That is correct @user-fa126e. If you delete a recording and then also permanently remove it from the Trash, it can no longer be restored or used for analysis in Pupil Cloud.
However, as mentioned previously, there is no automatic deletion; your recordings will stay in your workspace unless you intentionally follow those two manual steps to remove them. Just keep in mind that once they are cleared from the Trash, they cannot be recovered.
Hi @user-f43a29, thanks for the idea, setting pyxdf.load_xdf(..., dejitter_timestamps=False) actually seems to completely resolve my issue! Now most of my samples have pretty much exactly 200 FPS instead of the slightly lower rate of 198 FPS is saw before, and the data I get from LSL stay perfectly in sync with the data I record through PsychoPy and Neon Recorder.
Just to clarify: does this mean that with dejitter_timestamps=True, the timestamps I'm seeing in the data loaded through pyxdf are not the actual timestamps saved in the file? I get why this might make sense, but it also seems a bit dangerous.. Are there any possible problems with disabling jitter removal I should be aware of?
But most importantly: thanks for helping me with this! I don't know how long it would've taken me to figure this out on my own, since I wasn't even aware pyxdf was doing stuff like this.
Hi @user-a6c56f , there is nothing exactly dangerous about this. Although, ultimately, dejittering or no dejittering is something that your team needs to decide as a group.
To clarify, LSL collects & saves data as follows:
The process is quite similar to our Time Echo & Syncing protocol.
That accounts for precise synchronization in most cases. Also, LSL can do dejittering, which is useful to account for noisy clocks. More can be found in the LSL documentation and in this LSL Workshop.
We have tested this in depth ourselves and our LSL integration is working as expected. Until now, we have only seen dejittering have an effect when using Neon in combination with one EEG device. If you wish to dig deeper into it, then I recommend reaching out to the EEG manufacturer and the LSL maintainers, who are quite active & helpful in their public Slack channel.
Hi there! I am currently testing the new Neon Player. I really enjoy the new design & functionality, but I have two questions about it.
It cannot seem to load the fixations. Could this be because it is an older recording done in the old fixation format? Is there a way to get the fixations that were calculated in Pupil Cloud into this Neon Player? 2026-01-19 15:00:00,851 - ERROR - app.py:271 - Failed to enable plugin <class 'pupil_labs.neon_player.plugins.fixations.FixationsPlugin'> Traceback (most recent call last): File "/Applications/Neon Player.app/Contents/MacOS/pupil_labs/neon_player/app.py", line 269, in toggle_plugin File "/Applications/Neon Player.app/Contents/MacOS/pupil_labs/neon_player/plugins/fixations.py", line 59, in on_recording_loaded File "/Applications/Neon Player.app/Contents/MacOS/functools.py", line 998, in get File "/Applications/Neon Player.app/Contents/MacOS/pupil_labs/neon_recording/neon_recording.py", line 150, in fixations File "/Applications/Neon Player.app/Contents/MacOS/pupil_labs/neon_recording/timeseries/timeseries.py", line 53, in init File "/Applications/Neon Player.app/Contents/MacOS/pupil_labs/neon_recording/timeseries/fixations.py", line 67, in _load_data_from_recording AttributeError: No fixation data found
Can I assign fixations to areas of interest using a plugin in Neon Player?
Hi @user-0cdc2d , was this a recording where Compute fixations was disabled in the app's settings?
Sorry, submitted accidentally before I finished writing. It could be, where could I double check this? I remember it being an older recording -> older format
If your recording is from v2.9.0-prod or later, then you should have non-empty fixations psX.raw files in the Native Recording Data folder, if Compute fixations was enabled at the time of recording.
No problem.
If it is from before version v2.9.0-prod, then you need to use version v4.x of Neon Player, as detailed here and found here.
If you have used AprilTags to define a Surface, then the Surface Tracking plugin assigns fixations to it during export.
Sorry for the late reply on this. Finally got around to testing it. Pupil cloud output fed into the Egocentric Video Mapper does work properly in terms of temporal alignment (though the output video is still swapped RGB->BGR in the output video for the Neon camera). I've watched many of the videos I passed through Neon Player/Player Beta and then into Egocentric Video Mapper. The output of egocentric video mapper's gaze position on the image tends to be between 1-1.5 seconds ahead of the video, but more importantly is non-constant. Though, looking at all of the .csv files, I cannot find a combination of erroneous difference in timestamps that arrives at this number via computation per video. I suspect this arises because I had to mix player + player beta due to each one individually not outputting all of the files I needed when I ran it (e.g. the very suspiciously named world_timestamps.csv was missing from the beta output), but I am open and very appreciative to any thoughts you guys might have on this.
I see, may I ask which version of Neon Player did you use, and which combination of files?
Hi, I'm trying to use the Pupil Labs plugin on psychopy 2024.2.4, but am unable to find it. All I see when searching up the plugin through the psychopy manager is the "Pupil Labs Eyetracker support" plugin. If this is the right one, however, after installing this, restarting psychopy, and setting the eyetracking device to "Pupil Labs" I see none of the additional components e.g. the Neon Events component
Hi @user-ad7ce5 , the PsychoPy team changed some features of their plugin system with the release of 2025.1.1. As a result, our plugin avaiable in the PsychoPy plugin manager only works with 2025.1.1 and newer (with the newest currently being 2025.2.4, but not the Beta version). Is it necessary that you work with version 2024.2.4?
Ah alright thank you for the information. I'll try updating to 2025.1.1. I don't think there's any immediate reason for me to stick to the 2024 versions
After updating to 2025.1.1, I see the options for the Neon. I selected it, changed the port number, and am connected to a separate hotspot containing just the companion phone and the device running psychopy, but the trigger to start recording and the events don't seem to be doing anything. I can access the recording through the URL fine, but nothing via psychopy. I'm not sure if I'm missing anything of the setup. My main goal with this is just to be able to send timed events when recording. For context, here is the eyeblink preferences setup I have at the moment:
Companion Devices
Hi, does Neon come with the companion device when you purchase it? I don't see it listed as part of "what's in the box" for Neon, but I do remember that it requires an external device (mobile phone) to function.
Hi @user-d6e6ad , Neon does come with the Companion phone when you order a full system and it is referred to as the Companion Device.
Pupil Core is not used with a phone but rather with a computer. It does not come with a phone and you would need to connect it to your own computer.
May I ask why you consider both Neon and Pupil Core?
I also have same question about Core, but don't want to ask same question again in separate channel.
Hi @user-ad7ce5 , you want to enter 192.168.2.101 as the IP address. So, remove the :8080 portion and then it should work as expected.
that worked, thank you!
Hello Pupil Labs Team, I have a just simple and basic question about fixation visualization. How long is duration for showing fixation history on video?
Hi @user-25d81e Are you referring to Pupil Cloud, or Neon Player?
Thanks for reply. Pupil Cloud mainly.
On Pupil Cloud the fixation history is 3 seconds. In Neon Player you can adjust it, but also up to 3 seconds.
Thank you very much!
Hi team
We're using the Motorola that came with the glasses but we'd like to use something more reliable.
Is there a recommended or better device for the neon glasses?
Hi @user-314b02! Thanks for reaching out! Could you clarify what you mean by more reliable?Are you experiencing any issues with your Motorola device? Note that at the moment, Neon ships with either a Samsung Galaxy S25 or a Motorola Edge 40 Pro, and we strongly recommend using these models due to their superior performance, edurance, and stability.
Understood. We had an error with a corrupted file which was causing the app to crash repeatedly. It was only after deleting it from the companion app that the issue got sorted
If you encountered an error during your recording, I'd suggest opening a ticket in our 🛟 troubleshooting channel so we can take a closer look and help ensure a smooth experience with Neon going forward 🙂
Hi Pupil Labs. This question is probably out somewhere but I cannot find it in the chat. Is there a simple way to export the timestamps AND the frame index from a recording? I am only getting the option of using the timestamps.
Hi @user-660f48 , are you using Neon Player?
Hi @user-f43a29 THanks for the quick response. Yes, I am using Neon Player
Thanks, and you want timestamps and frame index for which data stream?
For the scene video
Ok, as documented here, each timestamp corresponds to a unique frame index. So, the first timestamp is for frame index 0, the second is for frame index 1, etc.
Ahh this makes sense, I assumed there were several timestamps per frame. Now this makes things easy as you say, just a matter of adding a column in the world_timestamps.csv file. I am running a code we made for extracting the Regions of Interest from each frame in the scene video.
If you need the frame index as an explicit column in the exported world_timestamps.csv file, then you run a small script to add that column afterwards. May I ask why you need that column added to the CSV file?
Correct. To clarify, for the gaze data, many timestamps correspond to a single scene camera frame, since the gaze data are sampled at 200Htz by default, whereas scene camera frames are sampled at 30Hz by default.
Yes, thanks for the reminder and for all your help.
Hi! We have a Pupil Labs account associated with a gmail. We are now planning to use the Neon to collect data for a study and we would like to use a lab account that can be accessed by multiple lab members. Could you advice us as to the best way to switch our account?
Hi @user-e544ee 👋 ! Do you mean on the Companion Device? If so, you can log out via the Settings in the Companion app and then log in with the desired account.
Alternatively (and much easier), you can simply invite the account currently logged into the phone/ Companion app to your workspace, and then simply switch workspaces using on the top-left corner of the app.
If you are wondering how to share the Unlimited Plan benefits, have a look here
hello hello! Is it okay to ask a question about a behavior which we've noticed using the realtime-api in combination with pupil neon glasses?
Hi @user-1391e7 , you are in the right place 🙂 Feel free to ask questions.
would here be the right place, or is another place better?
thank you 🙂
So last summer, we did a study with ~60+ participants and in the course of this, rarely, but it did happen, we noticed that, when we sent the request to start a recording on the companion app
we got the okay, we saw in the companion app the visual update, the recording running
but no actual data was recorded
Since this was last summer, it was potentially an issue that has since been fixed in newer versions of the Neon Companion app. It is recommended to always keep the app up to date.
Otherwise, do you mean that you are still encountering this issue? Whenever you encounter issues, be sure to reach out to us right away, so we can assist and resolve it promptly.
no errors showed up and the live reception of data worked just fine, so we know the device was working, gaze data came in and was valid
just the local recording wasn't created
more a case of - if you haven't seen this pop up, to give feedback
it's difficult to reproduce, because in ... let's say 250 recordings, this happened maybe 4-5 times?
Thanks and understood. We always make sure to resolve such issues when they are brought to our attention and as mentioned above, remaining up-to-date with the Neon Companion app is a key way to benefit from that.
Otherwise, also be sure that you are using a compatible phone with a supported Android version.
May I clarify if you are up-to-date and if so, have you seen the issue since?
we had a special version, which was given to us as a result of another behavior we noticed, which was the calculation of blinks etc. stopping if you took off the glasses mid recording (in the context of an xr application, in which users sometimes take off the glasses to reorient themselves or ask question etc.)
since we didn't know if this was implemented in the regular releases, we stayed with that one
Yes, this was implemented in the release right after that. As such, I would recommend updating your devices to benefit from stability fixes since then.
device was always the companion device that came with the glasses, one of the oneplus phones
will do, thank you
was the behavior with regards to the start recording event not starting something that was a known issue and has been addressed as well?
Yes, it was. As you noticed, it was rather rare.
And, if you happen to encounter an issue, please reach out to us right away by opening a Support Ticket in 🛟 troubleshooting .
will do, thank you very much
You are welcome!
one thing that was a bit spotty, was device discoverability in a local network that has all the requirements set correctly, in the python realtime api
I'm curious if there is something else I could do to be able to find the companion app more often
it's not down to traffic, it's a strong wifi router, the system trying to find the device isn't blocking any traffic
the only feedback I get is that the device cannot be found
what is interesting, is that if I try to repeatedly find the device, it will never be found if that has failed once
but if I abort and try to find it again, that's what usually works
abort meaning kill the python script that attempts that
is that down to how asyncio works and out of your control?
nevermind, I see some things are different now, so maybe that's not even a thing anymore
Do your glasses for sports come with the headband included and has there been any progress with connecting the Neon Monitor app yet with the IOS operating system
Hi @user-057596 , if you mean the Ready set go frames, then yes, an adjustable headband is actually built into the frame, as depicted in the attached image.
Regarding Neon Monitor on iOS, I will check with the team, but I think Apple still has not corrected the bug that prevents it from running there.
Thanks for that Rob
Hi @user-057596 , an update: Neon Monitor is working on newer iPadOS, but still not on iPhones (iOS). Hopefully that is helpful information.
Thanks for that @user-f43a29 very helpful
@user-f43a29 when you say Neon Monitor is working on newer IPadOS are we talking about the latest test version of IOS or the latest IPads. I’ve just updated both of our IPads to the latest IOS software with one of the pads being 8 years old and the other one just 1 year old and both have the same problem as before with a frozen worldview scene on the Neon Monitor
Hi @user-057596 , iPads no longer run iOS, but rather run iPadOS, which is similar to, but different from, iOS on iPhones. The split happened in 2019.
I will check with my colleague about which iPad model and which iPadOS version they used.
Thanks @user-f43a29
Are there any tutorials or similar videos on using Lab Streaming Layer?I need to connect real-time eye movement data to a computer and use it in real-time together with data from another physiological sign detection device. Since I have never dealt with this before, is there any kind person who has similar usage tutorials or experience?
Hi @user-6ccf3a , all relevant tutorials are found in the Lab Streaming Layer Documentation and the Neon specific documentation is here. However, Lab Streaming Layer is not explicitly designed for real-time processing of data, but rather for accurate post-hoc time sync and source-agnostic collection of data. In othr words, these are two closely related, but distinct tasks. What exactly is your experiment and the analysis goals?
Hello, I have a pupil Neon, all is going fine so far. I noticed today that during the recording (using NeonPlayer 5.0.5) the red vis circle disappeared. See two images of two consecutive frames.
