Hi regarding using the Neon module with Pupil Capture is there a recommended adapter cable to connect the module to the laptop?

Thanks Miguel

hello Nadia...Thanks for support. . working well now.

how do I connect to Neon Monitor? I opened RTSP PI monitor screen but couldn't live stream.

OK...thanks.

Dear Pupil Labs team, we enjoy working with the new Neon, just ordered a second one.

One question came up: is there a possibility to also record a confidence value (as previously in the core)? We use the API to record the gaze_datum which contains only x[float], y[float], worn[bool], unix_timestamp[float]

as described in https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/async.html#pupil_labs.realtime_api.streaming.gaze.GazeData

Thanks and best, Daniel

Hi!! thank you so much for your continuous assistance! I'm currently looking at 'Marker Mapper' tutorial provided from Pupil Labs, and I'm curious it would be ok if I use different marker at every corner of the boundary, or should I use the same marker for every edge of the boundary? Thank you so much in advance!

thank you so much!

Hi PL team, I've just started working with the Neon glasses and was trying to pull the data directly from the phone (without going through the cloud) and importing it into the Pupil Player (so I can add the gaze overlay to the video). However, I get a warning in the Pupil Player saying that "This version of player is too old! Please upgrade." - I can't find a newer version of the Player than the v3.5 anywhere. The one I tried is from this Github repo: https://github.com/pupil-labs/pupil/releases/tag/v3.5

Hi Pupil Labs team, I sent email to info@pupil-labs.com for inquiry on 28th July and have no response. Please check it up.

Is it possible to connect the Neon module to a laptop to run with the Core software if the laptop doesn’t have a USB-C port?

Hi @user-ae0f24 👋 I'm replying to your questions (https://discord.com/channels/285728493612957698/633564003846717444/1136327339706437642) in this channel to ensure it's in the right place.

Yes, Neon is a good fit for recording eye tracking data in such dynamic and high-arousal environments. It uses deep learning to provide accurate, calibration-free, and slippage-invariant gaze estimation in all kinds of settings. You can wear and remove the glasses without needing to re-calibrate.

Neon comes with a Companion device (OnePlus 10 Pro android smartphone) running the Neon Companion App, where the deep-learning gaze estimation takes place. Neon requires to be connected via USB cable to the Companion device, allowing health professionals to move freely.

Pupil Cloud offers several tools to facilitate the analysis of complex data. For example, you can use tools like the Reference Image Mapper enrichment (https://docs.pupil-labs.com/enrichments/reference-image-mapper/) to map gaze and fixations, creating heatmaps on images of medical instrument monitors or operating table regions.

Just to note, while Pupil Cloud is GDPR compliant, it's not HIPAA compliant due to some incompatibilities between the frameworks. You can find more information about our security measures here (https://docs.google.com/document/d/18yaGOFfIbCeIj-3_GSin3GoXhYwwgORu9_7Z-grZ-2U/export?format=pdf)

If you want to learn more about Neon and schedule a demo call with our Product Specialists, shoot us an email at [email removed]

Hope this helps! Let us know if you have any more questions. 😊

Is it possible to connect the Neon module to a laptop to run with the Core software if the laptop doesn’t have a USB-C port?

Hi @mpk I have but the Neon module is not being registered when you click Active Device on Video Source

Hi @user-cdcab0 That explains it. I had read in the instructions as regards running Pupil Capture from source and the part as regards Linux/Mac but I mad the mistake of assuming that it would automatically run with Windows. Thanks for that.

Hi Pupil Labs team, I am having connections issues with my device again. The phone recognizes the Neon glasses, but the tabs for video and eye tracker will not show up. I can push the red button to record, but unable to play it back, and it won't upload to the cloud either. Everything was working fine yesterday.

Thanks Neil..

@user-594678, in answer to your specific questions: (1) Is there a way that I can retrieve the raw magnetometer values? No, but the heading is encoded in the orientation quaternion, and as the 'yaw' component of the Euler angles.

(2) How is the accelerometer computed to implement the magnetometer reading? Because on the manufacturer's website, it seems like there are separate accelerometers that only capture acceleration values. It isn't. Remember, the fusion algorithm computes orientation using three different sensors. Essentially, it comes down to correcting gyroscope drift errors. The gyroscope should 'conceptually' be able to measure angular rotations by integrating velocity to position. However, due to the finite sampling frequency, drift errors accumulate over time. Thus, accelerometry and magnetometer readings are needed to correct the drift and provide absolute orientation. This all happens in a fusion algorithm implemented on-chip.

(3) Do you have any ideas or suggestions on how I can compute velocity [and translation from your first message] from the accelerometer readings? Getting translation from an IMU alone is nigh-on impossible unless you have additional sources of localisation data. Similar to the gyroscope, if you just integrate accelerometer data, you will get drift errors that inflate exponentially over time. Thus, you'll probably want to fuse it with something like visual odometry or GPS. This is a common problem in robotics and navigation systems.

That doesn't mean you can't pull useful metrics from the linear accelerations. E.g. you could examine things like head movement variability in a given plane.

Thanks Neil

Hi! I would like to ask if the acquisitions require an internet connection or not or if the device is connected to the android app in another way

Thanks! Is there anything to keep in consideration if an android tablet is used instead?

Hi @user-292135 ! You should find the answer to your questions in an email reply, please check your inbox.

Hey Pupil Labs team. I have a question regarding head tracking. We are running a study that we need to know the location of the head. I was checking the documentation for the imu.csv file. For the pitch, roll and yaw it is mentioned the output is between -180 and +180 degree but the values are between -2 and 2. I am wondeirng if I misunderstood something. I also did a very simple task and recorded only doing yaw, pitch and roll. I will attach the screenshot to this message. Thank you in advance.

Hello again, I am experiencing some errors when running the surface marker enrichments on Pupil Cloud. There are 2 errors which have occurred, and that I should contact you guys for help.

Hello, I would like to map gaze over a cockpit during a flight simulator experiment. How should I go about it given the small space of the cockpit? Also I would like to develop an XR system for freely behaving macaques. Would it be possible to schedule a call? Thank you

Hi @user-068eb9! Please write to us at info@pupil-labs.com to request a demo & QA call with one of Product Specialists. For the first one, it should not be a problem. Reg macaques, Neon was not designed, neither tested with macaques.

Hi! I'm currently suffering some issues and I would like to ask a quick question on 1) where could I find 'blink.csv' file? (image 1) I found all files but 'blin.csv' 2) what does each column refers to (tl, tr, br. bl) in 'surface_position.csv'? (image 2) 3) If I would like to segment the eye fixation within viewpoint 1,2,3 in a single screen (as image 3), how should I analyze this data with marker enrichment? or would there be better way for analyzing the fixations on each area in the screen? Thank you in advance for your assistance!!

Hi Wee Kiat I m sorry to hear that you

thank you so much Nadia!

sorry for bothering you agian 😢 would it be possible to analyze the area of interest for the data collected without marker?? thank you so much for your help in advance!

Thanks Neil for your reply!! they are actually watching the display! this is the example image of what they're looking at!

maybe I could use reference image marker...?

Hi Pupil labs, the neon companion app no longer recognizes the eye tracker when plugged-in. It's a relatively new device (we have been using it for less than a month), and never had this issue before. Would appreciate if you can help us diagnose the issue. Thanks!

Neon Companion App + Connection

Previously with Pupil Cloud when you added the Gaze Overlay enhancement in a project you could have it applied in the cloud to each recording in a particular project whilst you played the recording in the cloud. However with the new analysis format you can only see the video render in the cloud on the recording that was selected to set the parameters and you have to download the file to see it applied to all the other recordings in the project. Is this correct?

Hi pupil labs, I am trying to get the gaze positions offline. Previously when I used pupil invisible I can do this by pupil player, but now it says a new version is needed. Is there any plan on this new version, and is there an alternative way that I can do this on my side (without using pupil cloud)? Thanks a lot.

@user-839466, to throw in a wildcard, we also have a guide that shows how to map and record screen-based gaze coordinates using the Reference Image Mapper. I'm not sure how useful that'd be, but you can read about it here: https://docs.pupil-labs.com/alpha-lab/map-your-gaze-to-a-2d-screen/#map-and-visualise-gaze-onto-a-display-content-using-the-reference-image-mapper

Neon exporter

Thanks @nmt glad to help out.

Thank you so much!! With using Neon by Pupil device, could pupil diameter change (size to check dilation) be exported?

Thank you Neil! it will be released in Q3 2023, and it will be available without any changes needed on the device's end, accessible via pupil cloud right 🙌 thank you!

Hello. We are collecting gaze data from the neon, attached to the bare metal kit, via your real-time API/python script. I am wondering if the neon companion app and/or anything in the API is already applying a smoothing algorithm to the gaze data or if what we are getting is the raw gaze data? It looks like raw data, but wanted to be sure. We have found that applying a smoothing algorithm (using running average of set amount of frames to display gaze point) in our python script improves the usability of our gaze data, but wanted to know if there are validated and/or recommended approaches to a smoothing algorithm. Our current algorithm was chosen without significant supporting data. Thanks!

Hi Pupil Labs Team, quick question: Is it possible to extract raw data between certain events on Pupil cloud? I know that we can download the full raw data, but I would like to extract only data within specific events which I've tagged in each videos. Thanks!

Hey Pupil labs team. I can't access the cloud. Yesterday, same thing happen. I wrote a message here but since I could sign in again, I deleted my message from here. Will the problem persist for today? (apologies in advance if this is not the right place to ask this Q. I didn't know where to ask it) Thank you in advance for your response.

Hi @user-a64a96 ! We are experiencing some issues with our "Login" system, I would like to apologise for the inconveniences this might be causing you. We will let you know as soon as we have the situation handled.

Hi @user-a64a96! I am glad to say that Pupil Cloud login issues have been resolved, apologies again for the inconvenience, and let us know if there is anything else we can do for you.

hi PL, I wanted to ask if it's possible to get the gaze azimuth and elevation with the realtime APIs?

if not, is it possible to calculate it with the IMU data?

Hi, I have a quick question regarding the data which could be analyzed in pupil cloud! if there are several AOIs in the same project, would it be possible to calculate 1) revisit count to each AOI, and 2) fixation count in each AOI, 3) dwell time or duration on each fixation ??

I believe https://docs.pupil-labs.com/export-formats/enrichment-data/marker-mapper/#aoi-positions-csv doesn't include this info, so any comments would be greatly appreciated! Thanks

Hey Pupil lab team. Our experiment needs to know the head position however from time to time when we ran the experiment the imu.csv file is empty but other times it records data. Could you please help me to figure out the issue? is it something on my end or something else? Thank you in advance.

Hello all, apologize in advance for a long list of questions. I am a user Researcher working at a Public Transportation Service Agency and have been using Pupil Invisible and the add-on for Vive before, and we are thinking about getting another eye tracker for in lab studies and is debating between Core vs Neon and would like to get some input from the community.

1. If connected to a computer, does Neon support using running with Pupil Capture and store/process all data locally?
2. (connected) Are there any difference in the data we can get? (I saw Pupillometry is not available?)
3. (connected) Can I use all existing Core plugins? Such as using markers to calibrate, streaming data through LabStreamingLayer, manual offset correction, or integrating with Psychopy or other experimental programs?
4. Regarding the tracking accuracy, how does it compare to the Core for in lab study (UI and content reading related studies)?
5. And given Neon's form factor can it be used with a VR headset, such as Meta Quest Pro or HTC Vive 2?
6. As Neon camera is fixed I am worry that for people with greater IPD they eyes cannot be picked up (I had a colleage with 70cm IPD that none of the glasses type eye-tracker works except Pupil Core)

Thanks a lot for the input.

Hi @user-a64a96 and @user-594678! Sounds like you have similar problem with the IMU not recording data reliably. Recent updates to the Companion app should have increased the IMUs robustness. Could you let me know what app versions you have been using? You'd find this info in the info.json file of a recording or in the "App info" menu you can access by pressing and holding the app's icon on the homescreen.

3) Unless you run Neon through Pupil Capture, you will not be able to use Pupil Capture's plugins easily. But there are solutions for all the things you listed: - LSL: There is a dedicated LSL relay for Neon, which you could use in combination with e.g. LabRecorder. https://pupil-labs-lsl-relay.readthedocs.io/en/stable/ - Calibration: with Neon's accuracy boost you don't really need to do offset correction most of the time. But offset correction is possible via the Companion app just like with Pupil Invisible. - Neon is about to get officially supported in Psychopy.

4) If your subjects are using things like a headrest and you put a lot of effort into controlling everything Pupil Core might provide higher accuracy for short periods after calibration. In more practical settings the performance is expected to be the same. During long recordings Neon might perform better because Pupil COre can suffer from the calibration deteriorating over time. Consider e.g. these two examples of screen related applications with Neon https://docs.pupil-labs.com/alpha-lab/phone-screens/ https://docs.pupil-labs.com/alpha-lab/gaze-contingency-assistive/

5) Yes, Neon can be used within a VR headset. We are currently in the process of implementing the software support to make this easier. A first release of this can be expected in the next couple of weeks. The first headset we will support is the Pico 4 and more will follow after that. Given the software support, anyone could design a 3D-printed hardware mount themselves for whichever headset they need though.

6) We are evaluating Neon on a very large population covering the full range of IPD values. 70 mm should be no problem at all! Small IPD values can be an issue in some cases because the glasses are not sitting firmly on the head anymore (e.g. when working with children). To solve that we offer child-sized frames and headbands to hold the glasses in place.

I hope this gives you enough input for your decision! Let me know if you have more questions!

What is the cost of a companion device for Neon and head straps?

Thanks @user-d407c1 💪🏻

Hey Pupil labs team. I have two questions:

1. We did some recordings and checked the imu file. The recordings were short, standing in each direction (North, west, east, and south). The imu file showed the same values for yaw for all of them although we did the yaw calibration before this experiment. Then seeing the data like that we checked the glass with the imu visualisation tool + the compass and it seemed that the sensor was "stuck" in the west direction; After doing 360 degrees with the glasses, The glass went back to "normal" and as we moved the glasses the coordinates showed the right direction. Is there a reason why this would happen?

2. There is always a slight drift in the roll. pitch and yaw even when the glasses are stationary. I am wondering why.

Is the world view camera on the Neon capable of receiving IR light? Or can IR light help illuminate surfaces for the worldview camera?

@mpk Even for the purposes of using the reference image mapper?

@user-ac085e in that case just decent visible illumnation will help getting good non-blurred images and improve tracking.

@mpk We are attempting to use the Neon in a low ambient light setting (as needed for the purpose of the study) and were hoping to improve tracking by utilizing some ambient IR light. So this will not work as we intend it?

I dont think out of the box, but some sample footage would help us give useful feedback.

illuminating the markers would help of course. it also depends on how fast the subjects move. Maybe reference image mapper will still work.

It is for use in a simulator

this one is tricky for reference image mapper because the big screens have content that changes. but adding illuminated markers should work.

By illuminated markers do you mean illuminated april tags? Or just unique recognizable features about the evironment?

ideally illumnated april tags. Adding other markers would also work but will required custom tracking code.

Ah ok. AprilTags only work within the marker mapper encirchment though correct? Or can they also assist in the reference image mapper?

that would be for the marker mapper enrichment. But there might be ways of adding other illumnated landmarks that will make reference image mapper work in your usecase. That would simply have to be tested.

Thank you for your assistance.

You are welcome, good luck with making this work!

Hi @user-d569ee 👋 ! Are you accessing the stream trough your browser? VLC? There might be some buffering setting or codec processing causing this delay. Although, there are things you can do like enabling hardware acceleration to improve the codec processing performance, my recommendation would be to use our realtime API. https://docs.pupil-labs.com/neon/real-time-api/introduction/

https://pupil-labs-realtime-api.readthedocs.io/en/stable/examples/index.html

Hello, what is the delivery time for Neon eye tracker now?

For most frames its 4-8 days. I think the only one with a longer wait is "I can see clearly now".

Thank you, Moritz.

Hello! If I'm showing every subject the same stimulus and starting the eye tracker when the stimulus starts and ending it when the stimulus ends, should every subject have the same number of rows? Indicating, same number of gaze outputs?

I have a question regarding the enrichment data. If there are multiple recordings with enrichments shared between them when you download the the enrichment data it puts all of the data across recordings into the same .csv but is there a way to download a seperate folder that contains only the enrichment data for a specific recording?

Hmm, so for our use case we want to use the same enrichments (marker mappers) across multiple participants as they will be looking at the same surfaces but because we are doing additional analysis in MATLAB it makes it hard to automate the importing of the files and associating it with our naming convention for participants, experiments and sources of other physiological data. Hence for our use case it would be more useful to have one folder per recording so that we can automate the navigation through directories to read in the appropriate data and will not have to redundantly load the entire data set each time we analyze the data for new participants. If we used events and each recording was associated with its own enrichment is there a way to keep the same origin so that we do not have to drag the corners of the surface for each enrichment we make?

Hi Pupil team, I'm trying to define a surface using marker mapper enrichment. I tried to detect the surface of the monitor in the picture, but when I move the lines to align with the monitor, it twisted weirdly and I couldn't make it fit to the monitor - and that happened no matter how many markers I selected. The problematic one is the right bottom corner. Do you have any idea why is this the case and how I can solve it?

Yes I have. But previously I didn’t set the surface area manually. For this recording, the enrichment still works if I don’t define the surface area. It just makes the boundary based on the marker’s outer-most corners which is a bigger area than the monitor, so I need to adjust it. The serial number is 917651. Actually our device has some other issue and your team is going to send us a new device. But the problem we have is related to the timing stamps, and I’m not sure whether this would be a relevant issue.

In addition to that, I wonder whether you realize the issue with the app. When I open the app, it pretty frequently shows a white blank screen and shuts down the app. Sometimes it says that the app is not responding and whether I’d want to wait, and if I wait, about half time it works but the other half times the app stays just frozen and does not respond at all.