Hi @user-ae2b7f ! The Python client for the Realtime API does not currently support audio streaming, if you would find this valuable, feel free to suggest it on the 💡 features-requests channel. Meanwhile you can access it direclty through the RTSP stream using other tools.

Hi @user-6e3b6d 👋 ! If you are looking to use Neon with Unity, but without XR, then that package can still be a good reference. For example, the code here shows how to connect via RTSP over WebSockets to receive data from Neon. This is a standard part of the Real-time API protocol.

Hi @user-46e202! You can get the saccade data within your recording folder (right click on the recording on Cloud > Download > Timeseries Data). You can read more about fixations and saccades here. The full list of saccade data is also available in our data format docs

Hi @user-fea107! Just so I'm sure I understand, are you looking for a way to automatically create events in parts of your recording that correspond to the appearance of a single AprilTag marker?

Hi @user-81ea10 👋 ! Could you develop what are you aiming for? Yes, you can wear them during daily activities and the battery can last you up to 4h of continuous recording. If that's what you mean.

Hi @user-0e6279 👋! The recording data, as you mentioned, still achieves the expected sampling rate, which is great. The streaming functionality is primarily intended for monitoring, and the performance might also depend on the capabilities of the Companion Device, but 60Hz is definitely low, so we’d like to investigate this further. Could you please open a ticket in the 🛟 troubleshooting channel so we can follow up on this? Thanks!

Hi @user-665dc2 , could you clarify which saccades specifically you are interested in? For example, do you want any saccades that cross over the AOI boundary or maybe only those that happen within the AOI boundary?

Hi @user-bda2e6! I think the same model is available on Amazon US 🙂

That looks correct. Although we haven't ordered one ourselves from the US store.

Hi @user-9e0d53 - Thanks for reaching out and for your feedback 🙂 I have a few follow-up questions to better understand your setup.

Regarding this:

It seems that the monitor is too far from the participant's eye, and the ET Neon is unable to accurately track the moving target's path, or more specifically, the participants' smooth eye movements, at this distance.

I understand that you use fixed distances between the monitor and the participant when recording data with the Eyelink. Are you planning to conduct recordings simultaneously with Neon and Eyelink, similar to the approach in Ehringer et al.? If so, is this why you need to maintain the same participant-monitor distance for the Neon recordings? May I ask also how do you evaluate Neon's accuracy in tracking the moving targets on screen?

Hi, @user-0cdc2d - something like this should work for you. Use it like so:

pip install msgpack
python recover_annotations.py "path/to/my/export/annotations.csv"

That will reconstruct annotation_player.pldata next to your exported (csv) annotations. You can copy it into the neon_player folder of your recording. Note that you may need to delete the annotation_player_timestamps.npy (Neon Player will automatically reconstruct it)

EDIT: Script removed from this post and a better version is available below.

Hi, @user-aafe26 - this is certainly possible! To start, we offer a realtime API that allows you to stream data from your Neon device over a network connection. This allows users to (among other things) stream the scene camera and gaze data to a PC. From there, you have a many options for detecting the target, like using AprilTag markers for example. If the wearer's gaze lies within the bounds of the target, an action can be triggered to do whatever you'd like

Hi @user-aafe26 👋! Our prices on the website don’t include shipping fees, as these depend on the shipping destination. You’re welcome to add items to your cart and request a non-binding quote that will include the shipping costs directly from our website. Let us know if you have any questions.

Hi @user-0001be 👋! For custom order queries, please reach us at [email removed] When doing so, including a brief description of the frame you have in mind can help us determine if we can fulfill your request.

Hi @user-ebe453 ! 👋 One of the simplest ways to synchronize multiple sensors over a single clock is by leveraging Lab Streaming Layer (LSL). LSL is an open-source networked middleware ecosystem designed for streaming, receiving, synchronizing, and recording neural, physiological, and behavioral data streams from diverse sensor hardware.

We support LSL directly from the Companion Device, so I’d recommend reaching out to MbientLab to check if they offer support for it for example via relay.

Alternatively, you could explore their Python API and integrate it with our Realtime API to send events triggered by specific IMU signals.

Hi @user-2b5d07! Would you be able to clarify what you mean by 'align'. Is your goal to essentially get a gaze circle, indicating where the Neon wearer was looking, on the external video?

@user-a97d77 Since it does sound like you are not using NeonXR but rather general use of Neon, I have moved the conversation to the appropriate channel.

As there are multiple questions, let's address them one at a time. But first, some general concepts:

Every data stream (eye video, scene video, IMU, etc.) is timestamped on the device following Unix epoch, at the sampling rate of the respective sensor: - Eye cameras: 200 Hz
- Scene camera: 30 Hz
- IMU: 110 Hz

When running the marker mapper, surfaces are detected in the scene camera, and gaze coordinates are translated into the corresponding surface coordinates.

What's the definition of surface true/false in the data output? (is it world camera or the marker mapper 2D area?)  This indicates only if your defined surface could been found on the scene camera (namely, markers are found and surface defined) and in the case of gaze/fixation, whether the coordinates of these samples fall within the coordinates. There could be instances where you look to the other places and the surface is not in the scene camera field of view or the surface is detected but gaze is not falling within.

What's the definition of surface true/false in the data output? (is it world camera or the marker mapper 2D area?)

This indicates only if your defined surface could been found on the scene camera (namely, markers are found and surface defined) and in the case of gaze/fixation, whether the coordinates of these samples fall within the coordinates. There could be instances where you look to the other places and the surface is not in the scene camera field of view or the surface is detected but gaze is not falling within.

Currently, we do not offer saccade information over surfaces. If you think this would be useful, feel free to suggest it in the 💡 features-requests channel.

Meanwhile you can apply the translation of saccade position to the saccades.csv file. However, keep in mind that while translating gaze and fixation coordinates to a 2D surface is relatively straightforward, applying this to saccades is more complex. Saccades data might not align well with traditional algorithms for 2D data.

Additionally, note that we do not use a dispersion-based algorithm for fixations. Instead, we employ an adaptive velocity-based approach. For more context, you can refer to this Discord message.

For further details, you can explore:
- Our fixation detector whitepaper (PDF)
- Our publication in Behavior Research Methods

With that in mind, you may want to employ your own saccade detector on top of the gaze surface data.

@user-a97d77 The timestamp[ns] values are not in random intervals. As mentioned earlier, they represent the exact time at which the sensor captured the data locally on the device.

It is expressed in Unix time (nanoseconds) or in other words, the elapsed time from midnight UTC on 1 January 1970 in nanoseconds.

The sampling interval, or the rate at which the data is captured, depends on the specific sensor being used, and settings .

Hi @user-e74b04 , while the gyro_x/y/z values are angular velocities and the yaw/pitch/roll are absolute angles, it is a general property of IMUs that you cannot directly integrate one of the streams to obtain an accurate representation of another. One reason is that they are not sampled within infinite resolution. You will quickly notice what is known as "exponential drift". Computing a derivative, however, is okay; e.g., the derivative of the yaw signal gives you gyro z.

May I ask why you want to derive the angular velocity, rather than use the values provided by the gyroscope?

When running a marker mapper enrichment with AOIs in the Pupil Cloud browser, can I run other demanding programs, or does my computer need to focus solely on this?

You can run other programs or even close the browser entirely. The processing happens on our servers, so your device doesn’t need to handle the heavy lifting. In fact, you can initiate the enrichment from any device, including tablets or phones. The processing time depends on the number of recordings, the total duration of the enrichment tasks, and how many jobs are running concurrently on our Cloud.

Should I add the AOIs before I make the marker mapper enrichment "run"? Or is it better to add them after (or is it even possible)?

It doesn’t make a difference. AOI metrics are computed on the fly, so you can add them either before or after running the marker mapper enrichment.

What is the fastest way to download? I have fast internet, but downloads seem slow (my tested internet speed is about 90 Mbps for both download and upload).

At 90 Mbps, downloads should be fairly quick. If you’re experiencing slowness, may I ask what were you trying to download? Are you noticing this issue consistently or sporadically?

Kindly note that our servers are located in Europe, so if you’re accessing them from another region, you might experience slight latency.

To test your connection to our servers, you can use this speed test tool.

Hi, @user-29a91e - yes, this can be accomplished. First, be sure your LSL inlet (e.g., LabRecorder) is configured to receive data from the ***_Neon Events Stream. Then, you should see LSL-timestamped recording.begin and recording.end events in your LSL data, and corresponding events in your Neon recording data. You can match these events to synchronize the data streams with the audio/video from the recording

Hi @user-a97d77! Responses below: 1. Gaze entropy - I'm not fully sure I understand what you're asking here. Could you please elaborate on your goals? 2. Enrichment time - Two hours seems reasonable for 40 hours of recordings. Do you need to map the entire recordings? Don't forget you can choose smaller segments of recordings to enrich by using 'events' 3. Download - Apologies, but I'm not fully sure I understand what you're asking here either. Can you elaborate here as well? If it helps, you can test the connection speed to our servers with this link: https://speedtest.cloud.pupil-labs.com/

Hi @user-aafe26 , wearing Neon under or over glasses is generally not recommended, as it will obscure the view of either the eye cameras (disturbing gaze estimation) or the scene camera. So, I can see clearly now is indeed the solution for people who must wear glasses. If you need a stronger prescription, then an optometrist should be able to cut a fitting lens. The lens holders have a standard shape.

With respect to Every move you make, yes, it is intended for use with motion capture systems, but you do not need to necessarily build such a system yourself. It is compatible with motion capture systems provided by other manufacturers.

Do I understand correctly that you would like a version of I can see clearly now with the motion capture attachment of Every move you make? If you are interested in custom frame design, then please send an email to info@pupil-labs.com and we can continue the discussion there. 🙂

Hi @user-bda2e6 , this is in principle possible. Do I remember correctly that you are using the experiment Builder of PsychoPy?

I see. Then, this would require a modification to the internals of the PsychoPy plugin to expose that information. My colleague who is responsible for that package is currently on vacation.

Are you experiencing trouble with the gaze mapping? Typically, if all of the AprilTags are well illuminated with good contrast, have equal size, and are visible to you in the scene camera preview, then if one is detected, you can have a fair amount of confidence that are all detected.

I see. I will confer with my colleagues tomorrow morning to see if there is an alternative solution in the meantime.

Hi @user-e74b04 , I'd like to update you. My colleague, @nmt , pointed out that you can derive the orientation values (pitch/yaw/roll) to get angular velocity and use that as another reference/validation method. It is integration where you need to be careful.

My apologies for any confusion!

Hi @user-bda2e6 , may I first briefly ask why AprilTags would be occluded during your experiment? Are your stimuli moving and would cover them?

Hi @user-cc1b01! This error means that the reference image or scanning video were not optimal and the mapping failed. Please review the documentation, and this relevant message on how to run RIM⁠: https://discord.com/channels/285728493612957698/1047111711230009405/1268083029411364927.

You can also find more details on our documentation: https://docs.pupil-labs.com/neon/pupil-cloud/enrichments/reference-image-mapper/

If you prefer, you can invite us to your workspace, such that we can review your data and provide more accurate feedback. Just let me know and I'll share with you the email you can use to invite us to your workspace. 🙂

Thanks for your questions; this is the right place to ask them! What you describe is feasible and sounds like an interesting teaching use-case. The statement about obtaining lawful permission before recording others' data is quite broad, so we'll try to provide further context.

The type of permissions you need will often depend on your location, institutional procedures, and how the data will be used. For instance, when data are collected and published, such as in scientific papers, rigorous procedures like ethics review panels are typically required. However, for general teaching, as described, it's likely to be far less stringent, e.g. just obtaining parental consent.

The best course of action is to consult your institution's data controller or equivalent for advice. If they need more information, they can check the data privacy FAQs, which explain how data are stored in Pupil Cloud, and so on. It's also true that recordings can be kept offline if you don't want to use Cloud. If there are any follow-up questions, just let us know!

Hi @user-ac085e 👋! Just to clarify a few things, unless you’re using Neon with Pupil Capture (in other words, traditional eye tracking pipeline), there’s no calibration involved. It would use NeonNet, our deep learning approach to estimate gaze. Since the gaze estimation is based on single-eye images, there’s no temporal drift either. Are you referring to some other type of drift?

Now, about the gaze offset—this is a linear correction used to improve accuracy by addressing a constant offset. There are a few ways to apply it:

• In the Companion Device: In which case, the offset is stored in the wearer’s profile, and automatically carried over to future recordings for the same person.
• In Pupil Cloud or Neon Player: Where the offset only applies to the specific recording you’re working on. and the downstream data.

Now, why might the offset differ for the same person?

• Central FOV vs. Peripheral Areas: If you make a correction in the central field of view, this offset might not be the equal on the lower bound of the peripheral areas. Check out the accuracy whitepaper or this previous message on this topic https://discord.com/channels/285728493612957698/1047111711230009405/1281140902760550430.
• Different Frames: Switching to a different frame might change how the eye appears in the eye cameras, if this change is drastic, you might need a different offset correction.

Hope this clears things up! Let me know if you have any other questions. 😊

Hi @user-6e3b6d , currently, it is not possible with a single Marker Mapper Enrichment to define multiple surfaces, although you can define multiple AOIs within a surface.

As alternatives, you could:

• Run two Marker Mapper Enrichments, one for each surface, and temporally correlate the data post-hoc.
• Place enough AprilTags around the edge of the screen + writing area to define one large surface and then define AOIs for screen and hand.
• If you go this route, it would be worth a quick test to find an optimal AprilTag layout. More than 4 AprilTags will increase mapping success when others are not visible/occluded.
• It could also be worth trying Reference Image Mapper here.

If you absolutely need two surfaces, then Neon Player's Surface Tracker plugin allows that.

As a final fallback, there is also Manual Mapper.

Hi @user-dcc847! Could you please clarify what’s being plotted here?

Hi @user-83d076 , since you asked about Pupil Cloud, I have moved your message to the 👓 neon channel.

With respect to data privacy, Pupil Cloud is GDPR compliant and has passed an external compliance audit on 2024-06. You can find the certification here.

If you would like more details, you can check out our Data Privacy FAQ.

By "offline app", do you mean Neon Player? Are you looking to map gaze in real-time?

Hi @user-83d076! I'll step in briefly here for Rob. I assume you mean this tutorial in the Alpha Lab section of our docs. That does build on top of Pupil Cloud exports. However, you can achieve something similar without Cloud – use Neon Player's Surface Tracker plugin. You essentially affix AprilTag markers to your screen, and these are used to map gaze from Neon to your screen.

Hi @user-d78df3 ! Could you kindly create a ticket on 🛟 troubleshooting and provide the affected recording IDs such that we can have a look at what happened and reprocess them (if needed)?