Neon Companion has not detected a disconnection with the glasses.Also,I can't see the image of the eye camera from the Neon companion, I don't use a third-party USB when connecting, and I haven't updated the version of Android. Please let me know if you have any suggestions on how to deal with it.

is it possible with the player to have the visualisation available on the cloud? (heatmap, aoi heatmap) even after correcting an offset, trimming a vid, ... or else?

Hi @user-bdc05d 👋 ! Albeit different from those in Cloud, if you use the Surface Tracker in Neon Player, you can obtain heatmaps. Regarding, the AOIs, there are no AOIs like in Cloud for Neon Player, or Reference Image Mapper, but you can build custom plugins for your use case.

May I ask what is the reason for not using Cloud?

currently I have an offset on my recording data but I would use the cloud with pleasure if I could modify this offset it's too bad as it is just a tranform to apply to y and x 😦

I already see the surface tracker in neon player but it does not work well on my data, I would have prefer just on a static reference as in the cloud :/

Applying an offset over the data.

Hello, I stumbled upon your website. The Neon module seems amazing, but I couldn't find where to purchase one. Is there any shop page that I'm missing?

Hi @user-fcc31d, thanks for showing interests in our products! 🙂 Click on this link to access our shop page.

What headsets is it compatible with? I can't find a clear answer on the website. I understand that mount is offered just for Pico at the moment but I can build my own for others. Just to clarify i'm interested in the Neon XR module

Hello,is there a way to change the view from scene camera to eye-camera in pupil cloud to set events?

and another question came up: is there any way (boolean e.g.) to understand if the testperson has his/her eyes closed or open to make sure the data we collect e.g. pupil size is valid? As during or experiment the testperson might have to close the eyes for a bit

We currently only sell mounts for the PICO 4 headset. For other mounts, you'll have to 3D-print and build them yourself. We're working on adding more mounts in the future, but I don't have the estimated time for it.

Just out of curiosity, which XR headset are you using? You might also want to check out the 🤿 neon-xr channel for all things related to Neon and VR/AR.

Hi @user-95d0dd, thanks for reaching out! Eye videos are currently not available on Pupil Cloud. Feel free to post a request for this feature in the 💡 features-requests channel! As for the eye openness, we're currently working on this metric. Unfortunately, I don't have a timeline on when it will be finished.

Could you re-post the question in 🤿 neon-xr? Thanks 🙂

Do you mean like this alpha lab article ?

Oh! I thought you wanted them over the reference image.

I don't think I have a snippet that plots fixations over a video, but the fixations CSV file contains the x and y coordinates for those fixations, and both Neon Player and Pupil Cloud (that's what you see in the link you shared) would offer you with visualisation like the one you are looking for.

Could you develop what you are trying to achieve?

Hi @user-42cb18 ! With that develop I did not meant to ask whether you can achieve your goal but whether you could explain in more detail what is your ultimate goal such that I can better assist you. But, of course, feel free to develop the code for it.

Hi there, my enrichment for qr code tracking looks like this and I'm not sure why - can you help? It seems completely misplaced (this is the case over all frames of the video and different videos, including videos where it used to look fine). Thanks

of course yes! sorry for posting in the wrong place

Hi all, we have a user in our team that is using the neon via a shared gmail account. We want to have them create their own account for Pupil instead. Is it possible to migrate the workspace they created using the shared gmail account to their personal account? I noticed that you can invite collaborators to a workspace, and make them admin, but is it possible to make them owner too? Furthermore, would this have any impact on the companion app? Can they login with their personal account and still have access to the previously recorded data (from the shared account)?

Ok great, thanks for your reply. I guess creating the personal account and giving them access to the existing workspace would be the answer.

Hi @user-876d7f, let me clarify about the blink detection. Our blink detection is based on duration thresholds. The min blink duration has to be 100 ms, but the max duration between blink onset and offset is 30 ms. You can read more about the blink detection white paper here.

In your case, your participants aren't blinking, but intentionally closing their eyes. This means that the time between eyes closed and eyes open is atypical, and they exceed the duration threshold to be considered a blink. Thus, I'd recommend that you first look through the blink detection offline guide. Then, you'll be hacking into the blink detection script, and things will get super experimental as you customize the script to your specific needs 😉 .

For example, certain lines of our our blink detector pipeline would be useful, since they help define the thresholds as described in the white paper.

Hi, i am about to use the pupil neon for my multimodal brain imaging study that also includes fnirs. I am an electronics engineer. What i really need to know to mitigate potential interference is a) the wavelength of the NIR LEDs in the Neons frame b) the illumination pattern, if there is one (constant, pwm, if pwm at what frequency ca?) c) if possible a ballpark estimate of the intensities used

Would it be possible to tell me any or all of these details?

An independent question is what synchronization mechanisms with other modalities exist when it comes to higher precision (e.g. with eeg in the ms scale)

Hello ! I'm trying to make a nice ui using the neon headset and python real time API. I wanna use the gaze data to make nice dynamic graphs. After some attempts, I came here to ask a few questions about gaze data variables in the async API, from eyeball_center_left_x to optical_axis_right_z. Here is the point : what all of these represent exactly ? in which unit ? Thanks a lot for your reply 🙂

Hi Pupil Labs,

We've found that the NEON recordings don't have any audio when the phone is locked and we use the recording start/stop API but the audio is there when recording is triggered with the screen turned on and companion app is active. This behaviour beats the purpose of remote controlling the recordings with the help of APIs, is there a known workaround for this?

Dear support team, is this question too technically detailed? It would be very important to me to be able to do joint eyetracking with fnirs

Hi Pupils labs, I'd like to know if the blink detection fit well with the async API ? Thanks for the help and thanks for your amazing work

Thank you @user-d407c1 !! Constant would mean contantly turned on, yes?

(LSL is great and solves a lot)

Thank you

To do so, using the Timeseries CSV files from Cloud or pl-rec-export would simplify things a lot.

I don’t have a snippet to share at the moment, but here’s a basic outline of the process:

1.    Read the timestamps of the world frames.
2.    Load the scene video into a PyAV container.
3.    For each timestamp, read the frame.
4.    Identify if there is a fixation using the start and end timestamps.
5.    Use the x and y positions of the fixation to plot it.

Let me know if you need more details or any help!

It stands for presentation timestamps, in the pyav/ffmpeg context you can find more info here

Good morning:) for our setup we inted to use the marker mapper and therefore show them in the corners of a screen. Unfortuantely i couldnt find in which size I have to present the markers for them to work but also not occupy more space of the screen than actally needed. In this setup our test person will not be further away from the Screen than 1 m. Thank you in advanve for your support

hello, i'm using neon player 4.1.4 to annotate videos and track surfaces, upon export the player crahses. in the log it says

2024-06-10 11:21:26,900 - player - [INFO] launchables.player: Created export dir at "XXXXX\neon_player\exports\001" 2024-06-10 11:21:29,760 - player - [ERROR] launchables.player: Process Player crashed with trace: Traceback (most recent call last): File "launchables\player.py", line 670, in player File "observable.py", line 367, in call File "surface_tracker\surface_tracker_offline.py", line 597, in on_notify TypeError: 'NoneType' object is not subscriptable

2024-06-10 11:21:29,780 - player - [INFO] launchables.player: Process shutting down.

can someone look in to that please? thanks in advance

Hi, @user-688acf - I'm unable to reproduce this error. Would it be possible for you to share your recording folder with us? If you aren't comfortable sharing it publicly, you can email it to data@pupil-labs.com

Hello, we are currently experiencing immediate app crashes after opening the companion app (V 8.15) on two seperate devices. Our 3rd device has Version 8.10 installed and works. Is there a problem with the new Version?

Hello everyone how are you? I'm trying to use the new Map Gaze Into a User-Supplied 3D Model feature , but I'm having some problems with the 3D model, since it's the first time I've used this type of data. I used the 3D Scanner app to create the model of my room and now I'm looking for the tag information. Any tips on how I can get the position and size in the output space units? Should I use Blender or onther kind of 3D software, for example? If so, maybe someone has a tutorial link they can send me?

Hello, how are saccade amplitudes in degrees calculated in Neon given that there is no object distance information?

Hi @user-ff2367 , great to hear that you are trying this! Am I correct that you put an AprilTag on the wall of your room and made a recording with it?

Instructions can be found in the repo. Please also see @user-cdcab0 's post here (https://discord.com/channels/285728493612957698/1250066927980642395/1250232074611200061) that shows an easy and effective way to get the the position, size, and rotation values of your AprilTag.

For context, @user-cdcab0 programmed Tag Aligner 🚀 ❤️

Tag Aligner is independent of 3D software. The units and coordinate system are completely up to you. You can place the origin of your coordinate system at the bottom left of the door or at the center of your desk. You measure the center of the AprilTag with respect to this origin in your 3D software's units, meters, feet, etc. The choice is yours. For example, @user-cdcab0 's post shows an easy way to specifying everything in Blender's system.

In the case of the the Alpha Labs article, meters were the units and a ruler was used (see attached image). The foot of the statue was chosen as the origin and the AprilTag was 0.705 m upwards from that point. The size of the tag's black side was measured as 0.135m. This was put into the tag's json file (see step 0.3 of the repo). If you decide to do it this way, then you should scale your model to the same dimensions. For the Alpha Lab article, Blender's measure tool was used as a guide. The top of the statue podium was about ~1.27m above the foot of the statue.

Please let us know if you have other questions

Hi @user-fcdbb6! The continuous radiance values measured at 0.2m are 205 W sr⁻¹ m⁻². Let us know if you need anything else.

Hi @user-ff2367 , you are welcome!

The tag orientation needs to be specified in the json file as a quaternion in the OpenCV coordinate system. How did you hang and orient your AprilTag?

If you have made a 3D model of your scene in Blender, then you can make everything much easier by following @user-cdcab0 's instructions here (https://discord.com/channels/285728493612957698/1250066927980642395/1250232074611200061). The process should be similar for other 3D modelling software.

[email removed] , you are welcome!

Hi I am new to Neon and I'm trying to use real-time-screen-gaze. I generated april tag markers on the four corners with marker size 128x128 and 16-pixel white padding. When I specify coordinates in marker_verts, should I include the white padding, or only list the corners of the markers?

Hi, @user-b7c82e - do not include the white padding. Only the the black square of the markers is considered

@user-cdcab0 Thank you for providing those Tag Aligner + Blender instructions!

For anyone who referenced my previous posts, I recommend to also follow @user-cdcab0 's instructions here: https://discord.com/channels/285728493612957698/1250066927980642395/1250232074611200061

You will have a much easier time determining the rotation and position of your AprilTag 😄

Hello! My name is Ryan, I am using Neon to track where, on the spectacle lens, a subject is looking through. I've used the optical axis vector in 3d_eye_states.csv with good success. However, its come to my attention that from a physiological standpoint, it makes more sense to track the visual axis. This is typically a few degrees off from the optical axis of the eye.

Based on what I've read on the Discord and on the neon website, the gaze information provided by neon player is the visual axis. The only problem is that its in terms of the world camera coordinate system. Is it possible to the gaze of each eye as a unit vector, like how the optical axis is provided?

"gaze_point_3d_x (and y, z)" in gaze_positions.csv is the closest I've found, but it is still in world camera coordinates. Thank you in advance!

Hi @user-429d6a 👁‍🗨 ! You are correct in that we provide you with the "optical axis" in the eye states and not the visual axis per eye, and that this would be more correct for your application.

Technically, you should be able to compute the "visual axis", you have two vectors.

• [gaze.csv]: V1 is defined by its azimuth and elevation angles with its origin at (0, 0, 0) (scene camera origin).
• [eye_state.csv]: V2 has a direction given by coordinates (x_1, y_1, z_1) and its origin at (x_0, y_0, z_0) .

To get the "visual axis" vector for the eye, you could convert the gaze vector from spherical coordinates to cartesian, then as they are normalised compute the dot product and finally compute the new angle.

But it can get a bit more complicate, and it would depend on your definition of visual axis, take for example the diagram I attach based on Bennets & Rabbets' clinical visual optics.

Also I think the pantoscopic angle on Just Act Natural is a bit more than the avge, but don't quote me on that.

Hi @user-d407c1, thank you for your helpful response. I was able to convert the elevation/azimuth to x,y,z. After thinking more about my problem, I've come across another question that I think you could help answer. Is the gaze data in "gaze.csv" calculated from the optical axis data? If so, perhaps there is some code I could reference for my own project.

Visual axis

Hello I have a question about saving events with custom fields over the Neaon API. I have just started experimenting with out labs new Neon and its API. I have been using the pupil core for many years and rely heavily on the API's ability to save annotations with custom fields enabling me to save data such as stimulus condition, order, and other settings within multiple columns alongside the standard 'label' column. Is it possible to also save multiple columns when saving events with the pupil Neon? I had assumed it would be possible given how easy it was with the core but I haven't been able to find any information in the Neon documentation on how to do this.

Hi @user-a09f5d. To send events using Neon's Real-time API, you can use the send_event function that takes as input arguments the event name and the current timestamp (see here for some examples).

Although more columns cannot be added, you still can create custom names (incl. experimental variables) within your experiment (eg like in this tutorial). For example, assuming you have a matrix with your experimental conditions and a script that loops over your experimental trials based on this matrix, you can easily assign the stimulus condition or order to your event's name

If I’m wearing the module in a sound booth (faraday cage) after I calibrate the IMU outside, will it be okay for it to detect the magnetic north?

Your approach would work; however, I am wondering, why would you want to code the subject ID, age, and gender in every event? You can simply save the final csv file with this information (e.g., ID1_33_F.csv). To keep things tidy and clean for your data collection, I'd simply add the event-related information to the event name, that is: {block}{TargetDistance}{TargetPosition}

I can't see a reason why character number would be an issue, but I can test it out on Monday and let you know 🙂

For all the info that doesn't change within a recording, you could send a single event at the beginning of the recording that contains all of that info. You could also send one event per piece of info (then you don't have to worry so much about the length)

How is missing data coded within the gaze.csv exported file? I am looking at a sample file of 10 minutes of recording and there are zero NA's or empty rows for the gaze position (i.e., gaze x [px]); it seems likely that there would be missing data for a few of the samples.

HI @user-07e923. Thanks for the response. I work in clinical research with children and in typical eye-tracking experiments there is data loss due to movement, closing eyes, rubbing hands on eyes, jumping out of the chair, etc. Using Neon is newer for me; I guess I just assumed there would be times when the eye-tracker could not find the eyes (above and beyond blinks). Ultimately, I am trying to calculate some type of data quality metric to demonstrate feasibility in my intended participant sample.

Hi Neon team, I'm using device = Device(address=ip, port=8080) to locate the Neon eye-trakcer. However, this code return the OnePlus phone that the eye-tracker is supposed to connect to. When the eye-tracker is not conected with this phone, the code still returns the device object as long as the phone is in network. Is there anyway to use the Device() function to tell whether the eye-tracker is properly connected and ready to send_event or stream the eye_video? I'd like to achieve the same effect as discover_one_device(), namely when the eye-tracker is disconnected, it returns None, but I need to use the IP address.

Hi Neon team, I was testing out the real-time-screen-gaze package and generated 256x256 markers with white borders (48 pixels). Visual stim were drawn using PsychoPy. gaze_mapper was able to create a surface, but it failed to produce surface_gaze coordinates. I tried to tag the markers offline using marker mapper on pupil cloud, no luck as well.

Hi @user-613324 , device.serial_number_glasses is for Pupil Invisible. For the Neon module, you want device.module_serial.

Hi @user-613324 , the timeout for ‘receive_eye_video_frame’ only applies when there is no cached data. If it is not timing out, then it should be because you are actually receiving data, or at least have not yet drained the queue of cached data. Those messages are not exceptions but diagnostics, and are in principle harmless. The “unable to create requested socket pair” message is typically only encountered on Windows. If you are receiving data and are able to remotely control the device, then you have a working connection. Regarding a timeout parameter for the Device constructor, I will check with the team and get back to you.

hi, is there a csv or a way to get back timestamps of event we annotated in the cloud? 🙂

Ah thank!

Hey pupil labs, is there any way where i can set a threshold for Fixations. and only identify fixation that are greater than certain seconds ??

Dear support team, I'd like to know more about the "worn" attribute of the gaze data within the real time api data. I'm using the neon headset for 3 week now and the worn attribute has always been True, even when the headset was on desk. Do you have any clue for me ? Thanks in advance.

Hello Neon Team, I am new to using Neon. I am exploring the real-time-screen-gaze package. I am wondering how to connect the device on my local machine (not remote) because I want to do the same task with setup 1 monitor

Hello, wanted to ask if it was possible to make multiple Marker Mapper enrichments on the same workspace with the same April Tags used per visual stimulus, but different defined surfaces so that I can analyse areas of interest across different stimuli all in one workspace. As of now, the same image appears as the defined surface for all the marker mapper enrichments. The only solutions I can think of are using Reference Image mapper or adding all the projects to new Workspaces each time I want to do a MM enrichment (I have to make 56 of these), thanks.

To answer your question: Yes, you can have multiple marker mapper enrichment using the same April Tags. It's probably helpful to use events to mark the recording section that you want map -->e.g., Image X appear -- Image X disappear. Then, run the enrichment for all your recordings that have these events.

Edit: If I didn't address your question, then could you please elaborate on what I missed? 😅

Hi we have just carried out a research project eyetracking Regional Anaesthesiologists performing regional blocks where we had a marker on each corner of an ultrasound screen but we had different machines from different manufacturers so the shape and size of the screens would be different.

If I’m carrying out a Marker Mapper enrichment for the project can it be applied to all the recordings with the different types of screens in the usual batch form of do I have to do individually for each specific machine type?

Hi @user-07e923 they are different sized ultrasound screens 4 of them but they all had different combinations of markers

The next question regarding this project is we have 42 recordings which we want to download from Video Renderer but the size of the processed video is 4.7GB and we will have to remove things from our laptop to do so. Is it possible to download them to an external drive connected to the laptop without having to remove anything from our computer

Thank you @user-07e923 that was really helpful. 💪🏻

Hi @user-324374 ! The Marker Mapper should take the reference image you see on the right from one of the frames within the defined scope. If that is the case, would you mind inviting me to your workspace for further investigation?

Great, just sent an invite. Also, if I wanted to calibrate gaze data from native recordings using Pupil Player, how would I do this after adding Enrichments to the recordings on Pupil Cloud?

Yes, the gaze offset would need to be corrected individually on Pupil Player, but how would applying an enrichment to the modified data work?

Hi all, I have the following problem: I want to synchronize the recordings from two Neons using the UTC timestamps. I tested it by tapping a plastic token on a table and then plotting the audio recorded by the microphones corrected for the difference in recording starting times. Still, there was an offset of about a second. I then used the pupil labs cloud to set a very precise event for when the token touches the table and indeed the UTC timestamps for exactly that event differ by about a second. I already turned the "automatic time" off and on in the phones' settings. Do you have an idea what might be the reason for the world timestamps differing so greatly? And is there a better way to align the recordings of the two Neons? Thanks in advance 🙂

Hi @nmt thanks for your response! I synced the clocks as described in your docs directly before I did the test, so unfortunately this did not solve the problem...

@nmt Hi, My Neon suddenly cannot get the eye picture, but the front RGB is good. What's going on? The image of the eye captured in the app on the phone is black.

Since this morning, I have problems to upload a recording to pupil cloud. The recording is indicated in the workspace, but the video is black and further actions, such as moving to a project or downloading, are unavailable. What could be the case here? In an attempt to resolve the problem, I deleted the recordings from the cloud and tried to upload them again. However, they don't get uploaded a second time. Not automatically and I can't find a way to do this manually. Any suggestions would be appreciated.

Hello @user-817cfd 👋! Sorry to hear you’re having trouble with a recording. Could you please create a ticket in 🛟 troubleshooting so we can assist you better?

When creating the ticket, please include: - Whether the Companion App shows the recording as fully uploaded. - The recording ID: You can find in the Cloud trash unless completely deleted. Please restore it, then right-click over it, and select “Recording Information,” and finally share the ID in the ticket.

For future reference, please do not delete recordings before confirming uploads, as they cannot be re-uploaded.

Hi! I am trying to download the enrichment from my workspace in the cloud, but I get this error:

{ "code": 500, "message": "internal server error", "request_id": "6d4465a4c397e53a8e943e4a8861ea3b", "status": "error" }

How can I download my data?

Hi we have just carried out research on regional anaesthesia last week in the cadaver lab in Dundee University using a combination of three Neon Glasses and one Invisible which were all controlled by the Monitor App. All the devices were checked before we carried out the recordings to ensure that we were picking sound as it’s an important aspect of our medical training research but of the 42 recording only 15 have sound on them. I wonder if this is common feature

Hello, we are trying to use MATLAB's "pupil_labs_realtime_api" function in order to send event markers using an ethernet connection from an Ubuntu machine (Ubuntu 18.04.6 LTS)

We are having two issues, the first is that we're unsure whether the ethernet connection is successful since Ubuntu 18 does not have an option to select "shared to other computers" (as we were advised to select).

The second issue (assuming that the connection is established) is that calling the pupil_labs_realtime_api function in MATLAB produces the error: "Could not access server. Host not found: pi.local."

Any advice on setting up an ethernet connection and sending triggers via MATLAB would be much appreciated.

Thank you so much - this information has been helpful.

We have now downloaded the pl-neon-matlab repository and we are using MATLAB version 2020b with a python environment 3.8. With these settings, calling pupil_labs_realtime_api now returns the error "could not access server. http://pi.local:8080/api/status". Calling the "Device" command returns an error "could not find any device".

Thank you for looking into any changes we may need to make given our version of Ubuntu. Hopefully it is just an issue with setting up the ethernet connection. Thanks!

Hi everyone,

I have a question about Pupil Neon. In the analysis section, when I need to upload a photo to generate the heatmap, does this photo have to be taken by the camera? Or can I extract it from my film? Thank you!

Will do @nmt

Hi! I need to extend the usbc cable of the Neon. I bought the UGREEN USBC Cable extender but that’s not working. Any tips on how to get it to work or any recommendations for the right type of extender cables?