Hi @user-ce065d , could you share the equation you are using? Are you using the optical axis values and eyeball centers?
Yes the equation I'm using takes those values, subtracts the center from the axis values, gets the magnitude of each eye, then creates unit vectors by dividing by the magnitude, before calculating the angle using the arccos of the vectors multiplied together.
Hi, I'm using the neon to record eye tracking in the operating theatre - I've had 2 cases where the phone app says it is recording but then doesn't actually record (or stops at some point). Both times I have visually confirmd the device is recording but the device is then placed into the surgeons scrub pocket during the case and at the end of the case when I have checked nothing has recorded (not even a few seconds) - can someone explain why this is happening? All software up to date and device fully charged both times.
Hi all, I’m strongly considering purchasing neon eyeglasses. I noticed that the purchase on pupil labs website includes a phone. Is there an option without the phone?
Hi @user-d00215 👋. Great to hear you're considering purchasing Neon! You're correct that we ship a Companion device (Android smartphone) with each system. May I ask why you're asking about an option without a phone? Do you mean to source one locally, or are you asking whether Neon can operate without a phone?
Hi @user-56fa85. That doesn't sound right. Can you please open a ticket in 🛟 troubleshooting? We'll collect some more information there to better understand the issue and see about a way forward!
Thanks will do
Left and right gaze vectors
I downloaded pupil player from the website, and I'm trying to view videos but it says this version of player is too old
where do I get the updated version
Hi, @user-632df5 - since you're posting in the 👓 neon channel, I'll assume you have a Neon device and not a Core or Invisible. Pupil Player only works fore Core and Invisible recordings.
For Neon Recordings, you need Neon Player
thank you
Hi, I'm about to start a study that will involve participants performing auditory tasks while they walk outside and we track their eyes with our Neon headset and brain activity using EEG. I'll be using a Surface PC Tablet (Win 11) with PsychoPy for stimulus delivery and response logging. When we're collecting data inside the lab, we send commands from the PC to the Neon/Cellphone via a dedicated local WiFi router. However, when we're outside, we won't have this luxury.
One solution is to use a personal cellphone as a hotspot. I've tested this before, and it seems to work well. However, I'm wondering if there's a better way, that wouldn't require relying on personal cellphones. I read in this forum that Windows 11 has the ability to be used as a mobile hotspot. This sounds ideal. Would it just be a case of firing up the hotspot, connecting the Neon device to the hotspot, and then remotely controlling the Neon device from the PC? Are there any issues that you guys are aware of this might cause problems with this setup? Thanks in advance!
Hi @user-e7102b 👋. Indeed the cellphone approach works for this. It should also be possible to set up a hotspot from your Windows Tablet. The same principles would apply as when using the cellphone, so I don't forsee any problems with the setup. Best to give it a try!
Thanks @user-4c21e5 , I tested out using the WIndows Tablet as a hotspot earlier and it worked great 🙂
Hello! I'm trying to find out where exactly recordings are saved on the companion device/smartphone? The reason I need to know is that I have to specify every place my data is stored throughout my data collection process for a data management plan. I could not find it in the documentation, but feel free to point me back there if it is described!
Hi @user-b57ada ! You can find your recordings in your phone's file manager, eg Files > Internal Storage > Documents > Neon. Inside this folder you should find one or more folders (if you've recorded several times) with a long string ID. Once you open one of these folders, you can identify each recording of interest by its creation timestamp. I hope this helps!
Yes, that is exactly the info I need! Thank you!
To add to Nadia's response, and for any readers of this, if you want to transfer recordings from the phone to your computer via USB (locally), don't copy them from this location. Be sure to use the dedicated export function instead - instructions are in this section of the docs.
Thank you, Neil - I will probably keep using Pupil Cloud but it's good to know about this option.
Hi Miguel, just wanted to see if you've had a chance to follow this up.
Is it easier if I send through an email?
Hi @user-fa126e , my colleague, @user-d407c1 , was away and is back tomorrow.
I'm using the Face Mapper enrichment to find fixations on a face during a section of the video (using custom events added in pupil cloud, conversation.begin and five.minutes, as start and end events).
When I look at the fixations_on_face.csv that the enrichment outputs, this includes fixations which both start and end before the specified start event. Is this normal? I'd like to not have to filter the csv file for fixations that happen between those two events if possible.
Hi @user-fa126e ! Could you please share your enrichment ID?
Hi Nadia, I'm not sure where to find the enrichment ID. I have the recording ID and section ID, but can't find the enrichment one.
Neon Companion cannot access Neon Scene Camera v1
Hi all, i want to use the eyetracking to study the UI in mobile application. I am reading this "https://docs.pupil-labs.com/alpha-lab/phone-neon/", but the setup would required "the phone in a static position on a desk". It wont happen in the reality, in my use case, the user will use the mobile application outside and standing. Do you have any idea on this ?
Hi @user-af95e6! The part of the Alpha Lab tutorial you're referring to demonstrates how our Reference Image Mapper enrichment can be used to map gaze onto a phone. However, this enrichment is designed for static environments.
For your use case, there are a couple of alternative approaches you could consider: 1. Attach markers to the phone and use our Marker Mapper enrichment in Pupil Cloud.
Hello, a quick question: To run any script in pupil-labs-neon-recording, are we downloading the timeseries+video data from pupil cloud and providing the directory for that or do we need to do any other processing? I keep getting errors when I provide the path to the downloaded folder
Hi @user-ebd8d5! May I ask - are you using Neon or Pupil Invisible?
To use pl-neon-recording you need to use Neon's native recording format. You can download it from Pupil Cloud (Download > Native Recording Format)
Hi @user-480f4c : Thank you for the reply. I am trying to understand the "map onto anything project" and have some data from a Neon system
Hi! I have a few Neon videos in need of processing on Pupil Cloud. I’ve removed excessive recordings tagged with this device’s serial but cloud is still suggesting that I’m over my cloud processing limit. Any suggestions? Thanks!!!
Hi @user-1423fd , are those recordings still in the trash? That is found by clicking the three button menu in the top left next to "Add filter".
If not, do you have multiple Workspaces under that account?
I don’t think they’re in the trash. They appear in as the first recordings within this workspace
This is currently our only workspace for this account
I don’t think they’re in the trash. They
Hi Pupil Labs, a reseacher in our group has a question regarding using Neon with myopia / hypermetropia: "would we observe a decrease in accuracy even with participants who have prescriptions around +1 or -1.5? I am wondering whether Neon can accommodate people within a lower prescription range, such as between -1 and +1 or -1.5 and +1.5."
Hi @user-23177e 👋. No decrease in accuracy is expected with that prescription. That said, even with stronger prescriptions, up to +/-6 (like with our extended prescription lens kit), if there is an offset, it can be corrected with Neon's offset correction feature.
I am trying to directly access the eye cameras on the glasses with Python opencv, but so far have only been able to find the world camera view. By directly I mean the glasses are plugged into my computer, not the companion app. How do I access the eye cameras?
Hi @user-eebf39 👋 You can access the cameras from your PC using pyuvc.
May I ask what you’re trying to achieve with that setup? Just keep in mind that gaze and eye-state inference are performed on the Companion Device, not on the PC.
If you want to receive the eye cameras on your computer along with the scene camera, you may want to use our RealTime API, check out this example.
Hi @user-fa126e 👋! Briefly stepping in for @user-480f4c here, you can find it by opening the enrichment’s page and copying the URL, or directly from the enrichment table view by right-clicking on the enrichment.
Hi @user-d407c1, I think the enrichment ID is bd2e8ed7-23d0-43b8-9a88-c03aecc462e1.
It's hard to tell because I had to remove those videos from pupil cloud to make space to process our second batch, but that should be correct. It should include recording ID aab8ccf4-ae60-4103-93bb-5fcd0f0aed9f which is the one that seems to include data outside of the specified start and end events
Thank you. We're trying to stream the eye images into an ML model that gives us simple eye directions to control our assistive robotic neck braces and we don't have any cameras attached to our braces yet so we're using the neon glasses for the time being. I'm aware I could use the API, but for a robotic system it may be advantagous to not have the phone middleman.
As of today, using the Companion Device is the only supported workflow to access all streams and use it without calibration, it is also the most straightforward approach.
I have one question. If I would like to map the gaze into a user-supplied 3D model, when I read the github, it said I need to measure the size, position, rotation of the AprilTag, but it doesn't describe how to measure the position and rotation... Do you have any suggestion on that? Furthermore, if I use that method, do I need to have a photo of my environment?
Hi @user-a0189b , to build on the message of my colleague, @user-d407c1 , that guide is not only about user-supplied models, but also user-supplied coordinate systems.
That is, you in effect choose the position and rotation of the AprilTag. This is because you choose the origin of the coordinate system (i.e., the point [0,0,0]), as well as the orientation and labelling of the X, Y, and Z axes.
For example, in the Alpha Lab guide, we chose the bottom front center of the statue as the origin, with XYZ following Blender's conventions, where X aligned with the front face of the statue. We then used a ruler to measure how far up the AprilTag was in the context of said system.
Other users choose the center of the AprilTag as the origin of their coordinate system.
The process is flexible enough that you don't even need to use meters, feet, inches, etc. You can specify everything in arbitrary units following the steps of my colleague, @user-cdcab0 , here -> https://discord.com/channels/285728493612957698/1250066927980642395/1250232074611200061.
Hi @user-a0189b 👋 ! You can find detailed instructions at the repo, including details about how the apriltag json file is structured and measured.
And yes, you would need a reference image and a scanning recording to perform a reference image mapper enrichment.
Hi @user-fa126e 👋 If you’ve removed recordings from a project, you might need to create a new enrichment afterward. Could you try creating a new one and double-check that it contains the correct data?
If the issue persists, it might be best to open a ticket in 🛟 troubleshooting so we can follow up, we may need an invite to your workspace to take a closer look or additional data.
Hi @user-d407c1, I can't make a new enrichment with the recordings because I had to delete them from the cloud. We only have two hours of cloud storage, so in order to process the second half of our videos I had to delete the first half.
hi Pupil crew! I just got set up with a Neon and had a quick question: the azimuth/elevation angles are coming from what point? Is it a calculated point between the two eyes? the world camera?
Hi. I have a Neon device with 2 hours of recording in the Cloud. I have 6 hours in total, therefore some are blocked. I just erased a couple of recordings to free 90 minutes, none of the already uploaded is unblocked. I only see processed recordings that sum up to 30 min.
None of my videos is that long ( all less than 1h), shounld at least one of them be unblocked so I can process it?
Hi @user-b55ba6 👋. When you delete recordings they go to trash. You'll need to delete them from trash as well to free up space. For the avoidance of doubt, deleting from trash is permanent - they can't be re-uploaded. So be sure to download a local copy if you need them 🙂
Thanks!
Hi @user-8bd1e2! The elevation and azimuth are angles of a 3D gaze ray. The origin of the gaze ray is the scene camera. If you haven't already, be sure to read more about the datastreams and data formats in the respective sections of the docs.
delayed thank you 🙂
Hi, I am working on the tag-aligner project but I am having some orientation errors: 1) I have followed the instructions on the following chain (https://discord.com/channels/285728493612957698/1250066927980642395/1250232074611200061), however, the gaze is not properly mapped. Do I need to use the Apply_transformation which has been recommended but not required? 2) Can I enquire how RIM calculates the poses? Does it use COLMAP or another pose detector?
Hi @user-ebd8d5 , the apply_transformation step is actually required. Can you point to where it says that it is only recommended so we can correct that?
With respect to axis orientation, that choice is yours. It does not need to point in any specific direction.
From the look of the render, I would first double check the quaternion values in your alignment.json file.
The below image shows the adding of the mesh to the table. Does the X-axis need to point towards the right, as indicated in the link above:
Hi @user-f43a29 , sorry for the misunderstanding. I meant the line 'Once you're happy with your model's, I suggest performing an "Apply Transformation" and setting your model's origin to the World Origin. This isn't necessary, but I find it helpful' in the attached discord link.
Thanks for the clarification. As @user-cdcab0 mentions, it does make the process easier, so I would suggest trying again with that and see if it improves the values in your alignment.json file.
Thank you
Hi pupil labs team- I have two questions about video recordings:
Thanks!
@user-f89b02 , may I also confirm that you have already tried the different auto-exposure settings?
Hi @user-f89b02 , the MP4 files are the raw data that we provide. If you glare is an issue, then check out our Map Gaze Onto an Alternative Egocentric Video Alpha Lab guide and let us know what you think!
Dear Pupil Labs. Our ethics commitee requires the CE certificate of the Neon, could you send it to me? Thank you
Hi @user-1fda89 , sure. Please send an email to info@pupil-labs.com with this request and a member of the team will reach out.
Hi Rob- yes i tried the different auto exposure settings but it was still too bright. Then as i mentioned, using the manual setting and putting it to 0 still yields issues if glare. Unfortunately, our worklow does not make attaching another device a good solution unless you have recommendations for very small and secure alternative cameras
Hello, sorry I have some additional questions regarding reference image mapper: 1) Is there any metric to determine the accuracy of RIM? For example, on cloud, when we perform RIM, does the blue bar ("Fixation on reference image"), indicate timestamps where the pose has been successfully computed (Refer to attached image)? 2) If yes, would a reduction in timeframes covered by the blue bar indicate that lesser number of poses have been computed or a reduced accuracy in the number of poses?
Thanks
Hi @user-ebd8d5 👋 ! There is no accuracy metric for the reference image mapper localization.
The dark blue line indicates only whether the reference image was localized or not, regardless of whether it failed due to not being on the field of view or because it actually failed to map it.
The light blue line indicates whether the fixation was on the reference image, as you can see during blinks it also shows as not on the image.
Hi @user-d407c1 : thank you. does that mean a pose is computed for every timeframe even if the reference image is not localised?
Hi there, Is there python packages available to do the functionality the neon player does? For example, when obtaining the data from the companion app, it is initially in binary format. I understand through neon player we can obtain the CSV files, but is there any way to do this just with python? Thanks!
Hi @user-6e77fd , yes, check out our pl-neon-recording library and the example scripts here, including the CSV export script.
Great thanks! Just one more question, is there an easy way to implement the surface tracker from neon player in python?
You could try referencing the code from the real-time screen gaze package. That contains all the core elements in Python code.
Hi! I am trying to create a visualization of the gaze using Video Renderer. I have the gaze overlay enabled and have specified the temporal selection using events to trim the video. When I download the visualization and open the videos, the red circle depicting gaze is not visible. I would appreciate any advise on how to correct this issue!
Hi @user-13d297 ! could you create a 🛟 troubleshooting ticket and follow with the ID of the visualisation (right-click on top of the video renderer)?
That's correct, it will try so.
Hi @user-d407c1: Sorry for going back to this topic. I have two questions: 1) When I run COLMAP on a video/frames, I get an estimate of how many frames I am able to compute a pose for? Does RIM output any of this? For example, if the video contains elements which are not in the reference image, it should not be able to compute a pose? Am I right? 2) I would like to clarify some questions about the IMU transformations. If we use the function eyestate_to_world (https://docs.pupil-labs.com/alpha-lab/imu-transformations/) on eyeball centers, what should the output be? I have used COLMAP to calculate my trajectory through a house using video from the scene camera and converted the poses to LiDAR scale (similar to tag-aligner) (Attached image). However, when I plot the 3D eyeball location (with Rotation matrix being an identity matrix), I do not get the trajectory. What am I missing?
I apologise for the inconvenience
Hello! I was trying to use the Neon Monitor on my laptop, but following the instructions here: https://docs.pupil-labs.com/neon/data-collection/monitor-app/ gives me a page not found error. What am I doing wrong?
Hi @user-b57ada 👋 May I ask, what kind of network are you using?
In many cases, Eduroam or other institutional networks have firewalls or settings that block mDNS discovery and certain types of traffic. Could you try one of the following:
A) Enter the IP address shown on the Companion Device directly, and if that does not work B) Use a dedicated hotspot to connect both devices.
Just ran an experiment and we lost the recording again! I started and stopped it from the companion app, but when I went to download it, it wasn't in the list of recordings. I was marking events throughout and there seemed to be no problems with storage. This is the second time this has happened to us. Can someone tell me how I might recover the lost video and how we can avoid this problem in the future? I can't rly understand what happened. Last time it happened I thought it was because I stopped the recording in the monitor, but later I didn't encounter the same problem until today
Hi @user-ffef53. Please open a ticket in 🛟 troubleshooting and we can look into this for you
I made a ticket, but would love to know if others in the community have also lost recordings and what they do to avoid this. I'm really hoping it doesn't happen again!
Hello, what is the best way to get CSVs (imu.csv, saccades.csv, etc) from the API for recordings? Would it be getting the raw files and then converting them to CSVs?
Hi, @user-3ff3c1 - we provide a few options for obtaining CSV files of your recording data so that you can pick the one that best suits your needs: * Download them from Pupil Cloud * Export them from Neon Player * Generate them using our Neon recording python library
That's correct. Our Python recording API pl-neon-recording can load the raw binary format and export to csv files. Check out this example script that shows how.
Hi, we are using a pair of I Can See Clearly Now and Ready to Go glasses.
We have observed an issue where, when using the Ready to Go frames, the wearer's gaze is frequently directed outside the POV camera frame.
Our thinking is that this relates to the positioning of the frames, specifically on the wearer's nose. We found that if the head strap is too tight, the frames tilt upward, causing the POV camera to look further up and misalign with the wearer's line of sight.
We reviewed the documentation but did not find specific guidance on positioning the frames on the nose for optimal alignment, but we could have missed it. Has anyone else had a similar experience?
Hi @user-f7408d 👋. Your thinking is probably accurate on this.
The Ready Set Go frames often need some positioning before you tighten the strap. For example, if the module is too high when you tighten it, the eyes might only be partially visible to the eye cameras.
When putting the headset on a new wearer, navigate to the eye preview in the app so you can see this for yourselves. Be sure the eyes are centered in the eye camera field of view before tightening the strap.
Conversely, I Can See Clearly Now frames don't need adjustment they just sit into place.
I hope this helps!
Thanks Neil 🙂
Good morning, I have a doubt: this page https://docs.pupil-labs.com/alpha-lab/imu-transformations/#application-example says that the IMU reference system has the z-axis pointing upwards, but why is the acceleration value due to gravity positive in the data? Thank you very much
Hi @user-a4aa71 , that is because the accelerometer in the IMU is measuring g-force, as documented here, which is opposite gravity when standing still on Earth.
Hello. I am trying to estimate the 3D gaze point from optical axes. Neon is very often false about them, leading to the 3D gaze point (geometrically calculated as the nearest point of the optical axes) lying behind the eyes. That means, optical axes are like this \/, instead of converging like this /\, as if eyes were looking in different directions.
Is there any way to fix it, e.g., specific calibration?
Hi @user-fa126e ! As promised, although a bit delayed, here is the gist.
https://gist.github.com/mikelgg93/940e19f6a0ee8552066147e140fff732
Hi @user-69f927 , could you share the equation you are using?
I implemented the formulas from this page https://medium.com/data-science/3d-ray-intersection-closest-point-dc8c72122224
def calculate_viewing_distance(o1, d1, o2, d2): b = o2 - o1 d_cross = np.cross(d1, d2)
t1 = np.linalg.det([b, d2, d_cross])/(np.square(np.linalg.norm(d_cross)))
t2 = np.linalg.det([b, d1, d_cross])/(np.square(np.linalg.norm(d_cross)))
p1 = o1 + t1*d1
p2 = o2 + t2*d2
point_of_regard = (p1 + p2)/2
return point_of_regard, p1, p2
Thank you for the responses to my last question! For another question, is there a way to exclude all .mp4s when downloaded zip files using the recordings:raw-data-export V2 endpoint?
Hi @user-3ff3c1 👋 ! You can exclude the .mp4 when downloading files, similar to how you download Timeseries through the UI.
Have a look at https://discord.com/channels/285728493612957698/1047111711230009405/1419978809737740338
There’s no limit on the number of recordings you can download at once, but if you’re doing it programmatically, I’d recommend working in subsets, or even one by one to avoid very large downloads that might fail midway.
And is there a maximum number of recording IDs that can be used as a param for that endpoint?
Hi - is there a way through the real time API or neon monitor to adjust the exposure time setting that can be done in manual mode when using the companion app? Thanks!
Hi @user-f89b02 👋 ! Unfortunately, that's currently not possible through the API, if you would find this useful feel free to create a feature request at💡 features-requests , in doing so, it would be helpful if you can describe scenarios where you would find it useful.
Thank you! Last question for now, is it possible to programmatically download the mp4s of the recordings with the graphic overlay of the red gaze circle? I see that the in the zip is the raw video
Gaze overlay, you would need to create a visualization with Video Renderer.
So the only workflow available to download the videos with the gaze overlay is a manual one?
May I ask what you’re aiming to achieve that requires fully programmatic downloads?
You’d need to define the visualization properties yourself, it might be technically possible, but not documented, and at that point it might actually be much easier to render the gaze overlay locally. Just curious about your use case 🙂
Hi! I have a question related to the cloud storage add-ons. I have recording from multiple devices but some of them were lended to me from other labs, is it possible to transfer them to the cloud of my device? This would be really helpful since the clou storage will be for each device. Thank you in advance.
Hello! Is it possible to temporally synchronize data obtained with Optitrack Motion Capture System and the Neon glasses?
Thanks so much for this, I really appreciate it.
When doing a calibration for this with Neon, is it still recommended to get the participant to slowly move their head around while gazing at the center of the target?
Hi @user-fa126e 👋 ! Let me clarify this a bit. There’s no calibration per se on Neon, you only apply a linear offset, if needed.
Given your question, I’m assuming you’ve worked with Pupil Core before, is that correct? Pupil Core 3D calibration uses a bundle adjustment method to fit the eyeball model (pye3d) from 2D eye images (see more here). This is not happening on Neon, thus you won't gain anything from rotating the head while gazing a point or doing so before. Every gaze estimation is based on a single set of eye frames.
Now, the snippet I shared simply identifies the marker, measures the gaze-to-marker distance over time (euclidean and horizontal and vertical), and computes the optimal offset that minimizes the Euclidean distance to the marker’s center.
It can be helpful if you aim to remove the operator from the equation and perform the offset definition programmatically, but as I mentioned earlier, this usually brings little to no advantage over simply asking the participant to fixate on a target and manually defining the gaze offset on Cloud., plus usually entails more work.
Hi @user-cc6fe4 👋 ! If you need to transfer workspace ownership, please contact info@pupil-labs.com and include:
- The workspace ID
- The email addresses of both accounts
- A note indicating which account should become the new owner
Note that we would need confirmation from both accounts recipient and former owner.
Hi @user-d407c1, I don't need a workspace transfer, I think. We have recordings with multiple devices in the same workspace, but now the free cloud space is ending and since not all devices are from my lab, I was trying to understand if it is possible to change a recordings from one device to another. I did understand correctly that the cloud storage add-on os for each device, not for workspaces, correct?
Hello! Is it possible to temporally
Sort of! The add-ons are tied to the account/device, meaning they’re linked to both the workspace owner and the specific device.
As for changing the device, it’s not possible to change the device associated with an existing recording.
Ok, thanks for the clarification. One more question, if I have multiple workspaces with one device, how would it be? This could only be a problem if the workspace owners wouldn't be the same?
That's correct! If you have multiple workspaces with one device, to ensure all of them benefit from the add-on, the owner needs to be the same.
Hello! I'm building a experiment with a Neon and an other sensor with LSL library. Before starting my set-up I need to know the latency between an eye movement (image taken) and when this event is updated on LSL library. Did you already measured this lantency? Thank you for your help.
Hi, @user-cab5f3 - by far the biggest contributor to latency is going to be your specific network configuration/environment, which of course isn't something we can measure universally.
Are you using the LSL data streams for real-time data processing or just for recording synchronized data?
@user-cdcab0 we are using LSL data streams for real-time data processing.
Ok. In ideal scenarios the latency can be very small, but again the biggest contributor is going to be your network/environment. You can minimize the effect of network overhead by using ethernet, but this requires additional hardware
A coaching tool for athlete feedback
Hi @user-3ff3c1 👋 ! Are you looking for immediate playback with gaze overlay so the athlete can review the recording right after the session?
If so, I’d recommend using a local workflow rather than using Cloud for that purpose, which will inevitably have a bit more latency.
Just to clarify, gaze pipeline and transcoding statuses are internal fields used by the system to track whether the scene camera video has been transcoded and if the gaze pipeline has been reprocessed (for example, from 30 FPS to 200 Hz on Cloud).
For quick playback, the fastest option is to set up an FTP server on the Android Companion device and share the export folder (Internal Storage > Documents > Neon Export).
After finishing a recording, simply go to the list on the Companion app, tap Export, and the recording folder will appear in that directory. From your computer connected to the same FTP server, you can then load the recording directly into Neon Player for instant playback.
You can still use Pupil Cloud as a backup or sharing solution, for instance, to give athletes access to their recordings later or for more in-depth analysis.
Old thread. Yes, I maybe have my own android device. And I'm following other threads here that a Rasberry Pi might be able to interface with neon?
Understood. The Companion device is essential to operate Neon. Supported devices are listed in this section of the docs. But you would ideally go for the most recent devices, e.g. one of the Motos or Samsungs listed there. It's not possible to tether directly to a Raspberry Pi. Although it's possible to stream real-time data from the Companion phone to other devices.
Really thinking about saving that $700 or so and focus on the research
Thank you for your recommandations. I was wondering if there is a difference in term of latency between the use of the real-time API and the LSL library for data processing and data collection? I have seen that we could improve the latency with the async mode. Do you have an idea of the order of magnitude of the benefit with it ?
difference in term of latency between the use of the real-time API and the LSL library for data processing and data collection We don't have any hard data on this, but most people only use LSL when they specifically need to synchronize data from different sources (e.g., eye tracking + eeg)
I have seen that we could improve the latency with the async mode. Do you have an idea of the order of magnitude of the benefit with it ? The simple API is built on top of the async API, so it does have some additional overhead. I imagine the difference should be very small, but I am curious to hear what you've observed in this comparison
What exactly do gazepipeline_status and transcoding_status mean? In the context of fields being returned by the recordings endpoint
Hi, is there a way to change gaze circle dimension and color in the companion app ?
Hi @user-2eefe1 , there is not, but you can change it on Pupil Cloud. If you would like to see this added, feel free to post it as a 💡 features-requests
Thanks for your answer Rob
I can't post in that channel
When using the New Post button in the top right?
it says i need a tag
You can add Neon as a tag.
I did it. Sorry it's my first time on discord
No problem!
Hello! I'm trying to use the monitor app and I have the companion app open and the devices are on the same Wifi network, but I'm getting a "site can't be reached error" - are there ways to resolve this issue? Thank you!
Also, I've tried putting all devices on a hotspot and still no connection.
Hi! We are currently planning to use the Neon with an experiment in PsychoPy. However when selecting the Eyetracker device in the Builder, the “Pupil Labs Neon” selection appears to disappear the majority of the time and only shows “Pupil Labs Core(iohub)” and “Pupil Labs”. I have reinstalled the plugin and ensured it is the latest version of both PsychoPy and the plugin, is there anything else I should check for?
Hi again, @user-05ed40 - I was able to replicate your issue and released an update for our plugin. Can you let me know if it works for you now?
Can you share your PsychoPy log? Also, can you tell me what operating system are you on and the exact PsychoPy version number you're running?
I've had 2 instances now that while recording an "update FPGA" screen shows up and does not allow anything except downloading it. Is there a new FGPA update coming for the glasses? Ive only had the popup show up during video recording, but the FGPA related update choices in the settings on neon remain grayed out.
Hi @user-a84f25 , could you open a Support Ticket in 🛟 troubleshooting about this?
Hi @user-2798d6 , before you tried the hotspot, may I ask what kind of network you were using? Also, what is the exact address that you are typing into the browser tab?
Hi Rob, I'm using a wifi network - is there something more specific to tell you? I'm using the link on the Pupil website to link to the website. http://neon.local:8080/
Hi, I have a problem with Pupil Cloud. My maximum free storage was full so I had to delete a number of files, but even after deleting them on the Neon Companion App and on Pupil Cloud on my computer, the storage capacity does not seem to reset and reduce. Therefore, the recordings I realized today are not uploading and there is still a message staying my storage is full. Could you help me with this please ? Thank you.
Hi @user-9f34b6 👋 ! There’s no need to delete recordings from the Companion App unless you’re running out of local storage, and doing so won’t affect what’s in Cloud or the storage limits within Cloud.
On Pupil Cloud, when you delete a recording, make sure to also remove it from the trash, otherwise it will still count toward your storage quota.
Also note that recordings exceeding the quota are still uploaded, but they’ll remain inaccessible until you free up space or expand your storage.
Thank you Miguel for your switft reply. I didn't think of deleting the recordings from the trash, I'm sure that will help. I just emptied the Pupil Cloud trash but the storage space doesn't seem to be updating. My new recordings are indeed still uploaded but inaccessible. Does it take a bit of time to refresh ?
Are you the owner of the workspace?
Hi! Is it possible to apply a gaze offset correction from one recording to another recording in Pupil Cloud? For instance, a participant completes a validation procedure in a recording and an experiment in a second recording. Can gaze offset correction conducted posthoc in Pupil Cloud during the validation recording be applied to the experimental recording? Relatedly, is it possible to extract the size of the offset correction from Pupil Cloud? Thank you!!
Hi @user-13d297 ! This is currently not possible in through the UI, but we are looking into this. In the meantime, you can use this utility, which let's you copy one gaze offset from one recording to multiple ones via Cloud API.
We have some videos recorded on devices that ran out of cloud storage space. Are we able to pull data from those or are they essentially blocked for now until we pay for more storage?
Hi @user-3ff3c1! If you want to work with those recordings in Pupil Cloud, you can either purchase an add-on, or delete recordings from the free 2-hour quota to free up space. Alternatively, we have free offline workflows. Firstly, you can export the recordings locally via USB. From there, you could use Neon Player, our free desktop software. Just drag and drop them onto the software to playback and process them. We also have a Python recording API if you wanted to work with the recordings programmatically.
Are we able to use the API to pull new recording data if we've run out of cloud space?
Hello - I'm dealing with an older Neon model (2022) that hasn't been used in a couple years, so much so that the corresponding Android didn't even have the Neon Companion app installed. I went through these steps: - Sufficiently charge the Android - Enabled OTG - Installed Neon Companion app from Play Store - Logged in to app with my Google account (may be different than what was used to purchase the headset / was originally logged in) - Plugged Neon into Android via black USB-C cable and my symptop is: - Companion app does not recognize that Neon glasses are plugged in
Hi @user-1ffe68 ! Could you create a ticket at 🛟 troubleshooting ? Please ensure you follow up with a serial number, companion device model, and android version.
BTW OTG is not necessary
It will be subject to the same limits as using the web interface, meaning you won’t be able to download anything until enough space is freed up or an unlimited storage is applied.
EDIT: Sy! I mean download through the Cloud API, the pl-neon-recording API can be used without any restrictions.
Gotcha, thank you!
Hi,is there a way to adjust the field of view or resolution of the world camera? Current FoV is too large that we cannot capture the close objects very well
Hi @user-45f4b0 👋. Adjusting the field of view or resolution of the scene cam to focus the FoV is not possible. However, we do have a way to map gaze onto third-party video recorded from other cameras, e.g. a gopro. If you're interest, check out this Alpha Lab article: https://docs.pupil-labs.com/alpha-lab/egocentric-video-mapper/
Hi @user-2798d6! Yes - can you share what kind of wifi network, e.g. is it an institutional network, a local router, or something set up from your phone?
Yes! It's an instituaitonal wifi network BUT I will be doing research in the field and may need to use a hotspot. Is that possible?
@user-2798d6 , using a hotspot is possible, as well as Ethernet. Note that we typically do not recommend institutional WiFi, as it places restrictions that can block communication with Neon or the amount of traffic from other users can interfere with the signal. For WiFi, a dedicated, local router is an effective solution.
Thanks! The hotspot didn't work either when I put the android and my laptop on the same hotspot. Is there any troubleshooting I can try?
Hello,my Pupil cloud storage is full. After deleting some videos, new videos still can’t be uploaded. How can i upload new videos?
Hi @user-3d0d4d , I've moved your question to the 👓 neon channel.
Recordings will still upload to Pupil Cloud. Do you rather mean that you cannot access the additional recordings that have been uploading since? If so, then you may need to check in the Trash of your Workspace. Please note that deleting them from Trash will permanently remove them from Pupil Cloud and it is not possible to re-upload recordings. So, make backups if necessary.
Hi- do you have any recommendations for how to mitigate the phones getting hot for longer recordings? We have some recordings that are lasting a few hours which is when this seems to occur. Is there a case you would recommend that could mitigate this? Thanks!
If it makes a difference, my personal phone I was using for the hot spot and my laptop are Apple products whereas the Pupil phone is an android. Are they having trouble communicating?
Hi @user-2798d6 , the Apple hotspot is compatible with the Android phones. That should not be the issue. Can you try again with the direct IP address listed in the Streaming section of the app? So enter:
http://IP_ADDRESS:8080/
Hi! I am trying to perform eye contact detection between the wearer and the person recorded; given I estimate through DL models the gaze of the subject recorded from the scene camera, I want now to compute the deviation between the gaze vector I estimate and the optical axis of the wearer. Do you think is there any way to do that? Should I use data from 3d_eye_states?
Hi- do you have any recommendations for
Hi @user-5c48a3 , do I understand correctly that only one person wore Neon?
Yes, one person is wearing Neon, the other (in front of the wearer) is not
Hi @user-5c48a3 , we don't provide a turnkey solution for such a situation. Since you are already using tools to estimate the other person's gaze from Neon's scene camera image, then you would probably also need some estimate of both person's 3D positions to solve the problem with some accuracy.
Otherwise, if you had two eyetrackers, then you could run our Face Mapper on both of them and follow the Map Gaze Onto Facial Landmarks Alpha Lab to arrive at an answer. Then, you don't need estimates of their 3D positions.
But given that I can estimate the other person’s pitch and yaw (and the z coordinate is given by the distance-which I can estimate), this would give me a vector that I could compare with the optical axis of the wearer, wouldn’t it? My question was more related to whether I could extract the optical axis of the wearer. Do you think there could be a better way with the current setup?
You mean the optical axis of the person wearing Neon?
yes
Ok, then that is already provided by Neon by default. It is indeed in the 3d_eye_states.csv file, if using the Timeseries data from Pupil Cloud or the exports from Neon Player. A high level overview of that data is here. Let us know if you need any help parsing it.
Is there any way I could somehow project the optical axis so to have it with respect of the scene camera? (sorry for the many questions, but I am kind of stuck with this 😄 ) Otherwise, do I have to use the optical axis of the camera directly?
Feel free to ask questions. So, as documented here, the optical axes are already specified in the scene camera coordinate system (so, with respect to the scene camera).
thank you🫶
Hi, can I transfer a recording from one Workspace to another in order to put it in a single project in PupilCloud?
Hi @user-3c26e4 👋 ! Currently, you can not transfer recordings across workspaces. You can add it to any project you want within that workspace.
Hi, I have a query about Neon eye tracker storage on the PupilCloud. I'd appreciate some help!
Hi @user-9c4952 ! What question do you have?
Thanks, I already created a support ticket!
Hello,
I am working with NEON to study golfers’ visual behavior.
In my recordings, I created custom event markers such as “first shot,” “second shot,” etc.
I have exported the gaze, fixation, saccade, and blink data to CSV/Excel files.
I would like to identify all gaze and fixation data that occur within specific time windows relative to these events, for example 2 seconds before and 2 seconds after each shot.
Could you please advise:
what is the best approach to align event timestamps with gaze/fixation timestamps in Excel or another tool, to extract those ± time-window segments manually?
Thank you.
Hi @user-58c1ce 👋 ! All streams are already aligned, in UTC nanoseconds, so you can directly match data across files. So you can open gaze, fixations and events in a worksheet and refer to apply an > and < to the event timestamp of interest.
Hi, is neon possible for realtime data acquisition and processing sync with data stream from other platforms? If so, how to realize it? Thanks!
Hi @user-45f4b0 , yes, both of these are possible and are fundamental features of Neon. To best assist you, may I ask what your research goals are and what hardware do you want to synchronize with?
Hi @user-f43a29 , thanks. I want to do experiment on a real car. So the signals may include CAN, ROS, and some other physiological data
Hi @user-45f4b0 , then you have a few choices:
These methods can also be intermixed and they work over WiFi, hotspot, and Ethernet connection. All are available programmatically in different languages via Neon's Real-time API.
Oh, that's a pity. Do you plan to arrange this transfer being possible?
Hi @user-3c26e4! Feel free to upvote this in feature requests: https://discord.com/channels/285728493612957698/1212410344400486481
I'm still getting this message. Both my laptop and the Pupil device are connected to the same hotspot.
I've also tried putting the Neon ip address in and it doesn't work either
I also tried using my personal phone for the monitor app instead of my laptop and it says "Video Preview may not be available in IOS"
Hi @user-2798d6 , that is a current limitation of browsers on iOS and iPad. Apple has already been notified about that and should eventually correct it.
Regarding the URL, yes, I meant to replace the text, IP_ADDRESS, with the numbers shown in the Streaming section of the Neon Companion app.
May I ask if you are running latest MacOS? Did you potentially ever tell it to not allow Chrome to search for devices on the local network?
I'm trying to put together a simple Python script that will search for the two sets of Neon glasses we have (by IP address), start the recording on both of them, send an event to both recordings indicating when they're synchronised (accounting for device offset), then let me stop the recording on both glasses at any point.
I'm having issues with getting it to discover the devices sometimes, and also with understanding how the offset should work. Could I get some help with this?
If you know the IP addresses, then you don't need to use discovery
With regards to the offset, don't think about clock differences between the two Neons at recording time. During the recording and event generation, you only need to consider the offset between the PC and Neon 1 and the offset between PC and Neon 2. When you are ready to send an event to both Neons, generate a single timestamp in the PC's clock, and send that to each Neon with their respective offset (to the PC clock) applied.
The events will be recorded with different timestamps within each device's recording, but since you know they came from the same original timestamp when you recorded them, you can easily determine the clock offset between the two Neons post-hoc
Hi, I would like to know whether it is possible to define areas of interest in neon player not with the mouse but with coordinates so that the areas of interest of all tested subjects would be exactly in the same place.
Areas of interest within Neon Player only exist in the form of the Surface Tracker plugin
As long as your AOIs exist on flat planes that can have AprilTag markers affixed to them, then the Surface Tracking plugin will work for you. You still define the AOI using the mouse, but the corners of the AOI are defined in relation to the AprilTag markers in the scene. So if the eye tracker moves or the surface moves, the AOI moves as well
Hello, my lab's subscription ends on October 31 and we're waiting for the renewal, but it may take a few more days. Can we still use our neon and make new recordings after our subscription runs out? Do we risk losing any kind of data from those new recordings?
Hi @user-e33073 👋🏻 No worries if you don’t purchase a new Unlimited Storage add-on immediately after the current one expires. You won’t lose any data, as you will still be able to create and upload new recordings to Pupil Cloud, although access to them will be restricted, as your account will revert to the free tier of 2 recording hours until the new add-on is purchased.
Alright, thank you!
Thanks for adding audio support to the latest Realtime API. I have been exploring the audio support. I am able to 1) show live video preview with audio playback 2) save a singular mp4 file using pyav with the audio data muxed and 3) successfully playback the mp4 file with audio. However, the audio playback is delayed in the preview and further delayed in the mp4 file by about a second when I playback the file. Is there a way to reduce or eliminate the audio delay for live playback and also when muxing with video? I do see there is a timestamp associated with each audio sampling. I believe it may be possible to accumulate the video frames and gaze data, then match then up to the audio frame, but I wanted to get your thoughts. Thank you!
@user-f43a29 SUCCESS! I had to go change the settings to allow Chrome to search for devices. THANK YOU!
Hello again, there is a difference in the position of every fixation between yellow circle and red dot. How is this to be interpreted (for example Fixation No. 64)?
Fixations are described with a point and a duration, but the scene can move during that time (e.g., the wearer moves their head while fixating on a target).
Neon's fixation algorithm accounts for head movement by integrating motion data from the IMU. These data are incorporated in the calculation of the fixation point and duration.
The visualization of the fixation (yellow circle) is rendered at the fixation point but then adjusted for scene-wide optic flow.
That isn't always going to align well with the gaze point, and the difference is likely especially pronounced in dynamic scenes/situations
Has anyone else who uses the Python API been having an issue with recordings failing to stop and save correctly? When I run the simple api command device.recording_stop_and_save(), the companion app seems to pause but not stop and save the current recording. The app gets ‘stuck’ and thinks it is still recording, based on device._get_status(). This has only been happening in the last couple days, so it may have something to do with the recent update to the companion app (now 2.9.20-prod). I opened a ticket about this, but since I planned to collect data tomorrow I also wanted to ask if anyone else has experienced this issue or might have a quick fix. Many thanks!
Hi @user-5bd924 , we have left a message in the Support Ticket. We can continue communication there. Thanks!
Hi @user-937ec6 👋 Great to hear you’re already experimenting with the new audio support!
A few points that might help clarify and optimize your setup:
Audio latency: The audio stream is multiplexed with video and encoded as AAC at 8 kHz, with 1024 samples per frame, which alone introduces ~128 ms of unavoidable latency, that’s the minimum you’ll get even in ideal conditions without considering processing or audio driver latency). Additional delay can come from playback buffering or your system’s audio driver. Are you using the bundled AudioPlayer class from the library? That one uses a single-frame buffer and usually keeps things near real time.
Live preview delay: Do you experience it on the example codes or somewhere else? If you’re rendering both video and audio live, most of the perceived desync often comes because we try to consume it as fast as possible. Typically on modern computer both would play almost at same time (see note above), but as you mentioned, you can use the timestamps and slightly delay video to match audio, if you experience a more significant delay.
Muxing and post hoc alignment: For recording, similarly each audio frame has a precise timestamp , so, in principle, you can use those timestamps to align or resample when muxing with PyAV. However, if you’re after minimal latency and best sync, I’d recommend instead starting a recording directly on the device via the Realtime API remote control endpoint, and using the recording from the device. Those recordings use 48 kHz audio and are perfectly synchronized internally, eliminating the ~1 s drift you noticed.
That should give you near-optimal performance both for live and saved playback.
5) Calculate the fractional base for the video (1/30) and audio (1024/8000) and set that in each video and audio frame via time_base.
audio_stream_fraction = Fraction(1024, 8000)
video_stream_fraction = Fraction(1, scene_camera_frames_per_second)
6) Get the most recent video frame and gaze data so as not to use a now old initial frame 7) Offset the first video presentation time stamp (pts) by an appropriate amount such that the first video frame pts is roughly synchronized with the audio.
outside the loopaudio_delay_seconds = 0.3
video_pts = int(
audio_delay_seconds / (1 / scene_camera_frames_per_second)
) # calculate starting video pts to align with audio delay and apparent misalignment
audio_pts = 0
```pyav_video_frame = av.VideoFrame.from_ndarray(array=bgr_buffer, format="bgr24")
pyav_video_frame.time_base = video_stream_fraction
pyav_video_frame.pts = video_pts
for packet in video_stream.encode(pyav_video_frame):
container.mux(packet)
video_pts += 1```
while not queue_audio.empty():
_ts, audio_frame = queue_audio.get_nowait()
av_frame = audio_frame.av_frame
av_frame.time_base = audio_stream_fraction
av_frame.pts = audio_pts
for packet in audio_stream.encode(av_frame):
container.mux(packet)
#there is only one packet per frame
audio_pts += 1
Thanks for the response. I hope the below helps others, and I am posting my solution here for the edification of others and also to get feedback so that I can refine it. 1) Audio latency: Yes, I read the documentation. I am using the bundled AudioPlayer for live audio. 2) Live preview delay: Yes, the live preview delay is present in the example code. I have pretty much accepted this as unavoidable given #1. Delaying the video to align the two does indeed work but comes at the expense of delayed video. 3) Muxing and post hoc alignment: I wrote some simple code (see below) to delay writing the video until the audio aligns. Aligning audio and video I was able to get the audio and video mp4 synchronized by doing the following: 1) Create separate audio queues for live preview and saving the mp4 file 2) When an audio chunk is received, place it into both queues 3) When the first video frame is received, note the time stamp 4) Drain and wait on the mp4 audio queue until the timestamps for the first video frame align with an observed audio chunk. Note that even doing this, the sound is still significantly delayed by a few tenths of a second. Timestamp alignment alone simply does not seem to be sufficient for synchronized audio and video.
if not audio_is_matched:
# wait for the first audio frame
video_datetime, video_frame = await get_most_recent_item(queue_video)
_, gaze_datum = await get_closest_item(queue_gaze, video_datetime)
audio_datetime, audio_frame = await queue_audio.get()
while audio_datetime <= video_datetime:
audio_datetime, audio_frame = await queue_audio.get()
print(f"Matching audio at {audio_datetime} to video at {video_datetime}")
audio_is_matched = True
queue_audio.put_nowait((audio_datetime, audio_frame))
Dear all, We are interested in purchasing an eye tracker for our research lab, which focuses on software engineering. Could you please provide information on the differences between the Neon and Core models, and advise which one would be more suitable for our research needs?
Thank you for your time and assistance.
Hi, could you please advise how I can lit up the QR markers for a night driving? I used small LED lamps from above but it didn't work. Is there a way to lit them from behind through the whole area?
Hello,
I work in a research lab for HCI embodiment research and we’re interested in the Neon. But currently, the price is quite out of our reach.
I was curious if Pupil Labs are open to, or willing to discuss/negotiate prices for the Neon? Do you guys still offer demo devices?
Hi @user-4a1dfb 👋 ! Thanks for your interest in Neon and for reaching out! We don’t typically negotiate on pricing, our prices are already highly competitive for the level of accuracy, robustness, and usability Neon offers. But we do offer discounts for academic and research institutions, as well as volume-based options depending on your needs.
We currently don’t loan demo units, though we’d be glad to arrange a guided demo call, and help determine the most cost-effective setup for your research.
You can book a slot here
Hi @user-7fe82b 👋 ! We’d generally recommend Neon, our latest system. It’s more portable, robust, and easy to use, and it includes additional sensors compared to Pupil Core.
If you’d like to explore the differences in more detail or discuss how Neon could fit your specific research setup, you can visit our website and schedule a chat with one of our product specialists.
Hi @user-3c26e4 👋 !
Lighting AprilTags (the type of fiducial markers we support) for night driving can be quite challenging, they need even, diffuse illumination without glare or specular reflections.
A few ideas you can try:
Unfortunately, there’s no perfect off-the-shelf solution for this setup, but testing a few lighting configurations should help you find a workable balance.
Thank you @user-d407c1 , it is something I am trying to do for a long time. Maybe you can consider making such QR markers like Dikablis do.
Hello, at the moment after importing csv data from enrichments, the only way I can identify my participants is via the 'recording id' column. Is there a way to also import in the wearer id? I'm trying to sync my participant's responses on Qualtrics with their data in pupil cloud through subject id
Hi @user-30f8d5 ! We aim to keep the CSV files minimal to avoid cluttering with excessive columns.
That said you can get easily that information, do you do it manually or programmatically? I can definitely guide you on that.
How would I do it manually?
Manually you can cross check recording IDs with wearer in the UI and change it on wherever you are going to work with, but that's a tedious work and prone to errors.
Instead, you can:
A) If you have the native recordings you can check the info.json which contains both recording IDs and Wearer IDs
B) Leverage the Cloud API to get the wearer of a recording.
Thank you so much. could you offer us academic prices for neon model?. because there is academic prices for core model
Hi @user-7fe82b ! We do offer also academic discounts for Neon as denoted in our webpage.
@user-30f8d5 I have a little gift/snippet for you.
If you follow the README, you can run that script to append a new wearer name column to any CSV file from Cloud with a recording id column.
It leverages the Cloud API to fetch the wearer id from that recording and then the name, in case you have modified the wearer on Cloud. Thus, you would need to get a token.
Hello, Which chip model is installed in the Neon? Does it have enough power to analyse more data? Is the data analysed in real time on the device itself, or is it sent elsewhere, e.g. to a phone or PC? I would like some additional equipment to go with the NEON setup in my glasses, which would require NN and CV, but it would need to work in real life without any additional devices. Is it possible to easily integrate NEON with other equipment?
Hi @user-f6d4a6 , if you are looking to integrate Neon into your system, then please check out our Integrators page and schedule a call there.
If you are not looking for such a deep integration, then you can reference Neon's Real-time API. It is network enabled, while being network- & programming-language agnostic. You can connect to Neon via WiFi, hotspot, or Ethernet cable.
You can even run an additional Android app on the phone that locally receives Neon's real-time data for processing. A basic Kotlin implementation of such an app can be found here.
To clarify some points:
Hi, we have a question about pupil cloud. We created two different accounts, as we had issues with the verification for one Google account. For one of these accounts, we bought more storage. Would it be possible to merge the two accounts or transfer data from one to the other account?
Hi @user-80c70d , could you send an email to info@pupil-labs.com about this? Thanks!
Hello, I am connecting the device via IP address and I am purchasing a router. I wonder what WIFI frequency band is preferred. Thank you.
Hi @user-d4c059 👋 ! All Companion Devices support Wi-Fi 6 and can connect over the 5 GHz band.
The latest model, the Samsung S25, even supports Wi-Fi 7, so if your router provides it, you’ll definitely benefit from the faster and more stable connection in busy areas. 🚀