Hi @user-f0ea5b , would you be able to provide some more details? That would help us provide better recommendations:
Hi, I am new to the Neon glasses, so a little help would be appreciated! When generating scan path I realized all recordings in the project would be automatically included (e.g., the videos of all different wearers). Is there a way to control this where I can select a certain recording? Or do I have to create each project separately?
Hi @user-94700a , you can change the recordings that are used for the Scanpath by clicking Edit next to X/Y Recordings in the left hand bar of the Visualization editor.
Otherwise, have you or your group already had your free 30-minute onboarding call?
Oh, I see the problem! I tried to select the recordings in enrichment instead of the visualization section. Thank you for the help! 🙂
Hello @rob, I sent you some DMs as I attached images from my MSc work that I do not want to share in a public forum.
Hi @user-f0ea5b , understood. No need to share any photos here. But, if possible, could you repost the text of the question here? We typically keep such discussions to the main channel here, rather than DM. We can start a thread, if you like.
Hello @rob Rob, I used the pl-neon-recording library and adjusted it a little bit to run my analysis natively. I set up my python script using Opencv to customize everything to my liking. I sanity checked using the neon player provided in the pupil cloud as well and I am attaching some images to show you what I am looking at. This is at 18 seconds into the recording where I have paused the player to show the frame at which the baseball is hitting the wall which corresponds to the audio wav showing the impact. However, as the recording progresses there is a systematic delay in the audio compared to the scene camera This is at after 9 minutes and I attached the frame at which the ball impacts the wall for the thump sound. The audio is now several frames late. The data I provided in the Neon server is consistent with what I see in the cloud player and it makes it difficult to get the onset timing for the sound of the ball launching out of the pitching machine. I am using an esp32c3 with a break beam to send a signal to the neon with timestamps but it only records the ball pre launch, not the exact exit. To test for the synchronization separately I setup a python script to play a sound at 1khz and send a timestamped signal to the Neon to get the exact timestamps for the audio event as well as the audio recording on the pupil. It showed the same pattern and the regression indicated that the drift is variable with each recording with the audio falling behind 24ms/min to 36ms/min during the recordings. I did this 10 times to get a good sample and using a drift adjustment of 28ms/min gets me within 1 frame of the event when using the audio to mark onset consistently. The way your script works in that it uses the first loud event to sync the audio to a frame timestamp but then plays the audio at a fixed rate suggests to me that the audio is not tied to the POV camera pick up in the hardware.
Hi @user-f0ea5b , thanks for the details. This is helpful.
May I first ask if you have your modifications of pl-neon-recording uploaded somewhere? To clarify, OpenCV does not process the video or audio streams of Neon's MP4's correctly. It is rather recommended to use pyav with the hardware timestamps saved within Neon's Native Recording Data, which pl-neon-recording is already designed to use & handle.
May I also ask which script is playing the audio at a fixed rate? Can you send a link to it?
Hello @rob Rob, I used both approaches to get the audio aligned and I did use the pyav as was recommended which decreased the offset slightly.
From what I can tell it is your script, the pl-neon-recording library which treats the audio sampling rate as a fixed value # Calculate the sample timestamp using the reference timestamp samples_after_reference = loudest_sample_idx - ts_lookup[lookup_idx, 0] loudest_time = reference_ts + (samples_after_reference / recording.audio.rate) * 1e9
From what I can tell the recording.audio.rate comes from the AVCodecContext.sample_rate in PyAV. So the neon is passing on the intended sampling rate statically which is assumed to be the ground truth for the sampling rate rather than accounting for variable sampling rate. This leads the audio to drift over time as there are probably either dropped samples or the audio is not actually recording at the fixed sampling rate expected by PyAV (which makes sense because even minor differences will lead to major shifts at a 44.1k SR).
If you can look at the screenshots I DMed for reference it makes clear that it is not from my script as the pupil cloud has those same drifts which become very apparent at 10 min recording length.
Understood. I just want to be sure first that your script is working as intended so that there is no potential for additional discrepancies, since that script is the main element you use for analysis.
I assume then you mean the find_clap.py script? I wanted to be sure that we were speaking about the same script, as it does not play audio, but I see now what you mean.
I need to step out now and it is currently after-hours for the team, but I will aim to have a response for you later today or early tomorrow.
Hello @rob, I am adding a link to my function load_audio https://github.com/KlausErikAlexander/Pupil-Audio-Call/new/main. Yes, I am talking about the find_clap script.
Dear all, today we were noticing that some batches of recordings were silently failing. The light on the glasses were turned on and after ending the recording we got the notification that it was saved; there was no sign of failure. However, the records on the phone and in the cloud show a duration but 0MB of video data. Selecting the records on the cloud interface returns an Internal Server Error and Recording - ID Does Not Contain Scene Camera Details. After restarting the app it still did not work but rebooting the phone made it function again. Is there any way to recover that videos?
Hi @user-ecd7e7 👋 ! Sorry to hear that. Unfortunately, if the recordings show a duration but contain 0 MB of scene video data, there is typically no way to recover the missing video afterwards, as it was never properly written to storage.
A few questions that would help narrow this down:
To streamline our responses could you also please open a ticket in 🛟 troubleshooting and share the answers there along with the serial number of the device/module, affected recording IDs, Companion App version and the phone model.
In the meantime, could you try the following:
The symptoms you describe can sometimes occur when the app loses access to required permissions.
Hello! Is it possible to attach multiple google accounts if we purchase the unlimited cloud plan for pupil cloud? Or would we always have to login with one gmail account (i.e., the lab gmail we bought the unlimited cloud plan for)?
Hi @user-dbcae8 👋 ! Yes, that's absolutely possible. To ensure that all users benefit from the Unlimited Analysis Plan, the account that owns the plan (for example, your lab account) should be the one creating and managing the workspaces.
Once the workspace has been created, you can invite other users to it. Those users can then log into the Companion App with their own accounts, select the shared workspace, and any recordings uploaded there (with a linked device) will benefit from the Unlimited Plan associated with that workspace.
You can find more details here: https://docs.pupil-labs.com/neon/pupil-cloud/plans/#sharing-your-plan-s-benefits
Hi all. I have one question regarding the pupil dilation. Can we extract pupil dilation from Neon eye tarcker? how? and if we have some trial how can we marke the starting and ending of ech trial? Thank you very much
Hi @user-a9f703 👋! Yes, Neon can provide pupil size data. To record it, make sure Compute Eye State is enabled in the Companion App. If enabled the device will compute it in realtime. The eye state stream includes pupil size in millimeters, eyelid aperture, and eyeball position, with pupil size corrected for gaze angle.
You can access this data in several ways depending on your workflow:
Realtime: stream it directly from the Companion Device to your computer via the Realtime API, see here.
Post hoc: the eye-state stream is included in the time-series download from Pupil Cloud. If you're using Neon Player, simply enable the Eye State plugin in the sidebar and export the data from there.
In realtime streaming from the Companion Device to your computer.
Posthoc, that stream would be on the timeseries download in Cloud under the 3d_eye_states.csv, or if using Neon Player enable the Eye State plugin on the sidebar and export the data.
Programmatically: you can also access the stream through pl-neon-recording.
As for marking trials, yes, you can use events to mark the start and end of each trial. Events can be created during recording or added afterward, depending on your workflow, see https://docs.pupil-labs.com/neon/data-collection/events/#creating-events PS you can also add them in Neon Player.
Thank you very much for your clear explanation. I will try them 👍
[email removed] Ehinger , I am hopping in
Build software better, together
Hi!
I am using the variables from eye csv 'axis right x', y and z. I will use it for a science paper and would like to know
Thanks!
Hi @user-b55ba6 , if I remember correctly, you already have contact with my colleague, @nadia .
To answer your first question, the optical axis values are estimated by NeonNet, which is not publicly available, so we cannot provide further details about how they are computed.
With respect to your second question, it is first useful to clarify that the 3D eyestate estimates are rigorous. They are based on a few different streams of research and are based on currently known principles of the human eye. These measurements represent our current state of the art in ML-based as well as model-based methods.
For instance, pupil diameter is another component of the eyestate estimates. In short, Neon’s measurements accurately and robustly quantify pupil size in a person-specific manner, and are consistent with a personalised, realistic model of ocular geometry and optics. Among other features, the measurements take into account refraction occurring at corneal interfaces. Neon can be used to investigate cognitive load and the pupillary light reflex, and has been shown to as capable of classifying cognitive states. You can learn more about Neon’s pupillometry here.
Thanks Rob for the response.
I am specifically interested in understanding how accurate is the alignment of the optic axis with respect to the IMU axis.
Any other insights are welcome. Thanks
The eyestate measurements are made more accurate by setting the Inter-Eye Distance in each participant's Wearer Profile, so this would be a source of noise, if relevant to you. For instance, it is set to the adult human average by default, but children have smaller IEDs than adults.
Hi @user-b55ba6 , yes, the eyestate measures are in the scene camera coordinate system, but the optical axes will of course constantly move as the wearer moves their eyes. So, although there is an alignment between the scene camera axis and the IMU's heading, their is no "constant alignment" between the optical axis and the IMU's heading.
But perhaps I misunderstood the question. To help clarify, are you trying to study something like the vestibulo-ocular reflex (VOR)?
Regarding changing the IED for a recording post-hoc, this is currently not possible.
Also, be sure to check out our IMU Transformations guide. We already have a function to convert eyestate to the IMU coordinate system.
Yes. I am interested in something like VOR. If i see the optic axis is rotating upwards (pure pitch) . How sure can I be of how it is rotating with respect to the IMU signals.
If you just want to compare the physical rotation of the eye with respect to physical rotation of the head, then you technically don't even need them in the same coordinate system (although it does make interpretation easier). Since all data from Neon is timestamped by the same high precision clock, you can just convert the Cartesian representation of the optical axes to spherical coordinates and plot the pitch values for each eye and the pitch values from the IMU on the same time axis, especially since they share the same x-axis. There is also a function for that in the IMU Transformations guide.
However, I get the feeling that your question might be a bit deeper than that? Are you asking how can you know if the eye rotated with respect to the IMU because of a conscious, deliberate action on the part of the wearer or because of unconcious signals that stabilize the eye during gait?
My question is about the precision of the optical axis variable. If the optic axis in world camera coordinates is precise, and I know the world camera to IMU transformation, I should not expect much biases in my measures.
It influences my theoretical question. As an example. At frame f I see the head rotates vertically 10 degrees, and the eye optic axis rotates 9 and a bit tilted with respect to the vertical axes. Was this imperfect gaze stabilization? I just want to have an idea of how much i can trust the optic axes measures, not to extract wrong conclusions.
My question is about the precision of
How accurate is the positioning of the line of sight?
Hi @user-d50457, are you referring to the gaze calculation? Neon provides an accuracy of on average 1.3 degrees across the population. You can take a look at our whitepaper to see how we benchmarked the system.
I know this is a trivial question but how do I delete project in Pupil Cloud? I can't find the button deleting the project anywhere 😂
Oh nevermind, found it!