hey @user-d407c1 , I need more help, I guess. I'm trying to run a mount calibration for the PICO. I tried the following options: (a) running the apk for the PICO 4 -> visualizing the gaze rays indicates that there's no eye data being picked up or integrated (gaze is going just straight along my local z). Furthermore, when I proceed to the calibration scene, it shows all targets at once instead of sequentially (with pinch-confirm I can then deactivate the two along my local z-axis/gaze, that's it). HMD and Neon Companion are in the same WiFi (usual home setup; no institutional shabang). (b) Running the latest version of the neon-pico branch of the MRTK3 demo from Unity via PCVR (using PICO's business streaming) shows interactive gaze (i.e., ray moves with my eyes, but is clearly offset). However, in PCVR I do not have hand tracking, so I cannot move on to the calibration scene. The PC running Unity is connected via ethernet with the router. (c) Manually building the calibration scene from the latest state of the branch (incl the NEON XR prefab in the scene) again does not seem to pick up the gaze data (same WiFi setup as in (a)). In contrast to (a), it now shows one target at a time and lets me loop through them. However, at the very last target, it gets stuck and never reacts on my pinch-confirm (I assume some code which is triggered by the end of the calibration throws an error). Do you have a recommendation how to move forward? Debugging the apk is obviously painful, so before digging into that, I'm asking for directions first (and so will be modifying the PCVR version to be compatible with controllers; I read above that others went through that and suffered). Maybe, I'm overlooking something trivial (granting some permission?) which prevents the devices to find each other on the network?

Hi @user-f43a29 , thanks! I tried that for the original apk before; didn't help. Now I tried also for my rebuild of the calibration scene. Surprisingly, that worked! (Still painful to go through all targets with the poor hand tracking in the PICO, but ok.) Thanks!

Is the "I can see clearly now" frame roughly the same size and dimensions as the "just act natural frame" ? We are thinking about using the I can see clearly now frame for VR experiments so we can include participants who normally wear glasses. I just tried the Just act natural frame within the Quest 3 and it fits but it shouldn't be any bigger. Did Pupil Labs try/test the I can see clearly now frame within the Quest 3 for extended periods of time (+20 minutes) ? If so, what were your experiences?

Thanks! I assume you will be using an AMVR facial interface? We are currently using AMVR Q3FC3 FC4 for non eye-tracking vr experiments which probably is quite similar to the AMVR interface used for Neon XR

Hello hello, my name is Erik S, I am currently working on a project to support a dear friend and community member near our university city, and we are interested in purchasing the Neon Quest 3 Module, and have some questions surrounding the complete product package, and would greatly appreciate any clarification on the product layout - is the quest frame included, and is that the point of naming the product "Quest 3". Secondly, I have a Samsung Galaxy Flip 7 with Android 16, would I suffer any significant connection issues or graphical problems, because I have a high refresh rate, and well-above CPU hardware spec? Thirdly, our friend wears very thick lenses, and we are wondering the gap clearance between the Q3 bridge, and frame, if it would still fit his frames? Any support on this would be greatly appreciated! Thank you!

Hello! We moved the module to another set of frames. We're trying to access the scene camera but are unable to do so. Is it possible it was turned off because the module was in the Quest 3 gasket? Is there a way to turn this back on? Thanks in advance!

Hi @user-2ff673 , the scene camera is only disabled while using it in the VR mounts. It should record as normal after moving it to one of the glasses frames. Are you open to starting a Support Ticket in 🛟 troubleshooting about this?

Hello, I'm working on Unity with Neon XR and it can detect my phone ( Neon's companion app) easily in editor mode but it struggles to connect when creating a build. It even freezes at some point, which doesn't occur in editor mode where it works very well. Any suggestions to address this issue ?

yes that's what I meant, I use the ray pointer to check if the gaze is actually received and I can see this pointer following my gaze pretty well in editor mode but in the build that I created, it takes 20 to 30s to connect and once it's connected the pointer freezes at some point

So my glasses are connected to the phone with the usb-c cable, the phone is using wifi and the PC I'm running Unity with is connected to the same wifi

Tethering from my personal smartphone

Actually I don't use any headset, I only use pupil glasses to see that pointer in Unity. The only log given by Unity is that one "[RTSPClientWs] sleeping for 5 seconds before attempting to reconnect" and then I would try again

sure, build log : "[RTSPClientWs] using ip: 10.106.12.4 port: 8686 [RTSPClientWs] listening aborted Unable to connect to the remote server [RTSPClientWs] sleeping for 5 seconds before attempting to reconnect [RTSPClientWs] awake [RTSPClientWs] using ip: 10.106.12.4 port: 8686 [RTSPClientWs] sending: DESCRIBE rtsp://10.106.12.4:8686/?camera=gaze RTSP/1.0 CSeq: 1 Accept: application/sdp

[RTSPClientWs] received: RTSP/1.0 200 OK Server: Pupil Invisible RTSP Server Cseq: 1 Content-Length: 159 Content-Base: rtsp://10.106.12.4:8686/ Content-Type: application/sdp

v=0 o=- 0 0 IN IP4 10.106.12.4 s=Unnamed i=N/A c=IN IP4 10.106.12.4 t=0 0 a=recvonly m=application 0 RTP/AVP 99 a=rtpmap:99 com.pupillabs.gaze1/90000 [RTSPClientWs] sending: SETUP rtsp://10.106.12.4:8686 RTSP/1.0 CSeq: 2 Blocksize: 64000 Transport: RTP/AVP/TCP;unicast;interleaved=0-1

[RTSPClientWs] received: RTSP/1.0 200 OK Server: Pupil Invisible RTSP Server Cseq: 2 Content-Length: 0 Transport: RTP/AVP/UDP;unicast;destination=10.106.12.4;client_port=0-1;server_port=44886-48042;ssrc=b624b455;mode=play Session: 1185d20035702ca Cache-Control: no-cache

[RTSPClientWs] session: 1185d20035702ca [RTSPClientWs] sending: PLAY rtsp://10.106.12.4:8686 RTSP/1.0 CSeq: 3 Range: npt=0- Session: 1185d20035702ca

[RTSPClientWs] listening aborted Aborted [RTSPClientWs] sleeping for 5 seconds before attempting to reconnect [RTSPClientWs] awake [RTSPClientWs] using ip: 10.106.12.4 port: 8686 [RTSPClientWs] listening aborted Unable to connect to the remote server [RTSPClientWs] sleeping for 5 seconds before attempting to reconnect NullReferenceException: Object reference not set to an instance of an object at PupilLabs.RTSPClientWs.Stop () [0x0000b] in <dcaeb709c7c04e10a650d9b19d464d45>:0 at PupilLabs.NeonGazeDataProvider.OnDestroy () [0x00008] in <dcaeb709c7c04e10a650d9b19d464d45>:0 "

And editor mode log : "[RTSPClientWs] sending: PLAY rtsp://10.106.12.4:8686 RTSP/1.0 CSeq: 3 Range: npt=0- Session: 1185d20035702ca"

Hi @user-7292c5 , in case it was missed, I responded here: https://discord.com/channels/285728493612957698/1472972651239702653/1472987249082892389

Hello, I am conducting research on vergence and depth perception in augmented reality. I would like to ask whether the Pupil Labs Neon module can be integrated into an AR headset in a way that allows accurate estimation of gaze direction for each eye separately. Specifically, would such integration enable reconstruction of the optical axes of both eyes in 3D space, so that vergence could be reliably computed?

Hi @user-78d6c8 , yes to both questions.

• Neon is modular. We have open-sourced the 3D geometry of the eyetracking module and one of the frames for reference. If you are using a Pico 4 or Quest 3, we already offer mounts for those. Otherwise, you can follow this guide to make your own mount.
• One of Neon's default data streams is the optical axis and center of each eye, allowing you to estimate vergence.

When it comes to gaze direction though (i.e., the visual axis, rather than the optical axes), we currently only provide a single gaze vector, estimated either binocularly or monocularly. However, it sounds like the optical axes are more important than dual gaze vectors for your work.

Hi everyone, I’m using the NeonXR plugin because I’m developing a game in Unity, and I also need to save the eye tracker data to a file. Is there any way to get the data at 200 Hz, or at least at the 90 Hz update rate? I’ve tried saving the data to a file in the Update() method of the script NeonGazeDataProvider.cs, but it only saves at about 20 Hz. When I try saving it in GazeDataVisualizer.cs, inside the OnGazeDataReady(GazeDataProvider gazeDataProvider) function, it saves at around 40 Hz. Am I doing something wrong? Where should the data be saved to achieve the highest possible frequency? Thanks a lot!

Hi @user-a4aa71 , if you want to save at the update rate of the display device and/or the Unity app, then use the udp branch of the Neon XR Core Package. You can follow the same installation instructions, but use this URL:

https://github.com/pupil-labs/neon-xr.git?path=/com.pupil-labs.neon-xr.core#udp

If you want the data saved at 200Hz, then as documented in the tip here, you should run a Neon recording at the same time and post-hoc sync the data, followed by post-hoc applying the mount calibration to transform the data to VR coordinates. I actually made a basic script for this yesterday. I will do some more work on it today and can post a version for you to try here by Monday.

I saw that there’s also a branch called “ichair”, what’s different about it?

Hello everyone, I'm using NeonXR to receive gaze data in Unity and I was wondering if it was possible to start/stop capture and send event to Neon from Unity. Without having to manually start recording on the phone with Neon Companion app.

Hi @user-7292c5 👋 ! You can do so by sending a POST request in Unity to the following endpoints .

E.g. to start a recording you would like to send a post request to http://{IP_ADDRESS_OF_COMPANION_DEVICE}:8080/api/recording:start

Hi @user-7292c5 , can you try the instructions here, with the URL exactly as typed there: https://discord.com/channels/285728493612957698/1187405314908233768/1474378303828922560

I'm now able to start and stop capture using this type of URL http://10.106.12.4:8080/api/recording:cancel but I'm struggling to send events. It seems that it requires a different JSON body but I can't figure out how to implement it in my Unity script...

Hi everyone, I’m using the Neon together with a Meta Quest 3 for a gaze-contingent experiment. At the same time, I’m recording eye movements via the Companion App on the phone. However, within about 10 minutes of starting a recording, the app consistently throws an error and stops the recording. My suspicion is that the eye tracker may be overheating inside the headset - the eye-tracking module was very hot. Is this a known issue at Pupil Labs? Have others experienced similar problems when combining gaze-contingent displays with mobile recording? Any tips or suggested solutions would be greatly appreciated. Thanks in advance!

Hi, I'm working with Unity + Quest3 + NeonXR to conduct gaze-based XR research. Today the eye-tracking function suddenly does not work within the apk which is built and worked yesterday, offering only head pointing ray&pointer as a fallback. The GazeDataReady event is never triggered. I checked the log and it says the communication between the Companion phone and the headset is successful. I can also see the gaze point or adjust offset on the Companion App. It's only the headset cannot get any eye-tracking data. Could someone help me with this issue? 🙏🙏🙏

Sure, thanks alot ! I'm wondering is there any way to access saccades and fixations files the same way as events ?

Actually, since the data received would depend on Unity frame rate (which is lower than Neon), I would only like to get access to the timestamp of fixations and saccades to compute their relative time (eg starting to 0 at the start instead of a huge number). My final goal is to get the number of saccades and fixations (as well as other data related to this ; eg azimuth angles, etc...) between two stimuli in a certain window time in my Unity scene.

Same issue, eye data is no longer streaming using the NeonXR non-udp implementation. Update to neon companion app broke implementation.

Hey everyone, I have a question regarding the IPD Setting inside NeonCompanion. Does the IPD need to match the wearers IPD or the actual value that is set inside of the VR-headset? Usually both would be identical, but in some cases our study participants have rather small IPD values (i.e. 55) that force us to use larger settings on Meta Quest 3 (min. 58).

Hi @user-e060fb 👋 ! That should be the wearer's IPD, and would only affect pupil size and eye pose estimations. You can see more here https://docs.pupil-labs.com/neon/data-collection/measuring-ied/

It's 2.7.8

Thanks. This is a very old version of the app. It would be highly advised to update. In latest versions of the app, the scene camera is disabled in VR contexts, eliminating the warmth issue that you are experiencing.

Hi, I'm trying to get access to the first Neon timestamp corresponding to the Neon recording start in Unity. Since the http://neon.local:8080/api/recording:start does not return the timestamp, I'm trying to get it from http://neon.local:8080/api/status as soon as possible after recording began but this isn't working out so far. Is there any other methods ?

Thanks! what does this EstimateTimeOffset() method compute exactly? Is it the offset between Unity's clock and Neon's clock? Does it consider the latency due to Unity frame rate or even the WIFI connection?

Okay great, but the timeoffset must be updated sometimes I guess right? Also, is there a way to check if the offset is correct ?

Hi @user-7292c5 , yes, checking the time offset periodically is a good idea and is also recommended in our Time Sync guide. For reference, the Lab Streaming Layer checks the time offset every 2 seconds. Depending on your situation, every 10 or 15 seconds could be sufficient. Since checking the time offset is a somewhat costly operation, you will probably want to do it in a separate thread or Unity Task.

With respect to correctness, offset estimation is an established and very common procedure in computers since decades. It is actually the foundation of the common NTP protocol that syncs the clock in your computer, cellphone, and other devices, and our offset estimator follows essentially the same algorithm. Just make sure to use RTSPServiceWrapper.UnixTimeMs() when computing timestamps for events in your Unity app, as shown in the code that was linked above (https://discord.com/channels/285728493612957698/1187405314908233768/1476211119671480482).