Hi, I've saved gaze data using the neon-xr core package inside the Meta Quest 3. By any chance is there any way to apply your algorithm on these data to extract blinks, fixations and saccades?
Hi @user-3e88a5 , did you run a recording in the Neon Companion app at the same time?
Not for these recordings. In general yes, but in some tests I only have the raw gaze saved from the VR application using the neon xr package.
Thanks. In that case, you would need to at least convert those data to Neon's Native Recording Data format and then you could run them through the earlier versions of our blink & fixation detectors on Github. However, this is not an optimal solution.
Going forward, if you need such data, you can either:
Hi Rob! Would it be possible to try out the code you mentioned? So in a way, this would allow you to record an eye-tracking data stream synchronized (in post-processing) with, for example, the movement of an object in the virtual scene, right? It could be a way to avoid using LSL? Thanks!
@user-a4aa71 , can you give this script a try? To do time sync, you can make use of the SendEvent command in this PR. For example, you can send Events during specific moments of your experiment, like when a stimulus appears or a trial begins/ends. Depending on your experiment though, you may want to run Event sending in a separate Unity Task or at specific moments, since it can take up to 100 ms, depending on your network.
router
Have you made any changes to the UDP package in NeonXR? Because with the plugin I imported last week I was getting data at 90–100 Hz. Today I imported it on another PC and it’s back to 20 Hz. Thanks
Hi @user-a4aa71 , is the other PC running an app that is still using the previous form of the config.json file? You want to update it to include the useUdp and timeEchoPort lines.
And, yes, I can upload that script soon. It does not require LSL and yes, you would do a post-hoc timesync.
In the Addressable folder they are correct. I also modified them in the project’s persistent path because that’s where it loads them from. However, on this other PC what happens is that it keeps connecting to and disconnecting from the Neon, so it’s not actually sending any data (in fact, the gaze ray isn’t working either).
Are you running this through the Unity editor? Can you check the Microsoft Firewall settings for Unity?
Oops, no , the persistenPath it’s where it saves them… then I don’t know why it’s behaving like this
I was completely unaware of this! In fact, Unity had incoming communication blocked… I usually check that because I know that when I’m not receiving anything, 87% of the time that’s the issue, but it had never happened to me that a device would connect-disconnect-connect and again.. for that reason. Thank you so much! I was going crazy trying to figure out the problem. 😅
You are welcome. Happy to help!
Hi, I'm using NeonXR and I'd like to send POST request in Unity to start/stop capture. Due to security obligations, I can't use internet so I was wondering if starting/stopping capture and sending event is possible through the RTSP client instead of HTTP POST request ?
Hi @user-664cca , it does not require Internet access to send an HTTP POST request. Do you mean that your local network has a firewall blocking that? Are you using a University or work network?
I was thinking to share the connection from my personal phone to Neon device and the PC I'm running Unity. Naive question but : is it possible to share the connection from Neon device to the PC directly (without internet since it's not necessary)?
Hi @user-664cca , Neon is a networked device, so you need some kind of network connection, with or without Internet, such as WiFi, hotspot, or Ethernet connection. So yes, if you use a hotspot from your personal phone, then you can send HTTP requests, as well as stream data over RTSP/UDP.
May I ask for clarification on what you mean by "share the connection directly"?
I meant instead of using my personal phone hotspot as intermediate, using the Neon smartphone hotspot and connect the PC to that hotspot.
Yes, that should work fine. If you have the phone available, a quick test with Neon Monitor will confirm if it works with your setup.
Hi, Neon Monitor does not work for me when using the Neon's phone hotspot... I can't figure why though.
Hi Rob! I tried the code: from Unity I can send the events, but the start recording function doesn’t work. In the Python code I only had to change gaze_datum.time and .point to gaze_datum.ts and gaze_datum.xy, because the variables are called in this way but it runs. About timing, do you mean that up to 100 ms is the time between when I send the event from Unity and when it actually arrives at the Neon?
Are you using an older version of pl-neon-recording? It would be worth updating.
And depending on your network, it could take that long for the Event to reach Neon, but this has no influence on the precision of the resulting time sync, since the Unity code is internally making use of our clock offset method for precise sync. In other words, transmission latency is mitigated. The details are here; it is very similar to NTP, a long tested and widely accepted method to synchronize clocks.
Thanks, I upgraded the library! But why doesn’t the function to start the recording from Unity work for me?
I’m using a dedicated router that isn’t connected to anything else, so theoretically there shouldn’t be any delay (the only delay should be the time Unity actually takes to read the line of code that sends the event and transmit it).
@user-a4aa71 The code that starts a recording is not exactly an Event... Ah, apologies, I see. I had made a typo. We are correcting it.
I'm using Unity 2023.2.20f1 (with the URP rendering pipeline), for me the code entered the function but then got stuck, and I couldn’t understand the problem.. But if you’ve fixed it, then I’ll try it and give you some feedback hahaha. Thanks a lot!
@user-a4aa71 The Neon XR team pushed the changes. Please reference the new code in the udp branch. We will do some extra testing once everyone is back on Monday, but let us know if you encounter any issues in the meantime:
Thanks. I will let you know when the corrected code is on Github.
Hi! I have Neon glasses and a Meta Quest 3, and I can't afford the official Quest 3 mount. I want to:
Questions:
When I unscrew the module from the glasses, does the nest PCB (USB connector) also come out, or does it stay glued in the frame? Do I need a separate nest for the printed mount?
Is there a ready-made Quest 3 mount STL file available anywhere, or do I need to model one from scratch using the files at github.com/pupil-labs/neon-geometry?
Has anyone here already printed a Quest 3 mount and can share their STL?
Thank you!
Hi @user-7c37b1 , I can at least answer the first two points:
thanks! then if I get the quest 3 Frame mount (accessory) , could I move the module from the current glasses to the quest3 mount?
Yes, of course! Neon is explicitly designed to be modular. This is what was meant by "removing the module is normal". You can see how in the video that I linked above.
Hello, Can i wire connect neon xr to pc directly with usb rather than through mobile phone networking with app.
Hi @user-63c8a8 👋 ! NeonXR cannot be connected directly to a PC via USB for normal operation. The system is designed to work through the Companion device (phone), which runs the NeonNet estimation pipeline and manages the recording and streaming.
May I ask if there’s a specific reason you’d like to connect it directly? For example, are you experiencing issues with streaming?
Hi Rob, sorry for the delay. I tried the code and it seems to work perfectly. Thanks so much again! 😊
I will pass the kind words onto the Neon XR team 🙂
Hi Pupil Labs, finally I got hands on a Neon XR for a Quest 3. The Unity integration works out of the box. That is cool. Just a question, under NeonXR>PupilLabs>NeonGazeDataProvider the simulation details, do you have definitions for the values there? Simulated resolution, should that match the Quest 3's resolution? eye position is probably for the visuals x == IPD/2 and y and z the offset to the pupil? or the eye's rotation centre? Reason why I am asking is that my gaze positions are a bit off. Central works fine, but peripherally falls short in any direction.
Thanks for your help.
Do I understand that correctly, in post processing the gaze data might be differently mapped to the visuals in VR as they do in real-time?
And the mount calibration, just need to be done for custom mounts? not for the PL provided Quest3 mount?
Hi @user-a79410 , great to hear that you are using Neon XR!
Here are my responses:
With respect to accuracy in the periphery, could you try running a mount calibration and a Gaze Offset Correction and see if that improves it for you?
Hi Pupil Labs, I am actually try to make the Mount calibration for the PICO 4 ultra following the procedure on the official website. I download and open the MRTK3 template project on Unity but i can't find the scene PL_Calibration that should be use to have the config.json file. Also I wish to know if using scenes from the MRTK3 template ,I still have to install the Neon XR core Package in the unity project? Thanks for your help
Hi @user-4c288b , if you only want to run that scene, you do not need to build it from scratch in Unity. You can download the pre-built version here. If you want to run PL_Calibration as a dedicated scene, then see this part of the Documentation.
To answer your last question, our fork of the MRTK3 Project comes with the Neon XR Core Package already included.
Thank you for your Reply @user-f43a29 , Initially I downloaded the pre-built version on the Pico 4 ULTRA , but the gaze wasn't functioning . I thought it was because the apk is platform specific meaning, that of the Pico 4 couldn't function on the Pico 4 ultra , that's why I trying to run the PL_Calibration to extract the config.json file which is actually the file I need on my project in Unity because I am having an offset that am trying to correct. I followed the part of the Documentation you sent, and inside Unity project browser there is not the PupilLabs' folder inside the Assets.
The config.json file from running the mount calibration scene will be saved onto the harddrive of the VR headset, if that is what you mean. See here for details.
With respect to gaze not functioning, this sounds like a network issue, as the Pico 4 APK also works on the Pico 4 Ultra. What kind of network are you using?
I am using my mobile phone hotspot (5Ghz) to connect the companion phone and the headset , When i launch the apk in the headset the companion app shows ''1 Devices'' on Connected Devices But the gaze is till stuck at the center.
Hi Pupil Labs, I am using Neon to track eye gaze during a task that requires users to actively switch focus between two monitors. I want to know where exactly the user is looking at in each interface. The task is built on Unity and has two user interfaces, each being displayed on one monitor. I tried the Unity NeonXR package, but it seemed to only work for single-monitor tasks with minimal head movement. Am I understanding right? Now I am going back to using April tags and mapping the two user interfaces into the tagged surface. I wonder if there is a better way to do this. Thank you!
Hi @user-c14f83 👋 ! NeonXR is primarily intended for XR/AR use cases. It integrates a version of the realtime API and is mainly designed to map gaze from the scene camera onto an extended reality display.
For more traditional screen-based paradigms like the one you describe, I’d recommend taking a look at this Alpha Lab article. It walks through a gaze-contingent setup and uses the following library.
We’ve also recently introduced the IR plane tracker: https://github.com/pupil-labs/ir_plane_tracker This allows you to use alternative markers instead of AprilTags. However, it currently supports tracking a single screen. If you need to differentiate between multiple screens, you’ll still want to rely on fiducial markers like AprilTags, for example via: https://github.com/pupil-labs/pl-marker-mapper
Most of these libraries are written in Python, but the concepts can be translated and integrated into a Unity pipeline if needed.
Thank you!
Hey Team, I kindly request access to the Meta account associated with the Quest 3 headset that we are using for the experiment. To connect the device to my laptop(MAC OS) and enable ADB communication, I need to activate Developer Mode and authorize USB Debugging on the headset. This requires logging in with the Meta account linked to the device, which I currently do not have access to. Or if there's any other way to build the connection then i am happy to know.
Hi @user-d9920d , we do not sell or manage Quest 3 headsets. You will need to contact the person who setup and configured the account on the headset. If they are not available, then you may need to talk with your teammates about potentially factory resetting the device, but perhaps reach out to Meta's Support team before doing that.
Hi everyone, why are the camera matrix and distortion parameters slightly different when extracted from world.intrinsic in a recording folder compared to those used in the Neon XR plugin in Unity? K = [888.00501163 0 814.313144; 0 887.77337197 591.02475918; 0 0 1]; D = [-1.30726650e-01 1.08965294e-01 -5.99094933e-04 -7.71485819e-05 -7.90911587e-04 1.70053893e-01 5.18868831e-02 2.40532902e-02]; K_xr =[909.405 0 837.6172; 0 909.2214 598.2615;0 0 1] D_xr =[ -0.1305052, 0.1100388, -0.0001126339,-0.000484357, -0.001051797, 0.1672783, 0.05867504, 0.02264213] The first set comes from world.intrinsic, while the second comes from Unity Neon XR.
Hi @user-a4aa71 , the Unity client currently uses an average camera intrinsics when converting to VR coordinates, as this is sufficient in that case. The Neon XR team is currently updating the package in some ways and they will add a feature to use the intrinsics provided by Neon's API.
In the meantime, you can replace the calibration.bin file in the Unity app's persistent Data Path with one from a Native Recording Data folder made with that device. See here for additional details about that path. If you encounter any trouble, just let us know.
I have forwarded this question to the Neon XR team and will keep you updated.
and with the new version of the Neon Player, is it no longer possible to download the world.intrinsics?
Hi @user-a4aa71 , the calibration data are still accessible in the calibration.bin file in the Native Recording Data. If you run the latest version of Neon Player from source, then it outputs the scene camera calibration data to a calibration.json file.