Hi @user-f43a29 , We got the ready set go frame and are planning to use them inside Quest Pro. I was able to get the eye state visualization working in my Unity project (built on OpenXR) using Neon XR Core package but it is not calibrated properly. So I followed the Mount calibration (https://docs.pupil-labs.com/neon/neon-xr/build-your-own-mount/#calibrating-the-mount) . I cloned your fork of the MRTK project, checked out to neon branch, created the Addressables settings, but I cant get any data in any of the scenes. I get the following errors in the image in PL_calibration scene.
Another query, how should I incorporate this Calibration to my application? Please correct if I am wrong: We get the config file from performing calibration using the MRTK sample scene. And use this config file in my Unity application in the addresable group. Do it for each person or whenever the sensor position changes.
Thank you
@user-0a9f16 , I re-read your message and I see that you will wear the Ready set go! frame within the Quest Pro headset. Have you seen our guide on prototyping and building your own mount for other VR headsets?
If you must wear Ready set go! within a VR headset, then you would probably want to run the mount calibration every time you put on the headset or at least every time a different person puts it on (https://discord.com/channels/285728493612957698/1187405314908233768/1290966087353110609). Basically, it is unlikely that the module will always sit in the same pose relative to the headset as different people remove and put on the headset and "glasses frame". Any relative shift of the frame or VR headset while wearing the devices will also alter the mount calibration. However, whether the difference is enough to be relevant for your case can be assesed with a test.
Hi @user-0a9f16 , great to hear about your progress!
With how our fork of the MRTK3 Template Project is designed, you need to either:
Hand Interaction Examples Scene and open the Hand Menu (look at your open right palm) and then change to the PL_Calibration scene. This is the general entry point to the other scenesMRTK NeonXR Variant Prefab to the PL_Calibration scene and then you can start it directly (see this message for more details: https://discord.com/channels/285728493612957698/1187405314908233768/1308897181754851469)Once you run the mount calibration, the result will be saved in the config.json file on the headset's internal memory. You can then copy the values and integrate them into your application. An example of how they are applied in Neon XR is here, where the gaze data from Neon is transformed into VR coordinates, and you can see how our MRTK3 fork accesses those values here
Whether you want to do a mount calibration for each person or just one time whenever the sensor position changes is dependent on your goals. While it is not an eyetracking calibration, it is still somewhat dependent on where the individual looks. This is fine for many use cases, even when switching the headset between people, but you can also consider:
@user-f43a29 , thanks for the response. Yeah that is our initial plan: to calibrate for every individual and session to see if it can work out or else we will be considering the custom mount. I got the calibration scene working .
I got few more questions: 1.We will be using PCVR and so the config file is getting saved in AppData folder in the PC. My silly question is: After performing calibration and saving the config, will the next session automatically use the last generated config for offset values? Even if I run my Unity Application (built using OpenXR) with Neon XR core Package after performing the Calibration from the MRTK scene? 2.We primarily want the pupil diameter value but also require the Gaze if possible. Initially, I was using Lab Recorder to get all the metrics for each eye and the XY gaze coordinates. But now I think those values except the pupil diameter wont be useful in VR context and with individual Calibration. What is the way to track the same metrics in the VR space form Unity? What all values can will I be able to record? Can I get an equivalent of Gaze co-ordinate in the VR space? 3. And I am thinking of creating a LSL outlet stream for the VR metrics as we want to synchronize with other sensors. What do you think of this approach?
Hi @user-0a9f16 , you are welcome!
Here are my responses, split across two messages:
PCVR & config files: Not a silly question. Neon XR uses the config file that is saved in the app's Application.persistentDataPath. You can copy the saved config from the MRTK3 Template Project to the corresponding folder that is used by your project. So, it should work the same on your PCVR setup, although PCVR has not been explicitly tested. You could do a quick check by editing the Neon XR code to print the file name and path of the config file that it eventually uses.
Obtaining VR gaze coordinates: Obtaining Gaze in VR is certainly possible and is one of the main purposes of Neon XR. And, you are correct that the LSL data streamed from the Neon Companion app will be in Neon's scene camera coordinates, not VR coordinates. The transformation to VR happens within Unity, and yes, you can get those coordinates. For instance, they are the same values used in all the MRTK3 Examples.
Post-hoc VR transformation: You can also run a recording in the Neon Companion app in parallel. You could then save the pose of the VR camera in Unity over time. Then, after the experiment has finished, you could interpolate the VR camera pose and, using the saved mount calibration, apply it to the gaze data in the Neon recording, similar to how it is done in the link above and as linked in this message (https://discord.com/channels/285728493612957698/1187405314908233768/1326107256936337448). You would then have gaze in VR coordinates at the full sampling rate.
Unity LSL outlet: This is certainly a viable approach. It would require some coding, but there should not be anything that would hinder making an LSL outlet for the VR data. Note that also with this approach, you will be limited by Unity's sampling rate, if that is relevant for your use case.
Obtaining 3D Eyestate in Unity: This data, including pupil diameter, is also provided in Unity. The eyeball centers and optical axes are even converted to the VR coordinate system. Our fork of the MRTK3 includes a PL_EyeState scene, showing how to visualize this data.
Thank you for the answers. I was able to use the calibrated config files in my PCVR application successfully. I will follow and implement the steps for the VR data. Thank you
Hi @user-f43a29 , thanks for the answer and excuse my very late answer, we focused on different things for the project before the holidays. Happy New Year by the way. 🙂 I've taken a look at the MRTK3 Template Project to adapt how it integrates Gaze Tracking into our own, already existing Unity project, I think I got a bit confused during setup. Maybe I am missing something, I'll try to clone the MRTK3 repo (https://docs.pupil-labs.com/neon/neon-xr/MRTK3-template-project/#other-platforms) again and setup everything like it's detailed in the manual and come back to you with my results.
Regarding Quest 3, I'll try and test it with your suggested procedure, but to answer your questions: We will not be using SteamVR, we'd actually like to have the project run on the headset in the all-in-one configuration, so without a connected PC. If that doesn't work, we'd use Quest Link for the Gaze tracking setup (we will also have one without Eye tracking).
Hi @user-9b48d1 , no worries and Happy New Year to you, too!
Running the APK on the headset directly (i.e., without SteamVR, PCVR, or Quest Link) is actually the most tested configuration, including on Quest 3, so ideally, you should not encounter problems when collecting gaze data in VR that way.
Please note that if you want to run the PL_Calibration scene, the standard way is to start the PL_HandInteractionExamples scene and then open the Hand Menu (look at your open right palm) and navigate to the calibration scene. If you rather want to start the calibration scene directly, then see this message: https://discord.com/channels/285728493612957698/1187405314908233768/1308897181754851469
And of course reach out if you have any other questions or hit any issues.
Hello and good afternoon! Just trying my luck; is there a Neon XR custom mount 3d printable file out there for the Oculus Rift S? If not I can give modeling it myself a go.
Hey @user-aba646! If an Oculus file doesn't already exist, it'd be great if you could eventually share yours in 📸 show-and-tell! I'm sure you're probably already aware, but here's a link to the relevant Neon module and nest geometries that you'll need: https://github.com/pupil-labs/neon-geometry
Thank you. I shall do so!