Hello, I'm working with the pupil neon and the meta quest3, I bought the nest before that the adapter for the meta quest 3 was released. Now I'm trying to run the examples in the mrtk template project but they're not working, are there some particular setting that do I have to set in Unity? In particular I'm trying to run the PL_Calibration example
Hi @user-3e88a5 , may I first confirm the following:
neon-pico branch of the MRTK repository?May I also ask for more clarification about what exactly is not working? Do you receive an error?
Hi @user-3e88a5 and @user-386d26! Apologies for the delayed response. Our Product Specialists with more experience of Neon-XR will be able to assist you with your setup next week. Thanks for your patience!
thank you!
Hi @user-386d26 , no need to apologize!
gazeDataReady event. Our fork of the MRTK3 Template Project contains examples of monitoring this eventPlease note that streaming with Neon XR is meant for interactive use. If you need more thorough data analysis, may I ask what you plan to do exactly?
Thanks for the detailed response! I believe that what I'm looking for now is just storing real-time gaze data locally. I set up the MRTK3 sample project on my machine, but where can I find examples of monitoring the gazeDataReady event? I'm trying to search through the scenes to find any that use the NeonXR object like directed in the NeonXR Package documentation, but I can't seem to make sense of this. Please let me know where I can find examples of this to mimic. Thanks again!
I'm using the neon branch of the repository. I tried to start from scratch using just the neon-xr core package and in that case I manage to see the ray from the eye in my application but it's not calibrated. So I tried to use the MRTK sample for the calibration, there are some settings that I fixed in unity in order to make the application build and run on the meta quest 3. Then I can run the application but I something it's not working since if I pinch looking at some markers nothing happens. I'm using my personal network connection
Hi @user-3e88a5 , a new version of our fork of the MRTK3 Toolkit was just pushed on the neon branch.
Could you update to that and let us know how it works out?
Also, you might need to hold your hand a bit further in front of your face when doing the pinch gestures.
I see. Are you running this with the Quest Link or with SteamVR running at the same time?
In some minutes I'll send you soe screen of the projects settings in unity
I disabled googleAR core otherwise I cannot use Vulkan as graphic, and that's required by meta quest support
that's the message if enable google ar core
No, I'm not using quest link, I'm deploying the app as a standalone app on the meta quest 3
Hi @user-3e88a5 , thanks for all of this info. Let me confer with the team and get back to you.
Also in the calibration example I cannot see nothing that recall the Neon Gaze data provider, is that normal?
ok, thank you a lot for your time!
now I receive this error
Hi @user-3e88a5 , this could indicate that your system ran out of RAM or harddrive space during the build. Could you check that and then try again from a clean, fresh state?
that's the message in the console
ok, tomorrow I'll try, but this happens just if I build the project with unity 2021, if I migrate into unity 2022 it doesn't happen, but I have the same problem I told you before
Ok. You had mentioned that you had changed other settings in Unity. Could you list what they were?
well, I'm referring to the project settings. And it was regarding GoogleAR Core; so the thing is if I disable googleARcore and I enabke vulkan graphics (required from Optimize Buffer discard) the application start but than if I pinch only the center marker explode and then I can't continue. If I took the googleARcore with the OpenGLSE3 graphics I still have some warning in the project settings, the program build and deploy to the meta quest3 but then it doesn't run
@user-3e88a5 Ok, could you then open a ticket in π troubleshooting about this?
that's the warnings, so I know how to solve the one referred to the speech interactor but I don't think is needed, than one is referred to the deprecation of hand tracking for android used by openxr and the other one is about the Vulkan graphics thing.
Update on this, I am looking through the PL_EyeTrackingBasicSetupExample scene, but I don't see any references to the NeonXR object mentioned in the docs, nor do I see any gazeDataReady event, but I think I'm just not looking in the right places.
Hi @user-386d26 , apologies for being unclear. Let me expand on that:
GazeDataProvider class, in particular the NeonGazeDataProvider sub-class, which is defined in the Neon XR Core Package.GazeDataProvider class check the gazeDataReady event for you. This automatically runs in the background asynchronously.Update method of a NeonGazeDataProvider object is called, then the latest gaze datum (as provided by gazeDataReady) is retrieved and stored within the object. Unity calls this method automatically every frame refreshGazeOrigin and GazyRay fields to work with the gaze data in a format that is compatible with your VR scene. An example is provided in the MRTKInputNeon class implementation.Note that is important to do the mount calibration for your Quest headset first, if you want your gaze data to be in sensible units for your VR experiment. Also, use of only the Neon XR Core Package by itself is meant more for users who plan to integrate with other SDKs or systems.
Hi Pupil Lab, I got a time urgent question if you can help me. I am planning to use NEON-XR with any of the HMD that fits. I have 3 qeustons on this: 1. How the data stream works? The eyetracking data will stream from Companion app or from the VR side? (I am using iMotions so wondering if the data streaming will just work as usual as per using NEON glasses itself with iMotions) 2. how accuracte the eyetracking data is, given that there is an additional screen from HMD itself? 3. Can your NEON-XR work with Varjo XR series? Like for XR4? Appreciate a quicker response, thanks!!
Hi @user-006317 , we also received your email and replied there. Thanks!
Hi @user-cad8c8 , yes, it is in principle possible. You will need to receive the real-time streamed data (see here for full details) in your code. The Neon XR code contains an implementation for getting the gaze data.
Essentially, you connect to the RTSP server over WebSockets @ ws://neon-ip-address:8686 (you can also use neon.local), setup the session, and decode the received data. WebSockets will probably be necessary here, when working in the browser.
You could use that as a reference to start. If you need increased accuracy in temporal synchronization, then you may want to add code for collecting & responding to RTCP packets.
Thanks for this! I am aware of it in being available in Unity, able to be exported to webxr. But I would like to have access to the pupil labs gaze data in (a custom?) XR compatible browser for Quest3 so that I can run any webxr website in this browser and make use of the pupil labs eye tracking on browsers.
Will this be possible? Some suggestions for development in this area could be really helpful!
Great! Thanks for the reference! I'll look into it!
@user-cad8c8 No problem, but please note that this does not account for mount calibration. This will provide you with Neon's raw gaze signal in scene camera coordinates.
You could run the mount calibration in Neon XR and then re-use the provided values in your code.
@user-f43a29 Do you have suggestions regarding translating/mapping scene-camera coordinates to immersive-ar boundary coordinates?
If you know the coordinate system of your AR simulation and you can easily cast rays in that environment, then you could follow a similar calibration process as used in Neon XR. The end result is a transformation from Neon scene camera coordinates to the coordinate system of your simulator.
If you would rather not extract and modify that code, it might be easier to determine the relationship between the coordinate system of your AR environment and the coordinate system of Unity for the PL_Calibration scene. It'll most likely be a composite scale, translation, and rotation. Then, after you've run our mount calibration, you get the gaze data, convert it temporarily to Unity's system, and then apply the transformation from Unity to your AR system.
It's just an idea and has not been thoroughly tested, so you'll need to test it to see which approach is easiest. And please note that the process might be a bit more complicated with AR.
Thanks for this!
Hello there! I would like to include Eyetracking into an existing Unity Project for the Quest 3. I find the MRTKDevTemplate a bit confusing though, and I have followed the instructions that have been posted here (https://discord.com/channels/285728493612957698/1187405314908233768/1273433381622911037) already, except for the last part about the listener, which I still need to implement, but I do not quite understand how, do I need to implement an event listener with the XR Rig (the player), which then holds the OnGazeReady function?
Hi @user-9b48d1 , if you are using the MRTK3 Template Project, then you do not necessarily need to write code to check gazeDataReady yourself. The GazeDataProvider class and the sub-class, NeonGazeDataProvider, from the underlying Neon XR Core package automatically check for this event in a background process. If using MRTK3, then it is already using those classes -> take a look at the MRTKInputNeon class for an implementation and see this message for more details: https://discord.com/channels/285728493612957698/1187405314908233768/1304037280390053928
Regarding Quest 3, we would recommend first getting your experiment working in Unity and test with the MRTK3 Input Simulator in the Editor, without the VR device interface. Then, every now and then, build it and load it to the headset for a more authentic test. May I ask, do you plan to also use Quest Link and SteamVR?
Regarding the Neon XR Core Package, using that directly is meant more for users who want to integrate with other SDKs or systems. In its default state, it will provide you with gaze data in Neon Scene Camera coordinates, but typically you would want gaze data in the coordinate system of your VR simulation. The PL_Calibration scene of the MRTK3 Template Project shows how to get a mount calibration file for this. You will want to do this at least once for your Quest 3.
The MRTK3 Template already bundles the Neon XR Core Package, so you can remove it for said project if you installed it extra. After following the instructions for Other Platforms, the scenes are found under Assets/PupilLabs/Scenes as shown in the attached screenshot. When you double click on PL_Calibration, the scene should load, as shown in the second screenshot.
Hi, I'm try to calibrate the neon eye tracker with Gooivs headset, this headset is just a external monitor, I download the neon-branch and try to modified the project so I can run the calibration. Before change anything in the project, there are two errors in the project. It seems the Gaze Data Provider and Storage are None in the script. How I suppose to fix this and how should I connect the Neon to unity?
Hi @user-c9caf9 , may I first confirm that you followed the steps here? Thanks!
Yes, since I'm try to do calibration with other platforms so I followed the 'other platforms' setup steps and do all six steps. The second picture show the current addressables groups and settings, Could you take a look and tell me is there anything wrong? Thank you!
@user-c9caf9 Let me take a look here and check with the team and update you.
Hi, I also noticed that I could connect the device to computer via ethernet? https://docs.pupil-labs.com/neon/hardware/using-a-usb-hub/ I bought the exact same Anker hub and connect it to computer, but my computer doesn't identify it. Is there any DHCP or internet setting need to be down with the phone or computerοΌ I'm using win11 and the phone is OnePlus 10 come with the neon tracker.
Hi @user-c9caf9 , yes, this is possible. If you are using Windows 11, then check these instructions if you want to directly connect to the Ethernet port of your computer: https://discord.com/channels/285728493612957698/1047111711230009405/1272483345137139732
It will be easier though to connect your Neon and your computer via Ethernet to the same router. Then, you have to do less configuration and you will still have essentially the same low transmission latencies.
Hello! I just got the neon package with the quest 3 attachment, is there any guide online on how to put the tracker itself in the glasses? I can't seem to find the leaflet guide
Hi @user-11c6b0 , are you referring to how to mount the Neon module in the Quest 3 mount or how to place the mount itself in the Quest 3 headest?
Thanks, we have figure out the ethernet connection. But the Unity problem still exists.
You can also try going to Window->Asset Managment->Addressables->Groups
and clicking Update a Previous Build. If you receive an error message, then it is advised to delete that folder and try from step 1 again.
Hi @user-c9caf9 , you may want to delete the AddressableAssetsData folder and try those steps again. You can find that folder as shown in this screenshot.
I tried, got build success, but when I run the pl_Calibration scene, it sill has the same issue
Are you trying to run the scene via a cable link to the headset? Is Goovis an Android-based headset that can run untethered?
Are you testing this standalone or can I just confirm that your is Neon powered up and connected to the same network as that computer?
Do I need to change the ip address in config files so unity can find the neon device?
By default, no, as the autoIP option should be enabled. However, if your network is not supporting automatic device discovery for some reason, then yes, you can try entering the direct IP address, as shown in the Stream section of the app.
The device is connected to OnePlus 10 phone, the neon compasion is running
No, that headset is just an external monitor
I think the neon can be found by my computer, I just run the simple python code use discover_one_device() and it can return gaze information. For now I did'nt change the ip address in the config file
Hi @user-c9caf9 , I have an update from the team:
PL_Calibration scene by itself, then you should manually add an instance of the MRTK NeonXR Variant prefab to the scene.PL_HandInteractionExamples scene, which already has an instance of this prefab. This particular instance is by default not destroyed when you load new scenes.PL_Calibration scene.Please also see the attached images.
Ok, I am checking with the team and will update you when I know more.
@user-386d26 and @user-9b48d1 I believe the above info will also be useful to both of you.
@user-386d26, you may also want to check this message: https://discord.com/channels/285728493612957698/1187405314908233768/1308815595030052985
Thank you so much!!! It works
Hello hello! We have recently bought two mounts for the Meta Quest and with the eye tracking sensor attached, we've noticed that the sensor itself causes a noticeable amount of discomfort. Do you have any experience with ways to alleviate this?
Hi @user-1391e7! Can you elaborate on what you mean by discomfort?
Hi guys, is there any way to automate the process of loading a recording in pupil player and generating the fixation/saccade data for export? We are a bit limited in that we can't be uploading our data to the cloud (silly IRB rules), but Id be interested to know if we can do it with or without pupil cloud.
the sensor presses into the bridge of the nose and forehead, part of the weight of the HMD also comes to bear on the nose now
Thanks for sharing. Have you tried extending the face shield maximally? It should also be possible to get a thicker pad - that should alleviate the issue.
Hi @user-d086cf , if I remember correctly, you have a Neon, yes?
If so, you can check out:
Would those work?
This is exactly what I was looking for! Thanks!
Hi, I just tried to do the calibration in pl_Calibration scene, does those red dots means the calibration is very bad? The second picture show's my current setup
Hi, I couldn't find information on the weight of neon-XR frames. How heavy would for example the frame for the Quest 3 be? Is the frame just attached to the VR headset or is it swapped with some other part of the headset?
Hi @user-6d7fe6 , the weights of the Neon XR Frame mounts that we provide are:
With respect to the Meta Quest 3 Frame mount, you would remove the default face shield and replace it with the one that we include in the package.
Hi @user-c9caf9 , that calibration could be significantly improved and probably should not be used.
I've attached an example of a calibration result that is desirable. You can see that the green and red dots come close to laying on top of each other.
Considering your setup, you had mentioned that you use the Goovis headset as if it were an extended/external monitor? This could potentially lead to problems, as it breaks some of the assumptions that are made in VR/AR contexts.
The relevant team asks if you also have a reduced Field of View (FoV) with your setup. Were you able to easily see and look at all the calibration targets in the periphery or were they located at the edges, potentially even cut off at the edge of the screen?
I just wondering why the algorithm can't calculate the pos?
I was want to say that my calibration result is very bad, not the algorithm. I can see all the targets in the goovis, and I was able to see the edge of the screen.
Hi @user-c9caf9 , the algorithm is actively estimating pos(ition), but the algorithm works under the assumption that you are using your headset as a 3D stereoscopic display; in other words, as a full VR device.
Using the device as if it were a flat extended monitor breaks some of these assumptions, which relate to the positioning of the Neon Scene camera in relation to the VR camera.
I could imagine that this plays a role.
Is it possible to do a brief test where you use the headset as a VR display, rather than as a flat display?
I think in default the headset will run as a flat display, left and right view will look same. I make a little change in the game scene, put two camera and generate a side by side view. Red dot is the visualized gaze point for testing.
