Hi, Pupil Labs! I have some questions regarding app development using the Neon XR Core Package and Unity.

・Is it possible to display the Neon scene video along with an overlaid gaze circle on a Unity scene using the Neon XR Core Package (or MRTK)?

・If it is possible, what kind of customizations are required within the Unity scene? So far, following the instructions on this page, the console shows up as seen in the screenshot, but the gaze is not displayed in the scene. It is counted as a connected device in the "Stream to Pupil Monitor" section of the Neon Companion App.

・Additionally, can the gaze information (and scene video) transmitted from the Neon Companion App be received by Unity apps built on multiple different devices?

Hi @user-7df65a , to be sure I understand: - A person will be wearing Neon and walking around a room or outside and you want to display the scene video in real-time within a Unity scene? - Can you clarify a bit further what you mean by "can gaze information be received by apps built on multiple different devices?" Do you mean if the data from Neon will be different if the Unity app is built on Linux or Windows?

Thanks @user-f43a29 ! Can you point to any specific basic one in particular which just gets the gaze positions from the camera? I can't find exact documentation on it that can help me get started. I'm sorry but I'm having a hard time connecting it to any template once I hit play.

Hi @user-7df65a , thanks for the explanation. I understand now.

So, Neon XR is designed for mapping gaze to VR/AR scenes, with the idea being that one person is wearing a single VR/AR headset.

If you would rather stream data to a Unity app running on a phone(s), then you might be better off using your own implementation of the real-time API, since Neon XR makes assumptions related to VR/AR and does not include functionality for streaming the scene camera (e.g., the scene camera is not relevant in VR).

The real-time API can be used from many languages and you can use code from Neon XR to get a headstart. You should actually need less code overall for what you are looking to do, since you do not need to map gaze to VR/AR.

You might also want to use the Python implementation of the Real-time API as a reference.

Hello Pupil Labs, I have worked with both the normal Neon and the Neon as part of the Pico 4 bundle. With the Pico4 bundle Neon, I can see a VR headset image in the Neon Companion App. Are there differences between the Pico4 neon / bare metal and the normal Neon / bare metal? Are they fully interchangeable?

thanks!

I'm trying to interact with the MRTK demo scenes with the pico 4 controller instead of finger pinch gesture (since I also haven't figured out yet how to make hand tracking working with pico 4 in editor), do you have any tips for a MRTK and new input system novice? There are so many places to adjust input/actions/controls so I'm now doing trial and error but perhaps you can give me a quick tip on how to interact with any of the interactibles through a button press on the pico 4 controller?

for example, I tried to add a joystick clicked action to the Activate Action Value in the MRTK RightHand Controller , but this didnt work

When I play the scene with pico 4 controllers, I can see the controllers act as virtual/simulated hands and I somewhat can interact with the buttons, but for example with the piano buttons, I can half press them, but they are never fully pressed/activated (e.g. they don't make a sound)

I also tried adding primary and secondary button mappings to Select Action, but this doesn't work either

@Rob Thanks for the great support in July. https://discord.com/channels/285728493612957698/1187405314908233768/1261254975586177064

We successfully got Neon to send data to Unity in real time using the USB-Hub. The OnGazeDataReady event is updating variables in the Unity script, but we are having troubles with two things.

1. The frequency at which the OnGazeDataReady event is called is about 25Hz, which is too small. The refresh rate on the CompanionDevice is set to 200Hz and the data on PupilCloud updates the values at 200Hz. UnityEditor has a frame rate of about 200Hz.

2. measuring latency is too large Compare 3d_eye_state.csv stored in PupilCloud with the CSV created on Unity, latency was determined based on the pupil diameter change graph. Then we found that the latency was about 500 ms. The attached image shows that “the graph shape matched after moving the pupil diameter change graph for about 500 ms.

Code to get the timestamp on the Unity side :

DateTime utcNow = DateTime.UtcNow;
long unixTimestampMillis = ((DateTimeOffset)utcNow).ToUnixTimeMilliseconds();

Both the ping value between the PC and the router and the ping value between the CompanionDevice and the router are less than 1 ms. Considering the possibility that the clocks of each device are off, we checked the following sites to see if the clocks are off. The computer running Unity Editor was 0.1 second slower and the companion device was 0.8 seconds faster. (apparently not an exact value). What methods do you use to verify latency? How do you deal with the clock deviations?

Thank you in advance !

Hi @user-1ec436 , that's great to hear! May I ask for a clarification first and then we can proceed to the OnGazeDataReady code?

• What USB hub is that? I cannot make any guarantees about devices other than the tested USB hub.

Regarding the transmission latency and clock offsets, you can reference the TimeOffsetEstimator code of the Real-time API. It is a reliable way to measure the transmission latency, provided that the network is stable (that could depend on the quality of the USB hub).

You can also try our other sync tips.

Regarding the pupil diameter data, is there a reason that you prefer the data collected via Unity as opposed to the data recorded on the phone (i.e., the data that gets uploaded to Pupil Cloud)?

Hi is it possible to use neon with htc vive? If yes, how would I go about setting it up?

Hi @user-3f7dc7! Conceptually speaking, yes. Neon can be mounted into various HMDs, as long as there is space to accommodate the module. Although we don't have a pre-built mount for the HTC Vive, and so a mount would need to be prototyped to fit the geometries and constraints of the headset.

I also recommend heading over to this section of the docs to read about how to get started with integrating Neon into your XR environment. Let us know if you have any specific questions!

Thanks! I finally managed to do an eye calibration using the PL_Calibration scene.

After importing the Pico Live Preview Package and enabling it in the Windows XR Plug-in Management and enabling the Far Ray gameobjects, I had working controllers with aim and select functionality.

Then, I added the MRTK NeonXR Variant prefab to the PL_Calibration scene. I also changed the ip adres in the config.json file (in appData folder).

For the calibration to work, I had to gaze towards the targets and simultaneously aim and select at the targets with the controller. This was a bit cumbersome but it worked in the end.

I do feel that using the MRTK framework costed me a lot of extra time troubleshooting and seemingly its hard to add controller functionality, is there any benefit to you or me compared to using just the XRI toolkit?

Also, since I'm on a institute WIFI, I have some extra difficulties to connect devices which is why I preferred to do the calibration through PC VR / Unity Editor since I have previously figured out connections issues between Neon Companion App and my office pc with the network admins (devices needed to be whitelisted and proxy cache servers needed to be disabled).

Next step is to implement the Neon Core Package in our VR Experiment which should be easier as it doesn't rely on MRTK. Thanks for the support .

Thanks for the clarification! I personally found it hard to adjust the calibration scene to my needs as I couldn't figure out a way to replace the finger pinch gesture by a controller button press (or even keyboard button press). I feel I would need to study MRTK and the calibration scene more to figure this out.

I hope that we are able to buy the Anker USB ethernet hub soon, so we can circumvent WIFI issues altogether.

@user-1ec436 , great to hear!

As you've found, when you compare the data streams from Unity and the Neon Companion App, it is important to account for latency and clock offsets.

With respect to the 25 Hz sample rate in Unity, can you share the log file that is generated? You can share it via DM or send it via email to [email removed] if you prefer

Thank you for your support ! I will contact you againt after investigating a problem which we found with the output CSV data.

Hi @user-f43a29 , We want to use the neon module along with the quest 3 to measure the pupil diameter. We will be triggering some events within the VR scene and would like to track the pupil diameter values from that moment along with the time stamps. Will it be possible for the timestamps of the events in the Unity scene and the pupil data timestamps to be recorded with the same clock or synchronize?

@user-f43a29 , We are looking for maximum of 20 ms error.

I have tried going through the documentation. As we will be running the experiment on Unity, One approach would be using Unity neon XR package. We can extract the realtime Gaze data using this but not the Pupil Data .Because I couldnt fine any reference for that in the unity package. Please correct me If I'm wrong. But if I can get the pupil data realtime from unity, I have thought of mapping the required unity events timestamps with the Neon Module sensor data directly in the host computer.

Another approach would be just collecting the sensor data independently from the unity events' timestamps. We would be following this : https://docs.pupil-labs.com/neon/data-collection/time-synchronization/ for syncing our host systems clock with UTC before the experiment and collect the unity event timestamps in UTC itself. And then mapping them later. (Could you explain how the TimeOffsetEstimator works and how to apply in my case)

Which approach would be better or result in more accuracy?

If I decide to write my own client for the Real Time API in c#; are there methods to get the Pupil data?

Hi @user-0a9f16 , thanks for the clarification.

• Neon XR provides 3D Eye State & Pupillometry data in real-time. Are you using the latest version of Neon XR and the latest Neon Companion App (v2.8.25-prod)?
• I assume it was a typo, but if you are following the Time Sync guide, then it helps to sync the clocks to the same NTP server. UTC, rather, is a format for specifying a timepoint/date.
• Details of the TimeOffsetEstimator are contained here.

You can send events with custom timestamps (see part 1 of my colleague, @user-cdcab0 's, post here (https://discord.com/channels/285728493612957698/1047111711230009405/1288404861649424404) for additional details). As long as you account for the clock offsets, then you can theoretically reduce error to the point that it is avoided (i.e., ~0ms error).

If you want to write your own implementation of the Real-Time API in C# for Unity, then Neon XR would be a good resource to start from, but aside from events, I think it already provides what you need. It seems to me that it might be easier to add event sending via HTTP Post requests to your own fork of Neon XR, rather than re-writing parts of it from scratch.

You can also make a 💡 features-requests or inquire about a custom consultancy, if you'd like us to add this feature.

Hi @user-0a9f16 , yes, that's a valid workflow. Just note that the pupil data and events are contained in two separate files:

• 3d_eye_states.csv
• events.csv

Based on the events, you can filter the pupil data stream or align it to some other data stream.

You can of course check and update the time offset periodically (as often as you need), in case rigorously controlling for any clock drift is essential to your experiment.

Hi, any idea if the 3D model for the Quest 3 mount will be published ?

Hi @user-36c24b 👋! I’ve moved your message to the appropriate channel, 🤿 neon-xr .

I guess you’re referring to the mount available here ? Please note that the model isn’t just a 3D-printed piece but a custom facial interface, with cushioning.

You can purchase it directly here

Oh I thought it was just a small printed mount. Thanks for the clarification

Hey everyone, sorry for the long message:

I just received my neon and meta quest 3 mount/interface, but I am not sure how to install the neon into the interface. It appears as if I need some sort of screwdriver which did not come with the device, despite the website citing tool-free installation. Please let me know how to install the device, and if I am just not understanding how the mount is set up.

I know there is supposed to be an onboarding meeting but unfortunately I was not the one to make the purchase, I am working under a research scientist who handled the purchasing but also is unavailable for the next month. So I don't have any information about the purchase or any correspondences with Pupil Labs, just the device.

Please let me know how to install the device into the mount. Hopefully this is not a stupid question and I haven't missed the intuitive way to attach it 😭 .