@user-006924 I can see that you are running the hololens demo. This will diplay stimuli in the holo display and calibrate in this space. The calibration will not be correct for the world camera view. If you want to calibrate against this view try the sinble marker or screen marker calibration

For calibration, why do we need to send the size? The returned gaze is not from 0 to 1?

What does the size mean?

@mpk

@papr

Hi All,

I'm having trouble with HoloLens 3D calibration. I am using Unity 2017.3.0f3 and when I run 2D Calibration Demo with holographic emulation it works fine. However when I do the 3D calibration demo, calibration proceeds normally but when the Sharkman appears it throws an error and Unity execution pauses. The error is in the next post.

I did some digging and it looks like the error is caused by UDPCommunication.cs. Specifically the InterpreteUDPData and FloatArrayFromPacket functions. During 2D calibration the length of the byte array received from UDP is consistently 12 (4 bytes for an informational header, and 8 bytes for the 2D gaze position – split into 2 four byte floats). However, during 3D calibration the length of the byte array is most often 17 and sometimes 16.

16 bytes would make sense (4 byte header, and 12 bytes for the 3D gaze position – split into 3 four byte floats). When the byte array’s length is 17 it throws this error because parsing and converting the byte array fails in FloatArrayFromPacket. For now I’ve coded a workaround that gets the project to compile. It tests whether the byte length is 16 or 17 and changes the offset parameter of FloatArrayFromPacket to 5 if length=17 (treating the 5th byte as part of the header). Obviously this workaround isn’t ideal. Is this a bug with the HoloLens relay plugin? I’m not sure why the byte array should be switching between length = 16 and 17. Thanks for any help in fixing this issue.

@mpk Is the angular accuracy that I get through pupil capture is also for the world camera view or is it for the holo display?

Hello @user-e04f56. Like @user-aa0833, I do not receive any blink mesages in the blink demo (using 1.7 of the pupil service). Normal pupil messages work fine.

@user-006924 that depends on the calibration and test setup.

@mpk by calibration you mean 2D or 3D? I don't think I quite understand what you mean by test setup..

@user-006924 are you calibrating using a marker or using the hmd eyes routine?

I'm using the hmd eyes routine

Hey @papr Any idea how to get at the HMD distortion mesh stored in the firmware?

...when using an HTC VIve. I've seen some helpful tools on OSVR...

@user-8779ef nope. :/

@user-8779ef it might be worth stepping through Oliver Kreylos' VRUI. His LensCorrector appears to load a distortion mesh. Perhaps you can write the texture out to a file???? https://github.com/Doc-Ok/Vrui/blob/b9333e7b8b44a535f25a6e9b0c82f72b7ed7ebfa/Vrui/Internal/LensCorrector.cpp

Hello, for some reason after calibrating (2D) in my Hololens and opening my scene, it's not as near as precise as with the Pupil Capture GUI on PC. Also, the gaze constanstly twitches, if I don't move my eyes it will appear a bit to the left then a bit to the right for example. Any ideas ? I should note that I get the gaze data from ZeroMQ rather than UDP, would you think it's a better solution ?

@user-a6a5f2 Thanks very much for the suggestion!

@user-8779ef sure. We are leaning towards buying the vive add-on. Do you know the fov of the eye cameras? We want to get a sense of the range needed for eye relief/z distance to handle different sized eyes (looks like roughly 30mm to 40mm is the range of width for adult eyes)

The depth of field on the 120 Hz cams is quite poor, and I find that the optimal eye location for a good pupil image is often outside of the eye box required by the Vive's optics.

The 200 hz cams are a huge improvement in that regard (have a much larger depth of field).

...however, PLabs has not yet released the 200 Hz integratin.

Do you know what is max distance on Vive eye relief adjustment?

Yes, we may wait for the 200 version

I don't, sorry. Know that the max distance is going to result in some very poor images on the vive. Unusable.

Are you familiar with the concept of the "eye box?"

yes, Oliver Kreylos has nice analysis showing peak viewing for Vive is at 8mm z distance

Yes, he does, doesn't he. He's a great reference.

at 25mm, there is 25% loss of fov of vive imagery

...but, more importantly, eye will no longer be near to the focal point of the optics, and the image will be very blurry.

That's a larger issue than the loss of FOV.

makes sense

we are thinking of using the vive add-on in a different device

not vive

not even an hmd

Ah, Ok. ... keep in mind that it is specifically made to snap into the vive. The shape is very particular.

yes, looks like shape will work, just trying to engineer the eye relief, to confirm that it's compatible with our current hw design

have emailed pupil with fov question

Ok! Good luck.

they seem fairly responsive

they are.

thanks

👍🏽

Hi @user-a6a5f2 we got your email. Sorry a bit of a backlog at the moment - but will try to respond tomorrow

Thanks, @wrp. That's fine.

Hi everyone, Does anyone know whether the accuracy results I get in pupil capture are for the calibration I do on the holo display or not?

I'm not sure if I'm calculating visual angle correctly but using the definition of visual angle and angular accuracy in Pupil Capture, I'm calculate the distance between where the gaze point should be (location of the target) and where the gaze point is(dist1), and the distance between the plane where the target is and the main camera(dist2) and I plug it in this formula --> visual angle = 2arctan(dist1/2dist2)

I'm using the raycasting code of the 2D market scene in the HoleLens

The angular accuracy that I get is very different than the one shown in the accuracy visualizer in the Pupil Capture plugin.

Can anyone help me out with this issue?

so i tried a usb3.0 extension cable so that i could actually reach the computer, and it wouldnt work. take it out, use the supplied cable only. now it works. so i bought this thing and the supplied cable is not actually long enough to reach the PC. what gives? using a htc vive, so the vive HMD connection is usb-c but the provided cable has the standard usb port end so it must go back to the pc. has anyone got it working with a specific brand of usb extension cable?

vive pro*

@user-24270f you can get a USB c-c cable in plug the eye tracker directly into the vide pro. The alternative of longer cables usually works. But not all cables do :-(

cheers

@user-24270f one of the Pupil Labs members suggested the cables in the link below. I tried them and they do work but they are not that long either. You can check whether the brand makes longer cables. https://www.amazon.com/CHOETECH-Charging-Syncing-Nintendo-ChromeBook/dp/B0176BSAWU

May I ask that for 3D gaze position, we get a 3D gaze position from left eye, we also get a 3D gaze position for right eye. How do you get only one 3D gaze position in 3D VR Demo?@papr@mpk

@user-b91aa6 We use the average

Another problem is that I find the 3D gaze position for left eye is stable but the 3D gaze position for right eye jumps a lot. I checked both the calibraion process, the 3D eye tracking model performs well, what't the reason for this?@papr

Hi everyone,

I'm trying to add a pupil manager prefab to my scene in unity. The developer doc says:

"If you use the prefab for your own scene, please specify the scene/scenes you want to load in the Inspector GUI for Pupil Manager

Add the name to the Available Scenes list"

But I couldn't find the Available Scenes list. Any idea where it can be found?

Screenshot of the inspector of Pupil Manager prefab

Thanks in advance!

Which version of HMD-eyes do you have ?

Hi everyone, I tried to pupil capture's calibration, and calibration screen was displayed. but, it sound of alert with executed and it is not proceed. My hardware don't have world camera, is it related to that?

@user-ed61d7 The way I got this to work was to drag all the game objects in the scene (including main camera) into the the "game objects to enable" list

after calibration the game objects in the list will be set active and the scene will render as normal with gaze tracking active

the 3D pupil detector is supposed to enable slippage compensation (through model refitting). I've been using it with the HoloLens which is prone to slippage. However, it doesn't seem to work well. Even with mild slippage (ex. jaw clenching, yawning) the calibration goes out of wack and doesn't recover in HoloLens apps. Anyone have this issue and have insight?

@user-babd94 Pupil systems relies on the world camera in order to calibrate;

has anyone here looked at removing the htc vive lenses and replacing them with superior gear VR lenses, to get rid of that fresnel lens edge blurring? https://www.thingiverse.com/thing:2837804

3d printed adapter makes it slot in really easily apparently. it looks like pupil labs eye trackers will still fit, but i cant be sure

@user-cd9cff Thanks. I bought Moverio Add-on, but it did not contain world camera. I will wait for world camera arrives.

@user-58d5ae I downloaded Pupil.Import.Package.HoloLens.unitypackage v0.5.1

@user-d3a1b6 I tried it, but it didn't work for me 😭 I guess it may be because that I'm using an ARCamera instead of a normal main camera?

And just to specify that I'm only using the eye cameras to monitor pupil diameters, not really for gaze tracking at the moment.

Or is it because I'm building a VS solution and deploying to Hololens, but HMD-eyes only works by remote emulation in unity?

I'm working with Vuforia&HoloLens

I had the same problem as you @user-ed61d7, the link in the readme links to an old version. Download the master directly and it will be there.

So, recently started using the Pupil Cameras with the HTC VIVE, everything appears to work expect that it never recognizes any scene .exe's other than calibration. Is there a way to see what the eyes are looking at inside of an executable? I have the capture selection setting set to Local USB. Any tips?

@user-7bf7f5 I assume you have tried the demoes from here: https://github.com/pupil-labs/hmd-eyes/blob/master/README.md ?

Yes, that was the calibration scene.

I just cannot get the pupil capture to display what the HTC VIVE is seeing is the issue. The only thing that did work was the calibration scene.

@user-7bf7f5 ah ok. Pupil Capture does not display the vive stream. Instead we report the gaze data in the unity3d plugin.

Ah! Thank you.

I'm sorry, would you mind clarifying is this Unity3d Plugin for Unity or for Pupil Capture and also where to obtain it or is it pre-installed

@user-7bf7f5 this plugin is for unity3d

Thanks

@user-ed61d7 Have you tried with doing the calibration scene in the shark demo folder instead of the plugin and then adding you own scene in the pupil manager instead of the shark 2D scene?

@user-5ca684 hi Kornel , can you pls assist with calibration using htc vive pro

Hi @user-58d5ae , I did what you said and the option is there now. Thanks! And did you manage to add the prefab to your scene to make it load first? Is it okay just to add the name of the scene in the list and set the current scene index to 0?

@user-58d5ae @user-006924 It works fine now. I added my scene to the pupilmanager prefab in the calibration scene provided in the master brach. Thank you both very much!

Hi guys! Is there any neat way to integrate pupil-labs with UE4 using Vive? I saw there is instruction for unity. Do you have something similar for UE4?

@user-2feb34 there are some members of the community that are/were working on UnrealEngine support. However, I do not know how far along they are and/or if they are still working on this.

Ah, ok. So there is no official supprot for UE?

@user-2feb34 that is correct - hmd-eyes only targets Unity3d at this time

it is the only official framework supported for VR/AR development with Pupil

Thanks

welcome @user-2feb34

Hey everyone, Does anybody know if this "PupilData._2D.RightEyePosition" is the correct way to access position of the right pupil in pixels in the eye camera feed? I'm getting (0,0) for some reason.

@user-006924 I think this is right (?) did you make sure that Pupil reports decent confidence?

@mpk I was not recording at that point and I thought it's similar to gaze data that I have access to without recording in pupil Labs. Should I record in order to gain access?

I am trying to use the unity plugin with another device on the network doing the eyetracking. I am using remote settings but its not picking up the other device, any advice ?

Does pupil unity plugin currently only work with version 1.4.1? or is that the plugin version? in PupilSettings on Unity the pupil version is labeled as 1.4.1 but I am using 1.7.71, I have tried to change it but nothing happens

Hi all--we are using Pupil Labs cameras inside of a Vive Pro. I'm trying to find the optimal settings for Pupil Labs--the settings for things that you can't necessarily change within the user interface but can change using msgpack. I'm curious as to if anyone else has spent time figuring out what the best settings are within the VR environment.

For reference, we are using msgpack to read the settings found at /pupil_capture_settings/user_settings_eye0

@user-988d86 Its worth nothing that everything is also accessible through the source code, which is open source. If you want some insight into how specific variables affect things ,you can actually look at the code/algorithms that they affect.

Thanks @user-8779ef we are also going through everything ourselves, but I thought I'd check in to see if anyone else had already figured out some of this.

In addition to my above question, we have noticed a regular problem where Pupil will read eye values from the cameras even if nobody is wearing the headset/no eyes are present. Has anyone else noticed this problem, and does anyone know of a solution?

@user-988d86 Are the confidence values any good? I can imagine that the eye tracker accidentally finds a pupil in the noise ...especially if your gain is up high, but the confidence value should be junk.

I would like to use the 3d pupil detector with 2d calibration in the hololens. The 3d detector is supposed to permit "slippage compensation" (https://github.com/pupil-labs/pupil-docs/blob/master/user-docs/pupil-capture.md#pupil-detection). However, when I set up Pupil capture with a 3d detector and run my hololens app with 2d calibration, the app resets Pupil Capture to the 2d detector (via the Hololens relay). Is there a way to use the 3d pupil detector with 2d calibration in the hololens?

@user-988d86 you will get false positives sometimes when the headset is not worn. However they should be few and usually below 0.8.

Do we need to install any dependencies when eye tracking remotely on the machine using Unity? I am having trouble establishing a connection, I have a Pi doing the eye tracking (slow for now) and a machine with Unity running the calibration.unity .

Has anyone made a tutorial for setting up the dev environment for unity applications remotely? I having a lot of trouble getting pupil to connect and I think It may be because I am doing something wrong

I use Unity and Pupil on the same machine, and I have no issues, can you elaborate ?

https://github.com/pupil-labs/hmd-eyes/releases/tag/v0.5.1 download the VR unity package, connect HMD (vive/ocu) & pupil add-ons to the laptop & calibration should flow smoothly from there. As Tonzofdamage requested, can you pls elab ?

User error I have it figured out now, I didn't realize the Pi uses a different IP format than the windows device I was running unity from

Aside from minimizing obscuration of the subject's view, what are some of the advantages to placing eye tracking camera below rather than say to the left or right of the eye? It seems to be a very common placement across many products. Do you get less interference from the lenses on eye glasses, for example?