Hi. For some reason my Unity hmd-eyes application is no longer receiving any gaze data from Pupil Capture, even though I am able to get regular pupil data and blink events. I have not changed my code at all so I am guessing it is something to do with the settings in Pupil Capture itself. Are there any setting configurations in Pupil Capture that would cause this to happen?

I have also tried again with the hmd eyes default starter project and reinstalled Pupil Capture, and the issue is still happening. I'm very confused since now I know all the configurations and settings are at their default values--not sure what could be causing this

@user-32853a Instead of reinstalling Pupil Capture better "restart with default settings".

@fxlange I have tried that too and nothing changed

@fxlange let me know if my Pupil Capture log output would be helpful

Problem solved after I performed a few calibrations in Pupil Capture and then closed the application. I was previously able to get gaze data in Unity without calibrating in Pupil Capture so I'm still a little confused, but the issue seems to be fixed at least

Hi, I'm trying to use Pupil (along with the Pupil Core headset) with Unity to identify where a user is looking at on a monitor. The major road block is that eye/gaze positions are relative to the rotation of the headset (or what's referred to as being relative to the VR camera space). Is there any way to maybe using the surface tracker data to determine where the user is looking on the screen? Or any other solution? Thanks.

Hi, my team is about to purchase the htc vive add on. The idea is to integrate this into a custom VR headset. I would need a few info about the camera fov and focus. What is the average distance from the add-on’s camera to the user’s pupil? If we integrate the add on into our prototype at a increased distance (for example, 1cm further away from the pupil than usual) would this still work?

Hi @user-c40893 - I'm not sure if I fully understand your usecase. If you want to visualize "non VR gaze" (="Pupil Core") inside Unity you should calibrate via Pupil Capture and not via hmd-eyes and just listen to gaze via hmd-eyes afterwards.

@fxlange Thanks for the response. That's the approach I'm currently using, but the values (GazeDirection and GazeDistance) still change when rotating your head. For example, if I keep my eyes focused on a single point on the monitor while turning my head, the GazeDirection and GazeDistance will continue to change. My goal is to have these values remain the same (if eyes remain focused on a single point) regardless of head rotation.

Hi, we've been using the Pupil Capture v0.9 with the 120Hz HTC Vive eyetracker successfully, until now, when we've tried to upgrade to a 200Hz eyetracker and Pupil Capture v1.12. Unfortunately, when we try to calibrate, there are no markers shown. Now, when trying to use the 120Hz eyetracker, we get an error "unexpected keyword argument 'exposure mode'" or "unexpectede keyword argument 'check stripes'". Could you please advise us what could we look into? We are not using the hmd-eyes v1.0, because our experiment has been running on Unity 2017

hi @user-c40893 - turning your head while focusing on a single point results in your eyes moving - so definitely gaze direction and distance will change. But if you combine head and eye movement (eye movement in head space) the gaze target will remain at one spot (more or less due to latency). If you don't have the head pose due to using a normal pupil headset you could use a tracker instead - yes. The GazeVisualizer for example allows for any Transform as its space origin. Doesn't have to be a camera / the VR headset (though it is called Camera Transform...).

hi @user-1164a3 - the eye cameras of the new vive addon (200Hz) have a fov of 70° and a fixed focus with a depth of field of roughly 15-80mm (so 1cm extra distance shouldn't be an issue).

hi @user-0cf021 - 1) running the old hmd-eyes v0.62 + pupil capture v1.12 + the new 200Hz vive addon should definitely work. but you don't see the markers during the calibration routine? which scene are you running in Unity or are you running the demo? 2) where and when do you get the errors when trying 120Hz again? in Unity or Pupil Capture?

@fxlange Thank you for your answer! 1) I'm not sure I understand your question about the scene, so I will try to describe our setup. We have build a city in Unity, and the author wrote some scripts that take the eyetracking data from Pupil Capture and do raycasting and other stuff. 2) In the Pupil Capture cmd. If it helps, I can send you the log or the full error message

@user-97591f - we are looking into the timestamp issue and the pull request but from hmd-eyes perspective we have to wait until this issue is resolved. as an alternative to your "hack" and only subscribing to gaze.3d.1 you can filter gaze data by GazeData.MappingContext.

@user-0cf021 2) might be an issue in Pupil Capture as a result of switching hardware (120vs200). maybe restart with default settings after switching hardware but this is not related to 🥽 core-xr and better asked in 👁 core. 1) okay, so the calibration routine doesn't work in your custom unity project. no markers visible during calibration? please test the demo scenes (as part of the hmd-eyes vr *.unitypackage) after switching to 200Hz before you continue debugging your own project.

@fxlange I apologize for so many messages, but I just want to make sure I fully understand. Yes I am using the normal pupil headset so, as you mentioned, I do not have the head pose. But I do not understand what you mean by "use a tracker instead"? If you are referring to the surface tracker data, how would I access that data using the hmd-eyes package? Or are you referring to having to use an external tracker in addition to the normal pupil headset?

So, I'm currently trying to add pupil to a pre-existing Hololens scene, I've imported the "pupil manager" into the scene, is there anything else I ned to do? I see that it's got a camera built in but the Mixed reality toolkit already supplies one.

Should I use the calibration scene provided then load my main scene?

I've loaded it into my scene but nothing seems to be happening

Hi @user-c40893 - I was referring to external trackers yes. The HTC Vive Tracker came to mind or you could even strap the Vive controllers to someone heads as a prototype (don't do this at home...). But I was assuming that there is still some VR setup involved which is not the case right?

Surface tracker plugin could be an option but I'm not sure if it is accessible via the network interface ( ping @papr ).

@papr ```So, I'm currently trying to add pupil to a pre-existing Hololens scene, I've imported the "pupil manager" into the scene, is there anything else I ned to do? I see that it's got a camera built in but the Mixed reality toolkit already supplies one.

Should I use the calibration scene provided then load my main scene?

I've loaded it into my scene but nothing seems to be happening ```

@user-adf88b sorry, I do not have any experience with hmd-eyes on Hololens

Anyone I can ping for help?

@user-c40893 Subscribe to the surfaces topic to receive surface data

@fxlange is there any chance you could help with hmd on the Hololens?

Also I'm getting NullReferenceException: Object reference not set to an instance of an object HoloToolkit.Unity.Buttons.Button.get_handVisible () (at Assets/Plugins/HoloToolkit/UX/Scripts/Buttons/Button.cs:105) HoloToolkit.Unity.Buttons.Button.LateUpdate () (at Assets/Plugins/HoloToolkit/UX/Scripts/Buttons/Button.cs:126)

Hi @user-adf88b - which Unity, hmd-eyes and pupil capture versions are you using? We only support Hololens until hmd-eyes v0.62 with Unity 2017.4LTS. But hmd-eyes v0.5.1 (https://github.com/pupil-labs/hmd-eyes/releases/tag/v0.5.1) is our last release including a Hololens package and needs PupilCapture 1.7 (https://github.com/pupil-labs/pupil/releases/tag/v1.7) to work.

Please start by making sure that the demo scenes (shark demo for hololens) are working. After that I would indeed recommend to use the calibration scene to load your main scene via the Pupil Manager (https://github.com/pupil-labs/hmd-eyes/blob/master/Developer.md#pupil-manager-prefab).

Oh I was using version 0.62 and the latest pupil capture

@fxlange I made the changes and when I run the project on the hololens I'm stuck at "trying to connect" is there any way to debug the problem?

I've made sure to configure the settings scriptable object

Oh the connection over wifi works when launched within the editor but on the hololens

I'm also getting Exception thrown at 0x77472DC2 in BackdatedHeartRenderer.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x01E2F51C. Exception thrown at 0x77472DC2 (KernelBase.dll) in BackdatedHeartRenderer.exe: WinRT originate error - 0x80070005 : 'A network capability is required to access this network resource'. in Visual Studio. Is it related?

Also just running the default calibration scene

Exception thrown at 0x77472DC2 (KernelBase.dll) in BackdatedHeartRenderer.exe: WinRT originate error - 0x80070005 : 'A network capability is required to access this network resource'.

@papr You have any idea?

I followed the insructions, when I run the program in the unity editor it connects/disconnects fine but when on the hololens it's stuck at 'connecting'

No, sorry, I do not have any windows or hololens programming experience. @user-adf88b and these errors look not related to Pupil Capture.

Right, I'll look into it thanks

@user-adf88b might be a visual studio setting and not hmd-eyes related. found this https://stackoverflow.com/questions/33235131/universal-windows-project-httpclient-exception

Thanks I'll look at it

I know it's something to do with compiling to the hololens just not what

Hi, I am having an issue with pupil capture. Somehow now Unity cannot establish connection between eye tracker and the program. It has worked previously but now is not working anymore. This is the message I get in debug.log What might be the reason? Any advice would be very helpful. Thanks

@user-0eef61 please check if pupil capture/service is still running on port 50020. might have changed due to starting multiple clients. which version of hmd-eyes are you running - latest beta release or develop branch?

Hi @fxlange yes the problem was that the port was changed in pupil capture. I changed it now to 50020 and now works fine. Thanks for your advice. It was very helpful.

Hey, is there any know issue with Win 10 Enterprise Version 1809 and the pupil cam drivers? I previously managed to launch HTC Vive Addon successfully but since the Update Cam ID1 isn't able to connect anymore (in Device Manager it is shown as Sonix Technology Co., Ltd.). Driver reinstall wasn't successfull.

@user-095e7e to rule our an hardware issue, can you connect the vive addon to a different computer and see if that works?

@mpk I've tested it on a different computer (same Win Version) and got the same problems

Hi I'm loading up the blink example on the hololens and I'm getting a message "PupilTools does not contain a definition for Send"

Is there a link for an older working hololens blink script?

For ones having problem with running the Blink Demo for HMD-Eyes. These bellow steps are what I were doing to get it running as following the instructions from https://github.com/pupil-labs/hmd-eyes/releases

• Get the Pupil Labs add on your HMD and connect with your PC via its cable
• Install Unity3D ver. 2018.3.14
• Download the beta release of HMD-Eyes from the link above
• Set “Scripting Runtime Version” set to .NET 4.x Equivalent, and “API Compatibility Level” to .NET 4.x in your player setting
• Add VR support for your Unity project
• Start capture interface
• Play your Blink demo scene

Good luck

Hello got probably a pretty basic question, I installed the hmd-eyes unity plugin v.1.0 and wanted to use the plugin in my own unity script. But when adding "PupilTools.OnConnected += StartPupilSubscription;" to my c# script it just failed because "PupilTools" doesnt exist in the current context. What am I missing?

just for the record, had the issue only with the v.1.0-beta.unitypackage when importing "properly" with "Import Package>Custom Package". Just drag and dropping from "unity_pupil_plugin_vr/Assets/pupil_plugin/" to the project asset dir seemed to work

Hello all! Is there a software with which you can record what happens on the VR scene when working with glasses (as it is implemented for 2D mode in the pupil_capture.exe)?

hmd-eyes do not allow VR scene recording

Any timeline on implementing streaming of Unity's game scene in VR to pupil capture?

Anyone have an issue where calibration is interrupted by a steamVR menu flashing in and out?

This happens in the demo scene provided as well as my own that I made, any time calibration is called it seems to hang a bit sometimes in the middle of calibrating

Hi @user-8414e7, it looks like that you are mixing up hmd-eyes versions. PupilTools doesn't exist anymore in hmd-eyes v1.0 (beta) - instead you would use for example the SubscriptionController. Please refer to the release/beta branch for checking the readme and documentation:

• https://github.com/pupil-labs/hmd-eyes/blob/release/beta/README.md
• https://github.com/pupil-labs/hmd-eyes/blob/release/beta/docs/Developer.md

@user-98f03c Unity has a package (currently in preview) for recording your VR scene. but as @user-bd81b0 mentioned we are instead planning to stream the VR scene to pupil capture and record everything there. no timeline yet but it is planed for the final v1.0 release and we are currently working on it

@fxlange Is this only for Unity? If we want to play, say, movies or videos in third-party applications, will it be possible to capture the scene?

@user-98f03c we are currently developing it for "hmd-eyes -> pupil capture". but the network protocol is open and not limited to unity. depending on your use case it could be possible to stream other content as well - but not supported out of the box.

Thank you very much for the information!

[email removed] thanks a lot, apprechiate your answere. Yes was looking into the wrong version of the documentation and there was aparently nearly a total overhaul 😃 but anyway, it works smooth now

Hi, I am trying to record gaze and pupil data from a unity session. I have tried the "Data recording" scene in the Demos and successfully integrated it with the calibration procedure. But how do I export that calibration so that I can used it when exporting the data in Pupil Player?

Thanks

Does the vive's black nosepad being in the camera view affect accuracy?

It seems to get painted blue by the algorithm as though it could possibly be a pupil

For best performance, the screen should be extended far away from the face, so that the eyes look small in the camera, right?

Also I noticed when recording some data that the timestamps seem to not be at even intervals. Is it like, the pupil's frequency varies and is not always 200hz?

I just brought a pupil-lab eye tracker for HTC VIVE but I meet some difficulties to integrate the eye tracker to the headset. There are less spaces for me to insert the eye tracker. Did anyone meet this problem before? In addition , my boss plan to buy a HTC vive eye, he is wondering if the pupil lab can integrate with the new coming HTC vive eye headset. thank you

Hi, you can adjust the distance of the screen in HTC VIVE. Simply pull the circle on the side and rotate it.

@user-6320cf Refer to this video for extending the depth adjustment. https://youtu.be/oGcPyWlEZS0?list=PLi20Yl1k_57r4j0LXDfo6IYXAKTp_FIKf It’s in the playlist of instructions you should be following when installing the headset: https://docs.pupil-labs.com/#htc-vive-add-on

hi @user-5df199 regarding the tracking performance: with the FOV of the 200Hz cams there is no need to increase the screen/cam to eye distance anymore. and yes, when using pupil capture the frequency varies as pupil capture sends out data packages in bundles. for even frequency and lower latency per message you should switch to pupil service.

@user-97591f Thank you very much for your response. I am wondering if this pupil eye tracker is detachable since I want to use it in multiple headset.

@fxlange Does pupil service support recording data (as pupil capture does) ?

Hi everyone I'm having quite some tracking issues in my current project using the 120hz Vive Pro Add-on and hmd-eyes (v1.0-alpha-2). This might very well be due to my limited experience with eye-tracking - so any advice is highly appreciated. Usecase: using ET for fixation control in an EEG experiment (i.e., ideally no eye movements into the periphery / I only really care about central fixations); stimuli are shown at a fixed distance, therefore also my calibration is not going into depth. I observe 3 scenarios: (A) in ~50% of cases everything works neatly. (B) In 25% there are massive problems with pupil detection (also w/o make-up, ... and for same subjects for whom it worked in earlier attempts). This means only meaningless values are extracted.

(C) In the remaining 25% the pupil seems to be detected acceptably (confidence is mostly ok) but the mapping is off. In these cases the gaze is (more or less consistently) displayed (e.g.) 5° off the true point of fixation (sort of constant offset). Probably (C) cases can be reduced to pupil detection flaws during calibration. So (C) and (B) might be related. I attach a link to a video of a calibration that went wrong (it can be clearly seen that tracking is lost multiple times during calibration. Any ideas how I could improve on this? I also attach a video (with the same subject) in which everything went kind of smoothly. I cannot really see a difference between the two scenarios. (Btw if everything works, a grey dot overlayed on the "bulls-eye" in the bottom right corner of the game view window indicates the gaze direction in relation to the fixation cross. Green circle: 2° radius, red circle: 4°) There is a weird fast flicker visible in the eye videos (clearly visible in the vid of eye1 in the first video at seconds 15-20). It looks like there is a flickering light source that even reflects in the cornea (left of pupil in eye1). Could this be a problem? Any idea what this could be?
I'd be very grateful for input from someone with more hands-on experience who could maybe point me into a direction of how to fix this. Of course I'm happy to answer all questions. Thx! Videos: Bad calibration: https://bit.ly/2SBJuHl Ok calibration: https://bit.ly/2Z8bAwq

@fxlange Awesome response! that makes things really clear! Thank you!

still wondering about why the steamVR menu is persistently popping up during my calibrations

(C) In the remaining 25% the pupil seems to be detected acceptably (confidence is mostly ok) but the mapping is off. In these cases the gaze is (more or less consistently) displayed (e.g.) 5° off the true point of fixation (sort of constant offset).

I've seen this too

Hi, we're new to the pupil labs community and are trying to get the Gaze demos working on the HTC Vive Pro. The scenes load fine when played through Unity, but it just says "Not connected" on the screen and doesn't run any calibration. We have PupilCapture running in the background as well. Any thoughts as to what we might be missing? Apologies if this has been asked before. We searched through the forum and couldn't find anything.

hi @user-5df199 - which versions of hmd-eyes and pupil capture/service are you using when steamVR is popping up?

hi @user-141bcd & @user-5df199 - regarding C) at what distance do you place your calibration and reference markers?

• @user-141bcd - I had a quick look at your videos. the bad calibration video has no valid 3d model for the one eye (no reasonable green circle). a) you should definitely try to improve the pupil tracking (red circle is jumping quite a lot) and b) make sure that the 3d model is build before starting the calibration - "warming up the eyes" by looking in all directions shortly before running the calibration helps a lot.

for a) please check the fine-tuning section of https://docs.pupil-labs.com/#pupil-detection

@user-52f2e6 - no Pupil Service doesn't support recordings - but depending on your usecase you could record data directly in Unity via custom scripts or third-party plugins if Pupil Capture is not an option.

Solved it. Had to update the scripting runtime to .NET 4.x - For future reference this article helped us figure out how to do that: https://docs.microsoft.com/en-us/visualstudio/cross-platform/unity-scripting-upgrade?view=vs-2019 although the setting was under Edit -> Project Settings -> Player -> Other Settings

hi, is there any way to start calibration without using the keyboard? we would like the calibration and experience to be initiated with one click.

hi @user-644a65 - which version of hmd-eyes are you using? in v1.0 beta you can hook the calibration controller to your own scripts or for example to a unity ui button. via ToggleCalibration you can then start and stop the calibration process.

Hi, I've been using Pupil with Unity and I've been looking at trying to find the IPD of the wearer. Is there anyway that Pupil Cameras can help with that?

We've been struggling with the calibration a bit. We are triggering it programmatically from our unity code. We are having a problem running the calibration in 2D mode. I've set:

calibrationController.settings.mode = CalibrationSettings.Mode._2D;

in the code, but it still seems to be running the calibration in 3D mode. Are we missing something? Do you have to change the settings in Pupil Capture as well?

a bit more information on the above: we're trying to track a dot on a 2D canvas in unity and having trouble with the accuracy of the tracker. it sometimes recognizes the eye landed on the dot, and sometimes takes several seconds before recognizing.

Thanks @fxlange We managed to record data using pupil capture and unity but we are facing some issues regarding the pupil_position.csv data confidence levels for one of the eyes, which is consistently worse than the other one (0.8 vs 0.6 average). Could you advice on why that could be and what we could do to improve?

Hey everyone, Can PupilLabs work on the Hololens when we deploy it in VS ( with Debug/x86/Device) ? It works on Holomographic Emulation but not in VS, there is no connection with PupilCapture. Thanks !

@user-52f2e6 we are getting something similar -- our position readings for the eyes are skewed right, off by 200. have you noticed that the center of your calibration seems to be more towards the right side of the screen?

hello guys, I've a question about hololens + pupil labs, how can I calibrate the eyes rotation from pupil, and head rotation/position coming from hololens ?

I want a setup where I can record the head position/rotation + both eyes rotation for a certain amount of time, and then playback this record against any 3D space, doing a heatmap

and eventually if its possible, using multiple records at once, and produce a combined heatmap for that space

but I'm worried about the accuracy of the eyes position and rotation in realtion to the head tracking data

is that a built-in feature from hmd-eyes, or has anyone done something like this before

Hello everyone, I want to use pupil lab to monitor eye movement while playing a VR games. Is there an easy way to recording hmd images as world and sync it with data?