@user-07ff17 Pupil diameter is listed in the csv files as well. Using the 2d model the dimension is pixel, using the 3D model it is already in mm. See https://docs.pupil-labs.com/ -> detailed data format or the csv info file

@fxlange I tried the marketplace 2D demo. It worked past the calibration stage but no laser is showing on the screen afterwards. Do you have any suggestion? Thank you

Can I use DIY cameras to run Unity VR Project?

@user-94ac2a you can certainly give this a try, however the space constraints inside many HMDS can be a challenge unless you have very small cameras. You will also need to provide sufficient illumination in the dark VR environment. Cameras must be UVC compliant (same as DIY section in the docs notes).

@wrp I just want to use my monitor as the screen to play the unity game instead of VR headset. Does that work as well?

@user-94ac2a if you are using the monitor and want to treat it as an HMD, then you would need to assume that the head pose is fixed relative to the monitor

otherwise, you would need to use markers to track the screen realtive to a moving head

@wrp I got Display 1 No Camera Rendered Error after calibration

@user-14d189 sorry i don't understand , with the package HMD i don't obtain a pupil_position.csv ... I think to load the folder, obtain from the package, in pupil player thant export, can it work? Even thought i don't see the video of the world? Sorry but i don't understand how the output of the package works...

hi @user-94ac2a please see my message above, you seem to have the same issue as @user-a39804, try the market demos first. in general you need to set up the PupilManager in your scene properly (https://github.com/pupil-labs/hmd-eyes/blob/master/Developer.md#pupil-manager). specifying, which scene you want to load after calibration and making sure it is available for Unity.

hi @user-07ff17 , you want to access pupil data like diameter inside hmd-eyes, right? please check out the Pupil Scene ,which runs the PupilDemo script. It showcases how to subscribe to and access pupil data.

hi @fxlange Now i try to load the Pupil scene, thanks..... I load the file obtained from the package, running spherical video scene, in pupil player and I export the raw data. It's wrong?

Hello again, i try the 3d calibration mode. I load 3d calibration scene but after I making the calibration it doesn't give to me any message (successfully or unsuccessfully), seem to be that the pc is locked... does anyone know ?

After the calibration, since the last version, it seems to have a delay between the end of calibration and scene loading. It was already mentioned a couple of post above and Pupil Labs said they would look at it.

Depending on your computer speed, it can take many seconds... maybe try wait a very long time to see if it loads at some point. You can look and the console screen too if the calibration point calculation is still processing...

@fxlange I tried the market demo. It did load the market scene after calibration but it didn't show the eye tracker. All I could do was to navigate the market scene.

@user-a39804 Are you using the latest 3D Scene v0.61 with Pupil Capture 1.9 / 1.10 ? If you try using old 3D Scene with Capture 1.9 / 1.10, there was a bug and you could not see the gaze in the scene...

@user-a39804 the 2d or 3d demo? I would recommend 2D for now - the Market with 2D Calibration. In 3D, depending on the quality of your calibration, you might end up seeing nothing.

@user-8b1528 oh I used the old one. I will try the new one because I am using version 1.10. Thank you

@fxlange I tried the 2D version.

Yeah, I had the exact same problem and Pupil Labs released a corrected version with 3D Scene v0.61...

the only way I could make the old 3D scene work was with Pupil Capture 1.6 I think

@user-8b1528 I can only see the 2d calibration and import package https://github.com/pupil-labs/hmd-eyes/releases/tag/v0.61

Can you drop the download link? @user-8b1528

Hmmm... Guess I imported the Pupil.Import.Package.VR.v0.61.unitypackage and the 3D scene was in it.... Try and confirm please...

@user-8b1528 it worked. Thanks

Great, happy I could help you !

😊

I saw 3 trackers. Blue, red and green. What does this color mean?

red is gaze combined of the 2 eyes...so basically where you are looking at, blue and green represent left and right eye gaze...so red is the closest intersect of left/right eye gaze

Oh. That make sense. Thanks man

I did a couple of test with a scene with 3D object placed at 1m, 2m and 3m and seen that calibration and just small movement of the face on the helmet greatly change the values ! Just press a bit the Helmet on your face (even just a tad) and you will see it affect a lot the gaze Z position...

Just move your eyebrow up and down and it will affect the Z position... Was a bit surprised how sensible it is...

Hi guy's ! what kind of Unity Version are you use? Because i have a lot of issues on Unity 2018.3.1f1 :/

I'm using 2018.1.8f1 right now

what issue you have ?

i have 78 issues.... for example ( Assets\pupil_plugin\Scripts\MessagePack\Internal\AutomataDictionary.cs(175,31): error CS0246: The type or namespace name 'ILGenerator' could not be found (are you missing a using directive or an assembly reference?)

@user-924d7d I had the exact same issues. I just changed my unity version. Mine is 2018.2.8.f1

With Unity 2018.3.x I was having a lot of issues 😦

On Unity 2018.1.0f2 only 3 issues (Assets/pupil_plugin/Scripts/MessagePack/Internal/AutomataDictionary.cs(238,20): error CS0227: Unsafe code requires the `unsafe' command line option to be specified. Enable "Allow 'unsafe' code" in Player Settings to fix this error. )

oh ok 😃 good to know ^^ i'm trying this

Ha this error I already posted a solution: see my post of 21/01/2019

@user-8b1528 thanks bro 😄

Do you have a stable version of unity to work on ?

@user-8b1528 I was not able to calibrate and record pupil data on my project yet 😦 It just stops working and shows no error 😦 I might be missing something 😦 Do you guys have an walkthrough example or some simple toy program ?

When you say "stop working", can you be more precise ?

After done the calibration it does nothing. I Pressed "R" to start recording the data but nothing happened

It just stuck on calibration

@user-8b1528 Thanks for your "Unsafe Code" seems to work on 2018.1.0f2

@user-3c9008 You sure you have no error... usual problem is either the program cannot load next scene or no scene to load...have you checked the place where we specify the scenes to load ?

Hello, I am trying to get the gaze coordinates in my project scene using Pupil Labs binocular add on trackers with HTC Vive and Unity. Is there any example guide that I can follow to get my project communicate properly with the eye tracker and get this information? Thank you. I would really appreciate any kind of help.

Yes, I did add the scene on the pupil-manager

Sometimes the scenes are in the config section but not in the scene selection of Unity. You can check by going like if you want to compile the code and check the scenes are in the list...

which version of the pupil plugin are you using?

I'm using 3D scene v0.61 and Pupil Capture V1.10

Ok let me change my version and try again and also make sure the scene are loaded.

Just to make sure.. Do I need to add any scripts to record the data?

I was following the developer documentation, seemed like it was a simple plug and play

Check your build settings to see if your wanted scene is in the top of the window...

@user-a5f66b Just make sure you use latest Unity Scene V0.61 and Pupil Capture 1.9 or 1.10 and follow tutorial from Pupil Labs. To make everything run, as stated in the tutorial, make sure you start either Pupil Service or Pupil Capture so Unity can communicate with it.

If you stomp on the "unsafe code" error, just look a post above this one

My pupil capture and pupil service is running properly. I can run the demo market scene as well. I am looking for a way now to create my own scene and get the eye tracking work in it. Do you have any suggestions for that?

I believe I should create my own scene and place it in Available Scenes list in the 2D calibration Demo scene's Pupil Manager. Do you think it's the proper way to get it work? Then I would have SteamVR camera rig in my scene so may be I need to add the CalibrationDemo script to my Camera(eye)!? Am I right?

What I was try to do is add the pupil -manager prefab under my Camera(eye) on my custom scene and added the next scene on available list. But seemed like it didn't work.

I was planning to do the same thing as @user-a5f66b

I need to get the gaze coordinates of the user and write it to a file. Should I fetch gazePointCenter = PupilData._2D.GazePosition in my class; and then write gazePointCenter.x, gazePointCenter.y to the file?

@user-3c9008 great to hear that! Good to know someone's there in the same boat as me!

What I was wandering is that do I need to use the 2D- Calibration demo for just the calibration and then load my next scene? My application changes scene after some certain amount of time and there are 3 different scene for my application. So in that case do I need to add the Calibration Demo Scripts on all of the Scene under the Camera(eye) of Stream VR? or add the Pupil manager prefab under the Camera(eye)

"What I was try to do is add the pupil -manager prefab under my Camera(eye) on my custom scene and added the next scene on available list. But seemed like it didn't work." Oh my god! I tried the same thing and yes it didn't work! What a coincidence!

@user-a5f66b haha. I am trying every possible way to get it work. Honestly I did not found the developer documentation that helpful. A simple toy and example program would really be helpful.

Yes I think we need to I need to use the 2D- Calibration demo for just the calibration and then load our scene. Can anyone confirm this?

@user-a5f66b @user-3c9008 Here is what I have done to use my own scene:

• Open the 2D or 3D calibration scene (depending if you want 2D or 3D)
• In the "Pupil Manager" object, add your scene name (exact name) to element 0 for the 2D calibration scene, element 1 for the 3D calibration scene (or put your scene in whatever element # you want and change the "Current Scene Index" to that number)
• Load the scene you created into the Unity editor (don't change project, just open the scene)
• Go in "File-->Build Settings" and add your current scene ("Add open scenes") to the list , you can just close that window after.
• Load back the 2D or 3D calibration scene and run it (with Pupil service or capture running)
• It should run the calibration and after it should load your scene

Great, now your scene is loading !!! But wait... I don't see any gaze balls floating around ???

Normal, you don't have any GameObject in your scene for that ! Let's add that now:

• Browse to "Assets/pupil_plugin/Resources" and drag drop the "MarkerObject" on your MainCamera GameObject.
• Create a new script (name it for example "PupilTracking") and attach it to the MainCamera object.
• Edit that script and put this (WARNING: note that I use the "PupilData._3D" data, adjust it to "PupilData._2D" if running in 2D):

======================================= using System.Collections; using System.Collections.Generic; using UnityEngine;

public class YOURSCRIPTNAME : MonoBehaviour {

public Transform marker;

// Use this for initialization
void Start () {
    PupilData.calculateMovingAverage = true;
}

void OnEnable()
{
    if (PupilTools.IsConnected)
    {
        PupilGazeTracker.Instance.StartVisualizingGaze();
    }
}

// Update is called once per frame
void Update () {
    if (PupilTools.IsConnected && PupilTools.IsGazing)
    {
        marker.localPosition = PupilData._3D.GazePosition;
    }
}

• Save the script and back in Unity
• If you go check the script attached to MainCamera in the INSPECTOR section of Unity, you will see that your script now has a field called "Marker" empty, just drop your MarkerObject GameObject into that field.

You should be all set now ! Way to go !!!

Try running the calibration scene now and see when your scene loads if the gaze balls are there...

@user-8b1528 thanks a lot! 😄 I really appreciate your help 😄 You should try writing a blog post about it. I have found no good online resource about the exact steps. This will clearly help a lot of us.

try it and we can fine tune the details and maybe yes, I could create a blog post with it ! Took me a while to figure that all out !!!

Hello @user-8b1528 I saw the post that you mentioned, i try and i am waiting for 1-2 minutes, pupil capture is locked and i had to force the pupil to stop running. Had to do something?changing script? a lower frame rate, i dont know.... any suggestion?... thanks

Have you tried to create a new fresh Unity project from scratch and just add the v0.61 scene and see if it still hang out ?

Thank you @user-8b1528 for this huge help! I had a question though. When you say add "MarkerObject" and "PupilTracking" script o the main camera.. Do you mean Camera(head) or Camera(eye) in SteamVR? I assume it's Camera(head). Am I correct?

is this normal?

@user-a5f66b When I create a new Unity project, it creates me a MainCamera automatically, that's where I add the marker and script...

oh okay. In my case, I have to disable the default main camera and add the [CAmeraRig] from SteamVR to use with HTC Vive. I tried adding them to Camera(head) and it seems to work. A little jittery in gaze tracking(those 3 circles) but at least working. Thanks a lot.

@user-a5f66b did it worked. I haven’t been able to try out yet. But as you are doing it with htc vive i am really interested to know.

@user-3c9008 it's working. I can see the 3 circles in my scene (which starts after calibration) but now I need to see why it's not tracking my gaze properly. My guess is that for a virtual reality environment we might use PupilData._3D.GazePosition instead of PupilData._2D.GazePosition!? Don't know if my guess is correct! Next thing I would try is writing those gaze coordinates in a file.

Why does Display Eye Image is two Questions Mark for the unity project example?

Hi guys. Does anyone experience calibration failed while using Hololens add-on

@user-94ac2a is it an exe or on unity?

@user-07ff17 on editor. But for exe I am not able to see that also. It will show an grey image without anything and press c does not work for exe.

@user-94ac2a I had had a similar problem. 1. You can try to download source code and copy and substitute the two folder Project setting e unitypackmanager (\hmd-eyes-0.61\unity_pupil_plugin_vr) in the folder of unity project.

If it doesn't work, another way can be this : go to EDIT-- PROJECT SETTING -- GRAPHIC--- in always Included Shaders insert Unlit/texture

Let me know if it work... i hope to help you

@user-07ff17

I think Unlit/texture is already included

I also used the source code to open the unity project

Still the same

After i have load the scene tell to Pupil Manger, i can load other scene with the same calibration..

Hi pupil people! Has anyone tried to use the AR/VR addons with Google daydream?

Does anyone faced any problem while recording data ?

First I got a compile error becuase FFmpegOut was missing

I imported it and then it shows error like this

I changed the .net version to 4.x on player settings and still having this error

Assets/FFmpegOut/Runtime/FFmpegSession.cs(130,14): error CS0246: The type or namespace name AsyncGPUReadbackRequest' could not be found. Are you missingUnityEngine.Experimental.Rendering' using directive?

Hello, Did anyone noticed any change after l2D calibration ended and successfully loaded next scene? Seems like all my lighting and color changed after pupil_manager loaded the next available scene. Any solution to this:

@user-3c9008 I got similar grey and blue screen when I tried to do post-processing with OnRenderImage() function. Are you trying to do that?

Hi all, It seems that 3D Calibration does not work well in unity 2018. I am using package 6.1. After 3D calibration, the yellow ball in the scene just not moving. It was shaking in the same position when moving eyes.

0.61

Since it was shaking, I assume it receive some data... My first guess is a very bad calibration maybe... If you run the 2D calibration, everything work OK ? Are you using Pupil Capture V1.10 ?

I can run the 3D calibration OK on Unity 2017.3.1f1 and 2018.1.8f1

I seen some people saying that the very latest version of Unity (2018.3+) seems tonot work great with Pupil Lab, but I have not tested myself...

Hey @fxlange, we can't seem to get the eye trackers working with the 3d detection and mapping, even for the 3D demo. After we calibrate, the dot seems to dance all over the place, often not even in view, and usually not well connected to the eye movements. We are using Unity 2018.2.3f1, the eye trackers for the oculus DK2, and the latest releases of the pupil plugin and pupil capture/service. Thanks for your help

Hey all, we've developed an app with eyetracking integrated but we are finding an odd interaction where the calibration doesn't start. PC is running windows 10 and the latest pupil software (v1.10). When the vive pupil inserts are not plugged in pressing 'c' does start the calibration routine (which fails due to there being no eye inputs) however, when the pupil usb cable is attached and eye video is received, pressing 'c' appears to do nothing.

any ideas on what could be the issue or something i may have missed?

Hey @user-23e10d, I personally haven't worked with the eye trackers for the oculus as we are not selling them in the store anymore. I'm not sure if the 3d accuracy is any different compared to the Vive version but I'll check back with the team. But independent of the kit: In 3D the gaze direction should be stable but the depth error is expected to vary quite a bit - especially for higher distances. I would recommend normalizing the gaze vector to a fixed length or projecting it into the scene, which should give you a mush more stable visualization - given a high quality calibration. We provide a more stable visualization in one of the next releases. On top you should make sure to archive a stable pupil detection for both eyes - what do the confidence values say in PupilCapture after the Calibration?

Hey @user-8a8051, do you get any feedback in the console when pressing C? Are you running your tests from inside the Unity Editor or from a build? Which hmd-eyes version are you using and have you tried the Market Scene Demo/2D Calibration Demo? Please make sure to focus Game View in Unity when pressing C.

hi @user-3c9008 , I also noticed unwanted visual side effects after switching scenes. I'm suspecting PupilManager and how it handles the scene switch to be the root of these problems. We are currently reworking the plugin and will definitely look into the Calibration -> Application transition.

Thanks @fxlange, no feedback in the console when pressing C, running from a build, regarding which version of hmd-eyes i personally didnt build the app so i will try and find out, i have been speaking to those who built the app (located in another city/timezone) when possible and they have not managed to replicate this issue testing across multiple computers while this issue appears on two computers that i have tested on. I have tried the market scene demo and this calibrates fine, and yes the game/build is always the focus when pressing C.

Additionally, i have figured out a bit of a work around. It seems with pupil capture/service open but the camera inserts (usb cable) not plugged in, pressing C initiates a calibration attempt. I can then plug the usb cable in and complete the remaining part of the calibration (which, understandably, gives poor results due to having eye video input for half of the calibration routine ), following this i can press C again to recalibrate.

It's not ideal but seems to get the job done for now. I found it very strange that it is only seems to happen on my computers and only when the camera inserts are plugged in. Thanks for your help,

also any recommendations on purchasing a usb extension cable for the inserts to bring the overall length more in line with the vive headset cable?

Hey @fxlange thanks. I have made a way around to work with this now. But yeah sure please fix the issue.

anyone noticed that the eye-tracker gets hot? I run it for 10mins and it was hot. I worry that my participant may feel real discomfort due this heat generating from eye-tracker. If I touch the eye-tracker I can feel the heat.

Hi @user-3c9008 regarding heat produced by the cameras, I have responded in the core channel.

@wrp thanks.

Welcome @user-3c9008

Hi! I would really appreciate if you could clarify to me if the following is correct: I record both eyes, and I get two gaze positions in viewport coordinates (e.g., viewportPointLeft and viewportPointRight). I want to cast a manual ray (vector), in order to see where the left eye is looking. Is it safe to assume that the origin is the camera, and the vector direction is defined by the camera origin and the ViewportToWorldPoint(viewportPointLeft)? Or should I consider the corresponding frustum offset for the left eye? @mpk @wrp

Coming back to a project using Vive addon, Unity, hmd-eyes. I have yet to get a working demo scene that performs (2D) calibration, then passes to a scene with head/controller tracking via Steam plugin, while maintaining retinal tracking. This seems like a trivial setup for hmd-eyes: is anyone aware of a demo scene or tutorial as a baseline to get things working?

Hello everybody, when we start one of the calibration scenes and the pupil labs are successfully connected in Unity the pupil dector settings (min, max and intensity range) are overwritten. Is there any way to stop this or are we even doing something wrong? Thanks

@user-6997ad i have the same problem

@user-07ff17 I finally got things sorted out actually. The main barrier wasn't actually pupil it was getting controller tracking working in our scene via the new SteamVR Input 2.0 system. The actual handoff from 2D calibration -> our scene is working fine based on the instructions from @user-8b1528 above. What's your current issue?

HI @user-6997ad I make a scene with some buttons and the player of steam. The click event (hand event on click by from steam) work without pupil but if the calibration process is runnig it doesnt work. Do you have any advice?

Ah, are you trying to do click handling during calibration?

Yes exactly

I see. That's actually a more complex setup than ours. Is it possible to delay any click interaction until after calibration completes?

I try to explain my project... I load the calibration scene, finished the calibration i load the scene with click interaction (load scene additive without destroy the calibration information) . The interaction save a txt file and load another scene but it doesn't work... I hope I explained myself

it's necessary form me the gaze tracker during the scene with intercation

That sounds like our use-case too. So there's no click interaction until the calibration has ended. Did you try the instructions above? For me calibration data is conserved when the new scene is switched, and click handling through the Steam plugin works once actions are configured

I don't find the instruction above, i look for but i don't find... could you indicate to me?

the scene with interaction works in the same project if i don't made the calibration.. so i think that the actions are configured well ...

On Feb 1st from P Perron

is this?

Yes, that worked for me

thanks very much.. i try and i say to you.. thanks thanks

@user-6997ad When i use the player of steam the main camera is the VRcamera right?

@user-07ff17 In my scene I have the Main Camera (the default that comes in the initial scene), and then a separate VRCamera within the Player object

I attach the MarkerObject within the Main Camera, though you may not need tracking markers

Good morning, we were trying to narrow possible problems down. So we found out that the demo build demo scene works for us. So the .exe you offer for testing is working fine. But we cant finish the calibration in our own unity projects, even when we built them. Is that a common issue? Could it be a version problem? We are using Unity 2018.2.17f1

Hi @user-97bf6d, regarding your two questions: The issue with overwriting settings in Pupil Capture on starting a calibration is known. In the future you can set most of these values in hmd-eyes directly. Regarding your calibration issue: Could you please provide more details - what do you mean by can't finish?

@fxlange Thanks for your answer. The calibration itself works. It shows white markers most of the time. Some slighlty yellowish or pink but not that bad. But the calibration will "fail" all the time. We cant get it to suceed even when I change the needed confidence in pupil capture.

Hi guys. Is there a way to show world camera feed in unity remotely?

Hello there. We use the eyetracker for the HTC Vive. However, we have significant problems in tracking accuracy when the user wears glasses. Is wearing glasses under the HMD not intended when using the eyetracker, or are there ways to improve the performance? Thank you in advance.

We would recommend not wearing eye glasses within the HMD

@user-168d91 FWIW, we find that it performs well with contacts.

contact lenses.

@user-8779ef Thanks for the information. Unfortunately this is not an option in our case.

K. Also fwiw, no eye tracker that I've ever worked with will work through glasses.

Mostly because they smear the corneal reflection/glints from the LEDs that pupil detection algorithms rely upon.

Good to know. I am new to the field of eyetracking. So thanks a lot for this hint!

No problem! Best of luck.

Hi there. Is there a way to access base_data from gazeDictionary in Unity? I saw that there was a person with the same question on this, but no response. Thus, I am not sure if this issue has been addressed.

Hi @user-294ca7,

we are currently looking into it the base_data of the gazeDictionary. In the meantime you can get the data you are looking for by subscribing to pupil - please checkout the pupil_plugin/Pupil demo scene.

Hello Guys,

we're planning to use the hmd-eye Unity plugin with a custom (Android based) AR headset. Our setup currently consists of a notebook with the Pupil connected and software running and an Android smartphone running the Unity application and connecting to the notebook (tested wireless and wired).

At the moment we're struggling to get the Pupil connection running stable. When starting our application we connect to the Pupil (StartCoroutine(PupilTools.Connect(retry: true, retryDelay: 5f));), on success we subscribe to the gaze topic and on each frame we call PupilTools.Connection.UpdateSubscriptionSockets(); to process all new info which calls our custom method (PupilTools.OnReceiveData += CustomReceiveData;).

This works fine and we receive all the data we subscribed to and in the log we also see the "Succesfully connected to Pupil!" message. After 1-2 minutes or so it stops receiving data without any error or log entry generated. The Pupil still sends data (tested locally) and the network is also not the issue (monitored the connection, tested wireless and wired).

I think the problem lies somewhere within NetMQ, because as soon as we stop receiving data the if-condition that checks for new data on the socket in Connection.cs line 210 (v0.61) isn't validated true (subSocker.HasIn()).

Did already anyone experience something similar or has an advice? At this point I'm not sure how to continue.