hello everyone does anyone have the link of pupil detection algorithm used by pupil-labs?

Hello, I am working with HTC Vive and unity. I am trying to create a calibration scene(with pupil manager prefab) before loading my actual scene(added it's name in the Available scenes). My trouble is that when I execute the calibration scene it gives me error, please could anyone help me with these? The errors are: ArgumentException: The Object you want to instantiate is null. UnityEngine.Object.CheckNullArgument (System.Object arg, System.String message) (at C:/buildslave/unity/build/Runtime/Export/UnityEngineObject.cs:239) UnityEngine.Object.Instantiate[Transform] (UnityEngine.Transform original) (at C:/buildslave/unity/build/Runtime/Export/UnityEngineObject.cs:200) PupilManager.InitializeCalibrationPointPreview () (at Assets/pupil_plugin/Scripts/PupilManager.cs:66) PupilManager.OnConnected () (at Assets/pupil_plugin/Scripts/PupilManager.cs:51) PupilTools+<Connect>c__Iterator0.MoveNext () (at Assets/pupil_plugin/Scripts/Networking/PupilTools.cs:314) UnityEngine.SetupCoroutine.InvokeMoveNext (IEnumerator enumerator, IntPtr returnValueAddress) (at C:/buildslave/unity/build/Runtime/Export/Coroutines.cs:17)

and

NullReferenceException: Object reference not set to an instance of an object PupilManager.OnDisconnecting () (at Assets/pupil_plugin/Scripts/PupilManager.cs:42) PupilTools.Disconnect () (at Assets/pupil_plugin/Scripts/Networking/PupilTools.cs:572) PupilGazeTracker.CloseShop () (at Assets/pupil_plugin/Scripts/Pupil/PupilGazeTracker.cs:156) PupilGazeTracker.OnDisable () (at Assets/pupil_plugin/Scripts/Pupil/PupilGazeTracker.cs:141)

Hi I am working on an AR application with HTC Vive, I run the 2D calibration script and unity and then demo it works. When I run the 2D script and then my custom AR camera with 2 cameras. i attach the calibration demo script to the left eye (camera) and marketwith2DCalibration with GazeHighlight. It works and runs through calibration, but when switches to AR the eye tracking dots are distorted and don't track well and seem like they are on a different scale/coordinate plane. Again I have the market demo working great. Any thoughts on how to migrate the script to a stereo camera in AR?

Hi, I have a problem with the 3D Calibration Demo sample scene, I can complete the calibration process fine and when scene load, I see the 3D Market scene OK pour the yellow ball that is supposed to move to where I'm looking at remain at position 0,0 (directly on the camera). This happen when I use Pupil V1.9 and even the new V1.10... I tried using the Pupil Service or the Pupil Capture but both don't provide the yellow ball moving. If I use old Pupil V1.6, it works fine and the yellow ball follow where I look at.

So the 3D demo scene seems to work fine, only not with the new Pupil versions...

Any idea what could be wrong ?

Using the 2D demo scene (used the scene updated last November), it works fine with all Pupil versions, so only the 3D demo cannot work with new Pupil version...

Thanks !

hi @user-8b1528 - thanks for your feedback. You are right, the 3D demo in the current hmd-eyes release doesn't work with Pupil Capture V1.9 nor with V1.10. We will fix that and release a new version - ideally today.

OK great, thank you for your answer !

We just released an update for hmd-eyes (v0.61).

The 3d gaze demo included in the unitypackage is now compatible with Pupil Capture 1.10 (1.9 and up).

https://github.com/pupil-labs/hmd-eyes/releases/tag/v0.61

Thanks @user-8b1528 for pointing it out.

Thank you for the quick fix, appreciated !

Getting some strange behavior with the new 1.10 and the 0.61 HMD Unity plugin. I had all working for 2D in 1.9.7/0.60. Now trying to get the 3D going. In my 2D build, it cannot connect to Pupil Labs even though running and remote is publishing on 50020. I try the 3D versions and now it connects to Pupil Labs, but on the "Press c" screen nothing happens when you press 'c' and the eye images are not publishing. Just gray boxes. Thoughts on what is wrong here. Setup the same way as before? Thanks.

Also on my prior post - " Any thoughts on how to migrate the script to a stereo camera in AR?" Thanks!

The 3D calibration doesn't complete in either 6.0 or 6.1 (2D is working fine for me in both cases). The script reaches the point where it emits calibration.should_stop, but then the pupil service never sends back calibration.successful or calibration.failed and nothing happens in unity after that point (no text or scene loaded) - does this mean that the calibration just failed or is something else going on?

Anyone here worworking with the Epson bt300?

Hi @user-407ea7 I can't reproduce your issue but realized that finishing the 3D calibration in pupil capture v1.10 takes a fair amount of time with hmd-eyes v0.61 (+15-30 secs on my machine). this is much longer compared to 2d or to pupil mobile - we will look into it. in the meantime please give it another try and make sure the unity editor stays in focus while you wait for calibration.successful/calibration.failed

Hi @user-6b1b1f The setup didn't change with the new release and unfortunately I can't reproduce your issue. It sounds like that even in 3d you are not connected, that is why pressing c is ignored and the eyes are not published. As you are switching between versions I would suggest to restart 1.10 with default settings, make sure that only one pupil capture is running and check your PupilGazeTracker object in the 2D/3D Calibration Demo (settings like ip and port). Do you get the "Attempting to connect to: ..." log message?

@user-407ea7 in PupilSettings under Calibration/Calibration Type 3D you can find Samples Per Depth. It is probably set to 120, try something around 60-80 - which should already speed up the calibration calculations quite a bit. Be aware that we are ignoring the first samples (Samples To Ignore which is right underneath). I just tested it with 80 samples and ignoring the first 20.

I can confirm the delay when the calibration ends. I'm set tp 120 sample, ignore 20 and it takes longer then before to load the scene when last calibration point is finished. In my case , it's at least 10 seconds, not a big thing but good to know when trying to make everything works. I'm using pupil 1.10 with 3D scene 0.61.

For a good pupil detection, what sensor setting for resolution should I use ?

Default is at 320x240, but more pixels should help algorithm detect better, so I tried upping it to 1280x1024 but pupil detection seems not so good too. I seen playing with lens focus seems to be a lot sensible on the pupil detection too.

My biggest problem right now is not being able to get a good constant pupil tracking for both eyes so 3D marker moves correctly in the scene...

Hi all, we were wondering if anyone could explain the difference between the two functions GetEyePosition and GetEyeGaze. The most apparent thing to us is the "ApplyFrustrumOffset". If we are looking to get the coordinate where the eye is falling in the scene, which of these two functions is the best to use?

Sorry for the second question - in Pupil Capture V1.2, we were able to vary the calibration dots positions through a list. In the current version, this appears to not be possible. Would anyone be able to help us add this functionality to V1.9 so that we can change the dot positions at will?

@fxlange - ok yep, I should have been more patient. Everything working now, thanks

using hmd-eyes, am i able to drop a screen recording of the VR game into pupil player so i can see where they looked in the game?

hi @user-a18fb3 , you can find information regarding the different methods in the gaze section of the developer documentation (https://github.com/pupil-labs/hmd-eyes/blob/master/Developer.md#gaze-topic-available-after-successful-calibration). you probably want to use PupilData._2D.GetEyeGaze or actually the corresponding getters for left, right or center directly (PupilData._2D.LeftEyePosition, PupilData._2D.RightEyePosition or PupilData._2D.GazePosition) .

Be aware that these will return 2d coordinates. so you'll need to project them into the scene (by setting z ,ideally to the 2d calibration depth which is 2 by default). Check out PupilGazeTracker::VisualizeGaze as a reference.

@user-24270f , you can record a screen capture in VR + gaze data as a csv but these are currently not supported by Pupil Player: https://github.com/pupil-labs/hmd-eyes/blob/master/Developer.md#recording-data

additionally you might want to give https://github.com/fx-lange/unity-record-and-repeat a try. I'm currently using it to record and plot gaze data.

Hello, i make the exe of the heatmpa demo, all work. The output is a folder with two file heatmap.mp4 and a heatmaptimestmap.time, it's right or miss any files? How i can play this files?

@fxlange thanks... I guess there's no way to do it without unity? I'm going to have to screen record my non unity VR app, record eye positions, and combine them afterwards in my own software I guess

@fxlange thanks so much, this was very helpful. Is there a way to output the data from a calibration?

Hello, I am posting two photos below to help describe our current problem. The first shows the results of our fixations after calibration at a 0 degree heading direction, and the second shows the results of fixations (after the same calibration at a 144 degree heading direction). Our analysis takes into account pitch, roll, and yaw to plot the 2D points into 3D space as well as FOV.

We were wondering if anyone has ideas on why there is a shift downwards when rotating to 144 degrees (which moves back up when rotating from 144 to 216 degrees). We appreciate any ideas people may have to help solve this problem!

Hello I am pretty new in Unity and pupil VR plugin. I am working on getting the pupil VR plugin work in my unity research project where I should be able to capture the gaze of the users using pupil VR plugin. I am using unity version 2018.2.8.f1 and followed this documentation: https://github.com/pupil-labs/hmd-eyes/blob/master/Developer.md. But unfortunately as soon I import my plugin there are lots of errors. (e.g. unsafe code error) Even if I remove the code I was not able to get the pupil VR plugin work in my unity application. Can anyone please give me a working example on how to integrate the pupil VR plugin in a Unity application? I would really appreciate any help.

Please someone help me with a good walkthrough example.

@user-3c9008 Hi, i am pretty new in unity too. I try several times before to have a working exe, and i'm still having problems. I try to help you, but cosider that I'am a beginner. I import the latest package (https://github.com/pupil-labs/hmd-eyes/releases/tag/v0.61) in a new unity project (version of unity 2017.3 ) and i don't have any errors to import the package. In the tutorial page that you have post , it says "Load and display a 360 degree video based on Unity's 2017.3 implementation. Combined with Pupil, this allows to visualize what the user is looking at." I don't know, but it could be the unity version, i suggest you to install unity 2017.3...

Hello, i make the exe of the heatmpa demo, all work. The output is a folder with two file heatmap.mp4 and a heatmaptimestmap.time, it's right or miss any files? How i can play this files?. I try also to make the exe of Spherical Video demo, pressing r, it saves a folder contain 3 files (timestamp, mp4, csv) the mp4 it' s blue, why?..... It's making me crazy 😓

@user-3c9008 For the "Unsafe code" error, you need to enable the "unsafe code" in your project. To do so, go here: - Open menu "Edit --> Project Settings --> Player" - Open the "Other Settings" section on your right panel on your screen - In the "configuration" section, you will see the option "Allow unsafe code', you need to CHECK that option.

@user-07ff17 thanks a lot. I will definitely try that out. Yeah I also suspect it was a version problem.

@user-8b1528 Wow I didnt knew that. Thanks a lot.

With the Pupil data coming to Unity, can we get the angle at which the eyes are looking at (so each eye has it's own angle) ? If yes, how the angle is represented into the 3D space ?

Thanks !

I figure out that I can probably get the gaze position of each eyes and calculate resulting angle but was wondering if data is already available in the Pupil data coming into Unity.

I seen an ellipsis angle but I would have though to get 2 angles or something like that to know the horizontal and vertical angle in the 3D space...

@user-8b1528 if you use the 3D detector and the eye model fit is sufficient you can access "phi" and "theta" directly. This is in the pupil datum

Ho great ! Thanks !

Hey guys, just a note: When I imported the eye tracking into my existing Unity project and got everything working, the built project would crash when I loaded the calibration scene (using Unity 18.2.3f1), but not in the editor. After some digging in the logs I found that the Shader referenced in FramePublisher line 88 ("Unlit/Texture") will not be included in a built project by default unless it is explicitly referenced (at least in my project/version of unity). You can fix this by adding the shader under Edit -> Project Settings -> Graphics ->Always Included Shaders. Hoping this helps someone else who gets stumped on this problem, but also @mpk maybe this should be included in the hmd-eyes README if it is reproducable?

For DIY project, I exposed and developed black and white film. However, I can still see my screen reflected on my prescription lens. Should I change to a color film to avoid reflection?

thanks a lot for the details @user-23e10d , we will look into it for the next release. might be related to the unity version in use. we are still running and recommending unity 2017.4 (the latest LTS).

@user-8b1528 quick note on your question regarding the angles of the individual eyes: on top of the GazePosition (you might already be using) PupilData._3D offers gaze vectors in 3d: LeftGazeNormal and RightGazeNormal

I can't record (press r) skybox in the scene demo... it's blue.... someone can help me? it's not code and a it's a stupid question but i don't know and I cant go on..... @user-23e10d @fxlange I had the same problem. First, I resolve in the same way of Toona, but I discovered yesterday that in the source code there is a folder call project setting with all the Unity setting. So, i copy and paste the extracted folder of source code in a new folder project of unity and i resolve this problem..

I have resolved my problem. To record skybox pressing r , it' necessary to change the code of recorder.cs set " c.clearFlags = CameraClearFlags.Color;" in c.clearFlags = CameraClearFlags.Skybox.

Hi all, one of our cameras became out of focus and turning the camera around doesn't help, do you have any idea what could help?

@user-b7cf33 Have you tried to turn the focus ring on the camera multiple turn ? When I played with it trying to find the best focus, I seen that you can turn it many turns (I tried 3-4 complete turn at least) then turn it back the other way to understand what make it focus closer or farther...

Hi guys, is there a way to use normal glasses while using hololens? Thanks.

I am currently using a pupil with a world camera, and I want to implement it using Hololens.

Perhaps a DIY solution?

Hi @user-82e954 send us an email to info@pupil-labs.com and we can provide you with an option that will enable you to reuse your eye cameras.

@user-b7cf33 can you send a sample image of the video from the eye camera that you note is defocused to data@pupil-labs.com or post here if you don't mind? also @user-8b1528 offers good suggestion re focus.

@wrp Email sent. Thanks!

Thank you @user-82e954 - we will respond today

@wrp Thank you! I will send you the email today

@user-8b1528 Exactly, I've played with it for a couple of hours, going from one extreme to the other with tiny steps, but it didn't help. All our other cameras from our other eyetrackers work fine but there just seems to be something wrong with this particular one. Thank you for your answer though.

Hello everybody, I am new to networking in Unity. Would anybody share a code snippet on how to get the gaze position as a variable in realtime? I dont understand the subscription stuff. Thanks a lot! Any help is appreciated!

@user-97bf6d Here is a simple code to get the 3D gaze position, if you need 2D, just replace PupilData._3D with the PupilData._2D (I used a 3D text in the scene to display the x/y/z position of the 3D gaze with a 2 digit precision and update that text with the marker position, which you could just get from PupilData._3D.GazePosition):

using System.Collections; using System.Collections.Generic; using UnityEngine;

public class YOUR_SCRIPT_NAME : MonoBehaviour {

public Transform marker;
public TextMesh coord;

// Use this for initialization
void Start () {
    PupilData.calculateMovingAverage = true;
}

void OnEnable()
{
    if (PupilTools.IsConnected)
    {
        PupilGazeTracker.Instance.StartVisualizingGaze();
    }
}

// Update is called once per frame
void Update () {
    if (PupilTools.IsConnected && PupilTools.IsGazing)
    {
        marker.localPosition = PupilData._3D.GazePosition;
        coord.text = marker.localPosition.x.ToString("F2") + " , " + marker.localPosition.y.ToString("F2") + " , " + marker.localPosition.z.ToString("F2");
    }
}

}

Wow. Thanks a lot! I'll give it a try!

Yeah! It's working. Thanks a lot

Cool, glad that I could help you 😉

There is one issue with this. It only works in 2D. When I use 3D, pupil capture will automatically reset the capture mode to 2D so there is no 2D data available in Unity. BTW, is there any way I can save the config of the pupil capture? It seems to go back to default randomly. Thanks for the help

@user-97bf6d Depending on which scene you used for calibration (2D / 3D Calibration scene), it will use the settings from that scene for the data. So if you used the 3D calibration scene, you should have access to the _3D data and if you used the 2D Claibration scene, you should have the _2D data... never tried to access the _2D data while using the 3D calibration scene.

For the Pupil Capture, you need to close the correct window, else you loose your changed settings. You should have 4 windows: - Terminal window with debug info - Pupil Capture World - Pupil capture Eye 0 (if enabled in the config) - Pupil capture Eye 1 (if enabled in the config)

You need to close the "Pupil Capture World" window if you want your settings to be saved. If you close the "Terminal" window, it won't save the settings...

Thanks for the quick answer. So is the order in which I close the windows important or do I have to keep the Terminal window open for the time I want to keep my settings?

If you close the "Terminal" window, it will stop the Pupil Capture program (and not save the settings)... So closing the "Pupil Capture World" window should save your settings and close the Pupil Capture program.

If you happen to close an "Eye" window, Pupil Capture will still run but you will have shut down tracking on that eye... You can start it back from the general settings section of the "Pupil Capture World" window.

Awesome! That clarifies the most important issues.Thanks

@fxlange Hey fxl, @papr told me , that you might able to help me with configuring my own calibraiton scene. For our purpose it would be helpful to have more calibration points around the center and not only at the edges of the FOV. Is there a tutorial I couldnt find or could you give me a hint on where to start? Thanks a lot

hello, do you know the dimension of the fov in pupil lab?

Hello everybody, even after some suggestions for improvement from @papr I still can´t figure a working calibration out. Would anybody mind to share their configurations with me? What works better for you in VR? 2D- or 3D-detection? Do you have to adjust a lot of postprocessing variables? At the moment my main goal is to achieve a clear distinction of the lowest peak in the histogram of the brightness of the eye camera images? Is this the wrong approach? Any help and shared experience is appreciated. Thanks everybody!

hi @user-97bf6d, regarding the calibration points: you can indeed add more and adjust points for the calibration scene. in pupil_plugin/Resources/PupilSettings under Calibration/Calibration Type xD you can find the Vector Depth Radius. Its a list of circles with X describing the depth and Y the radius. All circles are centered around the Center Point.

You can add or adjust the circles but you currently need to restart the scene to see the changes. Try adding a circle with x=1 (depth) and y=0.05 (radius).

Hello, in the csv file recorded how it's calculate timestamp?

@user-07ff17 timestamp is a recording of the time at the time the image is taken. time in between timestamps indicate frequency. pupil_locations.csv is monocular, gaze_locaitons.csv is binocular. It might be useful to deduct the minimum timestamp from all timestamps - your starting point is 0 and relate better to the pupil player timeline.

Hi all, I purchased pupil eye for htc vive. I couldn't get past the calibration ended screen in unity scene. After I have the calibration ended canvas, I usually get no camera rendered error. I will appreciate if anyone can give me a heads up. Thanks

Hi @user-a39804, which scene are you running (and which hmd-eyes version)? After installing the hmd-eyes VR package of the newest release, I recommend running the Market Scene Demo/2D Calibration Demo and see if this works.

@user-14d189 thanks!

it's possibile to add to the script the diameter of the pupil?

I'm running the calibration scene. I will try and see if your method works. Thanks