Hello!
Greetings, I have question about wearing eyeglasses with HMD. From our experiments, the eye image being generally out of focus (blurry) and a lower contrast ratio with eyeglasses, plus reflections of the LEDs from HMD on the eyeglasses make the pupil detection accuracy drop significantly.
Since Pupil-Labs' doc claims detection works with users wearing contact lenses and eyeglasses so wondering if it's limitation of HMD device? Thanks!
To check calibration I am asking participants to look to some dots in a standard layout so that any eye-dot offset is noticeable.
@user-6e3ee7 for hmds tracking works well for contacts. Eye glasses are very difficult unfortunatelty.
Hello! Unfortunately I'm having the same problem as vivi, with only one eye appearing in the unity project, I am using the latest one from GitHub, any suggestions?
@user-64e12b did you try the dev branch with latest pupil capture or service (v096)?
Ah no, I was on master. Will check dev now. And yes v0.96, thank you
Hmm, now I seem to be getting 0's on both of the displays
The service is running, and when I turn on "print msg" and put on the rift I actually see the messages change to confidence:1 and the base_data variable change
Just wishing to run something to show me where I'm looking live
Are eye cameras feeds selected and visible in service or capture?
Yeah, they are both apparent
@user-64e12b - you should be seeing feedback in the plugin that you are connected and if both eyes processes are detected, then you will see two green icons in the plugin GUI
Hi hmd-eyes channel! I'm trying to get the pupil headset to work with the Epson Moverio. Has anyone tried this before? Right now, when I run it on the Moverio, it only shows a tiny green circle in the left corner. How would I know if both eyes were detected / calibrated?
@wrp
@user-64e12b - it would appear that both eye processes are detected/connected
On the plugin's code, I see that OnGazePacket is activated, and even though the _pupilDataDict has values like "confidence = 0.97, norm_pos = [0.49, 0.66]" the values returned in Pupil.values.GazePoint3D are still 0,0,0
Maybe a calibration issue?
I'm just trying to actually see the point that I'm looking at on the screen, to know if the device is actually calibrated..
@user-64e12b you have to calibrate to either 2D or 3D
you can set it up under the settings
sorry the documentation is not complete yet
@user-64e12b is it possible that you have not switchet to 3D mode for calibration?
@user-5ca684 I did, and I just performed another calibration but the results are the same I'm afraid. (and when support is nice and available the lack of documentation isnt that bad :))
In the code I also see CurrentCalibrationMode = _3D
while debugging i mean
sure
im pulling up the project now to see what can be the problem, but it seems to me that for some reason you are not getting the data from the service
Okay thanks. The weird thing is that when printing messages I can actually see stuff change, like confidence, and other variables when I assume are from the service
no its not wierd
if the 3D calibration is not complete or failed you will still get some data in
its just not 3D
the IPC bacbone is transmitting the pupil data either way
are you using sercive or capture?
currently capture
and I see, that makes sense now
could you check in capture if its on 3D mode?
detection & mapping mode?
yes
yes it is on 3d
hmm
Should I be seeing 3 windows?
One capture, one for each eye?
yes
oh and one terminal
that all sounds about right
yep
im going to check it right now on my machine locally, because lately I have tested it mainly with remote mode, however that shouldnt make a difference
except of course if there is an issue with the binary version
Does it make sense that I have to click "detect eye 1, detect eye 2" every time? the windows for the eyes don't open automatically
not that I really mind, just wondering if it might indicate something
hmm, no they should
in your version do you have an Autorun Pupil Service under Settings>>pupil app>>
?
yep
I toggled it on
try it off and run the service manually
I mean this is probably not going to solve the data issue, but this did bring the eye detection up for me just now
on "local" or "remote"?
im trying on local, because I see you are on local too
yeah
well this way I'm not getting "connected" even
that is kind of wierd
have you made sure that the Service port matches the one in capture?
okay im getting the data
okay a couple of things to check:
(Im taking a note to fix the issue with local automatic start, I believe with service it would work. I think the reason to this problem is that the request happens before your machine can bring up the Pupil Capture, so im gonna have to loop until the eyeprocesses are on)
Okay, gotta go for a bit but will do and update on results. Thank you!
allright, just send me a PM as well please
! I'm trying to get the pupil headset to work with the Epson Moverio. Has anyone tried this before? Right now, when I run it on the Moverio, it only shows a tiny green circle in the left corner. How would I know if both eyes were detected / calibrated?
@here - just a friendly reminder that Pupil software on Windows is only for Windows 10 (while you may get things up and running on Windows 8 - it is really not advisable to use anything other than Windows 10)
@user-efb835 - apologies for not responding earlier - could you please describe your setup a bit more? Are you using the Pupil Labs BT-200 or BT-300 add-on?
Universal Windows Platform (UWP) is not yet supported right? NetMQ seems to support UWP since a couple of days (May 6th) https://github.com/zeromq/netmq/commits/master . Are you planning to upgrade to this version soon?
I'm using Windows 10 on an Alienware machine. I have the Pupil Labs BT-200 and do not have the add on. I currently have the pupil headset plugged into the laptop and I'm trying to get the Unity Integration program to run without the headset - just to see if the ports work. I have Pupil Capture running on another machine with the Pupil Remote plugin. Right now, the errors I get in Unity on my Alienware machine have to do with Unity not being able to connect to the pupil remote server.
Both devices are on the same wifi network
I am also using Unity 5.5.1f1
@user-efb835 could you tell me the exact error message?
Dear @user-efb835 are you using the newest version of the unity integration?
And did you set the IP address of the computer running Pupil Capture in the unity integration?
Yes I set the IP address and Port in the Pupil Listener attribute
The error is Failed to connect the server; Context Terminate; Quit the Thread
@user-efb835 hmm that doesnt exactly sound right
I downloaded the project off of github's hmd-eyes today so I think I have the most recent verge
there should be a PupilGaze GameObject
yes, that is the newest version
It works if I have both the pupil capture and unity project on the same computer
I'm using the unity_integration project now; not the unity_integration_callibration
ahh I see
I'm using the IP and port from Pupil Remote
I have not updated the unity_integration at all
all the new work has been done under the unity_integration_calibration
try to connect with that
okay
so there should be a PupilGaze GameObject >>Settings>>pupil app
unity integration callibration works when the pupil capture is on the same computer as the unity program. Trying now on separate devices
Still same error
could you please copy paste the error
?
Just to clarify - are you using the dev branch?
Nope - master
Failed to connect the server; PupilGazeTracker:NetMQClient()
@user-efb835 could you please check out dev branch?
And try the unity integration calibration plugin
okay
I get a bunch of errors b/c i don't have an HMD headset plugged in. Is it possible to test the remote capabilities first?
yes it should be
I get Type "UnityEngnie.Camera" does not contain a definition for 'allowHDR' and no extension method can be found. Are you missing an assembly reference?
also "does not contain a definition for 'allowMSAA'
which version of Unity are you running?
this might be a Unity 5.6+ feature on the Cameras, if you do not want to update your Unity now could you just comment these two lines out? Later I am going to make this dependent on the Unity version number.
I am using Unity 5.5.1f1
im sorry for the trouble, didnt know 5.5 didnt have these features
No worries! Now I get OpenVR failed initialization with error code VRInitError_Init_HmdNotFound
that is fine
but the other two lines might prevent you from running
Okay - now I get NullReferenceException: PupilGazeTracker.OnDestroy()
without running?
when I start to run
or rather stop running
that makes more sense
Also - I don't get any Logs about being connected to server, etc
I will look into it!
nothing happens when I press play - thank you!
I seriously appreciate your help!
you might have to set the AutoRun Pupil Service under the PupilGaze
:
Local / Remote
if Remote, the IP address
you can choose to have AutoRun Pupil Service app or not, if you choose to do it manually, you can start / stop it with the two buttons under the toggle
I see - hold on
Still no luck - I've set the IP and reset the port on capture appropriately
how about the autorun?
yep that is set as well
okay, set to ON?
have you set it ON, before running the application or after? if you toggle it while running, it will take no effect
I have it set to on before the app runs
okay, and so nothing happens when pressing play in Unity
no error messages?
you are sure that its set to Remote?
the last thing I can think of is some kind of Firewall issue that blocks the connection
if you dont mind, could you PM me screenshots of your setup?
Hello! How do i apply the project "unity_integration_calibration" on hololens ?
Hello everyone! I'm newbie in vr and pupil development. I've carefully read all docs but still haven't a general vision how to sort out all of this staff. I'll be really appreciate for any help and tips! The background: 1) I have HTC VIve and Pupil's HTC Vive add-on. Add-on is connected to separate USB port beside Vive. The idea: 2) My goal is to get gaze direction/coordinates to some static picture in HTC Vive (it's first stage of experiment) The experiment: 3) I'm able to run Pupil-capture and see 2 window for 2 eye-cameras. But world camera doesn't show anything and in the list of capture source I see only 2 eye-cameras and several unknown source (is it vive's camera or my laptap camera or what?) The issue: 4) According to docs, first of all I have to calibrate eye-cameras with world camera. How to impelemt that with HTC Vive + Pupil's HTC Vive add-on? Is it possible on Windows 10 or have I setup all environment under Linux with hmd-eye blocks?
It's a common observe of my stuck ) I didn't dig deeply into development of my own plugin or something like this. I need to sort of that general issues first. Thanks in advance!
@user-f7028f Could you tell me which version of the unity_integration_calibration are you running?
@user-3ad5b4 I do not have experience with the hololens in Unity, but the plugin should work as long as you have pupil cameras connected to either local or on a remote machine
I doesn't running any 3rd-party apps but just pupil capture, i can't find any docs about it. Have I run some additional apps to make "connection" of HTC Vive and Pupil's add-on?
@user-3ad5b4 the documentation for the plugin needs a lot of work yet, but for now I can walk you through if you want
@user-f7028f Im sorry I might have misunderstood you
as I said, I'm very begginer in this area and have to learn from the very beggining how it works )
@user-f7028f I thought you are about to use the unity_integration_calibration
Hi all, I have some problems with the unity plugin. I am running my game and the data is stream quite well. But after a few minutes (it varies) the data stream just stops. the capture app is still working, the eye cameras are still working and getting good tracking, the game in unity still works, but no data is getting to the game. I basically know of this only when I stop the game, the unity app is not responding, and the eye processes are still on. I also collect the data for later use, and I see that the time course is much shorter then the game was. I tried to look at the capture log in the settings directory, but no notification marks the time it stopped. is this something that you know, or have seen and can tell me something about it? it is really crucial that I will be able to track for about 15 minutes straight. thanks
@user-1486c3 first of all, I want to make sure that you are using the correct version, which is in the dev branch (unity_integration_calibration)
@user-1486c3 are you recording with the plugin or you are just simply running it?
I am not sure what you mean by recording, but I am subscribed to the gaze messages through unity, and save the data that comes in in a csv file. I am not recording with the capture. I am not sure what version it is, as I have downloaded the plugin and made some changes to it, becuase the _pupilData object didn't receive all the data. the changes are these:
just added an envelop class so that the json string will be loaded properly, and not only the norm_pos
I see, well there is a development in the unity_integration_calibration, I believe the one you are using is not where the new work has been done
you could try that out, but if you have already made a lot of work in the customization, I understand that might be inconvinant
@here - lots of work has been done in the dev branch - https://github.com/pupil-labs/hmd-eyes/tree/dev/unity_integration_calibration
we plan to merge this to master after working out major issues and testing
feel free to test it out and give us feedback here
thanks, I will look into it. is there a way to have a look at the dev branch without using git? another thing, I have been using the capture app because while using pupil service, the configurations would reset every time i start the eye process. Is there a way to save the service settings?
@user-1486c3 you can just download the project as .zip
this I did, and this version is what I used, except for the envelop class that i added. but it says that it was last updated 2 months ago.
@user-5ca684 hi! I've made some steps to achive my goals ) I switched from Windows to Linux (Ubuntu 16.04), installed lates Unity3d, Pupil Capture. Now I follow hmd-eyes instruction and probably all going fine (PupilGaze shows eye position and etc) but I can't figure out whether should I see something on HTC Vive headset or not? Now I see there some part of Ubuntu desktop's background
@user-f7028f To be honest I have only tried local connection with Unity on Win10. I have only used Ubuntu for running the Pupil Capture while connecting to it via network. About Vive and Unity its another question, in order to use the Htc Vive in Unity you will need to have a plugin active, by default we are using the OpenVR plugin. Its should be included in the project we provide on git
yes, I added OpenVR plugin into project settings --> player ---> other
Am I right that thats plugin must translate Unity scene to Vive screen?
@user-5ca684 Hololens run an app on its own CPU instead of CPU on a local PC. So I plan to build a windows store app using unity and Visual studio. After building in unity, I get a .sln profile. Then I compile and release the .sln profile to hololens using Visual Studio. When I build a .sln profile in Unity, I choose the platform Windows Store.
But it fails with the error as the picture.
Do you have any idea on this error?
@user-5ca684 why so necessary to run Pupil Capture under Linux or MacOS? Why Windows isn't suggested? I asked this coz I failed to run calibration scene under Ubuntu because of Unity's SteamVR assets doesn't support Linux! >__<
You can run it on windows just fine. I dont think thats the issue.
@user-f7028f it is just an option! I have used two computers, one with Win10+Unity. the other one with Ubuntu+Pupil Capture
@user-5ca684 also be aware, that Pupil Capture is independent of your unity envorinment. They communicate over the network. There is no need to run both on the same operating system.
@user-f7028f I have done it so, because for some reason I could not get the git source version run on win 10
eh, I meant @user-f7028f , sorry
no worries 😄
I got it! Now I'm trying to run it under Windows. Bytheway, should I see colibration scene in Vive when I'll run it from Unity? Should I import SteamVR assets or OpenVR plugin in your project is enough for it? )
@user-f7028f As I mentioned the project should work as it is out of the box with the OpenVR plug that is included
@user-f7028f you might have to add it though in the Player Settings
@user-f7028f + yes, you should be able to see the calibration through Vive
If you do not see anything in your Vive, that there must be some issue with the way your Vive is installed on your system
Thanks! I worry because under Ubuntu callibration scene running from the box with checked OpenVR plugin didn't show anything in Vive (
@user-3ad5b4 The plugin uses Threading for the networking and the Recording. I have never tried to deploy as a Store App /Hololens . It seems like that there is a problem with Threading when you are trying to build it as a store App
@user-f7028f Frankly I have never used Unity on Linux
ok ) will provide you with my success\fail under Windows )
@user-f7028f I dont know much about Vive support on Linux, but you shouldnt worry if you are running it on win10 now
@user-f7028f thank you, thats great!
It's alive!!!! It's alive!!!! 😆
awesome @user-f7028f
Hello everyone. I was sent here by the support to ask my question. I'm using Windows 10 and the HTC Vive as my setup. Also, I had installed Unity 3D Version 5.4.0f3 and 5.6 seperately. I was able to run the Master-Branch Unity Scene on Unity Version 5.4.0f3 and the Calibration was successful. But I wanted to get information about the Axis, in perticular the z-Axis, and was told to use the Developer-Branch Scene. So I did and started the Scene. But as soon as the Trackers were connected, the marker for the left eye disappeared and the other markers remained static. Surprisingly, I could finish the Calibration process and got no error messages, neither in Unity nor in the Calibration App. But the blue marker was still missing and the other markers weren't moving at all, after the Calibration was finished. So I just wanted to ask, do you know, why this behaviour occured?
@user-a2fe55 As you said the dev branch contains the latest development progress. Please note that there are two calibraiton modes you can choose from : 2D and 3D. once you press the calibration button in 2D mode the markers should switch to a white marker that slowly dissapears with your gaze. After one is gone comes the next marker position. If this is not the behaviour you experience than something is going wrong
@user-5ca684 Thanks for your fast response. In that case, everything seems to work as it should. But I have another question. Is it possible for a simple Windwos Update to have an effect on the drivers, which are important for the Pupil Labs App?
@user-a2fe55 Frankly im not an expert on the Pupil App related questions nor on the driver of the camera, but I dont think it should do any harm. But... I guess you can always try to reinstall the drivers, re-download the latest binary etc. .. I heard regarding another case that updating to the latest binary actually solved some issues
@user-5ca684 Thank you, very much. This sums it up for me, right now
@user-a2fe55 - yes it is possible that a Windows update will clear user installed drivers. Please check the device manager before re-installing drivers to see if drivers are already installed when cameras are connected in the libusbK category. You may want to check "show hidden devices" just to see what is installed
@user-5ca684 Thank you. But I wonder if there will be solutions to deploying a Windows Store App.
Hey Guys, I am trying to integarte the Pupil Eye with Microsoft Hololens. Has anyone tried doing it ?
Hi @user-4a3b48 - I think we were in touch via email and I provided a response regarding the route some researchers currently take for Pupil + HoloLens development
I will post this information below here again for reference.
Researchers that are currently using the HoloLens have a setup that uses a separate socket server that talks through ZMQ Pupil Caputre and communicates using low-level UDP commands to the HoloLens. This is required AFAIK, because HoloLens will not allow one to use ZMQ on the HoloLens (unless there have been updates). Using this method one can send a command to the HoloLens so that it shows a calibration marker with durations for each sample site. The socket server sends the calibration timings using ZMQ to Pupil software. With this setup you can send a record command to your HoloLens setup and to start a synchronized recording on Pupil.
yeah the mail came after that..thanks for the info
@user-3ad5b4 I am looking into the deployment on UWP & especially hololens, there is a serious Threading issue with UWP deployment. I do have a lead, Im going to get back to you if I know more!
Hi! I'm investigating scripts of unity_integration_calibration and making some development for my own purposes. And I'm interested in PupilData3D class. I understand a meanings of most of class members but I would like to be 100% sure what they exact mean. Can you detailize these members?
Hi Everyone!
We have a need to plan our own camera mounts for Pupil's eye cameras as we are using Lumus AR glasses. We are trying to find some materials on the best angles and distance the cameras should be located at, in order to optimize the eye tracking.
Currently right&left movements works great, our issue is when gazing up&down.
Thanks!
Itay
@user-f7028f I am going to get back to you shortly!
@user-5ca684 Hi! I debug this parameters for one tracked eye and see that Diameter, Theta, Phi. Diameter_3D is always 0 whatever eye detecting well or not. Looking at the code I see that only norm_pos [0] and [1] is using to calculate avarage postion of eye's gaze. My goal is to obtain eye coordinates(x, y, z) relative to scene visible in hmd.
@user-f7028f the data received is depending on your calibration
so if you set the Calibration for 2D then after Pupil Service calibrates you will receive the 2D data
if you want to use the relative position after calibration, you can use
GetEyeGaze2D(GazeSource.BothEyes).x GetEyeGaze2D(GazeSource.BothEyes).y
this is NOT in world space
and is a normalized 0-1 value
Hi, I am trying to get calibration to work on my Oculus DK2. I attempting to use the pupil-labs/hmd-eye git repository, but I am pretty confused about what I need to run to make things work. So far, I have tried running Calibration.unity in Unity with Pupil Capture open. I am able to get a blue screen with to show up on my computer and on the oculus, but then nothing else happens. I have also tried playing around with running the python code in hmd_calibration, but I don't really see what's going on there. Would anyone have any more specific instructions on how to get a basic screen target calibration up and running on the Oculus? Thanks!
Hi, I installed the new pupil version 0.9.10 on a different windows computer (because working with unity and pupil in the same time lead to serious frame-drops of the Vive...). when I run pupil capture the frame rate of eyetracking does not go above ~65 for 1 camera and ~55 to the other, even if only one camera is recording. Its a strong computer, but it doesn't have USB3. the CPU usage is 60% at worst, and no more then 50% memory usage. I looked at the USB bandwidth used by the cameras, and it is fixed on 5% each, even if only 1 is recording. is there anything I can do, or it just wont work with USB2? did anyone else face this problem before?
@user-5ca684 thanks for this. Acctually I don't yet realize how to obtain data about eye coordinates relative to camera? PupilData3D contains only gaze coordinates but no info about detected eye. Have I switch on something in PupilCapture app to get this kind of info in unity_integration_calibration?
Additional note. In opened Debug window of Pupil Detector 3D I found text about Eyeball center and diameter in mm. Is it relevant info I search?
That what I see in Debug window
Hello all, Does the Hololens addon only work on Mac or Linux?
I believe that Hololens is for windows only
Am I wrong?
Microsoft built it but I'm not sure if other OS s can use it or not I was reading hmd-eyes' read me file in github and it only mentioned Mac and Linux.
Hummmm, I am not sure
I think that part is only for HTC vive probably.
Hey there. I have another question. Right now,
we are trying to calculate the z-coordinates with the information from the calculated 3D-Eye-Model. Or in other words, we want to get both line of sights from the virtual 3D-Eye-Models to get the coordinates for the gaze. Is there a way to access this information from pupil capture in the HMD-Unity scene (dev-branch)?
[email removed] if you run 3d calibration you should get a 3d gaze coordinate that contains the depth (magnitude of the vector)
@mpk Thank you, for your quick response. The thing is, we know about this and we are very happy about it. But we want to check the accuracy of the z-Axis for a university project. So, we figured, in order to check this, we get the intersection point of both eyes manually and build a seperate unity project in which we put objects with known distances to the camera. But as we looked deeper into the scripts, we got the impression that the Unity Scene can get the gaze-coordinates via json commands from the Pupil Labs App. That's why we wanted to know, if it's possible to get this information in Unity. I hope, I could clarify my objective.