I think we can help better if you upload a full recording. This might just be a calbiration or pupil detection setup issue.
Ok . I try to upload the full recording . Can you give me a e-mail to share the folder ?
Ok thanks. I will send you a email to share when , the folder is full updated
Good morning, @mpk . Do you have a few min to help me optimize our setup?
To recap from yesterday: I have a eye/head cam running pupil mobile with 'transcode to h.264' running on the world cam. That is streaming via local router to pupil capture on a mac. We calibrate that tracker using manual calibration. i have time sync running. Then I have a second tracker with just the world cam. That is running pupil mobile on an android with 'transcode to h.264' running. That is streaming to a second iteration of pupil capture on the mac. Time sync is also running there, and it appears that pupil capture has set this one to be the master clock.
You also mentioned pupil groups yesterday, but I'm not sure what to do with that (if anything)
Pupil Groups is mostly used to send notifications to multiple pupil capture instances at once
Our approach is to setup world/eye cam 1 on a parent. We hit record on pupil mobile and then hit record on pupil capture and do a manual calibration. Then we setup the second world cam on a child. We adjust the angle and hit record on pupil mobile and on the second version of capture.
You could use the Remote Recording plugin to trigger the pupil mobile recording
Thanks. Any reason to use pupil groups with our setup?
@mpk i sent you the folder .
If you want to start all recordings approximately at the same time, you could use Groups. But it is not necessary if you want to start them one by one ( @user-78dc8f )
ok. thanks. sounds like our approach is fine then? I realize we are getting redundant recordings, but sometimes the streamed data lags a bit, so seems like a good idea to have redundancy.
would you recommend the 2d or 3d algorithm for the eye cam?
Normally, there is no reason to record on Capture as well
recording on pupil mobile should be sufficient
Especially since we have offline pupil detection and calibration now
We were doing that because sometimes mobile freezes up. And since we are not able to monitor mobile on each participant, we felt this was useful redundancy
Oh, I see. What phone do you use and how long are your recordings on average? You are right, that this is a useful redundancy then
our recordings last about 10 min. We had a sense maybe this occurred as data accumulated on the phone. But this might have been an older version of mobile that was buggy (we've been at this for several months now)
Generally I would recommend 3d pupil detection.
I understand
Hi guys. I'm a little confused with the settings "target frame rate for H.264 transcoding" and "frame rate". The first defaults to 30 while the second defaults to 15. That seems odd to me, I'd expect that the frame rate should be a multiple of (or the same as) what is used for transcoding.
(I'm using pupil mobile)
Hey @user-fea4ed The frame rate
belongs to the native camera frame rate. Our eye cameras yield a MJPEG stream. This value can be as high as 120 Hz, depending on the resolution. Default is 120 Hz for 640x480.
The target frame rate for H264 transcoding
is the frame rate with which the hardware h264 transcoder tries to convert the images. The maximum transcoding framerate depends on your phones hardware transcoder.
It is called target
frame rate since it is not guaranteed that the transcoder is able to reach it.
We recommend to stream the eye cameras with a high framerate as a low resolution MJPEG stream and the world camera as a high resolution, low frame rate (30 Hz) h264-transcoded video
@papr Thanks for the detailed explanation. So I should set frame rate
to at least 30 when I want it to transcode with 30?
Not necessarly. But be aware that the transcoding framerate is the upper bound for the resulting video stream's frame rate. If the eye cams framerate is lower than the transcoding framerate, the resulting stream will have the eye cams framerate.
Also, the actual camera framerates are variable, i.e. they comply to the settings on average but might actually be a bit lower or higher than set.
This is the reason why each pupil datum has its own timestamp.
@papr That makes sense. Now, when I set frame rate to 60 and transcoding to 30, I'll end up with a video of (about) 30 FPS and pupil data of (about) 60 FPS, is that how it works? Let me know when I'm too demanding btw.
Do not worry about asking questions. That is what this channel is for.
"Pupil data" in the sense of pupil positions is not calculated on the phone.
This is only possible by streaming the eye videos to pupil capture or to run offline pupil detection in pupil player after the recording is done.
Example: 2x eye videos: mjpeg 120 hz, 1x world video: mjpeg 60hz, h264 transcoding enabled, transcoding framerate 30hz --> results in 2x 120hz mjpeg eye videos, 1x 30hz h264 world video
Right, I mixed the world and eye cams there, sorry. Is there a reason to pick 60hz for the world video when you transcode it with 30hz?
No. If transcoding is enabled, the transcoded video will be recorded.
This was not the case in earlier versions. Then you could save a 60Hz mjpeg video but stream a 30 hz h264 video. We simplified the settings such that the streamed video is also the video that is stored.
You mean, save as in save on Smartphone and stream as in stream to pupil capture/a laptop?
exactly.
Streaming and storing can happen independently of each other, btw.
@papr Okay, that already shed quite some light, thanks! The last thing I'm currently confused about is the bandwidth factor: From what I can gather from the docs, this should influence how high I can set resolution/FPS? Like 300% means this camera can (potentially) utilize three times more of the USB port than with 0%?
It is recommended to leave this option as it is. It influences how much bandwith is reserved for the different cameras. the defaults are set such that the eye cameras can deliver higher framerates than the world cam
Alright, thanks. One last thing: I noticed that the videos from world and eye cams sometimes get really inconsistent w.r.t. timing (e.g. a minute wall clock time passing by within say 30 seconds video time). The pupil player's synchronizing should take care of that, right? (I gotta wait for 0.9.13 for Windows to test it)
Yes. player sync will take care of that.
Wonderful. Thanks a lot and have a nice day! 🙂
To you too!
@everyone Mac bundles v0.9.13 have been fixed and uploaded!
Hey, Quick question. I see that you can access raw data and pupil_position.csv contains the position of the eye, but is there a file in the raw data that has images of the eye? If not, is there a way to capture images of the eye in the process?
@user-dbbc26 you can record the eye videos in Pupil Capture. In the GUI sidebar go to the Recorder menu and switch on record eye
Thank you.
@user-dbbc26 I would recommend recording eye videos especially now that Pupil Player has offline pupil detection and calibration features.
Thank you, I don't have the unit yet, I'm just doing research before I invest in one.
So far it looks promising 😃
@user-dbbc26 thanks for the feedback! Let us know if you have any other questions 😄
@papr I responded to your questions regarding the matlab commits
@user-ed537d Great, thank you. I will have a look at it.
Guys I am running the latest version (yesterday's version) of pupil_capture on MacOS and on the "Remote" plugin, I cannot choose which interface I want to listen to. I can either listen on my primary interface (which is the WAN interface), but not on my secondary interface. If I uncheck the "Use primary network interface" option, will it listen on all interfaces?
if both interfaces have different ips you should be able to set it via the ip.
interfaces are not listed direcly by capture.
they do have different IP's
ah ok so I must open an IP plugin or something? kk
then you should be able to use the one of choice by setting it in pupil remote.
eer... pupil remote has no option for IP as far as I can see...
ah manually
@mpk got it thank you!
@mpk I had the same problem with the IP/port in Unity side. These "text labels" don't look like they are editable. It's a usability thing :
: ) thank you
ok. good feedback!
@mpk as soon as I change the IP address in that field, the system automatically restarts the network interface or should I restart pupil_capture?
it restarts the interface automatically.
I can not open the recordings . like this
Did you drag the recording onto the window?
yes,i drag the recording onto the window
ok. I will try to reproduce the error
@wrp could this be a permissions problem?
@papr unicode...?
thank you 😬
the path does not include any unicode
ok - seems that changing permissions resolved this @user-abff35 ?
please let us know what resovled the issue you saw so that others might learn from this
I think the thank you was meant for trying to reproduce the error
= = ,I can not sovle the issue
Can you send the info.csv file?
@user-abff35 could you upload your info.csv here\: https://gist.github.com/
Thanks @papr
gist is better 😄
player_methods does not check if the exception is a unicode error or an ioerror. A unicode error is likely
yes, that was my thought
yep, seems like unicode error...
@user-abff35 can you try replacing your existing info.csv with this one 👆
ok, player methods try to load the file with utf-8 encoding. the error must have happend when storing it
it worked 😀 thank you
@wrp thank you
welcome 🙏
@papr could you make an issue for this if there is not already in the pupil repo?
Hello guys,
I am having a problem with my Pupil headset (binocular with high speed world camera) that I bought about a month ago : it is not recognized by my computer : Microsoft Surface with Windows 10 and Pupil Capture 0.9.12.2. At launch, i get world - [ERROR] video_capture.uvc_backend: Init failed. Capture is started in ghost mode. No images will be supplied.
. Same for eye0 and eye1, and same when I try it on another Windows 7 computer. On both computer it displayed the images correctly before.
I erased the pupil_capture_settings folder in my documents, re-extracted the zip folder that contains the software, and re-run the drivers installer (although it did not change anything as they are already installed. How can i uninstall the drivers to re install them ?), rebooted : none of these changed anything. However, when Capture is running, I feel that the cameras are hot, but I don't think it is as hot as when it was working (if this information is useful).
Any solution or idea on the cause ?
usually this is when windows updates and removes our drivers.
you will have to repeat the driver install procedure outlined in our dcs
more percicely: https://docs.pupil-labs.com/#troubleshooting
this part explains how to use our drivers again when windows removed them in an update.
Ah, I didn't see the troubleshooting part of the doc, sorry. I will try this, however i doubt it is caused by an update of windows : my Surface is not connected to any network...
do the cameras show up the with correct names in the dropdown menu in the sidebars of Pupil Capture or are they called unknown device
?
@user-8c0069 Do I understand it correctly that has never worked on the surface?
@mpk yes, there was 3 "unknown" in the sidebar
then its definelty drivers.
@papr No, it already worked on the Surface, until today.
we are working on gettings rid of this behaiviour but its really windows that is replacing our libusbk drivers.
So I uninstalled the drivers as indicated in the doc, rebooted, but I can't install them again : the driver installer instantaneouly switches to the "Finished" window (it appears they are still installed for it ?), although there are no libusbk
in the device manager (even with hidden devices shown)...
I redownloaded the drivers archive from the link on the doc, and tried with this one : no change
@wrp is the expert on fixing this. He is already out (different timezone) he should ready this in a few hours and can assist.
Ok, thank you.
@wrp FYI, i am in the CET timezone (UTC+2), so I won't be answering until tomorrow, 10:00 UTC+2 ...
So I tried with another computer (on which it also previously worked) with Windows 7 and Capture 0.9.12 : not working as well. (I know you don't support W7, it is just for testing). And I double checked : the surface did not get any update since the last time it worked (two days ago), and I did not install any software or driver on it since.
@user-8c0069 This very much looks like driver issues. You can get in touch via email at info@pupil-labs.com and I can set up a time to remote debug/setup - which should take no more than 15 minutes.
@user-8c0069 please also note that Pupil is not supported on Windows v7. We only support Pupil on Windows 10.
Side note - we are actively working on improving camera drivers for Windows.
May I ask a question? My team recently focused on capturing the attention of the human eye in a picture/video. So, can I transfer images from the HTC vive to the ‘world’ camera? I would be very grateful if you could give me some advice。
@user-abff35 the unity3d plugin should enable you to record what the participant is seeing (e.g. what is displayed to them within the unity3d environment)
@wrp OK,thank you. if I want get the gaze point in the picture. Should I use 2D or 3D ?
@user-abff35 please experiment with the detection and mapping method that works best for you. I would suggest starting with the 3d mode
@wrp thank 😬 you
@wrp Ok, thank you, I'll contact you via email. Regarding W7, I know you don't support it, I just tried to install it and it worked well (at least it detected the cameras, although it crashes frequently) until yesterday, so I just thought it might be relevant to say. If I can't get it to work on W7, no problem, I just need it on the Surface.
ok - thanks for the updates @user-8c0069 we will be on the lookout for your mail
Howdy, is there any way to get a menu back in Pupil Capture after clicking 'close'? The menus just disappear.
Look for open plugin s in the sidebar
I don't see a side bar in Pupil Capture.
Aha I see it now
Does the DK2 add on cups also have a world camera (i.e. the virtual world displayed on the DK2). In pupil capture I can see the eye cameras, but not the world camera. Any suggestions?
@user-65df3f for hmds I would recommend the hmds eye repo as well as.our unity3d plugin
Thanks! Know of any resources to use pupil labs with the vizard software?
No I don't think so what language is vizard in?
Python - can I only use pupil with Unity?
No. It can be used with more or less any framework. We just don't have parts build for that.
Sinces it's Python that makes is very easy. Have a look at the Python script that does a dummy calibration. Use that to get started in vizard. Post gists if you work and we can give feedback!
Sounds good, thanks for the quick responses, I will keep you guys updated
can you link me to the post that shows the dummy calibration?
this would be that script: https://github.com/pupil-labs/hmd-eyes/blob/master/hmd_calibration/hmd_calibration_client.py
Hi, I am having issues importing the zmq package. I downloaded the correct .whl file from the pupil developer docs and installed the package using the command line. However, when I run my code using my environment (vizard) I cannot import zmq_tools or hmd_calibration because I get the error that there is no module named zmq. Any suggestions?
Can the pupil be used on laptop, ipad, and iphone for eye tracking, for flat display?
I hope to get a definite answer
@xu_bigeye#6702 Pupil can run on Android Phones and Mac/Linx/Windows Laptops. You can track any screen and device with it. Since this is a mobile eye tracker it works differently than remote systems.
@Raz#7529 Maybe Vizard has its ons version of pyhton? You might have to install it using that interpreter.
Why does the ‘Main Camera‘ have an initial ’Position‘🙃
in unity_pupil_plugin
@user-ed537d @papr any look working with the Matlab code? I have a few questions
Hi, did anyone encounter problems with pupil mobile recordings? I wanted to test it out, recorded video, but when i launch player it tries to convert data and crashes
Can you share the recording? I'll test it out.
sure
I've got to say that i'm using self made tracker with off-the-shelf cameras
recording
I really appreciate you help!
That's crucial info. You will have to manually rename some files. I'll follow up with instructions tomorrow
well i've renamed cameras in the app (to eye and world respectively) now player loads world camera, but does not "see" eye camera
ah! eye camera should be named as eye0
thanks for the tip on renaming files!
Is pupil compatible with python 2.7?
AFAIR they've migrated to python3
so not possible to use with python 2.7? any plugins to convert between the two, or something along those lines?
you can look in github for a commit that precedes the one when python3 branch was merged
thanks man, appreciate the quick responses
Is someone out there?
I'm a German student, currently working with the pupil eyetracker for HTC Vive. We want to implement a HUD that is controlled with ones eyes. Right now we're having trouble mapping the eye data to locations within our Unity-scene. Our current attempt is to perform a raycast, where there start vector is the main camera and the direction is the detected gaze position, but with most of our attempts the raycast goes of to the wrong direction (behind the camera). We've come somewhat close by trying to manually calculate a camera-to-world transformation matrix, but even then it was still too far from an acceptable solution. Any ideas?
Going home now, it's late. Will ask again tomorrow
@user-e14fbd you're talking about gazepoint3d right? instead of using gazepoint3d I used new Vector3(-gazepoint3d.x, -gazepoint3d.y, gazepoint3d.z) and it works
that's how the script positions the gaze marker also
so I suppose this should be correct
Thanks alot for the reply! We're still really close to desperation an appreciate any advice. Will test that one out in a second
GazePoint3D is (0,0,0) for us :/
@user-e14fbd what did you use for the detected gaze position then?
I also used to get 0,0,0 for everything
@user-e14fbd have you had a look at the unity plugin? In general I would recommend using a polynomial mapping approach instead of raytracing.
We tried GetEyeGaze2D(GazeSource.BothEyes) and CalibrationMarkers.Where(p => p.nme == "gaze").ToList()[0]
@user-e14fbd actually I think this is a unity related question. I would recommend pinging @user-5ca684 and moving this conversation to our hmd-eyes channel.
@mpk can you please elaborate? maybe that would help improving my work also
@mpk We're working with the unity plugin. Haven't looked into a polynomial mapping approach though. Any suggestions where to start investigation that?
You are trying to map gaze onto a vitrual 2d plane in the hmd?
Then the 2d calibration is best. it will establish a polynomial that maps pupil data onto that coordinate system.
the general prpcedure works like this:
@mpk right now we're trying to just detect if the user is looking at a cube for initial testing. In the end we want to privide an eye-controlled HUD in VR
HUD is definetly a use case for 2d mode.
the general procedure would be like this.
We're working with 2d-mode already 😃
present a calibration stimulus in the hud to the user and report the location in the hud coordinate system to pupil service/capture. After calibration gaze will be repoted in the hud coordiate system.
@user-5ca684 is the one that writes the unity plugin. He can give more specific info to the unity side.
@mpk if that's the case, shouldn't calibration also offer a point in world space so that a raycast from camera to that point will result in hitting whatever one looks at?
there are no 2d points in the world. We need to define a plane to establish that.
if you calibrate with 2d we cannot report any 3d data.
Pupil does not even report 3d points in that case. I think what you are seeing is that the unity structure that contains the gaze data has 3d fields but they are zeroed.
@user-5ca684 can you confirm?
Oh, I think that explains some of the phenomena we've experienced. So let's say I calibrate to a plane (so far whatever plane is used by the default calibration). Will that plane move along with the camera movement? If not by default, how can we have it do that?
It will move with the camera. (think HUD)
gaze is always relative to the camera.
either 3d in the world camera space or 2d on a plane in front of the viewer but moving with the camera.
really sorry that this is not clear. its not that difficult but we are clearly missing docs for this.
@mpk don't apologize. It's actually quite exciting to work with that technology. I think part of the problem is our lack in experience with unity and 3d applications in general. We're basically learning everything at once ^^
We're actually going to try and do the plane-thing you suggested. Thanks alot for your advice!
As for a last question (for now that is): is the polynomial mapping approach relevant for the plane-solution? Just to verify whether or not I should read into that right now
the mapping is already done bt the 2d calibration in Pupil Capture/Service. No need to apply a seond map.
I m glad I could help!
@mpk Hello, this might be a dumb question, but I was looking to automate the recording feature of Pupil through an external script, and I was wondering how I could do that. Plugins seemed to be promising, but I'm not quite sure how to get started.
@user-e709ce hey :) You should have a look at the Pupil Remote section at docs.pupil-labs.com and the Pupil Helper repository
@papr Awesome! Thanks for the pointers. I'll take a look 😃
You are welcome :) don't hesitate to ask further questions
Hello @mpk . Could you please give more details on the libuvc performance improvements made in latest release (v0.9.13) for Windows? I have done PC power and CPU utilization measurements in the past and I am interested to know what to expect from these improvements. Thanks.
@user-f45055 I think about 20%
but we have to measure again.
the improvment was made in 3d detection but mainly in optimizing a part of libuvc
Thanks. If it is okay to share please let me know the CPU load measured before improvements.
will do
I appreciate
..
@user-efb835 @mpk I'm in my mid-50's and when I use the system my eyes get hot and dry out. Is there any way to turn down the intensity of the IR emitters? Did anyone test whether the intensity that's currently in use is necessary, or was it just convenient and didn't cause problems for the test users? For me, I can't use the system after a few minutes due to the discomfort from hot, dry eyes. The Vive itself creates some heat but I can use it for much longer without the tracker.
@user-b172cc thanks for your post! The dryness does not come from IR but most likely from the heat of the device. Our cameras run at high fps and its unavoidable to do so without generating heat. We are working on a new camera design where the hot components are further away from the eyes. Im sorry to hear that this is a problem for you! If we have a new design we will let you know and lets figure out a way that you can use Pupil in your HMD!
Cool, thanks for the response. It had not occurred to me that the heat might come from the cameras but that makes sense.
@mpk I encounter this problem of not finding the directory when I try to record. I checked and made sure that the paths are correct already, but it still happens. Do you maybe have an idea why that is the case? (could it be that there was no file created in the first place?) Thanks in advance!
can you paste the exception?
I mean the error message if any?
@user-b172cc Do you run from source or from bundle? And which OS do you use?
@mpk here it is: DirectoryNotFoundException: Directory 'D:/ohooo/Recordings/2017_08_15' not found. System.IO.Directory.GetFileSystemEntries (System.String path, System.String searchPattern, FileAttributes mask, FileAttributes attrs) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.IO/Directory.cs:514) System.IO.Directory.GetDirectories (System.String path, System.String searchPattern) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.IO/Directory.cs:275) System.IO.Directory.GetDirectories (System.String path) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.IO/Directory.cs:270) FFmpegOut.FFmpegPipe.GetLastIncrement (System.String path) (at Assets/FFmpegOut/FFmpegPipe.cs:129) FFmpegOut.FFmpegPipe..ctor (System.String name, Int32 width, Int32 height, Int32 framerate, Codec codec) (at Assets/FFmpegOut/FFmpegPipe.cs:49) FFmpegOut.CameraCapture.OpenPipe () (at Assets/FFmpegOut/CameraCapture.cs:216) FFmpegOut.CameraCapture.Update () (at Assets/FFmpegOut/CameraCapture.cs:108)
@user-fe23df Is this an exception that is thrown in your unity application?
yes, it is
Could you explain your setup? Do you run Pupil Capture on the same device? Do provide the recording folder manually to your unity application or is this given via Pupil Remote?
How do you initialize the recordings? Manually in Capture or via a notification?
I'm running it on local mode (HTC Vive); the recording folder is both given in the custom path part of the PupilGazeTrackerScript and in Pupil Capture
and the recordings are triggered by pressing R, just like the readme of the plugin writes
by pressing R I mean the keyboard R
And you cannot find the recording via the Windows File Explorer?
and before this error the debug prints out the name of the path, which is exactly the same as the path reported not found
@user-fe23df Isnt the path supposed to be D:\ohooo\Recordings\2017_08_15?
yeah I didn't find anything
instead of normal "/"?
Yes, please check the default recording location
is that so? I didn't type it down, I only clicked through the popped-up UI
there is probabaly a folder with the above path as its name
I do notice that the slashes are different in the Unity UI and in pupil capture
but that's how they're set without me changing things
but I can give it a try, hopefully it works
😃
@user-fe23df You could try that or try to recreate the "Recordings" folder manually
Oh I did though
I created recordings manually
I still have the same problem unfortunately
@user-fe23df could you please check the default recording location and see if the recordings are there. I want to make sure that the recordings are there.
They are not there either
The recordings just get saved by themselves right? (as in there's no stop recording button to save the data)
The recordings are recorded as long as the R button is red
if you press it again, the recording is stopped
Oh so I checked capture and there's another thing: it says the world capture is offline and I have to get a fake capture
The folder <recording location>/000/
should be there the moment you start a recording
maybe this is why no recording got created?
Ah, yes, that is correct
Because before I only pressed R in Unity and not R in Capture
And now when I press R in Capture the error shows up, and the button switches back to white
How do I switch to a fake capture then? Sorry for the stupid question, but I haven't yet managed to figure that out
Open the Capture Selection
menu in Pupil Capture
Select Test Image
from the selector named Manager
sorry, Fake Capture
is a legacy name for Test Image
thanks!
I swithed to fake capture already, but still get the same error message from Capture
Is there an error message in your windows command prompt?
from which you started pupil capture
or: that opens when you start pupil capture
Yes I have the windows command prompt here
I think that Pupil Capture cannot create recordings at path that you set . please reset your recording location settings and try to make a recording with the default location first
if this works try to change the recording path
and some errors would be about not being able to stop stream
[ERROR] uvc: Can't stop stream: Error:'Operation timed out.'. Will ignore this and try to continue.
for one eye
Okay I will do that then
That happens if the camera disconnects. it will reconnect automatically. this is not relevant for this issue
it's just that I think I also had this problem before that
This error is from the uvc backend. Its fixed in the next release btw.
but maybe that was because I didn't specify the path in Unity
Please ignore the unity part for now. Let us try to create a recording with Pupil Capture with default settings only. Afterwards we can change one thing at a time
Yes, I'm trying to do the default path instead
easiest way to archive is to close pupil capture, delete the user settings and to restart pupil capture. afterwards acticate the test image again and hit R
Or click the Reset to default settings
at the top of the sidebar if you are running the newest version
The fake capture error remains the same unfortunately, even when on the inspector it already shows that test image is being used for capture manager
So wait, does that mean that you see an image of random colors in your Pupil Capture world window, hit record, but the recording immediately stops?
the random colors image appeared that time when I first switched to fake capture, yes, but then pupil capture froze. Now I try to switch to fake capture again but no longer see the random colors image
Oh nevermind, I forgot to click activate fake capture
ok, next: click record.
yeah it works perfectly fine now 😃
Great
Next step would be to change the recording path and see if the recording is saved to the correct location
Afterwards you can get back to your unity application and set your custom path appropriately
yes, thanks a lot for everything! 😃
Hi everyone how is all going ?I hope fine. I am trying make an eye-tracker for myself, I had used a playstation eye camera (removed the IR filter). Now, I am trying running the recorder app of pupillabs but my camera does not appear into the devices. I am running on ubuntu 16.
Hey @user-e12082 your camera needs to support the UVC standard to be listed in Pupil Capture
Hey! Can somebody tell me how the captureRate works exactly? I have a good guess it just means the rate of which pupils are detected, but I want to make sure.... The reason for asking is that I want to check whether the pupils are checked consistently as well as their confidece. I found code which checks for confidence > 0.4, but this could also trigger objects in the distance of the camera, so I want to filter out flickering detections, is the captureRate a good way to approach this?
Hi, is there a way I can annotate the data from unity?
@user-72b0ef My intuition is that this is the frame rate of the eye cameras. Pupil Capture tries to detect a pupil in each frame. This means that a higher frame rate gives you more pupil positions.
I think the confidence is for blink detection, is it?
@papr I thought this was true in the start as well, but the captureRate sometimes fluctuates from 30 - 120
and is especially high if the pupils are tracked well
@user-72b0ef Since you cannot detect always a pupil (e.g. during a blink), the pupil datum has a confidence that tells you the quality of the datum. Filtering the confidence is the recommended way. I would filter with a confidence of >0.7 for 3d detection
@papr Do you have a suggestion on how to prevent the cameras from detecting pupils which arent pupils? (Sometimes objects in mty room get detected)
If the detection is bad, and there is a hard threshold at 0.4, then it is clear that less pupil positions are transmitted if detection is bad
@user-72b0ef Do you speak of reflections in the eye?
@papr no, I mean if I put down my headset on the ground
The pupil detecion algorithm is a best effort algorithm. It does not check for false positives
A lot of different data types in Pupil Capture have a confidence field. It is an estimated value of the data's quality. As @user-fe23df pointed out, we use confidence drops for blink detection as well.
So I should try to change this to > 0.7f?
Sometiems I get really anxious trying to fiddle around with values, since I highly believe you guys are experts and know what values are best practices ^^
I would add this as a post processing step in your application instead of changing the given code.
@user-fe23df Can you elaborate on your previous question:
Hi, is there a way I can annotate the data from unity?
I want to show the pupil data in the picture. I don't know the mapping between the panorama sphere and the image.Can someone give me some information.
@papr Hmmm, alright, in this case I wanted to make the confidence check higher for calibration, but I greatly appreciate your feedback
@mpk
@user-72b0ef All calibration plugins already filter pupil positions with a threshold of 0.6
yeah I was reading about the new version 0913 and there's the new plugin annotations for pupil capture
So I was wondering if I can send certain info to be annotated together with the recorded gaze data from Unity
@user-72b0ef this is usually enough. In my experience, once the detection is good, it stays good. Therefore just make sure that detection works well before calibrating.
@user-fe23df ah, I understand. This plugin already exists for quite a while 😄
@papr I see!! Thats a really good thing to know, thank you 😃
@user-fe23df Annotations are special notifications. This means that you can use Pupil Remote to create annotations
@papr oh really? I didn't see it in the documentation for 0912 and only discovered yesterday that 0913 has been out 😃
@user-fe23df We were quite behind in terms of documentation. We are actively working on improving it.
@papr So any Calibration plugin used confidence > 0.7, while normale gaze tracking uses confidence > 0.4, correct?
Gaze points asre based on pupil positions. They inherit the confidence of their base data
Up to two pupil positions (binocular) create a gaze position.
The problem with low confidence pupil/gaze positions is that there is no defined behaviour for their location. Therefore the data jumps around a lot. This is filtered by some amount in your screenshot.
I want to show the pupil data in the picture. I don't know the mapping between the panorama sphere and the image.Can someone give me some information. @papr
@papr Ah, Thats totally clear now :))
My recommendation stands to filter a posteriori with confidence >0.7
@user-abff35 Unfortunately, I have no experience with the unity code itself. I do not know which panorama sphere you are talking about. =/
But I can tell you that a gaze position has a norm_pos
field that gives you the normalized gaze position. This means that 0,0 is the origin of the coordinate system in which the reference points lie that you used to calibrate
What is the connection between the two pictures? Thank you very much
@here FYI there is a dedicated channel for the unity project over at vr-ar where @user-5ca684 is able to answer unity-related questions in detail.
@user-abff35 This looks a standard spherical image mapping. But I cannot tell you in which context this happens. Sorry.
@papr okay😬 😬
@papr I think he is on holidays, he didnt answer in a few days so thats why I came here 😄
same 😄
Do you have a quick snippet of code to read in the new-format msgpack pupil_data files? I'd like to just extract the pupil size... Thanks!
@user-1d8e87 You could take a look at: https://github.com/pupil-labs/pupil/blob/master/pupil_src/shared_modules/file_methods.py#L52
That looks helpful, thanks!
That works fine except that for some reason I had to move the csv writing code inside the run definition (otherwise it couldn't find 'l') Thanks!
Hi all, any idea why are those eye images are different? the image of the right eye is so dark and with low resolution... tnx!
12 Just Looks Out of Focus.
Otherwise fine
Hi
After updating to 0.13, command line pupil_player /recording/path/ will not open the recording
Am I missing something?
Hey @user-41f1bf I will try to reproduce this today
Thank [email removed]
@user-41f1bf I have reproduced your issue.
This will be fixed in the next release.
Please consider making an issue for this
Ok, thanks
hi all! Does anybody know of any way to synchronize the stereo world cameras? If i just do sequentially
frame_right = self.uvc_capture_right.get_frame(0.05)
frame_left = self.uvc_capture_left.get_frame(0.05)
frames get occasionally out of sync by 15-30ms, which makes the stereo camera rather unusable for anything except absolutely static scenes.
@user-c77dda With how many FPS do you run the stereo camera?
720p @ 60fps
@user-c77dda Could you try to grabbing the frames in two separate threads? This does not guarantee a synchronized image but should yield frames closer to each other than grabbing both in the same thread.
that would be my next step, yes. I just thought maybe there is a fork of the pyuvc somewhere that has this already implemented
@user-c77dda we have not implmented this as of now.
I recommend trying two threads.
Note that the cameras are not hardware synced. I m afraid you will have to live with a few ms.
but if this won't work out, is there a way to exchange the stereo world camera for the realsense version?
@user-c77dda Yes, I am sure we can do that. Please send an email to info@pupil-labs.com concerning this matter.
ok, will do the next week, thanks
btw, are you planning a version of the eyetracker with the new realsense zr 300? https://click.intel.com/intelr-realsensetm-development-kit-featuring-the-zr300.html it's basically like the old one, but has a IMU on board, should help with motion tracking
@user-c77dda We do not know yet. We will look into it.
cool, thanks
@wrp Windows Update just came by and now the drivers dont seem to be working, problem with the troubleshooting proces is there is no libUSBK category and all cameras fall under the imaging devices category removed and reinstalled the drivers to no avail
@wrp i noticed the Vive USB camera uses the same usb driver
@user-ef7690 Did you reboot after removing and reinstalling our drivers?
yeah i did
the installer seemd to skip installing the actual files
after uninstalling and disconnecting the HTC vive
then restarting
only after clicking the dspinst64.exe file did it actually install
now i do have a libusbk category
drivers are working again everything but ideal
@user-ef7690 ok, good to hear. We are actively working on a better solution than having to reinstall the drivers after each update.
would be good if we could specify which driver to use instead of windows forcing the USB video one 😃 look forward to seeing the solution
Hi everyone,
I keep getting this error : libav mjpeg: unable to decode APP fields: Invalid data found when processing input. But everything seems to work okay . Does anyone know how to resolve it?
Hello Pupil community, I hope you all are doing great! I have couple of questions that are related to the hmd-eyes unity plugin, I will be grateful if you could give me an idea of how to clarify them. 1- Do Pupil Service and Pupil Capture are the same? If not which one of them is more suitable to work with unity? as I’m running them and unity locally in windows 10.
2- What are the differences between (Camera Image, ROI and Algorithm) modes? Which one is more suitable to work with unity? Is it possible to modify the green circle that shows up around the eye’s boundary?
3- I have adjusted my hmd to fit my face properly, however during the 3d calibration I won’t be able to see the white marker properly when they are at the corners.
4- How could the fixation and blinking plugins work in unity?
Many Thanks
@user-006924 I have never seen this message. Do you see an error
or a warning
? What OS and version of Pupil are you using?
@user-1eecd4 Questions about hmd-eyes should be raised on the vr-ar channel. I will respond to these question on that channel
Hey there! Just received my hardware yesterday ... is it normal that I can only record gazing when the Eye Detection Dialogs are opened? ... for me, this results in a really high cpu load ...
Hi @user-a5527b Welcome to the Pupil community chat 😄 👋
@user-a5527b The eye windows (and eye processes) must be running. However, you can minimize the eye windows to reduce CPU load
You should also make sure that you are in the Camera View
mode and not the Algorithm
mode to ensure reduced CPU load
@user-a5527b out of curiosity, what is your OS and OS version and machine specs?
@wrp Thanks for your quick reply! ... will try that. I already switched to Camera View, using one eye only is fine with the CPU load, using 2 makes my mac book pro (i7, 3.1ghz, 6Gb ram) go really mad.
@user-a5527b I have a macbookAir 11" with i5 and 4gb of ram and CPU usage is reasonable (considering 3 videos are being processed) with the eye windows minimized
One other thing: I tried using it in sunlight yesterday and confidence was really low, so that I hardly got any gaze paths out of it ... Anybody has experiences with that and is there a way to get over it? ....
@wrp interesting! so minimizing the eye windows really might do it.
@user-a5527b eye tracking in direct sunglight is challenging for IR based eye tracking systems. You should certainly try adjusting the 2d detector parameters and setting an ROI to achieve higher pupil detection confidence
@user-a5527b you should also make sure that you record eye videos
so that you can adjust the detector parameters in Pupil Player (after the fact)
@wrp okay, sounds good, I'll further play around with it! thank you!
Just if other people experience this problem: As I want to record gaze behaviour while I'm moving around and I'm using a MacBook, I need to close it. Unfortunately, osX is going to sleep when you close the Mac (and no second display is connected) .... I now use the InsomniaX app (http://semaja2.net/ye-ol-projects/insomniaxinfo/) that prevents the Macbook from sleeping when you close it, so you can put it in your backpack.
@user-a5527b - Thanks for sharing. Yes, there are a few apps like this that prevent sleep
In regards to making recording on the move, for those out there with a Nexus 5x or Nexus 6p, I would highly recommend Pupil Mobile - https://pupil-labs.com/blog/2017-08/introducing-pupil-mobile/ 😄
Hi, I just have a quick question to the interpretation of the pupil data about the position of the pupil. In your documentation you say "We use a normalized coordinate system with the origin 0,0 at the bottom left and 1,1 at the top right." However I sometimes get negative numbers which a also bigger than -1. What does this mean and where does it come from?
@vivi, negative number means that points were out of the surface or world frame
Can someone confirm if pupil can report negative numbers on pupil positions (eye image)
I am guessing it can not, because pupil positions are the base data. But, it is just a guess.
is there a possible way to save an external timestamp into the pupil data that gets saved out when recording?
what i'm trying to ask is. Is there a way for me to write timestamps from another device into pupil so that it 1) saves out with the recording 2) possibly then pull that value via pupil remote
@mpk hi, is there any data about eye centers for 2D (similar to eyeCenters3D). A ray from eyecenters to gazepoints is the right gaze ray right? thanks a lot! 😄
@user-fe23df the 2d data only includes information about the Pupil position. There is no 2d estimation of the eye ball
@papr ah okay thanks, how would you construct a ray to indicate the gaze then?
as I understand it, the gaze point would give me the direction of the ray relative to its root, so I need to find the correct root of the ray, but I am stuck at this part
@mpk
@ohooo The 2d norm position is normalized to the given mapping frame
For a normal Pupil headset, this would be the world cameras frame
In the hmd case, it is relative to the space in which you calibrated
@papr so I just need to use norm_pos from the pupil_position.csv file for each eye right?
thanks a lot!
Wait, there is a difference between Pupil and gaze positions
A Pupil position is the Pupils position within the eye frame. This itself is mostly meaningless. You need to map the Pupil position into the subject's field of view. That is a gaze point and describes where the subject is looking at
What are you trying to visualize? The eye ball and its orientation? Or the gaze of the subject?
The first one requires the 3d model and cannot be archived with the 2d Pupil positions
@papr I need to make a ray with the origin being the position of the eyeball and the direction being from the eyeball to the gaze point, so that I can see what objects are hit by the gaze in Unity
I see. As far as I know the 3d eye ball position is relative to the eye camera. You will need the transformation matrix from each eye camera to the coordinate system in which you calibrated
are there any inverted surface markers? if so do these work?
and where are they located?
Surface markers are a complete different topic but yes, you can use inverted markers. Just set the flag in the surface tracker
how do i make inverted surface markers?
the only ones i can find are the originall black background with white
Yes, these are commended for print
There is a generator script in the Pupil helpers repository
You could have a look at that and invert the values
i'm looking at that code but cant find the invert option
ahhh ok
@papr yeah so I decided to switch to 2d because I saw your (or @mpk 's)comment about 3d still needing some improvement. so for 2d there's no way to know where the eyes are?
Correct
however, I'm getting much better calibration results from 3d than 2d (for the VR HMD)
which one would you recommend using, 3d or 2d?
3d because of the slippage compensation
thank you! so the problem with 3d is merely regarding calibration and depth levels right?
And cornea defraction but we are working on that
I guess I should stick to 3d then, thanks again! 😄
No problem
@papr updated the make_square_markers code to work w/ python3 and a line that allows you to invert the markers
would you like me to commit?
Yes that sounds very good!
ok will do in the next 1-
10
No hurry :)
👍
thanks for your help!
No problem
just sent the pull request
@papr any idea how i can send an external timestamp to pupil and have it save out in the recording files?
or is it possible to set two timebases for a pupil recording?
No it is only possible to set one timebase
is it possible to create two timebases from within the source code?
But you should always try to unify your clocks
i worry about lag time for sending messages
and it would be extremely difficult setting up a heartbeat type of setup for syncing the two
Actually we have a time sync protocol inplace
Nonetheless you can send annotation notifications via Pupil Remote
These will be stored within the recording
yeah those notifications are not to be sent at a high volume tho correct?
Normally not.
looking into network_time_sync.py rn
There should be a spec file describing the protocol along with the python file
wheres the spec file?
i think this network time sync might be the impetus for me to make those matlab python changes lol 😉
thank you!
No problem!
building windows from source and there is no c:\work\boost\stage\lib
folder thats created even though the command window says it is....
nvm figured that out now running into
raise RuntimeError('GLFW library not found')
Looks like the glfw headers were not found 😅 have a look at the glfw.py file and how it does the windows handling. This might help you to understand what the reason for the message is
thanks! looking into it now
python has import glfw
how is it supposed to import glfw when on windows source install we don't install glfw into python
where is glfw imported?
There is a local glfw.py in the shared modules folder
This is what is imported
This file tries to load the glfw lib
It uses the ctypes utils though. You will have to read the find_library docs on how it finds its libs if the Pupil docs don't help
ok
set it so it finds it directly and i get this now
OSError: [WinError 193] %1 is not a valid Win32 application
Traceback (most recent call last): File "C:\Program Files (x86)\JetBrains\PyCharm 2016.3.1\helpers\pydev_pydevd_bundle\pydevd_exec2.py", line 3, in Exec exec(exp, global_vars, local_vars) File "<input>", line 1, in <module> File "C:\Python36\lib\ctypes__init__.py", line 348, in init self._handle = _dlopen(self._name, mode) OSError: [WinError 193] %1 is not a valid Win32 application
wait please tell me i didn't download the 64 bit version
yeah it wasn't 64 bit
.....getting a pyglui error now
MainProcess - [INFO] os_utils: Disabling idle sleep not supported on this OS version. Process world: Traceback (most recent call last): File "C:\Python36\lib\multiprocessing\process.py", line 249, in _bootstrap self.run() File "C:\Python36\lib\multiprocessing\process.py", line 93, in run self._target(self._args, *self._kwargs) File "C:\work\pupil\pupil_src\launchables\world.py", line 76, in world assert pyglui_version >= '1.7' AssertionError
guess i need to install from source smh
pip wheel from within pyglui doesn't work for some reason
any idea why @papr?
i just get Could not find a version that satisfies the requirement pyglui (from versions: ) No matching distribution found for pyglui
lol nvm
distutils.errors.CompileError: command ['"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe"', '/c', '/nologo', '/Ox', '/W3', '/GL', '/DNDEBUG', '/MD', '-I.', '-IC:\Python36\lib\site-packages\numpy\core\include', '-IC:\work\opencv\build\include', '-IC:\work\Eigen', '-IC:\work\ceres-windows\ceres-solver\include', '-IC:\work\ceres-windows\glog\src\windows', '-IC:\work\ceres-windows', '-IC:\work\boost\stage\lib', '-I../../shared_cpp/include', '-Isingleeyefitter/', '-IC:\Python36\include', '-IC:\Python36\include', '"-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE"', '"-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE"', '"-IC:\Program Files (x86)\Windows Kits\10\include\10.0.15063.0\ucrt"', '"-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um"', '"-IC:\Program Files (x86)\Windows Kits\10\include\10.0.15063.0\shared"', '"-IC:\Program Files (x86)\Windows Kits\10\include\10.0.15063.0\um"', '"-IC:\Program Files (x86)\Windows Kits\10\include\10.0.15063.0\winrt"', '/EHsc', '/Tpdetector_2d.cpp', '/Fobuild\temp.win-amd64-3.6\Release\detector_2d.obj', '-D_USE_MATH_DEFINES', '-std=c++11', '-w', '-O2'] failed with exit status 2 all these paths exist i still don't understand why i'm getting errors building off source Windows10
Currently, I get an error when trying to use pupil capture. The message is 'Init failed. Capture is started in ghost mode. No images will be supplied.' All the cameras are not working and it seems to be the same reason.
Hi, are you running in Windows?
I work in Windows 7 Proffesional KN(64 bit). My working environment is as follows. CPU : Intel(R) Core(TM) I7-6700 CPU @ 3.40GHz RAM : 8GB
Yes
please note that we cannot officaly support windows 7. Can you use windows 10?
Ah... I should try on another computer.
Now I'm in the office. I should try on my home.
I'll ask again after trying on window 10.
Thank you ~
ok sounds good. Actually I recommend trying the bundle we are going to release tomorrow it has automated driver installers for windows.
Oh It sounds good. I should check in tomorrow.
Guys - just a heads up. There was lost of interest in the pupil tracker @ ECEM in Wuppertal. It's too bad you guys weren't there to represent!
Thanks for the feedback @user-8779ef - wish we could have been there!
(European Conference of Eye Movements) . I (Gabriel Diaz) taught a course to about 45 on eye tracking in VR. Sadly, I've only had a Pupil in my Vive for a few weeks, so I couldn't provide much input. I may teach more classes in the future, though, and hope to have good things to say about the tracker.
Well, MORE good things. Calibration is still a bit messy, and I think it's largely due to the off-axis eye images. Any hope of using a hotplate for an inline image in the Vive ? What was the constraint ? (sorry for this repost on this - I asked in the hmd-eyes text channel twice, with no response )
Is the constraint due to the distortions due to lensing? Space? The complicated installation process?
Thanks again for the feedback.
And apologies for not getting back to you with a response in the vr-ar channel
The angle of the cameras should not be contributing to reduced gaze accuracy
As long as pupil detection is robust
Hot mirror is not desirable for 2 main reasons (1) fresnel lens distortion (2) potentially non-reversible mods to the Vive
If you are seeing low accuracy gaze, the first place to look is in pupil detection. You may need to adjust the depth of the Vive.
The second place to investigate is the calibration procedure
In unity 3d
This is something @user-5ca684 is actively working on
Good to hear!
Off axis eye images in Pupil (non VR setup) are common actually and high accuracy gaze estimation is still achievable
I have both - the angle is much greater in the Vive
It sounds like you're suggesting that I find a balance between image quality (vive depth) and tracking quality.
Obviously, this isn't the ideal solution. I've also found tracking quality drops when looking at the top 1/3 or so of the screen, due to poor images of the pupils.
If you'd like to send us some eye videos via email - we'd be happy to take a look and give you more concrete feedback
So, your statement "The angle of the cameras should not be contributing to reduced gaze accuracy As long as pupil detection is robust" is true, however I have not found pupil detection to be robust.
wrp - thanks a lot for the offer
It seems that you may not be getting good eye images. Would be great to take a look and give feedback
🙂
I'll give it some more time and take you up on that if I can't seem to improve the track. At this point though, I think I'm close to squeezing all I can out of the current setup. I can send you a python notebook with the results of our calibration assessment ...let me get a fresh batch of data, though.
Maybe I can log the eye images as well.
Yes please capture eye videos as well if possible
Thanks a lot! Lots more success witht he mobile tracker
Very big supporter of your project, so I hope my input is constructive in the end 😃
Yes. Feedback is welcome - thanks
@here we just pushed v0.9.14 ! https://github.com/pupil-labs/pupil/releases/tag/v0.9.14
👆 lots of new features - check the release notes and download v0.9.14
😄
do you have some screenshots of the accuracy test at hand?
@user-02665a the results or the way the test works?
i just had a quick glance at the feature list and saw " visualize accuracy directly after a calibration."
which is what i would be interested in
(i just dont have the tracker here atm) just if you happen to have made a screenshot anyway
The blue dots are the detected marker positions, red the predicted gaze positions and in orange the error between them
@user-02665a ☝
interesting, thanks
Also thank you to keep supporting the eye tracker in such an amazing way (fast addition of features, very responsive to problems)!
@user-02665a thank you for the appreciation 😃
pyuvc not installing properly on ubuntu 16.04 i opened an issue on github