[email removed]
hello all...anyone know the best way to adjust the eye camera settings so it doesn't put blue on eyelashes (in algorithm mode)
@user-ed537d can you send a sample eye video so that we can give concrete feedback? The default pupil detection parameters have been optimized over the Labeled Pupils In The Wild Dataset. So, in most cases you shouldn't need to change the parameters. Often it is the case that poor pupil detection is due to the eye camera being out of focus or parameters being improperly set. Also - what version of Pupil Capture are you using?
i just used roi and that worked...i'm running into this issue where my calibration is always a few inches off
usin 0.9.1
i'm really at a loss right now i've tried everything as far as trying different calibrations and i can't get it
@user-ed537d are you using a 3d detection and mapping mode or 2d mode?
3d
binocular
@user-ed537d Please share a sample dataset so we can give feedback.
to dropbox?
Upload to cloud and share a link here or via PM in discord
I will be able to take a look at the dataset today (but not immediately)
uploading now
👌
i'm uploading my best dataset thus far you'll see that on certain parts of the screen its perfect but at other locations its off significantly. This dataset is the best i could get after 3-4 hours of messing around w/ the eye and world images as well as focus of the cameras....Thank you very much for your help.
uploaded dataset found above
@user-ed537d did you record eye videos as well? Are they included in the dataset?
Yes both are included
OK. I will take a look today. Thanks.
Thank you so much!!
Does someone have a ready C/C++ implementation of pupil remote client?
@user-d00e4f - unfortunately I do not. Have you looked at C++/C ZMQ examples?
@user-ed537d - thanks for sending the dataset
I have downloaded it and loaded the eye videos in Pupil Capture. In v0.9.1.7 I can see that the 3d detector confidence (with default settings) in many areas of the video is quite low below - e.g. less than 0.5. Which explains the poor calibration results.
@user-ed537d - switchign to detection & mapping mode
2d shows much higher confidence pupil detection (with default settings) - e.g. confidence never drops below 0.9. This is very helpful information because is shows that some of the changes made recently to the pupil detector could have had negative impact on the performance of the 3d detector. I will look into changes made between v0.8.7
and v0.9.x
. @mpk I suggest that you take a look at this dataset with MT as well.
@user-d00e4f - have you looked at: http://zguide.zeromq.org/cpp:syncsub - please take a look at the ZMQ examples and compare to simple client examples that we have written in Python in the pupil-helpers repo like this: https://github.com/pupil-labs/pupil-helpers/blob/master/pupil_remote/filter_messages.py
@wrp yup. Was wondering if someone had such I can use them straight away. Anyway, I'll upload them once my client starts working
@user-d00e4f - would be great to see other language examples like what is already on pupil-helpers. Please make a PR if/when you get around to it 😄
@user-ed537d - I just ran your dataset again on both macOS
and linux ubuntu from the most recent commit in master branch and was not able to replicate my previous observations. We get high confidence pupil detection throughout the video, except during brief periods when the eye moves rapidly, but then the model recovers. I will test again with the Windows bundle to see if I can replicate the issue there. However I noticed from your info.csv
file that you are using Windows release 7 v6.1.7601
- this is Windows 7 Service Pack 1 correct? Just to clarify - our bundles and source code targets Windows 10
@wrp @user-ed537d having looked at your recording. The eye videos and ran them. We conclude that this isssue is NOT releated to pupil detection. Pupil detection with default capture values was robust for us. Instead I beleive that you should use 2d mode for your particular use case. Or try running manual marker calibration. We are still finetuning the 3d detector so you might just not get the accuracy you are looking for in 3d mode. Futhermore I recommend doing a camera instrisics calibration and seeing if that helps.
@user-ed537d looking at your recording in PLayer again I also notice that gaze accuracy is great in the top half but bad in the bottom half. Please make sure that duing calibration you have good confidence when looking at all markers.
It seems to me that the bottom two markers may have been excluded because during calibraiton eye confidence for those two was low.
@wrp So I wrote a cpp pupil remote, and when I compare it with the data stream from python client, the order of data seems different. Is that normal? Or is there something I'm missing?
@user-d00e4f if you refer to the order of the fields in the dict. Then this is normal. They are not ordered.
Yes, its the order. Cool, then it work
works*
If somebody would be so kind to help me out find out how to find out which .dll is missing for Detector_3D described here.: https://github.com/pupil-labs/pupil/issues/653 that would be very very helpful. Messing with that since hours.
@wrp @mpk thanks for the input! I started to use the 2d detector and i got much better results aside from the slight deviation in certain locations. Yes I was using windows 7 on that system. I'm going to continue using the 2d detector for our experiments but will continue testing out the 3d detector.
any idea how i can get rid of this messing w/ image-post processing and pupil intensity doesn't seem to help much
Did you try reducing the roi?
@user-41f1bf yes i have but it hasn't worked all that well
also is there any possible way of increasing the number of calibration points?
You can implement your own calibration method
Inheriting from the calibration plugin class
Also, the 2d calibration uses 9 points
@user-41f1bf thanks! will try that out
BTW - eye tracking add-ons are now available for Microsoft HoloLens and The Epson BT-300: Checkout our post: https://pupil-labs.com/blog/2017-03/hololens-and-bt300-eye-tracking-add-ons/
cool 👍
btw, does the "gaze" in the tracked data mean the normal along which the eye is looking?
Coz in AR glasses like hololens, the normal direction would mean possibly changing images based on gaze (or something like that)
is there a plugin too, to interface tracking data with any application?
gaze is xy positions in world reference space, after calibration,
Raw data have normalized and scaled values
pupil positions are xy in eye reference space
gaze is xyz for binocular setups
so in an hmd, pupil positions would be xy wrt camera, and gaze would be xy on the display screen
?
Gaze in hololens do not refers to any eye tracking data, it is the center of their finger/ head calibration, afaik
okay
If I have to access specific data from the data stream, say norm_pos_x and norm_pos_y, should I have to parse the entire string for those values?
Or is there some format to the data? (struct /an object, etc.) P.S. I am using C++
@user-d00e4f The data is msgpack-serialized dictionary. See the hmd eyes proejct (https://github.com/pupil-labs/hmd-eyes/blob/master/unity_integration/Assets/Scripts/PupilListener.cs) for an example on how to deserialize these dicts with c++
I think there might be a 'collection' container you can jsr. Then you can just query the parts you need without having to define to full structure.
@wrp the release notes says that the 3d detector detects normal of pupil wrt the eye camera. Is there a way I know the transformation between the pupil, normal and the eye cam?
For reference @user-d00e4f is referrign to release notes from v0.7.4
where we introduce the 3d pupil detector and 3d gaze mapper: https://github.com/pupil-labs/pupil/releases/tag/v0.7.4
Yeah, sorry I didn't post the reference 😄
@user-d00e4f - take a look at the calibration visualizer: https://github.com/pupil-labs/pupil/blob/4db249a9e4e141edf6ed26650dd561621658b86c/pupil_src/shared_modules/calibration_routines/visualizer_calibration.py
Hi all I'm trying to run 0.9.1 on a win10 machine and upon loading I get the following.
yeah I had similar issue
I only faced this when I tried to run from my integrated Videocard from the motherboard
Actually I had a similar issue on Ubuntu but I blamed the graphics card high memory usage of an other program
I read that this is probably OpenGL driver issue
Do you have the same issue with 0.9.0?
Trying it on 0.9 I think it is he integrated graphics card. Going to try 0.9 then will try popping in a graphics card and seeing if it works with that.
@user-ed537d ok great. Let us know how it goes
@papr 0.9 doesn't work either
OK. I am curious if it is the integrated graphics card fault
I have a feeling that's what it is.
Will update in a few.
that was in my case
Video card fixed this issue.
New issue is im only getting 9 fps on windows for the eyes
@user-ed537d - what are the specs for your machine exactly (CPU, GPU, etc)?
@here - we should be releasing Pupil v0.9.2
today as in incremental release for bug fixes found in v0.9.1
Hi! I use PUPIL in physiological researches and it is critical to know the duration of eye fixation to measure emotional reactions and zones of interest. Where (and how) can I get/find this information after recording session?
For eye fixations, open the fixation detector plugin. After exporting, by pressing e, the rae data will be available inside the recording folder.
For Areas of Interest, you may use surfaces. Surfaces requires fiducial markers. You should take a look at the wiki in github.com/pupil-labs/pupil
I should try, thank you!
@wrp I'm pretty sure it was that computers processor tried it out on a i5 2500 and it's working properly now.
What version of Linux has everyone had the most success installing pupil from source? ubuntu 16.04? Fedora?
Ubuntu 16.04 LTS
I am also using pupil player in a Debian 8 distro, at home.
In a celeron, dual core, 1.2 ghz each. Little bit slower.
:)
Nice! Currently installing ubuntu 16.04
Yeah I tried dropping the pixels for the world camera but it starts to have problems with surface detection. Luckily we have a few computers lying around
Have you tried pupil service?
No gui should be faster
No I haven't how does pupil service work
Ports the video stream via zmq?
I guess you can open pupil capture, configure the way you want, close, open pupil service and control actions by zmq messages
Thanks for the heads up I never thought about that!
@user-ed537d For Linux Pupil Labs officially recommends Ubuntu 16.04 LTS and maintains instructions for dependencies for this version.
I also run Pupil from source on Arch Linux Antegros - bit this is only recommended of you really know what you're doing and are comfortable making changes yourself to dependencies.
Trying to build from source on Ubuntu 16.04 and running into an issue where it hangs on make test for ceres-solver test 10 is where it just hangs
Any ideas?
Wait for a bit. It might take a while
lol literally on queue
Sorry. I don't follow... I found the one off the tests takes a bit of time.
It worked after I posted the message
Ah OK.
The only thing I see when I start up pupil from source is
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
starts up fine however when pupil is plugged in
wow building on ubuntu 16.04 is flawless
easiest build i've ever had off source
👏
has anyone noticed that if you have surface tracker on and you try to change the resolution of the world view it will crash and give the following: world - [WARNING] plugin: Surface_Tracker: Update will be deprecated soon. Use 'recent_events instead.' world - [ERROR] uvc: Can't stop stream: Error:'Operation timed out.'. Will ignore this and try to continue. Estimated / selected altsetting bandwith : 773 / 800. OpenCV Error: Assertion failed (prevPyr[level * lvlStep1].size() == nextPyr[level * lvlStep2].size()) in calc, file /home/pupil/opencv/modules/video/src/lkpyramid.cpp, line 1365 Process world: Traceback (most recent call last): File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap self.run() File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run self._target(self._args, self._kwargs) File "/home/pupil/pupil/pupil_src/capture/world.py", line 453, in world p.recent_events(events) File "/home/pupil/pupil/pupil_src/shared_modules/plugin.py", line 66, in recent_events self.update(events['frame'], events) File "/home/pupil/pupil/pupil_src/shared_modules/surface_tracker.py", line 206, in update min_marker_perimeter=self.min_marker_perimeter) File "/home/pupil/pupil/pupil_src/shared_modules/square_marker_detect.py", line 329, in detect_markers_robust minEigThreshold=.01,*lk_params) cv2.error: /home/pupil/opencv/modules/video/src/lkpyramid.cpp:1365: error: (-215) prevPyr[level * lvlStep1].size() == nextPyr[level * lvlStep2].size() in function calc
pupil won't run from source now....
i keep getting this
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
The wiki from https://github.com/pupil-labs/hmd-eyes still links to the gitter and google group instead of discord
Thanks @user-eda5d2 I will correct that in a second
Apparantly I do not have rights to write into this repository. This means we need to wait for @wrp or @mpk to merge https://github.com/pupil-labs/hmd-eyes/pull/9
@papr I will merge this today. Thanks.
@papr - merged your PR
Wondering when will be released the new pupil osx bundle 🤔
Hi @user-99e72e - we are currently blocked on macOS bundles due to this issue: https://github.com/pyinstaller/pyinstaller/issues/2322
@user-ed537d the tracker thing is a bug. We can fix that. Please remove the settings file it should work again then.
Hi .. I just wanted to give some feedback on the Pupil Mobile app (I know it's still beta!). We got it working successfully with a Nexus 6P, and streamed the images over WiFi to another machine. All three cameras (world + 2 eye) worked fine. However the images seemed to be black+white only, and quite heavily compressed. I saw that you are working on H264 which might improve this. Other than that the app crashed every now and then, but looks now much much better than a few months ago. Good work, can't wait to see the final version 😃
That is great to hear. H264 streaming works already. We are working on being able to write the stream data to file in such a way that we can open the video later
Ive set up two PCs with one Eyetracker each and enabled the "pupil groups" plugin. I gave both eyetrackers the same "group" and different names. They connect to each other as seen by "other group members". When i start a recording at one eyetracker i was under the impression, that the second eyetracker should start as well - instead i get this error message on screen of the 2nd eyetracker.:world - [INFO] pupil_groups: Dropped garbage data by peer Eyetracker1 (a7c4b057-15c2-43b8-a0e6-2075c6e3fde9)
recording does not start on the 2nd eyetracker
thanks for that feedback. I ll investigate
thanks a lot
@papr at least we didn't get the H264 streaming working - when it was switched on, we couldn't retrieve any images
When switched off, the streaming was working fine
@user-dfa26d Did you use the h264 Pupil Capture branch and the h264 pyndsi branch?
There are 3 components to h264 streaming: The host app, pyndsi and Pupil Capture.
Ahhh no I didn't
I see, I see .. will try this the next time I get my hands on the Nexus 6P
See https://github.com/papr/pyndsi for the bleeding edge pyndsi versions
Great, thanks!
I guess the quality of H264 streaming is higher than the default (jpeg I assume)?
No, not necessarly. But the amount of data that has to be transmitted over the network is a lot smaller and therefore the stream should be more stable with less dropped frames.
bad img quality might be a usb transport artefact. I think you might want to adjust the banddiwth factor in the app gui for each cameras.
And yet another question .. when I try running the uvc-ndsi-bridge-host.py
example, I get the warning "Group pupil-mobile not found"; and pupil capture does not find any remote hosts
Do I need to run the example on a remote host? Right now, I try to run both the example and pupil capture on the same machine
No, it should work locally. Please take this file [1] and put it into your user plugin folder and uncomment the lines
[1] https://github.com/papr/pupil-plugins/blob/master/debug_logger.py
If done correctly you should see the line >> DEBUG_LOGGER: Enabled debug logging for Pyre and ndsi.
when starting Capture and a lot of debug messages that should help us debug the problem. Please put the output on http://gist.github.com
(Side note: looks like Discord team has not fully implemented markdown syntax 😿 )
@wrp Yes, would be great to use the link syntax
Hmmm I placed the script in ~/pupil_capture_settings/plugins
but it doesn't seem to get loaded
You did start from bundle, correct?
and uncommented the lines?
No, I start from source
Yep, lines are uncommented
~/pupil_capture_settings/plugins
is the folder when starting from bundle. Look for the pupil_capture_settings
folder relative to your source location
Got it
@user-dfa26d
- pupil/capture_settings
if running from source
- ~/pupil_capture_settings
if running from bundle
Might need to be properly documented in https://github.com/pupil-labs/pupil/wiki/Plugin-Guide
world - [DEBUG] ndsi.network: Dropping <pyre.pyre_event ENTER from 468f6eaff1d6464e943233e8a39c86ae>
world - [DEBUG] ndsi.network: Traceback (most recent call last):
File "ndsi/network.pyx", line 74, in ndsi.network.Network.handle_event (ndsi/network.c:2670)
File "/usr/lib/python3.4/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
@user-dfa26d - thanks for the feedback on docs. Plugin guide (and many other things) need better documentation. We are currently working on porting docs from wiki to our own docs site do we can properly version docs
and have better control of the way content is displayed in docs
@user-dfa26d Which commit of pyndsi are you using? cd
to your pyndsi direcotry and run git rev-parse HEAD
ca40906c6f38b791a16ee7ddc25a0e4003209686
And I copied over the changes for Python3 in the example script from your fork, everything else is untouched
I need to have a look at my current branch later. My current network makes testing very difficult though since I am on a conference.
Okay cool, thanks!
If you do not want to wait and do me a favor, I would appreciate if you could test https://github.com/papr/pyndsi/tree/master
Actually it was my bad
I had an old version installed with pip
It's working fine, both on the pupil-labs
and on your branch
Thanks a lot!
Great to hear that it works for you now
Except that I get Could not parse received cmd: b'{"action": "refresh_controls"}'
Must be an issue with the encodings I guess?!?
Fix:
diff --git a/examples/uvc-ndsi-bridge-host.py b/examples/uvc-ndsi-bridge-host.py
index d83d7fa..90a7189 100644
--- a/examples/uvc-ndsi-bridge-host.py
+++ [email removed] -94,7 [email removed] class Bridge(object):
while has_data(self.cmd):
sensor, cmd_str = self.cmd.recv_multipart()
try:
- cmd = json.loads(cmd_str)
+ cmd = json.loads(cmd_str.decode('utf8'))
except Exception as e:
logger.debug('Could not parse received cmd: {}'.format(cmd_str))
[email removed] -155,7 [email removed] class Bridge(object):
"readonly": False,
"map": [{
'value': idx,
- 'caption': '{:i}x{:i}'.format(*fs)
+ 'caption': '{:d}x{:d}'.format(*fs)
} for idx, fs in enumerate(self.cap.frame_sizes)]
}
})
For anyone who needs help with pupil remote and Matlab I just wrote a quick little python script that will filter messages and udp them to matlab
Are there any pros/cons to using the fisheye world lens?
Pro - Better field of view
Con - Impose excelent homography
Hi @user-ed537d - it woul dbe great to see the matlab script that you wrote and would likely be of help to others in the community. Perhaps you'd like to share it by making a PR to https://github.com/pupil-labs/pupil-helpers
I mean, it increases complexity
@wrp , some people asked me how to calibrate the camera. I suggested to follow existing documentation about using the opencv 11x8 pattern
@user-41f1bf - you can show the pattern from within Pupil Capture
on the screen
However, It is clear to me how to show this pattern in regard to the camera
Should I change the deaph?
Should I change orientation?
Should I change only the xy position?
I mean, pupil requires capturing the pattern 10 times
So how exactly show this pattern in regard to the camera?
to show*
How far the pattern should be?
And so on
@user-41f1bf - some notes on camera calibration via opencv: http://docs.opencv.org/3.2.0/d4/d94/tutorial_camera_calibration.html
@user-41f1bf - You should try to cover the FOV of the camera with the pattern and capture the pattern at varying orientations. You should not change the focal length of the camera (or other camera intrinsics) while calibrating.
Thank you Will, so basicly turn off auto focus and capturing the pattern as closest as possible while changing orientation.
@user-ed537d - wider lens = larger FOV (more closely approximates human FOV) as @user-41f1bf notes.
Whether a wider FOV is a pro or con, really depends on your research/application. Some people want a very narrow lens so that they can capture (less) of the scene but in higher resoultion. Example use case for smaller fov: capturing text on a document or details within the center of the FOV far away.
So, it really depends on the goal/aims of your research/application.
@user-41f1bf - right - turn off auto focus. Capture the pattern at various positions and depths.
@user-41f1bf - you should not capture all patterns as close as possible, but try to cover the majority of theFOV with the 10 captured patterns. Does that make sense?
@wrp and @user-41f1bf thank you. @wrp I'll try and get the code uploaded and cleaned up by next week
@user-ed537d sounds good. I look forward to seeing the code. There are a lot of people who would benefit from your example Matlab + Pupil helper script
pupil detection in 3d mode gives different values when run multiple times. Is there an inherent ambiguity in 3d detection?
@user-d00e4f - do you mean that 3d detector gives you different values when run multiple times on the same eye video data? How are you testing this? Do you have a dataset that you have recorded and/or can share and/or results to look at to validate this observation?
I have a fixed eye camera position and reading one frame at a time using pupil remote. So everytime I run the client, I get different data
I am pasting two such instances I got just now.
{"topic":"pupil", "diameter":87.0494, "model_id":31, "phi":-1.66338, "confidence":0.17573, "model_birth_timestamp":16925.2, "circle_3d":{"radius":0.702609, "normal":[-0.0911117, 0.169847, -0.98125], "center":[-1.37686, 0.543241, 10.0298]}, "diameter_3d":1.40522, "id":1, "ellipse":{"angle":-49.8168, "center":[235.145, 273.08], "axes":[83.1959, 87.0494]}, "sphere":{"radius":12, "center":[-0.283523, -1.49492, 21.8048]}, "norm_pos":[0.367414, 0.431084], "theta":1.74147, "method":"3d c++", "projected_sphere":{"angle":90, "center":[311.938, 197.493], "axes":[682.418, 682.418]}, "model_confidence":1, "timestamp":16939.8}
{"topic":"pupil", "diameter":0, "model_id":1, "phi":0, "confidence":0, "model_birth_timestamp":16890.6, "circle_3d":{"radius":0, "normal":[0, -0, 0], "center":[0, -0, 0]}, "diameter_3d":0, "id":0, "ellipse":{"angle":90, "center":[320, 240], "axes":[0, 0]}, "sphere":{"radius":12, "center":[-4.02659e+07, -8.55075e+06, 9.86045e+07]}, "norm_pos":[0.5, 0.5], "theta":0, "method":"3d c++", "projected_sphere":{"angle":90, "center":[66.8185, 186.235], "axes":[0.000150906, 0.000150906]}, "model_confidence":0, "timestamp":16937.8}
{"circle_3d":{"radius":0.285172, "center":[-0.766851, -0.192932, 2.97162], "normal":[-0.110369, 0.04773, -0.992744]}, "model_birth_timestamp":18613.1, "method":"3d c++", "timestamp":18643.2, "theta":1.61854, "confidence":1, "phi":-1.68152, "id":1, "model_confidence":1, "diameter_3d":0.570345, "norm_pos":[0.25095, 0.584436], "ellipse":{"angle":-8.22745, "center":[160.608, 199.471], "axes":[114.814, 119.331]}, "topic":"pupil", "projected_sphere":{"angle":90, "center":[343.225, 208.106], "axes":[999.695, 999.695]}, "diameter":119.331, "sphere":{"radius":12, "center":[0.557581, -0.765691, 14.8845]}, "model_id":1}
{"norm_pos":[0.292274, 0.48526], "confidence":1, "projected_sphere":{"axes":[858.421, 858.421], "angle":90, "center":[380.435, 208.2]}, "theta":1.65067, "circle_3d":{"normal":[-0.244008, 0.0797867, -0.966485], "radius":0.466818, "center":[-1.23844, 0.0683597, 5.73633]}, "phi":-1.8181, "model_birth_timestamp":18534.5, "model_confidence":0.673007, "ellipse":{"axes":[92.1003, 100.994], "angle":-13.6448, "center":[187.055, 247.075]}, "model_id":7, "id":1, "sphere":{"radius":12, "center":[1.68966, -0.889081, 17.3341]}, "method":"3d c++", "topic":"pupil", "diameter":100.994, "timestamp":18566.3, "diameter_3d":0.933637}
ignore the second datum, its id=0. The others all correspond to id=1 when I started the application fresh everytime, with the same eye and camera position
@user-d00e4f - Thanks for sharing this data, however, I am trying to understand your setup and how/what you are trying to reproduce from one trial to the next.
If your data is a video file source, then you should be able to reproduce pupil detection data data (assuming that detection parameters have not been changed).
If you are wearing the headset, the data the system observes will not be the same from one trial to the next. And therefore, you should expect difference in 3d model from trial to trial.
But perhaps I misundertood your setup/objectives.
Hi, I've been playing around with the trackers for the HTCVive, and they were working fine. However, I'm suddenly getting banding arccos the video stream, and I can't work out why. Any thoughts?
@wrp - If you are wearing the headset, the data the system observes will not be the same from one trial to the next. And therefore, you should expect difference in 3d model from trial to trial. So does this mean even for a fixed camera and head position, I need to calibrate every time before using it? Do you know why the system observation differs from trial to trial? I am wearing the headset and restarting the program multiple times to see if the 3d data is robust and it gives nearly same values every time. But the data differs each time I run the program, and I am trying to understand why
@user-2cdde0 This looks like an hardware issue to me.
@user-d00e4f The 3D calibration is done using bundle adjustment. Bundle adjustment is an optimization problem and it is very probable that the optimizer converges on different solution for different input data (even if the data is only slightly different). The only way to garantue the same input in all trials is to use the same video file source for all trials (as Will suggested)
And even then there might still be different solutions to the same input since the optimizer might use stochastic methods to find the optimum (even though I do not know if this is the case for the ceres bundle adjustment implementation)
@papr but does that mean I will have to calibrate everytime I use the camera headset?
@user-d00e4f You are able to calibrate, take it off and put it on again and it will still work reasonably well. Important is that the camera positions are not moved since this would break the projection of the pupil vectors into the world space
Yep, that is what I too am expecting. I kind of glued the cameras so they don't move, and am expecting to get more or less same values. But they differ with each iteration of program. I was trying to understand what the prolem could be and how to fix it
But that said, in your experimentation did you see similar values for fixed cam positions, when run multiple times?
To which values exactly are you refering?
the 3d pupil data stream as such
The pupil data stream does not carry any explicit information about the calibration but only model information. Each model refers to one specific eye ball position in 3D space in relation to the eye camera. This means that if you move the headset the eye will move in relation to the eye camera and a new model will be fitted. Actually there are multiple models fitted in parallel and the one with highest confidence will be broadcasted (identified by the model id)
So yes this is supposed to vary even you did not take it off and on again, since there is always some kind of slippage and this is our way to compensate for it.
@user-2cdde0 what you are seeing is the LIGHTHOUSE system in action. These stripes will not be there when you put on the headset.
@mpk Ah! So it's all fine then? I was just very confused because I previously was not seeing them in earlier testing. Thank you 😃
Yes. this is normal and an intersting artefact!
Great thanks. Would you mind briefly explaining why this occurs, or linking to something that does. It just means I can put future users in my department at ease when they see it and also panic like I did.
The Vive lighthouse system sweeps ir light into the room. Our cameras see this since we use the same spectrum.
Oh I see - thanks for your help today!
I keep seeing this error. Someone help me understand this?
you can ingnore that! its part of the 3d solver. It fails to converge and will retry.
@user-d00e4f we should get rid of the terminal outfut for this actually.
maybe we should just have it like a toggle option. Sometimes terminal output might be useful for debugging
We should just change the log level for these messages.
Hi everyone, I need some help. I trying install pupil on win10 using this guide: https://github.com/pupil-labs/pupil/wiki/Dependencies-Installation-Windows#system-requirements Now I'm on final step, executing run_capture.bat, but failed. Output:
MainProcess - [INFO] os_utils: Disabling idle sleep not supported on this OS version.
world - [INFO] world: Application Version: 0.9.2.12
world - [INFO] world: System Info: User: User, Platform: Windows, Machine: asus-pc, Release: 10, Version: 10.0.14393
world - [INFO] pupil_detectors.build: Building extention modules...
cl : Command line warning D9025 : overriding '/W3' with '/w'
cl : Command line warning D9002 : ignoring unknown option '-std=c++11'
cl : Command line warning D9002 : ignoring unknown option '-O3'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\link.exe' failed with exit status 1181
Process world:
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\multiprocessing\process.py", line 249, in _bootstrap
self.run()
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\work\pupil\pupil_src\capture\world.py", line 94, in world
import pupil_detectors
File "C:\work\pupil\pupil_src\capture\pupil_detectors\__init__.py", line 16, in <module>
build_cpp_extension()
File "C:\work\pupil\pupil_src\capture\pupil_detectors\build.py", line 25, in build_cpp_extension
ret = sp.check_output(build_cmd.format(sys.executable, install_loc), shell=True).decode(sys.stdout.encoding)
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\subprocess.py", line 626, in check_output
**kwargs).stdout
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\subprocess.py", line 708, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'C:\Users\User\AppData\Local\Programs\Python\Python35\python.exe setup.py install --install-lib=C:\work\pupil\pupil_src\capture' returned non-zero exit status 1
@everyone My eye cameras were working fine until today and they suddenly stopped capturing data and only displaying ghost images. What could be the reason?
@user-d00e4f what OS?
ubuntu
I was trying to extract eye images from the eye recordings, and I had installed ffmpeg again. Should an update in ffmpeg cause any such problem?
I think I was asked to ignore those white balance temperature, etc, warnings
@user-d00e4f please try deleting pupil_captute_settings
okay
Should I delete the entire folder?
Yep
still the same.. i see ghost images
You can always delete the settings folders. They will be recreated on startup of Pupil capture, player, service
And cameras are not showing up in local USB capture sources?
Or are showing up but no frames from backend?
It could be that updating ffmpeg requires relinking or broke other dependencies
ffmpeg anyways got installed in /usr/local i guess..
Also, I dont have a world camera.. but it sees my webcam as world camera. But it doesn't show any frames
What should be the names of the cameras if I do a lsusb? :p
@user-d00e4f - I'm AFK right now. Will respond in about 1 hour to messages.
okay!
@user-d00e4f - I think you may want to try rebuilding dependencies or checking the dependencies with their example files prior to rebuilding all dependencies
especially pyav
@user-d00e4f what setup do you have in terms of pupil hardware? If this is a hardware issue please write us an email.
@mpk I will try what @wrp has suggested. If I conclude there's a hardware issue, I'll drop an emial
Hi all, I tried to do calibration and when I got started the software got stuck on a black screen. Does anyone know what I should do to fix this problem?
@mpk @wrp I don't know what this means, but I plugged the device into another USB slot and now camera opens after 2-3 seconds delay
Hi @user-006924 - could you provide more info about your OS, machine, and version of Pupil Capture, running from src or bundle?
@user-d00e4f - this sounds like it could be a permissions issue
just to make sure - you edited the udev rules right?
echo 'SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", GROUP="plugdev", MODE="0664"' | sudo tee /etc/udev/rules.d/10-libuvc.rules > /dev/null
sudo udevadm trigger
Hi
Where should I put run time calibration plugins?
I copied the screen marker calibration, with class name changed, to capture plugins folder but it did not loaded
It did not load
@user-41f1bf Did you check the debug logs for errors during startup?
Yes, nothing showed up
So, you confirm that it should work the way I described? Putting it inside capture plugins with a different class name.
Even if it worked, it might crash the calibration settings Ui since your plugin would not be part of the system calibration plugins list.
I have never tried that. I will make a quick test later when I am at work.
Thank you papr
I am working on participant driven calibration
I will require a key press before every marker
If it work, somehow, It should appear in the calibration list shouldn't it?
@user-41f1bf - I believe that the plugin would get loaded to the plugin list.
Hummm, that would not be intuitive
true - but let @papr - give this a quick test when he is at the keyboard
Have a nice day @wrp
I need to sleep now 😪
ok - talk later @user-41f1bf
👋
@wrp is right. If it is loaded, it would appear in the plugin list.
@user-41f1bf Simply copying the screen_marker_calibration.py
file and renaming the class does not work since the class depends on other files in the calibration_routines
module. Please send me your calibration plugin such that I can test it.
I will send it to you when I am at university. Maybe copying the whole thing?
@user-41f1bf I am just wondering how you managed to not get any error messages
Well, I will double check this time
I am not running from source, I am using the 0.9.3 bundle
Maybe your logger settings are off
@wrp My OS is windows 8 .1 and I'm using v090 version for pupil.
This is the message I get after I'm forced to close the program
world - [WARNING] plugin: Failed to load 'participant_driven_calibration'. Reason: 'attempted relative import with no known parent package'
Yes, logging settings were off
Yeap that's what I got too
But I am pretty sure that Capture will crash if you get it loaded and select it from the Plugin list
@user-006924 I afraid you will need to update to 0.9.3 and idealy run it on win10
We need a good way to add runtime video source/calibration plugins to their respective lists. there is none yet.
I have posted the participant_driven_calibration in the participant driven issue
Not a big deal, just added a boolean and a onkey event
yeah, I saw that. you will probably need to add the dependencies for the copie to work at all
thank you @papr , I will figure out how to do it
some weird bugs where solved in 0.9.3 @user-006924
@user-41f1bf Thanks alot I make the adjustments,I did use the 0.9.2 version but it stopped loading my world camera out of nowhere which was working just fine and sent the error that the camera is blocked .
If I am corret, including the one you have posted.
Did you try 0.9.3?
No I haven't. I have to change my windows first.
@papr I updated the participant_driven_calibration.py code
Now it works 😃
running from source only
I will try some hours to make it happen for runtime
capture does not crash when selecting the plugin from plugin list?
I am running from source.
No, it does not crash if you update the __init_.py accordingly
It will appear in the calibration list
which init.py?
pupil/pupil_src/shared_modules/calibration_routines/__init__.py
just as a tipp, if you choose git
or diff
(not sure which one) as the source higlighting, you can easily post git diffs
hummm
a moment...
you added the plugin manually to the list, this is cheating 😄 this works of course. the actual challenge is to find runtime plugins that are calibration plugins and add them dynamically to the calirbations lists
Yep
you can test runtime plugins from source as well, btw 😉 you will have to put the plugin into a different folder though
I have been thinking about this question all day. I think best place to implement it would be the load_runtime_plugin function. it already does class checks. it could return a 3-tubple: normal plugins, calibration plugins, video source plugins
but the eye process needs access to the video source plugin list as well. this is still far from optimal
you mean import_runtime_plugins function
yes
What about a different folder for calibration plugins?
ah, you mean, that the calibration_routines init.py could load calibration plugins on its own?
Yes, but I am not sure where to implement this
thats actually not such a bad idea
you could modify import_runtime_plugins to take a folder
and a base class as filter
or only a base class
and then the calibration routine can reuse the runtime load function with the filter calibration_plugin
that is definitevely better than my idea with the 3-tuple
For the sake of clarity and safety I would prefer to create a specific function first
yeah sure, just copy the import runtime function
I guess I have enough information, lets see what I can do
Cool! Good luck 😃
anyone have any luck using the 3d model?
just analyzed some eye tracking data and I'm getting some low frequecy noise in the Y direction mostly and i'm assuming its due to the fact that the person using them is scrunching their nose and this can probably be fixed w/ proper 3d model calibration
Using for what @user-ed537d exactly?
using it to track gaze. Currently with the low frequency noise in the Y dimension its hard to use that data to determine gaze during observation of an action as this noise will show a different location as the position drifts over time (~6 min).
hummm
@user-ed537d can you provide a demo recording with eyes?
sure
uploading to dropbox now
I'll look at it when I m in the office tomorrow
thank you!
also has anyone noticed that the fps for world and eye drops during a calibration? is this normal behavior that i'm just noticing now? or is it possibly due to the fact I'm calibrating off a TV?
This is your CPU not being fast enough.
During calibration the world image is searched for the marker and this is cpu intensive.
i5 2500 isn't fast enough?
It depends on the scene image and your camera resolution. It's not too hurfull to have reduced frame rate during calibration though
ok I figured that its fine.
The marker tracker could be optimized for speed I m sure....
Circle marker tracker that is
ok
I did it
from calibration_routines.calibration_plugin_base import Calibration_Plugin
from calibration_routines.finish_calibration import finish_calibration
calibration plugins can be easily imported directly from calibration_routines
I am importing calibration plugins from inside plugins/calibration_routines
the import_runtime_plugin_function works without any change
calibration_plugins_path = os.path.join('plugins','calibration_routines')
runtime_calibration_plugins = import_runtime_plugins(os.path.join(g_pool.user_dir, calibration_plugins_path))
calibration_plugins = calibration_plugins + runtime_calibration_plugins
those lines in world.py were enough
cool! it would be awesome if you could create a PR for that to review and test the details
I am doing this
@papr done
great! thank you, I will have a look at it when I have a bit more time
@user-ed537d , fps drop is nothing compared to Ubuntu pop ups and resolution flicker.
😃
I have a dual monitor setup. All sorts of messages are popping up during calibration (about resolution, HDMI in , monitors IDs). What can I do??
what kind of graphics card are you using?
None. On board graphics.
@everyone any tips on avoiding Ubuntu monitor pop ups?
i don't get those popups and i'm using a graphics card...when i tried using a onboard graphics i ran into a ton of issues
Actually I am using a datashow as second monitor
I am not sure when those pop ups begin to showing up. It was just fine some time ago.
When calibration starts, it seems that the resolution is changed, forcing monitors to invalidate.
@user-41f1bf odd that doesn't happen to me on an ubuntu machine...if you have a spare graphics card i think its worth testing it out
@user-41f1bf [email removed] does anyone know how to go about creating a new calibration routine that puts points in the lower half of the screen....i really don't understand the whole setup of inheriting those calibration properties etc...
@mpk heres the dataset I was talking about that shows some low frequency Y-axis noise https://www.dropbox.com/sh/9f4ne1ap6lwxw03/AACTJ2tF28wqEcEsAXr9omi8a?dl=0
@user-ed537d What about checking the screen marker calibration code? Look for "sites". It is not that complicated and you don't need inheritence taking it as an example
ok sweet i'll give that a shot Thanks!
Just remove some of the items from the "sites" array to figure out how it works
I mean, remove some and one by one
Hopefully @mpk will release sooner a new version with runtime calibration plugins properly loaded
@pupil_labs why not including a participant driven calibration option? I have two arguments. First, participant driven calibration tends to produce better calibration results. Second, a new product may be included in the pupil labs list. I am thinking about a calibration button.
i just added like 4 spots will see if it works when i connect my pupil currently working remotely 😦
what do you mean by particpant driven calibration?
ahh sampling
so they say when they're looking and you can end it when you want
couldn't you put a pause in the calibration routine that waits for a key press before it starts sampling?
The participant knows best when he/she is looking at the calibration marker.
I just hacked the screen marker calibration to wait until a key is pressed before each marker
The pupil clip could have a button to serve this specific purpose.
I am asking why this functionality is not present by default!
Ahhh I understand now
Does the coordinate system use x from bottom left to the right and y is top to bottom? Also where'd you put the key press line? I'd like to try this out
You can found the plugin in my repository on github
capture_plugins
Sweet ty
diffs from different repositories are possible in github??
Hi .. I have a quick question. Is there any easy way to disable the IR lights? I assume it's somehow controlled using UVC?
Or alternatively make them less bright?
could add a resistor
but that might be tricky because its all smd
Ahhh I see. I thought there might be a way to control the LEDs, but it's okay if not
May I ask why reducing the led intensity?
In our setup there is already enough IR light, so we don't need the additional IR from the glasses
Hi There, I was wondering if someone from the community can point me to a place where I can download the Pupil Capture App for MAC? I saw a bunch of YouTube videos running it but can only find the Windows and Ubuntu Versions on GitHub. Many Thanks.
[email removed] ! I am looking for someone to help me with the surface-tracking export ouput. I got an understanding-issue regarding srf_positions_*.csv 😃 As I've found out so far, it's outputting the homography matrics between world camera's image space and the surface's coordinate space and vice versa; What I am kinda lacking is to understand how to extract the surface's position in the world camera space. Do I have to do basically take the corner points (0,0), (0,1), (1,0), (1,0) and use the m_to_screen matrix somehow... right? Just the question now is... how 😃 Normal perspective transformation? i.e. P' = M * (P_x, P_y, 1) ?
Morning, Has anyone some good tutorial on how to get pupil apps running on a MAC? Thanks for help in advance
@user-f68ceb - https://github.com/pupil-labs/pupil/releases
currently v0.8.7 is available for macOS - we are chasing down a PyInstaller issue with Python v3.6 which is why we do not have v0.9.3 bundles available for macOS
However, you can always run Pupil from src on macOS
Thanks @wrp – By source you mean launching it via terminal? Any instructions online on that?
And on another topic – does it work with a webcam, too? Or even the built in laptop camera?
@user-f68ceb - http://docs.pupil-labs.com/master/#macos-dependencies
[email removed] - new docs site is online (we will be taking the old wiki offline this week)
@here - the old Wiki is no longer - all docs content is now at docs.pupil-labs.com - please fork the pupil-docs repo and make a PR to add content: https://github.com/pupil-labs/pupil-docs
rip wiki
💯 👌
@wrp @mpk any update on the dataset I uploaded?
Rip wiki 2
@user-41f1bf i put your calibration plugin in pupil/pupil_src/shared_modules and i can't seem to find your plugin anywhere...
@user-41f1bf nvm i got it to run i just put it into the plugin folder but now i get an error
world - [INFO] plugin: Loaded: Participant_Driven_Screen_Marker_Calibration Traceback (most recent call last): File "pyglui/ui_elements.pxi", line 354, in pyglui.ui.Selector._on_change (pyglui/ui.cpp:18849) ValueError: <class 'participant_driven_calibration.Participant_Driven_Screen_Marker_Calibration'> is not in list
added this to my capture/world.py file
from participant_driven_calibration import Participant_Driven_Screen_Marker_Calibration
and it crashed when i try to load the plugin
Traceback (most recent call last): File "pyglui/ui_elements.pxi", line 354, in pyglui.ui.Selector._on_change (pyglui/ui.cpp:18849) ValueError: <class 'calibration_routines.calibration_plugin_base.Calibration_Plugin
Have you checked your dependencies? This plugin is 98% copied from the 0.9.3 screen marker calibration.
It seems that your pyglui is not updated.
I would like to thank you for trying it. Hopefully @papr @mpk and @wrp will give it a try also.
@user-ed537d sorry for this late reply. I was helping building my new house.
:)
@everyone I want to move the cameras out from the headset. Can someone point me to the place where I can get the connecting cables?
@user-d00e4f - send us an email sales@pupil-labs.com
we can ship a cable harness or cabling
cool, thanks @wrp
@user-ed537d - I looked at the dataset but have not yet had time to respond - apologies - will try to get to it by the end of the week
@wrp no problem thanks for looking at it either way. The only reason I ask is because the subjects using this will usually scrunch their nose because of an itch etc and once they scrunch because we are using 2d calibration methods the gaze point shifts up or down due to the fact of the world camera settling at a new point in the Y direction.
@user-41f1bf thank you for your help i'll try and figure this out sooner rather than later currently the computer i'm using is at a remote location has anyone had any success debugging w/o a pupil hooked up?
by hooked up i mean physically connected...
Depends on the subject matter, but it is possible to use video files or the fake capture as video source input.
I had some success, I do not transport my eye tracker, for me it is safer to leave it at the work. 😃
running into this error when trying to build opencv have googled a bunch anyone have any ideas?
In file included from /home/nps/opencv/modules/core/src/precomp.hpp:55:0, from /home/nps/opencv/build/modules/core/opencv_core_pch_dephelp.cxx:1: /home/nps/opencv/modules/core/include/opencv2/core/private.hpp:63:23: fatal error: tbb/tbb.h: No such file or directory compilation terminated. modules/core/CMakeFiles/opencv_core_pch_dephelp.dir/build.make:62: recipe for target 'modules/core/CMakeFiles/opencv_core_pch_dephelp.dir/opencv_core_pch_dephelp.cxx.o' failed make[2]: [modules/core/CMakeFiles/opencv_core_pch_dephelp.dir/opencv_core_pch_dephelp.cxx.o] Error 1 CMakeFiles/Makefile2:1490: recipe for target 'modules/core/CMakeFiles/opencv_core_pch_dephelp.dir/all' failed make[1]: [modules/core/CMakeFiles/opencv_core_pch_dephelp.dir/all] Error 2 Makefile:160: recipe for target 'all' failed make: *** [all] Error 2
this is what the cmake shows
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found suitable version "1.2.8", minimum required is "1.2.3") -- Could NOT find TIFF (missing: TIFF_LIBRARY TIFF_INCLUDE_DIR) -- Could NOT find JPEG (missing: JPEG_LIBRARY JPEG_INCLUDE_DIR) -- Could NOT find Jasper (missing: JASPER_LIBRARIES JASPER_INCLUDE_DIR) -- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.8") -- Checking for module 'gstreamer-base-1.0' -- No package 'gstreamer-base-1.0' found -- Checking for module 'gstreamer-video-1.0' -- No package 'gstreamer-video-1.0' found -- Checking for module 'gstreamer-app-1.0' -- No package 'gstreamer-app-1.0' found -- Checking for module 'gstreamer-riff-1.0' -- No package 'gstreamer-riff-1.0' found -- Checking for module 'gstreamer-pbutils-1.0' -- No package 'gstreamer-pbutils-1.0' found -- Checking for module 'gstreamer-base-0.10'
You must install intel-ttb that's whats missing
@user-ed537d - tried to download your files
but was not able to download. Dropbox message "The zip file is too large"
@user-ed537d can you remove the exports dir to make the zipped file smaller/possible for download or upload to another service/location?
@wrp will do so when I'm at a computer.
@mpk will try that and report back
@user-ed537d noted, thanks
@wrp i removed the exports folder from the share link
updated link is above
now running into this issue with source build on ubuntu...
Traceback (most recent call last): File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap self.run() File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run self._target(self._args, self._kwargs) File "/home/nps/pupil/pupil_src/capture/world.py", line 385, in world on_resize(main_window, glfw.glfwGetFramebufferSize(main_window)) File "/home/nps/pupil/pupil_src/capture/world.py", line 180, in on_resize graph.adjust_size(w, h) AttributeError: module 'pyglui.graph' has no attribute 'adjust_size'
i think i'm getting close to finding out what the problem is...my pip3 install of pyglui doesn't have a graph file located inside of it...
i've tried --upgrade still no graph file...
@Matiar this might be due to an outdated pyglui version. Independent of the graph part.
it says i'm using 1.3
when i look at the github master it has graph my dist packages in python3 doesn't have graph...
i'm so lost rn
Did you use the git+<github repo> option to install?
yes
Oh wait. This means you are using a version that is too new. :/
if i go to the terminal and load up python3 and type the following it yields what i think i've found that i don't have a graph module
from pyglui import graph Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: cannot import name 'graph'
to clarify i used "sudo pip3 install git+https://github.com/pupil-labs/pyglui" to install
This is my bad. I made two pull requests. One to pyglui one to Pupil. The pyglui one was merged I think. The Pupil one was not.
I am not sure why it does not build the graph module. This is a different problem.
This should be the correct way to install pyglui
this is what my pyglui folder in dist packages looks like
That is the correct content
It includes the graph*.so file
should i just try a restart?
The problem is that my pull request broke the old api. You cannot do anything right now to fix this. It is my fault. I will try to fix this as soon as possible
ok
Are you running from source or from bundle?
source
bundle works fine
i need source as our experiments require me to put more calibration points in certain parts of the screen
OK. Easiest way should be to checkout the previous pyglui commit instead of the current one.
ok would that just be based on tag?
I will provide you an exact command in a few minutes
thanks!