hi, is it possible to remotely trigger Reset 3D model ? (like it is possible to eye_process.should_start or to set detection_mapping_mode to 2d/3d)

@user-82e7ab yes just set detection mapping mode to 2d and then again to 3d it will reset the model.

perfect, thx

hi again, is it possible to cancel an HMD_Calibration (remotely), i.e. stopping without Pupil (Capture) calling finish_calibration()?

This is currently not implemented

ok

marketplace demo seems to be broken? I can start the calibration but never see anything using l,g keys. anyone with the same experience?

Hi, i've installed the lastest version of pupil capture v1.8.26 in windows. Then i performed calibration and a recording test, which lasted about 15 seconds. The question is, why when i play videos .mp4 ( i.e. eye0, eye1, and world) by using VLC i see them speeded up? i meant they last less than 15 seconds, except for audio.mp4, that one seems to be right. Could you suggest something, please?

@user-ba85a3 because we store the videos with evenly distributed timestamps and store the original timestamps externally. After exporting the video should play back normally

@papr thanks. but does it happen due to different sampling rate for eyes and world camera?

@user-ba85a3 no this happens due to how we store the video. This is one of the design decisions that let player work better.

Hi. I am using the DK2-add-on. I control it with a C++ program which works fine, except that I cannot send annotations.

When I call the Annotation_Capture the Pupil Capture crashes

@user-ec5b50 what message do you see in the log after the crash? My guess is that the message you are sending is mal-formatted.

seems you are right. I'm getting this:

Traceback (most recent call last): File "launchables/world.py", line 425, in world File "launchables/world.py", line 275, in handle_notifications File "shared_modules/plugin.py", line 318, in add TypeError: type object argument after ** must be a mapping, not str

Can you share what annotation you are sending?

the payload should look like this:

Yes. I have written a C++ - function to mimic what you do in python. It works for all other notifications as calibration_should_start...

eyeTracker.sendSerialized({"subject","start_plugin","name","Annotation_Capture","args","{}"});

I think it should be:

({"subject":"start_plugin","name":"Annotation_Capture","args":"{}"}

you need a send a mapping/dict

@user-ec5b50 I had that problem try leaving out args

yes, the function is doing that. It works for other mappings, like: eyeTracker.sendSerialized({"subject","calibration.should_start","hmd_video_frame_size","(1000,1000)","outlier_threshold","35"});

@user-ad8e2d, many thanks! I guess that was it. It doesn't crash anymore!

No bother! Glad I could help. I only use args when I have something to add and it works fine then.

Hi, does anyone know if there's the possibility to extract world camera stream and gaze, while pupil capture is running? thanks for you help.

@user-ec5b50 The exact issue is that you put the {} after args in quotes.

@user-ba85a3 Yes, this is possible. See this examples: https://github.com/pupil-labs/pupil-helpers/blob/master/python/filter_messages.py https://github.com/pupil-labs/pupil-helpers/blob/master/python/recv_world_video_frames.py

thanks @papr, i don't want to bother you anymore, but i'm stucked on PyAV. How can i install it? thanks.

@user-ba85a3 Am I guessing right that you are trying to run Pupil from source on Windows? Is there a specific reason why you need to run from source?

@papr No, actually i'm running Pupil Capture on Linux (Ubuntu 16.04) and it's working fine. Then, i'm trying to run the (https://github.com/jesseweisberg/pupil) jesseweisberg's work, but when i try to lunch main.py from terminal iget an error " no module named uvc".

Ah, I understand. This issue is not related to PyAV but pyuvc. Make sure that pyuvc was correctly installed via pip. The installation might fail if libuvc was not installed correctly.

i get this error now: what could it be? /usr/bin/ld: //usr/local/lib/libceres.a(problem.cc.o): relocation R_X86_64_32 against `gxx_personality_v0' can not be used when making a shared object; recompile with -fPIC //usr/local/lib/libceres.a: error adding symbols: Bad value collect2: error: ld returned 1 exit status error: command 'x86_64-linux-gnu-g++' failed with exit status 1 world - [ERROR] launchables.world: Process Capture crashed with trace: Traceback (most recent call last): File "/home/optolab/Pupil_Notte/pupil/pupil_src/launchables/world.py", line 113, in world import pupil_detectors File "/home/optolab/Pupil_Notte/pupil/pupil_src/shared_modules/pupil_detectors/__init.py", line 16, in <module> build_cpp_extension() File "/home/optolab/Pupil_Notte/pupil/pupil_src/shared_modules/pupil_detectors/build.py", line 25, in build_cpp_extension ret = sp.check_output(build_cmd).decode(sys.stdout.encoding) File "/usr/lib/python3.5/subprocess.py", line 626, in check_output **kwargs).stdout File "/usr/lib/python3.5/subprocess.py", line 708, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['/usr/bin/python3', 'setup.py', 'install', '--install-lib=/home/optolab/Pupil_Notte/pupil/pupil_src/shared_modules']' returned non-zero exit status 1

world - [INFO] launchables.world: Process shutting down.

For some reason ceres has not been installed in a dynamic fashion. Make sure to run cmake .. -DBUILD_SHARED_LIBS=ON [before building and installing ceres]

I don't understand why Ceres is needed. I've correctly installed pyuvc and so on, I go to the folder where the plugin I mentioned above is and try to run the main.py

when this runs, I get that error. Why do I even need to install ceres?

Ceres is required for the pupil detectors. They are compiled [when you run Capture/Player for the first time].

Hi, I'm using Pupil Capture from source on Linux MInt 18.2. I've been trying to extract gaze data. I've modified fixation plugin, with no effect. I tried to send some data to .txt file, but whatever I do, no effect. Then I tried to create offline data extractor, taking .npy and .pldata files after recording with Pupil Capture and read them, but I'm getting some unreadable data, encoded or something else I don't know. Can someone tell me more about this? And another question is how can I extract gaze pozition in world camera normalized coordinate system, online and offline using methods I mention?

@user-bb47e7 be aware that the fixation detector file has two different plugins, online and offline.

@user-bb47e7 I will send you examples and links to these things as soon as I am in the office

I didn't know that. Thank you.

Hi, I have a c++ function that correctly starts the annotation plug-in. But when I want send an annotation the annotation in the pupil_data file is malformatted. I send the annotation like this: eyeTracker.sendSerialized({"subject","annotation","label","new annotation","timestamp","1235.56","duration","1.0","record","True"});.

Does anyone know why this is malformatted?

@user-ec5b50 the numbers should not be encapsulated in quotes

but can I send integers with msg pack?

@user-ec5b50 yes you can send ints and floats.

OK, thanks!!

This is the full msgpack spec for reference: https://github.com/msgpack/msgpack/blob/master/spec.md

@user-bb47e7

Intermediate recording format: https://docs.pupil-labs.com/#recording-format Exported data format: https://docs.pupil-labs.com/#detailed-data-format Online data access example: https://github.com/pupil-labs/pupil-helpers/blob/master/python/filter_messages.py

Thank you very much. I was so focused on Pupil Capture that I forger about Pupil Player.

Hi, i'm attempting to test Pupil capture 1.8.26 and Matlab R2018a running on windows 10. I followed the guide line on https://github.com/pupil-labs/pupil-helpers/tree/master/matlab. First of all, i've installed ZeroMQ 4.0.4 on win, then matlab-zmq and matlab-msgpack packages, but i'm not pretty sure they are well installed. Is there a way to check it? Documentation combined with my poor software experience in most of the times not enough. Anyway i went on through steps described, launching Pupil and running pupil_remote_control.m, but i get this Matlab's error: Undefined variable "zmq" or class "zmq.core.ctx_new".

Error in pupil_remote_control (line 15) ctx = zmq.core.ctx_new();

Anyone can help me? thanks

Can someone explain me the purpose of this command? Because I did not understand the importance in the file 'mouse'

def move_mouse (x, y):      if system () == "Darwin":          sp.Popen (["/ mac_os_helpers / mouse", "-x", str (x), "-y", str (y), "-click", str (click)      else:          m.move (x, y)

@user-ba85a3 looks like the zmq Matlab lib is not in your Matlab path

@user-3f0708 in macos it is not possible to move the mouse using the python module. Therefore we had to create a small utility binary that does that instead.

@papr I was using the mouse_controll script normally, but now the mouse movements are not very faithful to the look tracking. The mouse is only moving in one direction, does it have something like a calibration process?

@user-3f0708 the mouse movement depends on a lot of factors. Pupil detection, calibration accuracy and surface tracking quality

@papr I'm guessing it could be surface tracking

@papr I'm doing a job where I use the pupil so the user can type words with a virtual keyboard, because this problem is making it difficult to perform my work. And I do not know how to solve

@user-3f0708 Could you make an example recording and send it to [email removed] It is difficult to judge what is going wrong exactly without it.

Hello, I'm trying to extract some of the pupil detection source into a standalone package for testing. The test script in pupil_src/shared_modules/pupil_detectors/Tests/DetectorTests/test_detectors.py imports the class autoCreateCapture, but this doesn't seem to exist anywhere in the repo. What is this class and is it still being used

This is deprecated code sorry

@papr Sounds good thanks!

I've got a different problem with detector_3d now: Seems cython can't link against google/glog: detector_3d.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6google21kLogSiteUninitializedE

However, I can't see anywhere in the source code that actually calls for google log? I've both installed glog from the ubuntu repo and from source, and added it as a library inside the setup.py but still no luck

Could be that ceres tries to link against it, but not sure

@papr Thanks. I did figure out that issue, you have a good suggestion for checking that ceres is being linked properly.

Is there any documentation on how your detector_2d/detector_3d classes operate? I think the pupil labs papers I've seen were mostly focused on the total hardware/software package instead of the specific techniques

@user-ff6db6 2d algorithm: https://arxiv.org/abs/1405.0006 The 3d detection is based on https://www.researchgate.net/profile/Lech_Swirski/publication/264658852_A_fully-automatic_temporal_approach_to_single_camera_glint-free_3D_eye_model_fitting/links/53ea3dbf0cf28f342f418dfe/A-fully-automatic-temporal-approach-to-single-camera-glint-free-3D-eye-model-fitting.pdf

Hello every dev here I'm building a eye tracking Android app using phone front cam. I can detect the pupil. Now I want to map the eye gaze into real world. Does anyone have any idea?

@user-eb5882 Hey, this type of eye tracking is called "remote eye tracking" and is different from "head mounted eye tracking". The Pupil software does "head mounted eye tracking" only.

@papr I see. So do you have any ideas or paper related to this

I cannot reference any specific papers but I would search for something like "webcam remote eye tracking". Or simply looking for remote eye tracking should yield a lot of information already

You will find that there is much more to do than just detecting the pupils. 🙂

🤗 thank you @papr

@papr is it possible to send notifications with another topic (not "notify.") , maybe "other."? I changed the sending code to the new prefix, but I get the error that no additional request can be sent. So I get no "acknowledging" as a response.

Are you sending the data via pupil remote?

yes

Pupil remote can only accept notify topics, yes

But what you should do is request PUB_PORT and create a PUB socket for that port. You can send any data on that

ok I'll try, thanks

If you use annotations. instead of notify. the data will be stored in annotations.pldata instead of notifications.pldata.

interesting!

The offline annoations plugin will not be able to detect the recorded annoations in v1.8

but we will change that for the upcoming version.

I might even be able to provide a temporaly fixed version that you can use with v1.8

@user-29e10a Actually, I was wrong. The offline annotations plugin already loads annotations from the annotations.pldata file!

@user-29e10a "But what you should do is request PUB_PORT and create a PUB socket for that port. You can send any data on that" - Even though this is technically possible, this data will not be recorded yet!

Sorry for the confusion. I am working an a fix for that

ok, the recording is not urgent, because I want to react on another way to this ... I created a socket with a PUB_PORT and sending something on this, but how can I subsribe to "other." for example? I get no data on the other side

maybe it's obvious, but I'm not into the PUB-SUB thing yet

To receive the data you will need a SUB socket that connects to the SUB_PORT

And then you subscribe to "other."

how do I get the sub_port from the new pub_port? I cannot subsribe on the same sub_port than the one on "notify."?

There are three different ports: - pupil remote, REQ: 50020 - subscription, SUB: request from pupil remote using SUB_PORT - publish, PUB: request from pupil remote using PUB_PORT

I am definitely doing something wrong. If I register my req socket normally and getting the PUB_PORT, I'm unable to "sub.recv_string()" with "sub = context.socket(zmq.SUB)" and "sub.connect("tcp://{}:{}".format(addr, pub_port))" .... no failure, but it breaks in a while loop with no exception... it hangs.

I subscribe with sub.setsockopt_string(zmq.SUBSCRIBE, 'other.')

it hangs even if nothing is sent

@user-29e10a Keep in mind that the subscription messages have two frames, one with the topic, one with the msgpack encoded payload

and be aware that calling recv() is a blocking function that only returns if something was actually received

@papr Thanks for the references and the great work

@papr Thanks, I got it, at least I have the feeling to understand zmq. The trick was, NOT to subscribe to the PUB_PORT 😉

May I ask that why the 2D pupil ellipase in 3D eye detection mode is not that stable compared with the 2D pupil ellipase in 2D eye detection mode?

There are some eye image frames 2D pupil is not detected correctly in 3D eye detection mode but correct in 2D eye detection mode.

What's the reason behind this? Is it because you are using different pupil detection methods in 2D and 3D eye detection mode?

@user-b91aa6 we would need to see an example video. Otherwise it is hard to tell.

@user-cd5907 Is the eye ball detection working correctly? It's the big green ellipse. You can turn on the debug window in 3D detection to overlay the eye model with the real eye. If the eye ball is not fitted correctly, the 2D ellipse is not stable

Here is what I don't understand. Why the 2D ellipse is not stable when 3D eye model is not fitted well?

@user-29e10a

because the location of 2D ellipse is fitted onto the 3d eye model. Understand this as a sort of "sanity check". it the headset position relative to the eye changes ("slippage") the model should be recalculated (or re-positioned). The only way to know this is to project the 2d ellipse onto the sphere and check for sanity (angle, size etc) ... correct me if I'm wrong but additionally there is a Kalman Filter which predicts the movement of the ellipse as a secondary step. The whole 2d detection is single-frame based, so the ellipse is not referenced by movement or eye model and therefore more stable, BUT has other limitations such as missing angle information or slippage detection

in fact, the 3d detection starts with the bare 2d detection and performs other steps on top

@papr Is what @user-29e10a says correct?

@user-b91aa6 Yes, @user-29e10a is correct. I was not aware of the Kalman filtering - I do not have a deep insight into the 3d detector code. I can ask my colleague tomorrow about details though!

Thank you very much. Wish to hear the details.@papr