is there a dedicated youtube series/documentation for how to actually plug in eye tracking information to projects

in code

I'm interested in using it in MATLAB, but anything would be helpful

Hi, Is it possible to use pye3d-detector without using Pupil Capture? Our project needs to develop our own UI to visualize both Pupil eyes video and other sensors. Thank you

In a stand-alone context, we allow the usage of pye3d only for (1) academic use or (2) commercial use in combination with the Pupil Core headset.

For now I’m using Pupil Labs Marker to calibrate the device... I suppose the calibration taken into account of the 2d image points and the 3d points of the marker to calibrate... I would like to know if it’s possible to replace the marker with my own calibration technique (that also gives me the image points and the 3d points)?

Ah, understood. Yes, it is possible since version 2.0. Let me look up the necessary class to subclass in a plugin.

Hi pupil labs team, I was reading your paper and I came across the system latency figure. What I understood from that figure is, gaze mapping only happens on every 4th frame, other three frames are ignored? I am confused because it shows the latency of 0.119 secs between every gaze estimation. Also it takes about world pipeline being 0.124 second. It means the gaze mapping result are available only 0.124 sec apart (gaze mapping result for every 4th frame) and in between world frames are ignored and are only available for offline mapping. Did I understand it correctly?

From the paper: Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction

https://dl.acm.org/doi/pdf/10.1145/2638728.2641695

Thanks @papr for the explanation. But what does here 0.119 sec refer to. I thought pupil data will be mapped as soon as pupil data is available to nearest world frame by comparing the datum. But why the wait of 0.119 sec? With exposure time of 0.033 sec then it waited (0.119-0.033) sec.

To be honest, I am not sure where that number comes from. I think it is important to understand the different components contributing to the pipeline delay: 1. Frame time (time it takes to expose the image + compress it) 2. Frame transmission (time it takes to transfer the image data via USB to PC) 3. [eye] Pupil detections (2d + 3d) 4. [eye] publishing results onto IPC 5. [world] queuing all available pupil data 6. [world] run pupil data matching 7. [world] gaze estimation 8. [world] publishing gaze results onto IPC

Delays 3-8 do not play a role for recorded data, as data is processed based on timestamps recorded during 1 (linux + mac) or after 2 (windows).

The world process always visualizes the most recent gaze and the most recent world video frame. It does not buffer frames.

Is there any example on how to read jpeg from frame publisher. I changed the python example frame format but doesn't work.

Other functionality of our concrete choreography implementations: Circle marker detection + visualization

I have Pupil Capture v 3.0.7. How can I activate the Marker Tracking plugin? The link here doesn't work: https://github.com/pupil-labs/pupil/blob/marker_tracking/pupil_src/shared_modules/square_marker_detect.py

The plugin is named "Surface Tracker". The linked file implements our legacy marker detection. I strongly advise against using our legacy markers but AprilTags as stated in our documentation: https://docs.pupil-labs.com/core/software/pupil-capture/#surface-tracking

Look under the 'Fixation Filter Thresholds' heading of best practices

Thanks. The maximum dispersion and minimum duration can be set posthoc on pupil player to decide preferences. I am currently going through these links. Thanks

It looks like we have to set up our timer to guide us on what filter to use. If the participant gaze for 400ms, the setting shouldn't be less.

Hi, I meet a problem while import Detector 2D from pupil detectors. Any suggestions? Thank you

here is the result

Thank you, I'll check my Anaconda env, maybe this is because I install opencv from pip.

In other words, the detectors would not have built if you did not have the requirements installed. But they did. So the necessary files are somewhere on your system but anaconda is not finding them.

hey guys, how many bits of data with above 0.8 confidence level should I gather before averaging and using them

I want to measure pupil dilation and eye virgence, but want to make sure the data is accurate. so what steps should I take in analyzing what's output by the system

Do I understand it correctly, that you were able to fix the problem by replacing [email removed] with absolute paths and it is now working as epxected?

@papr why is that subscribing to “gaze” slows down the application extremely? (6-12 seconds of delay)

Which application slows down? Pupil Capture or your own?

If you are rendering every gaze point one-by-one, it is expected to become slow as the rendering takes much more time (60hz for many monitors) than the incoming signal (up to 200hz).

Is there a way to get the pupil core time using a function in Matlab?

Using two loops. A render loop and and an inner processing loop.

while program_should_run:
  while socket.has_data:
    data = process_data_from(socket)
  render(data)

Consider this PI network example: https://docs.pupil-labs.com/developer/invisible/#network-api

while network.running: is the outer render loop with the cv2 calls at the end. for sensor in SENSORS.values(): ... for data in sensor.fetch_data(): is the inner loop processing data, storing it in the world_img and gaze variables.

Please try to reinstall numpy

Apologies for not being clear. By "consider this example" I meant looking at it, not running it. The example is meant for streaming Pupil Invisible scene video and gaze data. It uses a different API than Pupil Core. Nonetheless, it showcases how to use the two loops to display the most recent data.

I agree that the documentation could be structured in a more user friendly way. This specific issue is more of an software architectural problem though and is not necessarily related to the network API itself. If you want to read more about how the network api works under the hood, checkout this part of the ZMQ guid about the PUB-SUB pattern https://zguide.zeromq.org/docs/chapter5/

Could you please provide a skeleton loop for what a reading from matlab portion of pupil helpers would look like? what I've done is create a while loop that breaks when we the escape key is pressed, and that checks each bit of data for its confidence threshold. when 10 pieces of information are gathered that have a confidence threshold over 0.8, then the array with the data is averaged, and saved. the problem is, this results in the program falling behind at times. what should I do?

any conceptual help would also be greatly appreciated. For example, I am taking an array of size 10, and averaging out the pieces of data that fill it as data is being gathered (that has a confidence threshold above 0.8). after 10 pieces of data are gathered, I average out the values in the array, and use the information. Is that conceptually fine? note, I'm gathering information using the 2d pupil capture software.

furthermore, in my loop I wait until another 10 pieces of viable data is gathered before I average out the array, effectively meaning that each piece of data is used only once. should I change this? more importantly, am I missing the picture entirely?

Is this now how the pupil detector look like in v3.7?

I have read it before. I will go through it again. Thanks. Please another question. I am still yet to solve the issue of annotation notification display showing up multiple times. I just can't do without doing a prolong click on the keyboard

Can it be done in such a way that a prolong stay on a keyboard key doesn't increase the number of notifications?

I'm sorry for the late reply. Here's a quick summary of the problem and what is trying to be accomplished. We have is an issue where the data we are collecting through the TCP connection grows further and further behind real time. The way this code works is that 10 samples worth of data with 0.8 confidence is gathered, and written to a file. A function reads the saved file in real time and averages the results internally. The while loop is looping as fast as matlab physically allows it to, and yet the data output is falling behind. There is some backlog of data being stored in the buffer that is growing and growing, and the issue is that I need to know that the information I'm receiving is current, not from 10 seconds ago.

So I have a few questions, but would also appreciate thoughts and recommendations based on the above description. #1) Can we query from the top rather from the bottom? #2) Can we purge the buffer to keep it from filling up with old data. #3) How can we query faster in order to get all of the data. Can we grab all the data at once or does it have to be sample by sample?

I think the is that the buffer fills up and we fall out of time regardless of saving to disk or packing our array.

Is there a way to extract data from pupil capture without using zmq?

You can use LSL to receive data from Capture using this plugin https://github.com/labstreaminglayer/App-PupilLabs/tree/v2.1/pupil_capture

Alternatively, you can use the build-in hololens relay to receive the data via a udp socket

Any other option as this is under LGPL?

Yeah

Is it MPL or BSD? When i looked at pyzmq i think it says BSD?

Does using pyzmq come under LGPL or is it solely BSD? We don't want to use LGPL

Can using czmq be an alternative?

Technically, that is a possibility, too.

Only those that I have mentioned already.

Thank you for the reply! I have a quick followup: based on filter_messages.m, there is no way to clear the buffer or discard the information. What command should I use for that?

Oh I see, so the recv_message function clears it when being called. Alright, so every iteration of my while loop does [topic, note] = recv_message(sub_socket, 2048); which I'm guessing should clear the buffer. If that's not the problem, then the issue is probably in the way the while loop also writes to a file every 10 iterations

ooh. is there a way to clear it entirely, and take the most recent one?

and as a way of replacing the write to file process (to connect the reading of data to a separate MATLAB thread), should I use ports?

or what's the most efficient way of accomplishing communication via MATLAB, both from pupil-labs to the reading gaze file, and from the read gaze file to other files/processes in my project that need the information

got it. thank you sir!

I really appreciate the time you've taken in answering my questions

Hey there core community! Im having a hardware issue and I'm hoping someone on here can point me in the right direction: I'm trying to get pupil core to work on my Linux laptop. The machine detects the cameras in the usb port, but pupil software can't detect the cameras. I've put the headset on my PC and it works fine. I've tried rebooting, unplugging, different usb ports ( I have 2 on the laptop ), and reinstalling the software.

Hey, your user needs to be added the plugdev group. Afterward, restart the computer to apply the change.

The command to add your current user to the group is sudo usermod -a -G plugdev $USER (you will need the admin rights to perform that command)

Hi, does anyone happen to know of a specific functional alternative for the Microsoft LifeCam HD-6000? I've been looking into it on behalf of a student group project and we were told that the camera should have these specs: "Webcam - UVC compliant cameras are supported by Pupil Capture. If you want to use the built-in UVC backend, your camera needs to fulfil the following: -UVC compatible (Chapters below refer to this document) -Support Video Interface Class Code 0x0E CC_VIDEO (see A.1) -Support Video Subclass Code 0x02 SC_VIDEOSTREAMING (see A.2) -Support for the UVC_VS_FRAME_MJPEG (0x07) video streaming interface descriptor subtype (A.6) -Support UVC_FRAME_FORMAT_COMPRESSED frame format"

None of us have enough experience to be certain whether certain models meet these requirements and we haven't found any documentation that references these codes.

Hi, first time poster here: We have an HTC Vive style Eye-tracking system, that we are testing with our own Android app running on a phone (Pixel 2XL, Android version 10). I'm trying to detect when the cameras attach to the USB bus and am enumerating the PID:VIDs. Unfortunately the cameras come up with the wrong IDs (0x507:0x581), whereas on Linux and on a different smartphone (Galaxy S21, Android version 11) the IDs come up as (0x0c45:0x64ab). On all systems the Microchip hub comes up correct (0x04d8:0x00dd). Any guesses as to what could cause this discrepancy?

Hi, I have a ball on my screen which moving from left to right.. How can I detect that my gaze are focusing on the ball? Thank you.

Good morning! I have a problem with time syncronization... So, I'm using the core glasses to track eye's movements in specific tasks drawn in psychopy. However, in specific space/eye location, they don't have the same timestamp... There's always a (different) delay associated... Does anyone know what's the problem here? Thank you!

Hey 👋 What do you mean by "same timestamp"? What are you comparing it to?

I mean, when a dot (from psychopy) is in a place of the screen, the timestamp associated does not correspond to the timestamp of the glasses when the eye is looking to that position...

How can I check which clock is beeing used in psychopy? The delay between them has been between 8 to 16 seconds, more or less. This number is always different...

Also other thing that I noticed (not directly connected with this problem) is that the clock plugin in pupil player (vis_datetime) is giving me one hour less. I travelled between countries with 1h of difference... is this the reason? I mean, the clock in my computer is right...

Did you implemented explicit time sync with Capture in your experiment?

I'm using a python code to generate the timestamp from the core data Ah, ok. Let's move this to 💻 software-dev then. Can you share the mentioned code there?

Hi, thank you for the response. Surface tracking will tell me that my gaze are on the surface or not. But how can I detect that my gaze are at specific object inside the surface?. Thanks

Hi, does anyone know of a plugin or a simple python script for pupil data visualisation, e.g. pupil size over time?

Check out https://github.com/pupil-labs/pupil-tutorials/blob/master/01_load_exported_data_and_visualize_pupillometry.ipynb

Can you think of a reason, why my recording is saved like this, missing some values?

The rows with the "missing" values are 2d detections (see method column). This data does not have 3d information. Please be aware that this file include 2d + 3d data, as well as data from left and right eyes. Therefore, you will have to group by eye_id and method first, before you even can start visualizing data relative to your trial starts.

In order to do the latter, you need to slice the data into the block sections and subtract the first timestamp of each block from the remaining timestamps. This will give you relative timestamps since block start.

Is there a way to only record wih the 3d method?

removing rows from a pandas dataframe (as we use it in the tutorial) is easy. See cell 7 in the "Visualize Pupillometry Data" section of the tutorial.

Yes, my second paragraph of this message was meant as a response to your question: https://discord.com/channels/285728493612957698/285728493612957698/809419958261907529

Should my response not satisfy your question, then it is possible that I misunderstood your question. Feel free to tell me, if that was the case, or should it happen anytime in the future.

It is in the nature of events that they do not happen at the very same moment as the eye camera would record a video frame. Therefore, it is very unlikely that they would have the same timestamps. Instead, I would ~~filter between timestamps~~ slice data between two event timestamps.

Again, based on the tutorial, you can do:

trial_start = ...
trial_end = ...

mask_included = eye0_df['pupil_timestamp'].between(trial_start, trial_end)
trial_data_eye0 = eye0_df.loc[mask_included]

hi, why the eye tracking hardware is not working suddenly

Hello, I'm interested in IR safety of the PupilCore, as I have to get an Ethics Committee aproval. Is there any documentation where the IR safety is specidied?

What program are you using to process the csv files?

@user-98789c I would not merge the data frames. Since they have different columns, that is difficult to do. Instead, you can do this:

import pandas as pd

pupil = pd.read_csv("pupil_positions.csv")
# creates new column and fills with None
pupil["trial"] = None

# (1) calculate start and end timestamps based on
# annotions.csv
trial_1_timestamp_start = ...
trial_1_timestamp_end = ...
trial_1_label = ...
# (2) boolean mask that is true for rows that are
# between start and end timestamp
trial_boolean_mask = pupil["pupil_timestamp"].between(
  trial_1_timestamp_start,
  trial_1_timestamp_end,
)
# (3) set `trial` values to `trial_1_label` for rows
# belonging to trial 1
pupil["trial"].loc[trial_boolean_mask] = trial_1_label

# (4) repeat steps 1-3 for all trials

# (5) remove data that does not belong to any trial
mask_trial_not_available = pupil["trial"].isna()
mask_trial_available = ~mask_trial_not_available
pupil_clean = pupil.loc[mask_trial_available]

Hey everyone, I am trying to integrate a custom pupil detector into the Pupil Player. I followed the instructions given in the documentation and found the example file artificial_2d_pupil_detector.py on github (https://gist.github.com/papr/ed35ab38b80658594da2ab8660f1697c#file-artificial_2d_pupil_detector-py). Now to get started I wanted to see what this artificial detector does and put the file under ~/pupil_player_settings/plugins However the plugin does not show up in the Plugin Manager. I tested on Pupil Player v3.1 and v2.6 without success. In the log files it says that the python plugin file is imported and scanned but the plugin does not show up in the Plugin Manager. What am I missing?

Here is the log file from Pupil Player v2.6 after launch

Of course, I would recommend calculating the start and stop timestamps automatically. But that depends on your annotation data structure and it is therefore up to you to implement the start/stop timestamp calculation :).

@papr Ah sorry, you are correct. This plugin was meant to work with Pupil Capture I guess. Thanks. What I am trying to achieve is to run a custom pupil detection after data collection in the Pupil Player. Is this possible?

Okay thanks a lot! I was just using the wrong plugin template then 🙂

Hello, Currently, I am using Pupil Core for my research. During the last experiment, one of the cameras has been stopped working suddenly and it showed "Init failed, Capture is started in ghost mode". However, when I hold the wires in a certain position, it starts to work again. I think the possible issue is related to the disconnection of one of the wires in the attached image; so, I appreciate it if you could help me to fix the issue.

Please contact info@pupil-labs.com in this regard

Hi, does anyone happen to know of a specific functional alternative for the Microsoft LifeCam HD-6000? I've been looking into it on behalf of a student group project and we were told that the camera should have these specs: "Webcam - UVC compliant cameras are supported by Pupil Capture. If you want to use the built-in UVC backend, your camera needs to fulfil the following: -UVC compatible (Chapters below refer to this document) -Support Video Interface Class Code 0x0E CC_VIDEO (see A.1) -Support Video Subclass Code 0x02 SC_VIDEOSTREAMING (see A.2) -Support for the UVC_VS_FRAME_MJPEG (0x07) video streaming interface descriptor subtype (A.6) -Support UVC_FRAME_FORMAT_COMPRESSED frame format"

None of us have enough experience to be certain whether certain models meet these requirements and we haven't found any documentation that references these codes.

Good question! @user-a16c82 I am facing the exact same issue. Any help would be great, I'm located in Canada, so I would prefer a hardware option that is accessible from here.

Hi, what is the best way to get the camera pose and orientation relative to the eyeball sphere?

Conceptually, this looks correct.

Syntactically, this line looks incorrect:

for i in 1,all_trials: You probably meant for i in range(1, all_trials):.

How can I translate the timestamps that are given in my export file to real world time or timestamps in my video length?

Looking at my export file "Fixations on surface..." there is no frame pts in it? Only worldtimestamp, world index, fixation id, start_timestamp, duration...?

I want to know why this app can not work successfully like this.

In order to make sure Pupil Core actually tracks the pupil diameter : Is there some kind of plugin or any platform to visualise the changes in pupil diameter as a simple stimulation is being presented, like a simple flickering box, whose flickering frequency, duration, etc. changes and we can see the influences of these changes on the pupil diameter?

I have 20 trials here and I'm expecting all of them to be plotted in the same figure, but somehow only the last one is plotted. Python is supposed to hold the plots automatically, right?

The eyeball center (x,y,z) looks good but i could not find the information of the eyeball center orientation with respect to camera. I tried to find uskng solvePnP between projectedsphere(x,y) and sphere(x,y,z) but those values looks off. If this is already provided in the export files let me know. Thanks

Isn’t it the normal vector for 3D pupil? May be there is a misunderstanding I wanted to know where the eye ball is relative to camera (or say camera orientation) and I do not care about the gaze direction at this moment.

There might be a misunderstanding, indeed. Could you elaborate what you mean by

eyeball center orientation The eye model has an orientation: Its gaze direction. What orientation are you referring to?

Not the gaze direction but what the orientation of eyeball wrt to camera. Theta and phi in this case

The gaze direction is different I suppose. Phi_c and theta_c should be same for a video provided that there is no head gear slippage .

That would be a great help. Thanks

I tried to find using solvePnP between projectedsphere(x,y) and sphere(x,y,z) but those values looks off.

Hello I'm new to pupil (and to Discord as well, I am more used to Slack).

I am interested in pupil as an easy way to record eye movements, and I like very much the idea of 3d tracking. What is the most straight forward way to get it running and obtain data from it with the simplest setup? (so far I just have an ordinary webcam). Is there some docs that I can read to get myself started?

Hello @papr (I see you are the most active) is here the right place to ask for one just starting? Are there examples ready to run? E.g. one that use video snippets instead of a real time camera stream, just to be sure that everything is setup rightly?

Has anyone had an issue with fisheye model based undistortion with opencv for videos with 1920,1080 resolution? Because the pupil player export creates a rectangular video padding with black pixels, my undistortion function behaves badly horizontally, but very well vertically. It performs well in general for the 1280,720 resolution videos, but i need the FOV that the 1920,1080 provides. Would sincerely appreciate any tips or tricks people have found

hi, there is a blue circle and a red circle on the pupil, what is the difference? If I only want the blue one, how can I do?

Because I think the blue one is more stable than the red one.

hello, anybody here. Does is ablut pupil.0.2d pupil0.3d

hey, I realized i am not able to install Pupil Invisible Companion with my phone Pixel 3a Xl. Would you let me know the reason I am not able to install the app? let me know. thanks!

Also, am I able to connect my Core with Invisible app? let me know!

@user-82b4f1 To follow up on this note: Please understand that Pupil Capture only works for head-mounted eye trackers. It does not work for remote eye trackers (e.g. webcam placed in front of the subject).

Thanks, @papr, I will have a closer look at this. Additionally, I need the specification of the camera sensor (size, FOV) used for Pupil Labs core eye tracker (one with resolution 640 x 480). Is it possible to share the detail?

Hi, does anyone happen to know of a specific functional alternative for the Microsoft LifeCam HD-6000? I've been looking into it on behalf of a student group project and we were told that the camera should have these specs: "Webcam - UVC compliant cameras are supported by Pupil Capture. If you want to use the built-in UVC backend, your camera needs to fulfil the following: -UVC compatible (Chapters below refer to this document) -Support Video Interface Class Code 0x0E CC_VIDEO (see A.1) -Support Video Subclass Code 0x02 SC_VIDEOSTREAMING (see A.2) -Support for the UVC_VS_FRAME_MJPEG (0x07) video streaming interface descriptor subtype (A.6) -Support UVC_FRAME_FORMAT_COMPRESSED frame format"

None of us have enough experience to be certain whether certain models meet these requirements and we haven't found any documentation that references these codes.

@papr

I have msgpack=0.5.6 in a conda environment where I also have msgpack-python and msgpack numpy. Is there a way to make sure my code imports the right version of msgpack? I run into an error that I think is the result of my module importing the wrong version. As an aside, is it still necessary to use msgpack=0.5.6?

msgpack-python has been deprecated in favor of msgpack. I would highly recommend uninstalling msgpack-python. Also, starting with Pupil 3.0, we also support newer versions of msgpack

hi, blinking eyes have an influence on my detection, so how can I filter blinking operation

@papr

and how can I know the distance between my eye and the eye-tracking hardware?

the distance is one of the parameters in my project, I should test the distance in real time.

Thank you papr, I will think about your suggestions.

@papr we are building a test for our subjects to test the eye behavior. for that we have a video to show them. this video is about.. lets say 5 minutes, and in order to understand where the eye is looking in relation to a reference point we use the post-hoc method and create the reference points our selves. As you might imagine that's a lot of points(!). just to give an example - on a short video (1min) its about 800 points. Do you know of a better way to create the reference points?

sorry papr, my expression maybe not clear, I say again, I should know the distance after I move the relative positon between eyes and hardware.

the thing is we have a different "marker" that the software doesn't understand. Can we program a new shape to be our new marker and thus the software itself will create the reference points to us?

thank you papr

unfortunatly we don't have the time to write a new plugin, we're a bit low on time 😦

thanks anyway! if any idea comes to mind that will help us a lot! 🙂

@papr this is our "marker" in case it helps to understand:

hello! I have a problem with capture - the EYE 1 camera cannot be found :/

this is the error message I am getting eye1 - [ERROR] video_capture.uvc_backend: Could not connect to device! No images will be supplied. eye1 - [WARNING] camera_models: No camera intrinsics available for camera (disconnected) at resolution (192, 192)! eye1 - [WARNING] camera_models: Loading dummy intrinsics, which might decrease accuracy! eye1 - [WARNING] camera_models: Consider selecting a different resolution, or running the Camera Instrinsics Estimation! eye1 - [WARNING] launchables.eye: Process started.

ok thank you!

yes we have, why do you ask?

the thing is we don't really know the position in the surface coordinates because we don't use psycopy (it creates a really slow clunky video), so we generate the video using python and then record it and use the psycopy to manage the timing.

a) LGPL allows (commercial) use as is, see https://tldrlegal.com/license/gnu-lesser-general-public-license-v3-(lgpl-3) b) of course c) allowed under the conditions mentioned in a) d) Modifications to the headset are allowed (but looses warranty). You can also buy specific parts only as well. Contact info@pupil-labs.com in this case.

Please keep in mind that pye3d, our new 3d detector, is not LGPL licensed. Its use is allowed for (1) all academic uses, (2) when used as part of our bundled software releases, or (3) stand-alone in combination with official Pupil Labs hardware.

Correct. Any of the cases allows the use of pye3d

Hi! I have an question about the data output of the surface tracker plugin. For a research project, we want to use an apriltag marker as a reference point in the environment. Hereby, I need the position of the marker in the camera coordinate frame. The surface position output only gives the transforming matrix of surface from image to surf or surf to image. Is it possible to calculated the position of the marker in the camera coordinate frame soley with these matrices? It has been a while since I worked with matrices, so my mathematics knowledge needs recab/update.

Thank you!

There is a hidden feature where you simply drop a (Capture) recorded video onto the window and it will start using it as input. There is no support for synchronised playback fo multiple videos though. You can find an example recording here: - core example https://drive.google.com/file/d/1nLbsrD0p5pEqQqa3V5J_lCmrGC1z4dsx/view?usp=sharing - vr example https://drive.google.com/file/d/11J3ZvpH7ujZONzjknzUXLCjUKIzDwrSp/view?usp=sharing

Hi. Is there a way to better track surfaces that are distorted in the wide FOV camera? As you can see a lot of the screen is cropped out of the surface due to the distortion.

Hi, very silly question. Does the player only replays the recording or does it run the 2d and 3d detectors again?

You can run pupil detection and calibration post-hoc as well.

It will run the modified code. You can simply verify this by changing the detectors "method" output. That should reflect accordingly in the exported data

Hi, I'm trying to run player from source on Big Sur. I've installed all the dependencies and when I run player on I get following message File "/Users/yoncha01/Project/git/pupil/pupil_src/launchables/player.py", line 826, in player_drop from pupil_recording.update import update_recording File "/Users/yoncha01/Project/git/pupil/pupil_src/shared_modules/pupil_recording/update/init.py", line 15, in <module> from video_capture.file_backend import File_Source File "/Users/yoncha01/Project/git/pupil/pupil_src/shared_modules/video_capture/init.py", line 40, in <module> from .uvc_backend import UVC_Manager, UVC_Source File "/Users/yoncha01/Project/git/pupil/pupil_src/shared_modules/video_capture/uvc_backend.py", line 25, in <module> import uvc File "/Users/yoncha01/Project/git/pupil/pupil_venv/lib/python3.9/site-packages/uvc/init.py", line 4, in <module> import urlparse ModuleNotFoundError: No module named 'urlparse'

So if i tried to install urlparse, then I get ERROR: Could not find a version that satisfies the requirement urlparse ERROR: No matching distribution found for urlparse

Can you help me what I'm doing wrong?

Is it still possible to programmatically freeze the 3d model in the latest version of Pupil Capture? My notifications no longer have any effect. `{ "subject": "pupil_detector.set_property.3d", "name": "model_is_frozen", "value": True, # set to False to unfreeze

# remove this line if you want
# to freeze the model both eyes
"target": "eye0",

}`

It it possible that you did not install the Pupil Labs pyuvc? Please use pip install -r requirements.txt to install the requirements, or check its content if you want to install specific dependencies only.

{ 'topic': 'notify.pupil_detector.set_properties', 'subject': 'pupil_detector.set_properties', 'values': {'freeze model':True}, 'eye_id': 0, 'detector_plugin_class_name': 'Detector3DPlugin', } Is this the correct name for the new plugin?

I recommend subscribing to notify.pupil_detector.properties and sending

{
    'topic': 'notify.pupil_detector.broadcast_properties',
    'subject': 'pupil_detector.broadcast_properties',
    'eye_id': 0
}

to get an overview over all possible properties and detector classes.

correct. In an extra script perhaps. Also add a short time.sleep() between the subscription and sending the notification to make sure the subscription goes through.

Are there any studies/python scripts/plugins done on the frequency analysis of pupil diameter? for example, pupil diameter recorded while participant stares at a flickering stimulus with a predefined frequency, and then by the means of FFT, ICA, etc. on the pupil diameter time series, the frequency is extracted.

Hey @papr , could you point me to documentation which lists how the confidence/goodness measure is calculated. I'm trying to find the relevant code with no luck so far.

Is there a way to get 2d diameter of the Iris in addition to the pupil?

Which parts of the Pupil Capture software source code would need to change to use a different world camera via the USB-C mount? I'd like to use a zed mini (https://www.stereolabs.com/zed-mini/)

hi, I'm a complete noob 🙈 the logs show me this

anyone got an idea what I can change? I'm not able to see where I'm looking at, but eye0 and eye1 are calibrated and recognized

Hello, everyone, why does surface6 enter and surfce2 enters? Shouldn't surface6 come out first and then surface2 enter?

thanks, papr!

Any idea why I get these instances of zero pupil diameter?

Thank you ! what kind of situation will generate subset to overlap?

The diameter is zero during blinks or other low confidence values for example. I suggest filtering your data by confidence first, before processing the data further. See the Plot Pupil Diameter section of on how to remove low confidence data: https://github.com/pupil-labs/pupil-tutorials/blob/master/01_load_exported_data_and_visualize_pupillometry.ipynb

I have zero pupil size also in high confidnce instances. They must be blinks, right?

Looks pyav is not able to find your ffmpeg correctly. Have you installed ffmpeg via brew?

What is your output for brew --prefix ffmpeg?

/usr/local/opt/ffmpeg

Is it static vs dynamic library issue?

Please try running `FFMPEG_DIR=$(brew --prefix ffmpeg) pip install "av @ [email removed] /cc @user-764f72 please try this one as well

ERROR: Failed building wheel for av Failed to build av ERROR: Could not build wheels for av which use PEP 517 and cannot be installed directly

Is there more information that you can share with us. If the error message is very long, please use https://gist.github.com/ or https://pastebin.com/ to share the complete output.

macOS-11.2.1-x86_64-i386-64bit

Done, do I need to run the requirment.txt again?

ahh... it is not finding uvc this time...

Traceback (most recent call last): File "/Users/yoncha01/Project/git/pupil/pupil_src/launchables/world.py", line 140, in world from uvc import get_time_monotonic ModuleNotFoundError: No module named 'uvc'

This was the output re. uvc when running requirement.txt, seemed alright

Building wheel for uvc (PEP 517) ... done Created wheel for uvc: filename=uvc-0.14-cp39-cp39-macosx_11_0_x86_64.whl size=200040 sha256=c5368201f2ef0d6158dfc381476bf44e0a4c4a591a704eec059797038fce0d81 Stored in directory: /private/var/folders/l5/dxqs5b1x67309qvyym_8b9640000gp/T/pip-ephem-wheel-cache-k6zr8dx_/wheels/cb/a5/c2/4db7d009c02fa34f5b138db80bfe561442e29aebcff314c5e9

Danke Sehr!! it works. I had to install pupil_detector and ndsi again after the pyuvc. Thank you so much!! You really saved me today.

• I start pupil_capture, and I have two windows with pupil detection running, each using one of the supplied files eye0.mp4, eye1.mp4

• I have a jupyter session open where I test connecting to the service via the Network API (I think), where I test commands. I just suceeded to get some basic output but no pupil coordinates yet of any sort. More precisely I could follow the Network API guide but I am stuck here (https://docs.pupil-labs.com/developer/core/network-api/#reading-from-the-ipc-backbone) where I can't get no output (actually I modified the loop so that it is not infinite, but no luck).

...continued from above Assumes sub_port to be set to the current subscription port

subscriber = ctx.socket(zmq.SUB) subscriber.connect(f'tcp://{ip}:{sub_port}')

topic='pupil.0.2d.' subscriber.subscribe(topic) # receive all gaze messages

we need a serializer

import msgpack

i=0 while i<10: topic, payload = subscriber.recv_multipart() message = msgpack.loads(payload) print(f"{topic}: {message}") i+=1

(sorry, I am still not good at formatting code in messages in Discord...)

Hello EyeTracker, how did you generate this graph? I am trying to do something similar (from jupyter), but on pupil position, not diameter. But I'm still not able to read eye tracking data after the subscriber.connect(f'tcp://{ip}:{sub_port}') and subscriber.subscribe(topic). Maybe I'm doing it the wrong way. Any working example would greatly ease my start. Can you point me to some?

Do you need the data in real time then? @user-98789c is following the pupil tutorials that work with Pupil Player exported csv files https://github.com/pupil-labs/pupil-tutorials/blob/master/01_load_exported_data_and_visualize_pupillometry.ipynb

Yes, I found it myself yesterday. Kind of unexpected that the good things can be done only within code blocks, e.g. one cannot make bullet lists or tyope a CTRL+ENTER to make a newline within the message? I must study it better

👍 That's a combination I didn't try.

So a few additional questions: 0. I am interested in eye positions only (unrelated to a world view). Is starting pupil_service and query the server via the network API the right thing? Are there leaner options? 1. Is there any tutorial available for real time read/graph/analyze for pupil tracking data? 2. I use to switch on and off the two "Detect eye 0 / 1" buttons in the main window to start capturing the two mp4 files. This way I can only switch one at a time (you mentioned there is no way to sync the two playbacks at the moment). Maybe adding a "start/stop eyes together [ ]" control to that window would ease the sync. Can you point me to the right place in the source where that window is managed?

Is it possible at all with the PupilCore device to test vergence of the eyes - how well they work together? I am interested in testing for subtle skew deviations between the two eyes. I understand the calibration task is taking a constant average between the gaze points for higher accuracy, but is there any way around that? What would you suggest? Or is this the wrong device for that? Many thanks in advance 🙂

You are comparing the whole data frame to your trial label, instead of comparing it to the trial column

- annotations_pd_frame[annotations_pd_frame == str(trial_label)]
+ annotations_pd_frame[annotations_pd_frame.label == str(trial_label)]

Although I think a nicer solution would be to group by label:

annotations_pd_frame.groupby("label").timestamp.agg(["min", "max"])

and loop over that result instead of the indices.

Checkout this tutorial for more information on pandas grouping functionality https://realpython.com/pandas-groupby/

Thanks, I'm working on it. Is there a relationship between the indices in the annotations.csv and the world indices in the pupil_positions.csv? and can they be set to be syncronous?

@nmt thank you very much!

Hi there, is it possible to turn on surface tracker and add surface with Pupil Remote?

hello @papr , does the pye3d model publish also corneal shape data? Under which topic?

changed it according to your suggestion:

`import csv

all_labels = annotations_pd_frame['label'] label_names = list(all_labels)

eye0_high_conf_df["trial"] = None eye1_high_conf_df["trial"] = None

all_indices = annotations_pd_frame['index']

groups = annotations_pd_frame.groupby("label").timestamp.agg(["min", "max"])

for label in label_names: trial_label = label trial_start_timestamp = groups['min'].loc[trial_label] trial_end_timestamp = groups['max'].loc[trial_label] trial_boolean_mask_eye0 = eye0_high_conf_df['pupil_timestamp'].between(trial_start_timestamp,trial_end_timestamp) trial_boolean_mask_eye1 = eye1_high_conf_df['pupil_timestamp'].between(trial_start_timestamp,trial_end_timestamp) eye0_high_conf_df["trial"].loc[trial_boolean_mask_eye0] = trial_label eye1_high_conf_df["trial"].loc[trial_boolean_mask_eye1] = trial_label mask_trial_not_available_eye0 = eye0_high_conf_df["trial"].isna() mask_trial_not_available_eye1 = eye1_high_conf_df["trial"].isna() mask_trial_available_eye0 = ~mask_trial_not_available_eye0 mask_trial_available_eye1 = ~mask_trial_not_available_eye1 pupil_data_in_trial_eye0 = eye0_high_conf_df.loc[mask_trial_available_eye0] pupil_data_in_trial_eye1 = eye1_high_conf_df.loc[mask_trial_available_eye1]

label_names = list(dict.fromkeys(label_names)) for label in list(label_names): plt.figure(figsize=(16, 5)) plt.plot(pupil_data_in_trial_eye0['pupil_timestamp'], pupil_data_in_trial_eye0['diameter_3d']) plt.plot(pupil_data_in_trial_eye1['pupil_timestamp'], pupil_data_in_trial_eye1['diameter_3d']) plt.legend(['eye0', 'eye1']) plt.xlabel('Timestamps [s]') plt.ylabel('Diameter [mm]') plt.title('Pupil Diameter in ' + str(label) + ' sequences' )`

now the grouping is done correctly, but the same data goes into pupil_data_in_trial_eye0 and it's not sliced according to 'trial'.

So it does not really fit the corneal shape (a first model could be e.g. an off centred spherical surface whose main parameters are central protrusion and central curvature), it uses a fixed one from external data

hi, i have recently started working with the pupil core eye tracker for a psychological experiment. for that, i have to record the gaze positions of the participants on the monitor with respect to the monitor itself as we have to plot them on later.

i am able to extract data from the python api but i’m not sure what the data means or what would be the optimal data i should be using to carry out my experiment. can someone guide me as to what norm_pos_x, gaze_3d_x or some better data that will help me with the same?

Can pupil core output the log of the eyeball in and out of each surface? Thank you.

Hello, when I start Capture I am seeing this error and one of the eye cameras is failing to provide an image. It just started happening randomly. The eye camera window says the camera is disconnected, but everything seems to be well-connected.

Hey, is there a possibility to write the detected blinks onsets and offsets into the LSL stream as markers?

Hi, can I run the player from the command line giving path to the recordings as an argument. So the player starts immediately?

Yes, you can.

Does this difference in the y axis between the two eyes have any meaning? Can it mean that the participant was tilting their head?

do you know the how much white space should I need to have for detecting?

Hi all, does anyone have any data on how the new pye3d detector plugin compares to the old 3d model in terms of the accuracy of absolute units of pupil diameter? My initial impression is that it could be underestimating pupil size (in comparison), but maybe my pupils are really that small.

Hello @papr, I have gotten myself two videos of eyes and am trying to get them "captured" with pupil_capture, but I get the message _"Could not playback file! Please check if file exists and if corresponding timestamps file is present."_How can I generate such data files? Do I need more data files other than that? (in fact I supposed that was the purpose of pupil_capture to generate all files after actual video grabbed in real time, or, just for test purposes, from a video recording.)

The file backend is primarily used in Pupil Player and is meant to playback Capture recorded videos. These are meant to be recorded from a live video source. You can use the eye videos from this example recording https://drive.google.com/file/d/1nLbsrD0p5pEqQqa3V5J_lCmrGC1z4dsx/view?usp=sharing It includes the timestamps. You can also generate timestamps for the video. See the docs for the required format: https://docs.pupil-labs.com/developer/core/recording-format/#timestamp-files

In https://github.com/pupil-labs/pupil-tutorials/blob/master/01_load_exported_data_and_visualize_pupillometry.ipynb the only difference between the first and the second plots is that the second one only contains high confidence values. How come the y-axis label in the second one is pixels, while in the first one is mm?

That is a mistake. Both should be in mm. Thank you for pointing it out!

Hi @papr , I have questions around surface definition for multiple static face stimuli (e.g. 20) across participants (e.g. 20). I understand if the AOI is certain part of the face, eg mouth area, i can only do post hoc surface definition for each face and each participant which add up to 20 surfaces for each participant - I can’t reuse the surface define for the same face across participants? Is there a more efficient way to define the AOIs? Thank you in advance.