Hi, can Remote Annotations be used to annotate at a high frequency, say 100Hz? I'm using a PupilCore

Hi, yes, that is possible. You might want to use this custom plugin that does not log the annotation to the main window: https://gist.github.com/papr/7b940b2c02e05135f59d599a6a90c5f6

You will also need to install the custom plugin. It is not built in.

Forgot to say that I did install it

Great, thanks!

Hello, I have done the calibration with the glasses core and I would like to be able to get the point of the gaze in python. I am using that code https://github.com/pupil-labs/pupil-helpers/blob/master/python/recv_world_video_frames_with_visualization.py to get the video, but I don't know from where I can get the location of the gaze.

Someone can help me?

Hi again, what is the best way of adding a label to a recording (using the Network API)? The label is associated with the entire recording and not to a specific timestamp

hmm could work but the label is not unique

Can I get the auto generated name somehow?

Can a plugin save a file to the recording directory? How does it get the path in that case?

Awesome thank you!

What makes creates this pop-up window and how do i stop it?

Ohh interesting, is that saved to the recording directory?

The result is written to user_info.csv.

Nice how does it work can I modify the defaults? I can't see a save/confirm button

I mean can I modify the dictionary keys

When the recording is stopped, the current values will be written to the mentioned file. Unfortunately, there does not seem to be the possibility to set the values via the network api. You can add more fields by modifying the dict representation at the bottom

I just realized that the user_info.csv is always generated. So we could write a custom plugin that accepts values via the network api and sets these values in the recorder plugin

Let me draft something up. @user-74ef51 do you know how to send notifications via the network API, already?

Actually, having a pop-up in the GUI is better for my usecase then what I had in mind before, there is another problem however

The videos I collect are short ~20s and there is a risk that the person does not have time to fill it out since it disappears when the recording is done (which is controlled by the network API)

My usecase is this: The person responsible for the data collection starts a script on the computer that show an animation and at the same time starts the recording (via the network api) and annotates the recording with data about the animation. However I also want the data collection person to lable the recording with some personal data about the test subject

Is it possible to go the other way i.e. creating a plugin that shows the pup-up and a start button, which when clicked sends a signal over the network API to start the animation+annotation?

Yes, that was my initial idea which should work fine, just not as pretty. I have already started experimenting with a plugin for storing the labels to a json file by sending them as a notification and listening to "notify.recording.started" to extract the "rec_path"

Does it sound like a good idea?

What am i doing wrong here?

# Client
topic = "Add_Recording_Label"
payload = {"topic": topic, "label": "my label", "name": "Add_Recording_Label"}
pupil_remote.send_string(topic, flags=zmq.SNDMORE)
pupil_remote.send(msgpack.dumps(payload, use_bin_type=True))
print("Added label:", payload["label"], ",", pupil_remote.recv_string())

# Plugin
from plugin import Plugin
import logging
logger = logging.getLogger(__name__)

class Add_Recording_Label(Plugin):
    def on_notify(self, notification):
        logger.info(notification)

What do in need to change so that my notification shows up in on_notify?

Notifications have two requirements: - a subject field (string) - the topic field should be formated like this notify.<subject>

You could also use this library that I started developing https://pupil-core-network-client.readthedocs.io/en/latest/api.html#pupil_labs.pupil_core_network_client.Device.send_notification

Hi again, sorry for bombarding you with questions. Can I use the Network API to set resolution, frame rate and auto exposure of the eye recordings?

Did you find anything @papr ?

@user-74ef51 The corresponding notification should be

{
  "topic": "notify.start_eye_plugin",
  "subject": "start_eye_plugin",
  "target": "eye0",  # and "eye1" respectively
  "name": "UVC_Source",
  "args": {
    "frame_size": (width, height)
    "frame_rate": 200,
    "name": "Pupil Cam2 ID0",  # and ID1 respectively; Cam1 for 120 Hz models
    "exposure_mode": "auto",
  }
}

Just like this?

notification = {
    "topic": "notify.start_eye_plugin",
    "subject": "start_eye_plugin",
    "target": "eye1",
    "name": "UVC_Source",
    "args": {
        "frame_size": (400, 400),
        "frame_rate": 120,
        "name": "Pupil Cam1 ID1",
        "exposure_mode": "auto",
    },
}
pupil_remote.send_string(f"notify.{notification['subject']}", flags=zmq.SNDMORE)
pupil_remote.send(msgpack.dumps(notification))
pupil_remote.recv_string()

This caches the eye process

Is ID1 the same as eye 1? What is Cam1?

Alright, I think I have the new version (I got it last week) I usually record in [email removed] but it also works in [email removed]

I tried but it still crashes

Oh right the capture.log? Note that I first sett the values to [email removed] manually to check that it changes back. I do that before starting the Network API controller script

I am not sure how to work around this issue. I will have to try this myself.

How about sending this message, and then stopping starting the eye process?

Sounds like a race condition, i.e. an issue that is triggered by specific timing. Try subclassing UVC_Source like this in a user plugin:

import time
from video_capture.uvc_backend impot UVC_Source

class Wait_Before_Init_UVC_Source(UVC_Source):
  def __init__(self, *args, **kwargs):
    time.sleep(0.5)
    super().__init__(*args, **kwargs)

Then change the name field in your notification accordingly.

The added sleep will allow the system to release the prev. acquired resources before attempting to claim them again.

No, to the former. The latter sounds like the plugin was not loaded correctly. I did not test the code. Please check the debug logs. It will tell if the plugin was attempted to load and if it failed.

https://github.com/pupil-labs/pupil/blob/master/pupil_src/launchables/eye.py#L220-L227 looks like eye process plugin must inherit from PupilDetectorPlugin, too.

+ from pupil_detector_plugins.detector_base_plugin import PupilDetectorPlugin

- class Wait_Before_Init_UVC_Source(UVC_Source):
+ class Wait_Before_Init_UVC_Source(PupilDetectorPlugin, UVC_Source):

-   super().__init__(*args, **kwargs)
+   super(UVC_Source).__init__(*args, **kwargs)

But this multiple-inheritance workaround might have side-effects that I cannot predict right now. See also https://docs.python.org/3/library/functions.html#super You might need to overwrite multiple functions with their respective super calls or implement them with a pass.

mmh, I wonder if we had an easier time by monkeypatching the init function.

import functools
import time
from video_capture.uvc_backend import UVC_Source

@functools.wraps(UVC_Source.__init__)
def custom_init(self, *args, **kwargs):
  time.sleep(0.5)
  UVC_Source.__init__(self, *args, **kwargs)

UVC_Source.__init__ = custom_init

this instead of the subclass

ok, I will need to test the code tomorrow. Just wrote this from the top of my head 😄 But I think this could be a viable solution.

(also, it had a typo in the import statement)

No, because the notification causes the prev plugin to be overwritten. And you need the sleep between shutting down the old one and starting the new one.

Can someone help me? I have my scene with a 3D shape, and I want to Store data of each gaze point. I want to Store the timestamp, the coordinates of the 3d point of the shape I'm looking at, the camera position, the camera direction... To sum up, the variables in the script gazeData. How can I get them and Store them as I'm performing the experiment?

For example, in this case I want to store the point of the shape I'm looking, the timestamp in which I've looked at it, the position of the camera at that moment, the gazedirection...

How could I do it??

Hi, did you have time to look at the plugin restart problem?

The wrapped code called itself recursively 🤦 Please try

import functools
import time
from video_capture.uvc_backend import UVC_Source

UVC_Source__init__ = UVC_Source.__init__


@functools.wraps(UVC_Source.__init__)
def custom_init(self, *args, **kwargs):
    time.sleep(0.5)
    UVC_Source__init__(self, *args, **kwargs)


UVC_Source.__init__ = custom_init

I should put that code in a file under plugins called whatever I want right?

Okay, I still get errors

I still get the error but now it is delayed 10s

The capture.log?

Hi @user-f76ddf 👋. Relevant documentation: -Accessing data: https://github.com/pupil-labs/hmd-eyes/blob/master/docs/Developer.md#accessing-data -Recording data: https://github.com/pupil-labs/hmd-eyes/blob/master/docs/Developer.md#recording-data

In order to determine which object is being gazed at, you could use raycasting to find the intersection of gaze and objects in the scene: https://github.com/pupil-labs/hmd-eyes/blob/master/docs/Developer.md#gazedirection--gazedistance-instead-of-gazepoint3d

@papr Hi papr! We've been making progress on closing the difference between the realtime and post-hoc calibration routines for vr headsets running HMD-Eyes. We've been working off of the post-hoc-vr-gazer branch, and I noticed that in the gazer_3d/gazer_hmd.py file that contains the binocular model for the realtime calibration sequence (ModelHMD3D_Binocular), there is no equivalent model for the monocular models. The ModelHMD3D_Monocular's _fit() function just returns a NotImplemented. Are there plans to implement this monocular model? If not, what monocular model is being used for realtime HMD data in that branch?

I was not able to find anything helpful yet. In the meantime, could you try running Pupil Capture with sudo and try the replacement again?

Okay I see, unfortunately starting pupilcapture with sudo (like this sudo pupil_capture) does not seem to make any difference other than that it can't find the plugins anymore. Also, I noticed that this warning: eye1 - [WARNING] uvc: Could not set Value. 'Backlight Compensation'. keeps popping up regardless if the other problem occurs, do you know what it means?

Hello, I am working with capture version 1.14.9. I am wondering how I can make sure my pupil is beint detected using the pupil detector. With the new version of pupil capture I was able to see a video of each of my eye. I would then move my eye around to make sure my conf was around 1. Is this different for the older versions of capture? So far I have tried Activate source and I choose Pupil Cam2 ID1 (for Eye 1) and Pupil Cam2 ID0 (for eye 0). It's not a hardware issue since it works fine with the most up to date version of pupil capture. Thank you!

why detect is wrong?

Hello everyone, i m trying to make datalogger with arduino to get data from pupilcore and another sensor. Do you know if some people had ever done similar things ? Thanks for your help 🙂

I do not know anyone who used an arduino for that purpose yet. Please be aware that you will need a device that is able to run Pupil Capture. The arduino is likely not powerful enough to run this software. But you should be able to receive data via Pupil Capture's Network API https://docs.pupil-labs.com/developer/core/network-api/

Hi everyone! I wonder if you know the exact recording settings they had in the videos from pupil core example video 'sample_recording_v2' : eye0.mp4 ? Thanks in advance.

Oh, and another question: do you know a way to extract heart rate using the video recordings? Any one that have tried it out? 🙂

Thank you for your response! I was able to finally detect my pupil. What is the difference between 2D and 3D detection, or better which one should I be using? Thanks again 🙂

Hello, I am trying to analysis pupil invisible recordings in Pupil player and I have 2 hour, 1 hour, and 45 minutes videos to analyze. I am using the linux version of pupil player. My issue is that the 45 minute videos take about 20 mins to create the marker cache but the hour long video takes up to 4 hours to process the marker cache and the two hour video assuming it does not crash easily 12+ hours. I know I could use the cloud software for doing surface tracking but I do not see anyway to pull gyroscope data/head pose tracking data. I did some initial testing and I can easily pull the Gyroscope data out of the videos as long as I have the surface tracker plugin turned off but head pose tracking runs into the same problem as the surface tracking as it eventually slows to an absolute crawl or crashes the software. My question is there a way to cut the length of my videos or is the best solution using a combination of pupil cloud to do the surface tracking and simply pull the gyroscope data from the pupil player? I would like to cut the videos because then all export data is nicely organized by pupil player but I completely understand if this is not possible. I think the head pose tracking could be nice but I am fairly confident that the gyroscope data will work for our purposes.

That is great, I must have missed it when I was first reading this documentation. Thank you @papr

Hey, do you have something like a doc which is referencing all kind of notification that we can send to IPC Backbone ? I have some weird results sometimes, I think I am not understanding exactly what is doing my notification.

I was analizing the mean colour intensity for each frame and I could identify regular peaks in the example videos but not in the videos that I was recording myself.

I am using a Microsoft Windows 11 pro. Processor : 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz, 2304 Mhz, 8 Core(s), 16 Logical Processor(s)

ah, yes, I use the Pupil Core headset

yes, sure

https://drive.google.com/file/d/1vzjZkjoi8kESw8lBnsa_k_8hXPf3fMMC/view

Well I was trying things with remote recording (you send me the github link 2 days ago). The thing with this project is that I have some latency because in the video backend, the guy is sending image per images... I was wondering, is it possible to select a video source for eye0, eye1, world from an URL ?

Sorry for this naïve question, but I'm still confused. I'm checking this tutorial https://github.com/pupil-labs/pupil-tutorials/blob/master/07_frame_extraction.ipynb, on cell 10, to denormalize gaze coordinates, i.e convert them from normal scale (0-1) to frame coordinate with size H, W, should X, Y multiply with W-1, H-1 instead of W, H? An image with size W, H should have the top left corner coordinate to be (0,0) and the opposite corner coordinate to be (W-1, H-1), right? Another way to think about this is considering the first channel (red channel) of an image with the size (W,H). Then to access the corner element, it should be img[0,0], whereas for the opposite corner, it should be img[W-1, H-1]. Hence, to map the normalized coordinate to the array, we should multiply with W-1, H-1, right?

Yes, I agree with the height and width values, but that's not the problem I'm asking about. I'm asking about the mapping normalized coordinates to a correct index in frame image. Let's say the image has the size (image resolution) of 5x5. Undoubtedly, the height and width returned from "frame_index_image.shape[-1] is 5 and 5. But when it comes to mapping a normalized gaze coordinates (0.5, 1.0) to numpy array array with size 5x5. The question is where do we put this gaze? Without math, we know the image array has index from (0 to W (or H) - 1), i.e 0 to 4, and hence the index of the gaze should be at (2, 4) which is the same as multiply by W (or H) - 1). However, now, if we simply multiply tthe normalized gaze by the width and height, we actually get 2.5 and 5 (out of index value)

Yeah, I try to apply zero-indexing since I want to map it to the location on the numpy array. But I think it's better to just calculate without zero-indexing way (i.e just multiply with W and H), then -1 to get the index on the numpy array. That would be more straightforward.