Hi! Could you please release an updated client (https://github.com/pupil-labs/pupil-cloud-client-python) for api version2 (https://api.cloud.pupil-labs.com/v2) Or is it possible that I can generate the python client code myself with the swagger.json?
Hi @user-100d6e I have asked my colleague to reply over in the invisible channel
And is there any possibility to export videos via api? If not. Could you please implement this?
Hi, I have a question regarding blink correction of PupilCore data. I streamed the data via LSL and it contains NaNs. If I want to implement your offline blink detection I run into problems as the filter response is heavily influenced by NaNs. Any suggestions for a workaround?
Hi Jolius! The LSL relay for Pupil Core will produce NaNs for one eye where it is not possible to match pupil datums to gaze binocularly. The pairing of Pupil data to Gaze is accounted for directly in Pupil Capture. This could be affecting your use of the offline blink detector. You can read more about the matching process here: https://docs.pupil-labs.com/developer/core/overview/#pupil-data-matching
Hi @user-100d6e! Unfortunately we do not have the capacity to provide a client for v2 right now. Using the API via plain HTTP requests does work though.
What type of video are you interested in exporting? You probably have seen that exporting recordings including the scene video is possible. Are you looking for e.g. video with gaze overlay?
@marc Thank you for your feedback.
In fact as you suggested with your example: I would like to start the export for a video with gaze_overlay via API and download the file afterwards.
Basically I want to automate the process, so I don't need to log in to the frontend.
I see! This would require you to create a gaze overlay enrichment which currently is not possible via the API. We do have changes to how videos can be rendered and exported on the roadmap and I'd expect API access to become possible then, but its gonna be a while until we get to it I am afraid.
Thank you! Then we will continue to process the video export manually.
On the other hand I was thinking to download the raw files from cloud and export them via batch_exporter.py commandline. There shouldn't be any problem with that?
Therefor I also wonder if the export would be faster as in the cloud (oc depending on the local machine ressources)
Yes, using the raw files and the batch exporter should work. In terms of speed it should generally not be much faster. The computation that happens is basically the same in Player and Cloud, and in Cloud the degree of parallelization would be higher. For small job volumes things like initial waiting time in a queue in Cloud could make a difference though.
Thanks for the Info. How can the datums not be matched? I always expected them to the same recording onset and produce samples one after another
The cameras are free running. We use Pupil Time to assign video frames with a timestamp which is guaranteed to be monotonically increasing. https://docs.pupil-labs.com/core/terminology/#_2-pupil-time
Hi, in a PupilCore recording I took some time ago there is a synchronization error of about 120 ms between the right and left eye video. What could have caused this error? I do match the frames in eye0 and eye1 with the eye*_timestamps.npy files
What do you use to extract the images from the video? Note, that opencv is known to be inaccurate in that regard. See this tutorial for more details https://github.com/pupil-labs/pupil-tutorials/blob/master/09_frame_identification.ipynb
I will check that tutorial. However even thought I use opencv i can't see how that is the problem because my code simply selects the two frames that are closest to a specified timestamp according to the eye*_timestamps.npy files and reads them with opencv
Yes, exactly, opencv is known to not return the frames that you expect.
If i want to play the frames loaded with pyav as a video should i use cv2.imshow()? Or does pyav have any functionality for this?
pyav does not have image display functionality. You can continue using opencv for that. The ony thing you need pyav for is the frame extraction from the video file.
From the top of my head, the code should look like this
- frame.to_image()
+ img = frame.to_ndarray(format="bgr24")
+ cv2.imshow("Frame", img)
Hi guys,
First of all I would like to thank you for the support during the time of my master thesis and the many questions you have answered @nmt @papr .
Since I am now working as a research assistant at the TU Darmstadt, the projects continue directly.
In a future project, I would like to track the subject's gaze on other moving ROIs during night driving.
Would it be possible to place AR markers in the car that are always visible and then define a ROI on the road that you can also change during the video ?
Cobe ✌🏻
Is the code in the tutorial https://github.com/pupil-labs/pupil-tutorials/blob/master/09_frame_identification.ipynb outdated? I can't get seek to work
No, it should be up-to-date. Which OS, Python and pyav version are you using?
My code
path = DATASETS_ROOT.joinpath("2022_03_04/019/eye1.mp4")
video = av.open(str(path)) eye0_timestamps = np.load(path.with_name("eye1_timestamps.npy")) pts_of_1200th_frame = eye0_timestamps[1200] print(pts_of_1200th_frame)
requires primitive Python intvideo.seek(int(pts_of_1200th_frame), stream=video.streams.video[0]) frame = next(video.decode()) frame.to_image()
There is a difference between pts and timestamps. You are attempting to use a timestamps to seek, instead of a pts. Timestamp refers to the wall-clock time recorded by Capture. pts are presentation timestamps and refer to the media file's internal timestamps. You can easily extract them from the video using something like this:
with av.open(str(path)) as video_pts_extraction:
pts = [
packet.pts for packet in video_pts_extraction.demux(video=0)
if packet is not None and packet.pts is not None
]
Ubuntu 22.04, Python 3.10.4, pyav latest
My timestamps are in .npy format not .csv
I always get the first frame no matter what
and then use
- pts_of_1200th_frame = eye0_timestamps[1200]
+ pts_of_1200th_frame = pts[1200]
Using marker usually only makes sense if they do not move in relation to your ROI. You will need some other kind of way to track your ROI. Maybe there is software to detect street outlines in a video already?
Ok i'll have a look !
Is this not a problem From the docs:
After seeking, packets that you demux should correspond (roughly) to the position you requested. In most cases, the defaults of backwards = True and any_frame = False are the best course of action, followed by you demuxing/decoding to the position that you want. This is becase to properly decode video frames you need to start from the previous keyframe.
Can double check on Monday if this is an issue
Core recordings use mjpeg format and in mjpeg every frame is a key frame to my knowledge.
When I iterate over frames by seek as a test nothing happens for a few values and then there is a jump
Could you share the code with us?
Hi @user-b91cdf 👋. That's no problem at all, and it's great to hear you're continuing the projects! Just to unpack this a bit, is the aim to have ROIs 'earth-fixed' on the road, or have the ROIs moving with the car, e.g. covering a given portion of the road ahead? I know that both cases are of interest in this kind of research
Could be both. I'll write again after the kick off meeting with the customer. Currently i am thinking of doing a video annotation in the world video.
Wait, I noticed that this is not a problem on the original video only on the same video formatted with the recommended settings ffmpeg -i world.mp4 -vf format=yuv420p world_compatible.mp4
The converted video is h264 and has key frames. So that file might suffer from the issue note that you copied from the pyav docs.
from pathlib import Path
import numpy as np
import pandas as pd
import cv2
# This hack prevents av from freezing
cv2.imshow("temp", np.zeros((4,4,3), dtype=np.uint8))
cv2.waitKey(1)
cv2.destroyAllWindows()
import av
video_dir = Path("formated/rig_data/2022_03_04/019") # Formated video
video_dir = Path("sources/rig_data/2022_03_04/019") # Or original video
with av.open(str(video_dir.joinpath("eye1.mp4"))) as video:
eye1_pts = [
packet.pts for packet in video.demux(video=0)
if packet is not None and packet.pts is not None
]
eye1_video = av.open(str(video_dir.joinpath("eye1.mp4")))
def add_text(frame, text, pos):
return cv2.putText(frame, text, pos, cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)
pts_idx = 0
while True:
pts = eye1_pts[pts_idx]
eye1_video.seek(int(pts), stream=eye1_video.streams.video[0])
eye1_frame = next(eye1_video.decode())
if eye1_frame is not None:
frame = np.array(eye1_frame.to_image())
add_text(frame, f"pts_idx: {pts_idx}", (4, 40))
add_text(frame, f"pts: {pts}", (4, 80))
cv2.imshow("frame", frame)
key = -1
while key < 0 and cv2.getWindowProperty("frame", cv2.WND_PROP_VISIBLE):
key = cv2.waitKey(1)
if key & 0xFF == 27 or key == ord("q") or not cv2.getWindowProperty("frame", cv2.WND_PROP_VISIBLE):
break
elif key == ord("a"):
pts_idx = max(pts_idx - 1, 0)
elif key == ord("d"):
pts_idx = min(pts_idx + 1, len(eye1_pts))
elif key == ord("z"):
pts_idx = max(pts_idx - 10, 0)
elif key == ord("c"):
pts_idx = min(pts_idx + 10, len(eye1_pts))
cv2.destroyAllWindows()
And by nothing happens, you mean that video.decode() does return None, correct?
No, it just returns the same frame many many times
Interesting. Could you share the video with [email removed]
Sure,
I experimented a bit too and adding the flag -g 1 seams to solve the issue: ffmpeg -i eye1.mp4 -vf format=yuv420p -g 1 eye1_compatible.mp4 but I don't understand what that flag does
That flag defines the key frame interval. With an interval of 1, every frame is a key frame. So the issue that you are experiencing is definitely key frame related.
Do you not have this problem?
I didn't have any issues with it in the past. But I will revisit the tutorial on Monday and make sure it works as expected for both of your shared videos
Seeking within h264 videos
Hi @papr , I appear to be having an issue with the calibration sequence in my Pupil Labs Core command line pipeline. I have each component of PLC functioning in a way where I can feed in VR headset eye videos, pass through custom preprocessing steps, apply timestamped calibration points from HMD-Eyes, and get gaze data... but there is a strange accuracy problem that arises no matter how tight we can get the precision.
The image attached is a visualization of gaze data gotten with the HMD-Eyes calibration sequence, where the big blue dots are the calibration points and the smaller, colored dots are the calculated gaze locations associated with each calibration point. Each small colored dot represents one timestamp that was actually used for the calibration sequence. (Also, the two numbers above each large blue calibration point dot represent accuracy in degrees and precision in degrees respectively, from top to bottom.)
Do you know why there might be such an accuracy issue, despite this being the calibration sequence itself? As I understand it, this should be highly accurate, since it's the actual calibration sequence.
Pupil Core Calibration Pipeline
Also, for HMD-Eyes, was there a particular reason you chose the calibration point locations that you did?
We felt like they would cover the HTC Vive FOV fairly well. I don't recall any other specific reason.