Hey there!

I need a little bit of help, Im working on a personal project for school, and I am unable to determine whether or not pupil capture is connecting with my programs over zmq. My main goal is to create a plugin that will take https video streams and enable pupil capture to read and utilize them. I would love any assistance that anyone has to offer! Thanks so much.

quick question. I'm contemplating the Neon, but it is unclear to me if the data can be stored locally. My projects would prohibit me from using any cloud based storage / processing

Thank you! can we pull it in into our own analysis software same to the Core? We use EventIDE https://www.okazolab.nl/elements-library

Hey there, I planned on using ZMQ via python, and the video streams would be a camera feed which I would want pupil labs to take in. Another issue is I'm unsure what version to use and if I should use a dev version?

So the plan will be three video streams, two for eye videos, and one for the world camera video. I'm not completely sure if I have a stable ZMQ connection since I'm sending sample data instead of real data. Would you recommend connecting and using the cameras to send real data? Next, how do I add my plugin, and make sure my plugin is enabled?

Well I am currently a high school student and we are competing in a Regional Science Fair. Our idea is to create a pair of glasses that will take where you are looking, in our scenario using pupil capture, and determine who you're looking at. From that connection we would nullify surrounding environmental noises to help individuals wtih hearing disabilites. We strayed away from Pupil Core since we wanted to create the glasses from "scratch". My current objective is to connect two interior cameras, ESP32 camera, and one exterior camera Raspberry pi Zero 2W camera, over Wi-Fi. But I am having tremendous difficulty establishing a plugin to take the https web streams of the cameras and convert it in a way for Pupil Capture to take them.

Hello Pupil Labs team !

Our company is trying to use the Neon for eye tracking. But one thing that bothers us is the Companion App. It forces us to have a complex setup process that will have to be repeated over and over and introduces complexity in the long run.

Was wondering if there was a way for us to skip the need for an Android Phone. Could it be a Linux with needed libraries for example ?

Please don't hesitate to come to me in DMs for details and discussions.

What does it mean to be an integrator ?

Let's say we want to be an integrator (using the Neon as basis since we have already bought them), what should we do ?

I am wondering which would be better, if possible to extract all recording data from the neon companion app without first hitting the export button, onto another device. Would this be possible with pl-neon-recording or real time API? I have a way to pull videos from my device after I hit export in the companion app but want to see if it is possible to have it work before

thank you Rob! One other question, which might be a little bit stupid. We have 2 use cases. 1. Client does not want cloud storage at all due to privacy. 2. Client context prevents internet access, so we have to run offline. In the case of 2, would we still be able to use cloud analysis, uploading local files?

Hi Rob. I am a developer of Okazolab and I have a quick question about LSL. How does it work exactly? Do you have a LSL server app running on a phone and streaming data to LSL. Apart of a gaze position does it stream other data, e.g. timestamps or pupil sizes? In EventIDE we actually have support for Neon Companion app but only for stating recordings and sending sync markers. Adding data access, especially in real-time would be super helpful. Thank you in advance!

Rob, thanks for info! With LSL we can run it already but I agree that implementing the RTSP would be a better option, since it streams a video too. Will both RTSP and LSL work in a local WiFi network, with no cloud/internet access?

Thanks a lot, it's all clear now. Just a feature suggestion, it would be super cool to have the Neon video stream in LSL too. LSL becomes more or less standard in research, and it much easier to handle than RTSP.

Hi! I was wondering if there's a way to automatically upload the data collected in the Neon Companion app to a different cloud than cloud.pupil-labs.com. If this is not possible, do you have any recommendations on what the easiest way is to get this into another cloud (since we don't necessarily want to upgrade our storage of the Pupil Labs cloud)? Thanks in advance!

@user-f43a29 yes, that's what I mean. I guess it's easiest to just download it from cloud.pupil-labs.com and immediately put in our own drive?

Very clear, thanks for your help!

@user-f43a29 Hi, quick question My Reference Image Mapper enrichment is completed, but I can’t find where to download the gaze.csv (with the mapped reference image coordinates).All downloads only give me AOI or timeseries files. Where can I download the RIM gaze.csv? Thanks!

I've tried it, but I still end up downloading the AOI file.

oh okayy, let me try it now!

all good now! thank you for your quick reply!!

Hi, I'm having trouble downloading from pupil cloud. I've tried download 100s of times but it always seems to cut out (incomplete download) after about 1 GB of data. Sometimes (especially smaller files) it works. Is there another mechanism (API?) to request downloads from pupil cloud? This happens both for native format and for the post-processed ones. It happens both for recordings from invisible and from neon it seems. I've tried in both firefox and chrome. Specifically it is the scene video that seems to be truncated (if I force extraction using e.g. 7z e --force). The download through the browser does not "fail" per se, it reports it successfully finished. However, when investigating the ZIP file it is ill-formed.

[email removed] unzip Timeseries\ Data\ +\ Scene\ Video.zip Archive: Timeseries Data + Scene Video.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of Timeseries Data + Scene Video.zip or Timeseries Data + Scene Video.zip.zip, and cannot find Timeseries Data + Scene Video.zip.ZIP, period.

This happens both for downloading single recordings and also when I select multiple at a time.

Hi Rob, thanks.

Connection, you mean to the internet?

I've tried from both home (high-speed cable), and from the university wifi, and from university LAN connection...so pretty reliable connections. I've also tried VPN through university server.

This is not a recent thing, it has been continuing since at least Thursday, and I've had this problem on-and-off for the last year. However, it seems to be a lot worse now, or I am trying to clean out my cloud and downloading many large recordings...

By the way, I'm at Kyoto U in Japan (not sure if it could be relevant).

Yes, my local disks have space. Even when downloading two large files at a time (should probably be 2GB+), it always stops "successfully" after about 1GB.

Some more information: the downloads always stop around 1135610635 bytes (1.1GB). Sometimes 1140.... sometimes 1142, 1144... but always around exactly 1.1GB.

I want to note the downloads I'm trying are all older than 2 years old...from 2024. I can not try any newer ones because pupil cloud locks them until I delete the old ones. But I can not delete the old ones until I've downloaded them...but I can't.

Hi Rob, thanks.

hey there! my team is currently using pupil core. I only have the following data with me: eye0.intrinsics eye0.mp4 eye0_timestamps.npy eye1.intrinsics eye1.mp4 eye1_timestamps.npy gaze.pldata gaze_timestamps.npy imfo.player.json nofity.pldata notify_timestamps.npy pupil.pldata pupil_timestamps.npy user_info.csv world.intrinsics world.mp4 world_timestamps.npy

I wanted to draw the gaze onto the world.mp4 video that I have using python and the existing data. Is that a possibility or do I need some additional stuff for the same? I need the eye tracking data as something that can be accessed in code 🙂

Hi @user-3d30b4 👋 ! The standard workflow would be to open the recording on Pupil Player and export the data, which allows you to export the data into CSV files and also directly create an overlay as you mentioned.

But doing it yourself from the recorded format, is totally doable too. It just entails some additional work on your end.

Firstly, you would like to learn how to open the recording format, have a look here https://gist.github.com/papr/81163ada21e29469133bd5202de6893e

Also note that Pupil Capture and Player's code is open source, you can read it here.

Hi @user-d407c1 ! I have already extracted this data and it ready to go. Im not able to make sense of this data and how to use this to overlay the gaze points. I thought of using the norm_pos points in the gaze.pldata file to plot these points on top of the video (ps I use opencv). I was unsuccessful. I also tried using the camera matrices and projecting the gaze 3d points, but got the same results 🙁

So, you already have the scene camera timestamps and the gaze data with timestamps, is that correct?

First, then you have to find the closest gaze points for each scene camera frame (note that the cameras are free-running https://discord.com/channels/285728493612957698/285728493612957698/1344615386850660412, so there might not be a perfect match). There are many ways depending on your needs, but one way would be :

current_world_ts = world_timestamps[frame_idx]
    closest_gaze_idx = np.abs(gaze_timestamps - current_world_ts).argmin()

Then, you would need to convert normalized coordinates to pixels, just note the coordinate system conventions where Pupil Core (0,0) is bottom-left and OpenCV (0,0) is top-left, so something like this:

    x_pixel = int(norm_x * width)
    y_pixel = int((1 - norm_y) * height)

and then plot with cv2.circle.

One more thing, this will still not match Player, as you would probably want to get the confidence value, from the gaze and remove anything below 0.6.

You may also want to apply the camera intrinsics to undistort the fisheye lens of the world camera.

That said, let’s first confirm you can generate a clean 2D projection before diving into distortion correction.

By scene camera timestamps do you mean the timestamps for the eyes? I am assuming here the normalized coordinates here are normalized values of the gaze. I actually use a similar setup for this by taking the x pixel and y pixel as you mentioned here, but still get bad values. The confidence value for these values is 1

The mentioned delay is something I can work on when checking the pupil positions in the eye0 and eye1.mp4 videos. For projecting the gaze on world.mp4 is something im struggling with. Ps: i have gotten rid of the distortion for the video

Hello everyone,

I am encountering difficulties when trying to view recordings with the Neon Player (V5.0.7). Whenever, I am dropping a recording made with Neon, I get the message that the Neon Player only supports Neon recordings and I should try Pupil Player instead.

Does anybode got the same error?

Thank you in advance!

Hello. I am not sure if this is the right place to post. I have deleted some old recordings but I still get "no storage" issue on the cloud. Any idea why? Thanks

Hi everyone, I need some help with Neo Companion.

I recorded videos in the New Companion, but I didn’t upload them in time before the software crashed. After reopening the app, the recordings no longer appear inside Neo Companion.

However, I can still see the full recording folder on my local drive, and all the files seem to be there.

Is there a way to manually import these local recording files back into Neo Companion so they can sync to my pupil cloud? I need to process this data, but since the files don’t show up in the software anymore, I’m not sure how to push them to the cloud.

Any guidance would be greatly appreciated. Thank you!

@user-4c21e5 @user-f43a29

Hi! Noted! Thank you for the quick response! I cleared the cache and restarted the app, but the recordings still don’t appear in Neon Companion. The local recording folder is still on my device, but the app is not recognizing it.

Is there a way to manually re-import or re-link the local files so they can sync to Pupil Cloud?

Yes sure!

I used to be able to type custom names from "AOI" under Market Mapper. Now, it is all locked "Unlabeled". Any idea why?

Got it. Thanks!

Hi, separate question, what is the stance of pupil regarding development of third-party products using the open API (for example, we have developed a system which uses wearable eye-tracking to diagnose some disease and are trying to commercialize it). I see in the license that it prohibits "competition" with the companion app (I'm sure lawyers would hash that out), but are there other more stringent limitations? Have any other businesses successfully applied your systems?

OK thanks. So it seems creating apps which use data from pupil is OK as long as they don't compete with pupil's eyetracking business (e.g. glaucoma in Reyedar's case). Of course selling/bundling pupil's IP along with said apps is a no-no, but simply creating a system which makes the end user purchase a pupil neon and use it with the app is OK. Is this correct?

Of course details are important but if there is a blanket ban on selling any system that includes a pupil at any point, we will not investigate further.

I'm basing this on the https://pupil-labs.com/legal/neon-companion-terms-of-use

Also of couse they shouldn't share/upload data online or share with third parties etc. But we are aiming something like the medical diagnostic app you shared for glaucoma which would be on-site system used by doctors.