Hello, can anyone tell me if it is possible and if it is how to get data from the backbone without starting pupil service?

I would like the IPC backbone started and kept running from my c++ app

Thank you for your answer, the idea would be to have as little as possible, to have only the 3D eye tracking without the interface. The idea would be to have only the backbone running, to have a deamon answering messages sent with zmq, processing the camera images and sending the 2D/3D eyes information. If it can only be done through entry points throughout one of your dll, it might still interest me, even if it requires calling the processing functions without any IPC backbone. I just do not want to produce the image processing from scratch 😬

yes

You can also find 2D pupil detectors in the pupil-detectors repository.

Lastly, there is this third-party, user provided repository, showing a way to get the Pupil Core camera feeds in C++, but I cannot make any guarantees about it. The other community contributed software can be found in the Pupil Community repository.

So if I understand correctly the image processing is done through pye3d and then the service app is written in python and serves as the IPC backbone. The service app gets images through pyuvc and feed them to calls to pye3d to process the eyes pos which is then send with zmq? If so I will look into it, thank you very much.

I am already running a custom gaze estimate, because the objects are displayed in a headset and I could not find a way to feed objects position to the pupillabs calibration procedure. Is there actually a way to do it?

I am trying to use it for XR through a custom driver

Thank you a lot for your help, I will try to go through all this to see if it fits our needs.

I just realized. You could potentially also fork the code for Pupil Service (part of the pupil repository) and configure it to skip the GUI and just receive command line arguments. Just note that we would not be able to provide support for such modifications, but in terms of overall development, it should in principle be less effort. Then, your C++ app could still use the Network API, i.e., communicate with the IPC backbone via the standard ZMQ commands.

As noted in the README of the pupil repository, we target and recommend Python 3.11. Where did you see that an old version is required?

I’m not sure what may have happened in this case. I would recommend trying again with a fresh pull of the latest commit in a clean venv using Python 3.11. The README and Developer docs are also worth referencing during the install process.

Hello - I've been able to get most of the relevant data using the cloud API, but I'd like to access the timeseries files (before enrichments) like fixations.csv, saccades.csv, gaze.csv, imu.csv, but can't find the reference to thsese in https://api.cloud.pupil-labs.com/v2 Are these accessible via the API?

(I realize they can be downloaded in a .zip file but I am trying to access them directly on the cloud rather than download them)

Yes, I see a list of files that I can access in the recordings, including some related to imu and gaze but the file list doesn't include any of the .csv files that I mentioned (fixations.csv, saccades.csv, imu.csv, etc), but which are in the directory when I download the timeseries files. Can you help point me to the exact way to access those .csv files?

Here is an example of the list of files I get from the endpoint you suggest:

Vs. here is of the files I get in the .zip downloaded from the same recording:

Hi Rob - I'm still having trouble understanding how to access those timeseries .csv files. Is there more information that might help you track this down? Many thanks!

Hi @user-62b6a8 👋 ! I just quickly put together an script demonstrating the use of the proper endpoint to download the CSV files using python. Note that, you can not really select which CSVs to download, but you will get all of them. The only file you can exclude these [ scene_camera.json, gaze.csv, scene.mp4, fixations.csv, blinks.csv, faces.csv, events.csv, imu.csv, eye_state.csv ]

Hi Miguel, thanks for the snippet. What I was hoping to do is to directly access the .csv files via the API, not to download them. That is, the same way I can access say the video files or the other raw data files. This would be a big advantage over having to store them again in another location. Is this not possible?

May I ask what you are trying to achieve?

Many thanks! the gaze endpoint is just what I was looking for. I am writing python code to integrate the gaze data with other experimental data collected separately.

hi I'm actually searcjing if is it possible to modify the max and min duration of fixations (I'm using neon)? maybe with the neon_player? or directly in the src code (of the player, for my use?)? (it is because I have way too long fixations that interfere with my data analysis)

I think there was a response but I cannot see it

it is just that I had too long fixations that I think are in fact multiple fixation because it is widely over the litterature threshold (over 500-600ms) like some time 14000

and I think as they just should be separated and not deleted because they are present and revelant for my use

but just agregated in some way I thnik

think*

it is looking at an image and so what I am actually trying to do is determine time of fixation in some AOI I define (quite precise AOI)

and because of this I cannot have consistant result between subject and some are big 'outlier'

oh okay so it is not with the player?

I was on this deposit

ok! to run it is just: pl-rec-export /path/to/rec (with the appropriate path)? because I have decodeerror (parsing message)

Yes. Make sure it is Native Recording Data (unzipped, so the same directory that you give to Neon Player). If you still encounter errors, then we can proceed in a thread.

okay this is what I use and it make an "export" folder with events.csv and gaze.csv but no fixations.csv or blink.csv

Am i still doing something wrong?

pl-rec-export fixation detector modifications

Hi @user-c541c9 👋 ! The purposes of the two tools are as follows:

• pl-rec-export: This is a command line tool to export the raw data (i.e., the Native Recording Data) to user-friendly CSV formats and to run the fixation & blink detectors on your own computer. It runs a pre-defined set of algorithms and transformations. You could use it as a Python library, but when used as a command-line tool, it is non-interactive and could be used in a shell script for batch processing data, for example.
• pl-neon-recording: This is a Python library for programmatically accessing the Native Recording Data. It handles the repetitive routine of loading the raw binary data, while leaving the rest up to your creativity. You can use it to create your own pipelines that build directly on the raw data. In this sense, it is more interactive. It comes with helper functions, such as routines that make it easier to sample the different data streams at the same timestamps, but does not run pre-defined algorithms, such as the fixation or blink detectors. I recommend taking a look at the examples to get an idea.

Here are the differences between the two Android directories:

• Neon: This is the raw binary data in a folder structure that is relevant for the Neon Companion App. It is not what you typically work with, as an end user.
• Neon Export: This is the raw binary data, converted to a more user friendly directory structure, with a more human friendly name. This is equivalent to the Native Recording Data download option on Pupil Cloud. These are the directories that you use with pl-neon-recording or pl-rec-export. See the "Transfer Recordings via USB" section of the Neon docs for the export instructions.

Could you please share the error message?

I have not encountered an error with loading the IMU data from my recordings, nor I have encountered the error that you previously posted with respect to line 27 in the basic_usage.py example.

I just tried with a new recording here to be sure that it was not anything specific about my previous test recording.

Ah, I see. Indeed, I missed that the stream names are absent in the output from the basic example. Yes, you are correct that the pl-neon-recording library was updated to lazily load the data, so the stream needs to be accessed first, then it is loaded on demand, and then you can print out its name, as you have demonstrated.

The IMU error that you encounter (Found zero norm quaternions in 'quat') is not exactly coming from the pl-neon-recording libary. Rather, it could indicate that something is off with the raw IMU data itself for that recording. I will start a separate thread, where we can dig into this a bit further, if you prefer.

Problematic raw IMU data

Hi all,

I've been using pl-rec-export to extract the gaze from phone recorded data. It creates an export folder the folder has event.csv and gaze.csv files. I'm confused by the time stamps as the event recordstop recordbegin difference is greater than the last row first row timestamp difference of the gaze data by a few seconds. also the start time of the gaze data does not match the start time of event recordbegin for example: timestamp [ns],name,type 1717586382816000000,recording.begin,recording 1717587392364000000,recording.end,recording

file,timestamp [ns],gaze x [px],gaze y [px],worn,fixation id,blink id,azimuth [deg],elevation [deg] gaze ps1,1717586386994395346,820.7727,620.8837,True,,,0.69898784,-1.0643278 gaze ps1,1717586386999399346,822.5108,615.86975,True,,,0.8110849,-0.7404676

doing the basic calculations gives me: Event duration: 0:16:49.548000 Gaze duration: 0:16:45.358712

also the orignal recorded scene video is : 00:16:47 long

the original eye video is 00:16:46 long

i'm trying to match timestamp in order to get a gaze per frames of the video

so how can i determine the gaze matching with the scene video frame to overlay the data

further more: pl-rec-export always returns the following error and doesn't export blinks even if prompted:

hi rob, im on python 3.11.5

will do. I will keep you posted. Thank you

will it still likely to be the same problem regarding synchronisation of gaze data to video frames? or do you have a fix for that or even better a function

@user-f43a29 i bought 6 sets plus my client 2 on ym reocmmendation. Would it please be possible to have a one to one quick support outside here due to sensitivity of the data and the work im doing?

Thank you i sent an email

I have a question about sending events to Neon from Matlab. Not sure if it belong here or in 💻 software-dev . - The phone and my computer are connected to a LAN via ethernet dongle. - Neither phone nor computer are connected to the internet. - When opening the webpage of the glasses' phone from the Matlab Computer, it connects nicely and I see the video without delay. - Ping is also very fast. - I am following https://docs.pupil-labs.com/neon/real-time-api/track-your-experiment-in-matlab/

On that page I found "The average response time for HTTP requests to Neon in MATLAB is 0.33s" In my case, sending an event with pupil_labs_realtime_api('Command', 'event', 'EventName', ['test']); takes around 1.8 seconds, which is far too long. The Matlab Code is blocked while the request is being processed.

1. What could be the reason that the http request takes so much time?
2. Is there a faster alternative to set an event?

Hi @user-4b18ca 👋! As indicated on the repository that library is obsolete.

We are yet to update our documentation with the new library that my colleague @user-f43a29 referred on the previous message https://discord.com/channels/285728493612957698/1047111711230009405/1263072770619736114.

The first one used https://es.mathworks.com/help/matlab/ref/matlab.net.http-package.html to send the requests, while the latest, uses the new python integration in Matlab, and relies on the python library to make the requests and stream.

Why is so slow in your use case, well it could be that your code blocks the execution before, it's hard to say without seeing the code, but my recommendation would be to use the new package.

Ah, now I get the difference to the new package. I was following https://docs.pupil-labs.com/neon/real-time-api/tutorials/ where the "Track Your Experiment in MATLAB" section still follows the old library.

I timed the code execution around the api call, so the 1.8 seconds I'm seeing was really just from that request. I will report how things behave with the matlab python wrapper. Thanks so far.

When running https://github.com/pupil-labs/pl-neon-matlab/blob/main/matlab/examples/recording_with_events.m I get the following error

Searching for device... Device found! Phone IP address: 192.168.50.150 Phone name: Steven Hawking Battery level: 100% Free storage: 83.04 GB Serial number of connected module: 815614 Error: Python Error: ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/plain', url=URL('http://192.168.50.150:8080/api/event') Stopping run loop for rtsp://192.168.50.150:8086/?camera=gaze Stopping run loop for rtsp://192.168.50.150:8086/?camera=imu Stopping run loop for rtsp://192.168.50.150:8086/?camera=eyes Stopping run loop for rtsp://192.168.50.150:8086/?camera=world

Recording without sending events works as expected. I configured a python environment as suggested (python=3.10). System: x64, Ryzen 5 Desktop, Windows 10, Matlab R2023a

Could you share some additional information about what OS are you using, computer specs (is it ARM based?) and the version of Matlab?

x64, Ryzen 5 Desktop, Windows 10, Matlab R2023a App Version 2.7.10-prod pupil-labs-realtime-api 1.3.1

Thanks, it should work with these settings. Please allow me to consult tomorrow with my colleague @user-f43a29 who wrote and tested that wrapper. In the meantime, I strongly recommend that you use the latest version of the Companion App.

Thanks. Installed the lastest version (message is the same)

I believe I found the problem. In https://github.com/pupil-labs/pl-neon-matlab/blob/main/matlab/Device.m#L103 (and L105) the timestamp is of type double, while send_event expects an integer value. Thus it should be evt = obj.py_device.send_event(event_text, int32(current_time_ns_in_companion_clock)); and evt = obj.py_device.send_event(event_text, int32(timestamp));

Hi @user-4b18ca , I have a moment, so just wanted to hop in.

First, if that solution works for you at the moment, then please note that you want to convert to int64 to match the type parameter of send_event in the real-time API. Otherwise, you will lose a significant amount of precision and potentially send erroneous and truncated timestamps.

I'm a bit unsure by the error and the solution at the moment, as I did not encounter this on different operating systems or Matlab versions. At first glance, the error message seems to be related to something else.

The double type parameter on the timestamp argument was chosen because MATLAB's default data type for numbers is double. Since timestamps could come from any source (Neon, EEG, Psychtoolbox, etc), it was decided to go with MATLAB's default, so that users need less type conversion sprinkled throughout their code.

The MATLAB-Python boundary should do the correct type conversion for you automatically. In retrospect, you are correct that depending on the system to automatically do "the right thing" is probably not optimal and I will find a better way.

I will have time to look at this tomorrow

Maintainers take note. I feel, in pl-neon-recordings, this line self.stream["eye"] = EyeVideoStream(self) needs to be self.streams.... Then one is able to run examples/eye_overlay.py.

Hi @user-c541c9 , a fix has been pushed. Please update to the latest version and let us know how it goes.

Good morning, I can't do full screen calibration because the screen flickers, the display remains stable only by connecting a second monitor and using that for calibration or reducing the window. Why does this happen? are there any specific settings I need to set for my screen for this not to happen? thank you very much

I have an asus tuf gaming with window 11 and NVIDIA 4070, yes is a laptop

Hi! I wrote up some code to do real time data collection with the neon and saving it to a CSV file after running the script. We're finding that the frequency is a lot lower than the 200 Hz that the documentation mentioned pertaining to the gaze X and Y coordinates - I'm getting about 25 Hz of data. I think this may be one of the functions that I wrote that might be reducing the frequency. Any help would be amazing!

Thank you so much! Can I ask what the main difference is between async and simple? I think I understand that async is non-blocking but how does it differ based the data rate?

For real time data collection, especially the implementation of calculation metrics like entropy, would it be more advised to use async or simple for better performance?

Hi, i used the Invisible and we had to change to the Neon now for our experiment. I read that the parameters are basically the same such as 200hz for both eye camera, the scene camera. Just the world camera is much larger and captures more in terms of size. I wanted to create heatmaps via python and compare the patterns. However is this even possible ? I normally say what the original resolution (1088 x 1080px) was and scale it down. Wouldnt work complety if i switch between glasses right?

The Neon captures more than the Invisible can after all?

Thanks Rob! No worries.

Hi everyone! I am working on my thesis project with Pupil Labs using HTC VIVE on Unity. I was reading the documentation, particularly the section about the right gaze mapping pipeline. I read that it is recommended to divide the experiment into blocks. Does this mean that at the beginning of each scene in the project, it is advised to recalibrate? Did I understand correctly? When I record, should I record everything (even when switching scenes in Unity) and check if the error is high or not? How could one estimate the slippage error?

@user-480f4c I tried to do it, doing a reference image and later heatmaps but i get errors for the reference image. Also i am kind of confused about 3 minutes maximum. In the end we also want to statistically analyze the patterns and do a PCA for example and thats not (as far as i understood) possible with the predefined heatmaps?

@user-480f4c

1) I get the following errors. "Error: The enrichment could not be computed based on the selected scanning video and reference image. Please refer to the setup instructions and try again with a different video or image." The thing is for me. I wish to collect all recording information that belong to Condition X (defined by event stamps with a _start and _stop)from every video (i have 39 recordings. So that might be a problem) and in another reference-enrichment i would collect all information belong to Condition Y and generate the heatmaps over every participant

2)Thank you. I tried to do this with a video that was less than 3 Minutes.

3) we want to do a principal component analysis and see how much variance specific parts within the heatmap explain for the fixation (and their distribution). Ideally we want to compare if this pattern is differently between conditions that we have in our experiment.