Hey guys, as #dhennem on 31.05.2019 I have troubles to get the GazeRaysDemo to work. Its one of the Unity Demos to get in touch with the handling of the pupil software within the unity environment. I went through all the demos, but in the GazeRaysDemo the Calibration doesn't work.

I had a look into the code, but I coudn't find anything fixable for myself.

Do you guys have an idea what might be the problem?

@user-0eae33 Please reach out to the vr-ar channel for help with the unity demos.

@papr Hi, could you please tell me that, regarding the Network API, is it possible to trigger all the events through the PUB-SUB connection?

I mean, all the events stated in the source code

Such as Emits notifications: eye_process.should_start eye_process.should_stop set_detection_mapping_mode world_process.started world_process.stopped recording.should_stop: Emits on camera failure launcher_process.should_stop

Because from what I read in the source code.

elif msg == "SUB_PORT": response = self.g_pool.ipc_sub_url.split(":")[-1] elif msg == "PUB_PORT": response = self.g_pool.ipc_pub_url.split(":")[-1]

It seems to be possible, I suppose ?

thantks

thanks

@user-b8b425 Do you just want to receive the messages, or do you want to send a message to Capture that triggers the according events?

I want to control the Capture remotely, maybe even change the configuration remotely, like the resolution of the camera etc

@papr Trigger the events

Maybe by adding the [email removed] function in "pupil_remote.py" ?

@user-b8b425 Have you seen our helpers already? https://github.com/pupil-labs/pupil-helpers/blob/master/python/pupil_remote_control.py

oh, thx , seems the "def notify(notification):" function can emit all the notifications right?

correct

Danke schon

Hi Pupil Labs team, quick question: is it possible to use the Pupil Capture software to capture gaze overlaid onto a screen recording, as opposed to the world camera video? Thanks!

@papr hi, i'm trying to run from source code on raspi, i successfully installed all the dependencies except one, the pupil-apriltags. I tried to install locally, it said:CMake Error in CMakeLists.txt: Running

   '/tmp/pip-build-env-d6ksm0je/overlay/bin/ninja' '--version'

  failed with:

   No such file or directory

Do you have any idea how to fix this?

Hi @user-b8b425 please note that we do not officially support the raspi for Pupil. Regarding the error: pupil-apriltags is merely a Python wrapper around the official apriltags library: https://github.com/AprilRobotics/apriltag The CMake error indicates that the original library cannot be copiled on your system. I am sorry, but you will have to figure out for yourself how to make the apriltag library compile on your raspi. If you manage to get this working, we'll gladly accept pull requests into pupil-apriltags that update the dependency to a version where compilation on raspi is supported.

Ok, thx

BTW, a small question is, the dependency is requiring cmake 3.14.4

would it work on higher version ?

I mean, on x64 platform

@user-b8b425 where do you find this requirement? The cmake version should be independent of your processor architecture. Pupil is built for x64 systems only, so all requirements of Pupil should work on x64.

@user-c5fb8b Regarding the previous question, it seems, after I removed the remote folder and copy the files into a local folder. It compiles

@user-c5fb8b the requirement is in pupil-apriltags

in "pyproject.toml"

Ah right, as you see there's a comment on this. We had an issue on mac with 3.15 which was the newest version when we built the wrapper, so we set the requirement back to 3.14. Feel free to edit pyproject.toml and try out newer versions.

Hi! Is it possible to receive the normalized x and y coordinates from the surface tracker plugin in real time, through the API?

Ok, I found the answer to my question in the documentation for Pupil Capture! If I subscribe to the surface topic through the IPC backbone, how can I receive only the normalized x and y coordinates? Thanks in advance!

@user-65b830 Subscribing to surface is correct. Its payload contains a field called gaze_on_srf which is a list containing gaze that was mapped to the surface.

@papr thank you!! I am now able to stream real time surface gaze data 🙂 this is awesome, thanks for all of your help!

Hello! Is there a way to make Pupil Service publish "gaze.3d" topics? So far I have only managed to receive "gaze.2d" with it (using HoloLens add-on), although when I use Pupil Capture I can receive "gaze.3d". I tried switching the "Detection and mapping mode" through the UI, only change I see is the Pupil in the Gaze's base_data field has more information... Thanks!

@user-a51d13 this is matter of calibrating with 3d data. But I do not know if the hololens integration supports receiving 3d data. Please post your question in vr-ar for a certain answer.

thank you!

Afternoon, I tried to get some questions answered last week on vr-ar but this channel doesn't seem to be very active. It is possible to answer my questions here or is there another channel/source that I can access? I already tried to contact pupil labs by mail but have yet to receive a reply.

I'll just post it again in this channel.

Hi, I'm a game developer at the Utrecht University in the Netherlands where I build VR applications/prototypes for research. The Vive Pro Eye and the Pupil Labs Binocular Add-on for the Vive are currently on my radar. While I've read the documentation and github examples, I still have some unanswered questions. For context, I'm building an environment in Unity that is experienced from a seated position. I want to tracks pupil size and gaze(both timestamped) , I also want to measure saccadic eye movement. I'm currently researching VR eye tracking solutions for an upcoming project. My questions: 1. Are there events that indicate the start and end of saccadic eye movement or are is the prefabricated code available to measure this? 2. How does the data flow work from the integrated eye tracker into unity, is all raw data available in Unity or do I get data that is filtered for in game performance. 3. According to your documentation it should be possible to connect the Binocular Add-on can to the usb connector on the HTC Vive Pro. Does this mean that I wirelessly use the Binocular Add-on can in combination with the HTC Vive Wireless Adapter? If so, what kind of latency can I expect? 4. Is possible to add/remove the Pupil Labs Binocular Add-on multiple times or will that cause damage to the headset and/or Binocular Add-on 5. I've recently read an article on slippage: https://link.springer.com/article/10.3758/s13428-019-01307-0 How does the binocular add-on perform in this matter? 6. This question might seem a bit naive. Is there an active pupil labs developers community. This discord server doesn’t seem to be very active, I hope there are more sources out there. I hope someone here is able to answer my questions. Thanks in advance, I'm looking forward to our input.

@user-973483 Please check your inbox. You should have received a response from info@pupil-labs.com yester.day. 👍

thanks @papr

@papr Hi, i'm experiencing a weird problem. I'm running from source code (pupil core), sometimes the window would freeze (could be any one of the three windows), and if I move the mouse cursor onto the window, it starts to refresh again. Do you have any idea what is happening ? The code is not modified and the dependencies are installed according to the document, the OS is ubuntu 18

@user-b8b425 I have not experienced this problem before. Is your headset connected via a USB hub or via a direct connection?

direct connection

it happens both on a regular PC and a powerful workstation

Ok, I asked because I have experienced issues with some usb hubs in the past.

Is the problem solved as soon as the mouse is moved or the mouse enters the Capture window?

However, it would not happen with the released software

@user-b8b425 Which glfw version do you have installed?

let me check

it's libglfw3-dev

@user-b8b425 I recommend building the current 3.3 release from source https://www.glfw.org/docs/latest/compile_guide.html

AFAIK the apt release is out of date

the same as you suggested on the pupil/dependency webpage

ok i'll try that tomorrow.

glfw is just a guess on what the issue might be. It is possible that the apt release is fine.

OK. thanks, i'll keep you updated

Is there any built in way to use the Pupil Player do "replay" a recording over the network (ZeroMQ)?

@papr Hi, I compiled and installed the latest glfw lib, unfortunately, it has not been fixed

it still goes freeze after running for a while

do you think is it possible something wrong with the timer?

Is the problem solved as soon as the mouse is moved or the mouse enters the Capture window? Have you been able to test this?

yes

once the mouse is moved upon the window

it starts to work again

the cpu usage also decreased when the window get freeze

now all three of them are freeze, the CPU usage goes down to around 10%

0% now

in terminal, it still outputs "*** Correcting clock frequency to 14999909"

@user-b8b425 Are you running the master branch? If so, could you let me know the output of git describe --long

1 min

v1.21-5-ga9d492ec

@papr Are all the dependencies on the document updated to date?

Could you share the ~/pupil_capture_settings/capture.log file of a session where such a freeze happened?

sorry

this is the wrong one

This log only shows a session of 5 seconds. Could you please run a longer session, e.g. ~60seconds

And do I understand correctly that the video freezes immediately as soon as the mouse leaves the window?

no

the capture would sometimes freeze, when the mouse moves onto the window, it unfreezes

i found a suspect message

2020-01-23 10:49:52,658 - world - [WARNING] uvc: Turbojpeg jpeg2yuv: b'Corrupt JPEG data: premature end of data segment'

here is the full log

the Turbojpeg lib i'm using now, is what stated in the dependency document, i think it's 1.5.1

It is ok for this to happen, as long as it does not happen in high frequency.

ok

for this log

the world window freezes at nearly the end

@papr I just did some testing, it seems there is nothing inside the log when it freezes

E0123 11:15:04.243528 6251 trust_region_minimizer.cc:72] Terminating: Residual and Jacobian evaluation failed.

Does this matter?

I just did some testing, it seems there is nothing inside the log when it freezes I can confirm this. Terminating: Residual and Jacobian evaluation failed. Does this matter? No, it is a log message from the 3d eye model

@user-b8b425 So the only thing freezing is the camera video preview, the graphs etc continue working?

all the things freeze

including CPU graph

confidence

all things inside the window

How do you know that the cpu graph goes down to 10%/0% then?

i opened another terminal

and htop

ah ok, I thought you were reading the values from the graph in the top left of the window

@user-54376c Technically, there is a notification that can be sent to seek, but the issue is that Player does not run Pupil Remote, making it difficult to access the ipc backend. The ipc backend ports are allocated randomly on application start.

@papr Hi, how could I run the source code with a debugger ? Maybe if I can put a breakpoint somewhere or pause the programme when it freezes, I shall be able to tell what is the problem

if I run the source code with debugger, it says : "RuntimeError: already started"

Maybe my environment config is somehow wrong ?

@papr Hi, any updates?

@user-b8b425 @user-c5fb8b will send you his Visual Studio Code debug launch config.

Unfortunately, we won't be able to help effectively, since we have not come across this issue yet, and are not able to reproduce it on our Ubuntu 18.04 machines.

@user-b8b425 Hi. I'm using Visual Studio Code to debug Pupil from both Windows and Ubuntu. The problem is that VSCode currently has a problem with their python debugger, making it unable to debug multiprocessing code on Ubuntu. Unfortunately Pupil runs in a multiprocessing environment, so it won't work out of the box. I have a custom beta build of the python debugger running with is good enough for my needs. If you need to run on Ubuntu, I'd probably rather recommend a different IDE (maybe PyCharm?) But I can't help you with setting up the debugger there. I'll ask around though if someone from our R&D team uses PyCharm and can share a debug config. However, if you are able to switch to Windows or Mac, you can use the following config for debugging Pupil from Visual Studio Code:

OK thx. Does all the dependencies exactly the same as shown in the doc/dependencis ?

@user-c5fb8b OK, thanks, I'll try it first

@user-b8b425 Yes, the dependencies from the docs should be correct. In some cases, e.g. glfw, it is possible to build newer versions from source.

OK, i'll keep you guys updated once I found something

@user-b8b425 In PyCharm you should be able to also just select main.py as start script and make sure to supply capture as parameter in the debug settings in order to start Pupil Capture with the debugger attached.

@user-c5fb8b I'm currently using vscode to edit code, maybe I'll try PyCharm if I'm not able to solve the problem

Like I said there's an issue with the vscode python debugger on Ubuntu. You can follow the issue here: https://github.com/microsoft/ptvsd/issues/1706 If you are feeling adventurous, you can try building their custom beta version. References are in the linked thread.

So far, I can only tell, when the window freezes, the process cannot be suspended by the debugger, I just added some "print()" function to see what it is doing.

before freeze, the program was at the GL drawing

So it might be an issue with your graphics driver? I can only guess 😕

somewhere inside the "if window_should_update():"

idk, but this issue happens both on my regular PC and our workstation. So the chance that it is the graphics driver to blame is very small I think

Maybe I'll try to test it on a windows machine and see if the problem goes away

could this "glfw.glfwPollEvents()" suspend the thread?

yes, it blocks to call registered ui callbacks. If one of these callbacks blocks, it will block the main loop

I'll continue this maybe tomorrow or next week then, gonna run now

thx

@user-c5fb8b Hi

I've run some tests

it seems, when the window freeze, the process get stuck in

clipboard = glfw.glfwGetClipboardString(main_window).decode()

this function

GL drawing

Nice find!

could I just bypass all clipboard stuff ?

seems it's not vital

Yes that is indeed possible.

ok, i'll try

This only effects pasting text into text fields

it seems it is not freezing again.

I'll run it for perhaps for one hour or longer to see

@papr I think, basically, problem solved. But I'm still confused. Why it feels like this problem will not occur with the released bundle. Or the possibility is very small

Maybe your compiled glfw uses a different lib to access the clipboard than the bundled one

And maybe this non-bundled lib is not able to properly provide the clipboard while the app is inactive

idk, last time you mentioned about the version of glfw. I compiled& installed the latest version, the problem still existed.

I do not think that this is an issue with glfw itself but the lib providing the clipboard access. glfw is a cross-plattform lib and is just an abstraction of multiple way to access the OS's clipboard

possibly it's due to the python wrapper ?

Unlikely, since Pupil source and bundle use the same bindings for glfw.

OK, anyway, my problem is solved, and I have no plan to fix the clipboard stuff. LOL

thanks

Danke

Good luck!