Hi all!

I have a question about calibration. Calibration for AR VR happens without a need for a front camera. Is there a way to use the api to do calibration similarly but outside of the supported platforms (oculus, vive, etc) and using my own HUD? Basically, calibration using an HUD at a fixed position with respect to the eye, without using the front(world) camera.

Hi @user-ed87f1,

without fully understanding your setup, I'm confident that this should be possible. In the end our Unity plugin also just uses the network API of Pupil to send the calibration targets (3D coordinate and timestamp). We have a special calibration plugin in Pupil designed for this type of calibration, which can only be selected through the network API. You should be able to essentially mimik the messages of the Unity plugin and send the calibration data over the API.

Unfortunately there is no official documentation of this, but you should be able to figure this out by looking at the source code of Pupil and our Unity plugin: https://github.com/pupil-labs/hmd-eyes Feel free to ask questions if you don't understand something.

Some pointers to important code: - Unity plugin starts the Pupil Plugin: https://github.com/pupil-labs/hmd-eyes/blob/master/plugin/Scripts/Calibration.cs#L45 (defined here: https://github.com/pupil-labs/hmd-eyes/blob/master/plugin/Scripts/CalibrationSettings.cs#L17) - Unity plugin sends calibration data: https://github.com/pupil-labs/hmd-eyes/blob/master/plugin/Scripts/Calibration.cs#L83-L93 - HMD calibration plugin in Pupil: https://github.com/pupil-labs/pupil/blob/master/pupil_src/shared_modules/calibration_choreography/hmd_plugin.py

thanks a lot for the swift reply. I'll take a look and report back!

@user-ed87f1 if you are working with Python, our Python reference client for HMD-Eyes could also be interesting for you. This is basically just dummy Python code which mimiks the calibration workflow done by the Unity plugin: https://github.com/pupil-labs/hmd-eyes/blob/master/python_reference_client/hmd_calibration_client.py

Excellent. Yes, I can do both Unity and Python, so these are very useful links.

Hi again. Do you know if it would be possible to use the Unity plugin for calibration, but build for Android or c/c++ ? Mi situation at the moment is that to create my custom portable AR helmet with eyetracking I have a chipset with android (can work with c/c++) to render stuff on the HUD. So basically Python is practically not an option, unless there is a way for me to convert the entire python api to c/c++. Any ideas could be very helpful!

Hi @user-ed87f1, I can't say anything about building the Unity plugin for Android/C++ (probably this won't work, but honestly I don't know). However, I know we had a user working with C++ before and you can actually just use any of the available ZMQ bindings for C++ (https://zeromq.org/languages/cplusplus/) to directly access our Python API. You essentially rewrite the part of the Unity plugin that communicates with Pupil, but this actually should not be such a big project. You can try to reimplement the python reference client for a start (https://github.com/pupil-labs/hmd-eyes/blob/master/python_reference_client/hmd_calibration_client.py) You will also have to use a C++ binding for msgpack, which is we use as serialization format for transmitting structured data via the ZMQ messages. There's also official C/C++ bindings available: https://github.com/msgpack/msgpack-c

great, that will give me something to start working with. Thanks a lot for the swift reply again!

Hey all! I was wondering, if I purchase a Binocular add-on (Vive kit), do I need to use it on a VIVE? i.e. are there any restrictions that require it to communicate with the VIVE in any way? Just wondering since I'm planning to design my own HMD and build my own C++ project then use this add-on without anything related to a VIVE. Thanks!

Hi @user-8619fb, this should be possible, the VR/AR add-on works basically as a normal head-mounted eye-tracker, but designed to be placed inside the VIVE. It only needs a USB connection and can be used with our desktop software (Pupil Capture/Player/Service) as any other headset we sell. It seems that @user-ed87f1 also builds a custom headset using the VIVE add-on? Maybe you can share experiences.

Hello, have any of you encountered that windows can only detect one of the eye cameras?

hey everyone! i'm currently working with the pupil add-on for the microsoft hololens and attempting to setup the hmd-eyes project to work the same as it does for the HTC vive.

i've setup the hololens relay in pupil capture and have it running when i build my project, and the connection IP and port are correct in my unity project, but the hololens can't connect to pupil capture

i noticed that hololens support was missing in previous releases of hmd-eyes, and the documentation doesn't say anything about it, so if anyone has any info on how to properly setup the project it'd be greatly appreciated!

Hi @user-ac0304,

with the release of the Hololens 2 at the end of last year, we have decided not to port the Hololens 1 support of HMD-Eyes v0 to HMD-Eyes v1. The last version that supports the Hololens add-on is v0.62. You can find the v0.62 release here: https://github.com/pupil-labs/hmd-eyes/releases/tag/v0.62 Please note that there is also a specific hololens branch. It contains the same code as the v0.62 release (which you get on the legacy branch), but the repository's README has been adjusted with additional information regarding getting started with the Hololens. See the branch here: https://github.com/pupil-labs/hmd-eyes/tree/hololens Additionally, the latest versions of the Pupil software are not compatible anymore with these early versions of HMD-Eyes. Specifically, HMD-Eyes v0.62 only works together with Pupil up to v1.23. So you will have to install an older versions of Pupil to be able to work with the Hololens. You can find the downloads for Pupil v1.23 at the bottom of the corresponding release notes here: https://github.com/pupil-labs/pupil/releases/tag/v1.23

Previous releases of HMD-Eyes also contained a Hololens.unitypackage. The latest release to include this package is HMD-Eyes v0.5.1: https://github.com/pupil-labs/hmd-eyes/releases/tag/v0.5.1 This version of HMD-Eyes needs Pupil v1.7 to work: https://github.com/pupil-labs/pupil/releases/tag/v1.7

That's perfect! Thank you for the quick reply