Are there any tips or tricks how to improve 3d sphere center coordinates (camera coordinate space)? Even with optimal pupil detection these coordinates jumps ~2mm mainly on Z axis. Is this expected precision of 3d eye model?
Position on Z-axis is the most difficult to estimate. I think that the 2mm could be normal but I can't tell from the top of my head what the expected variance is
If the model is well fit, it should be fairly stable. You can also freeze it in place.
thank you, I mean for gaze estimation this does not seem to matter at all but I was trying to use 3D model data for rendering adjustments where I would need to get below 1mm
pye3d needs to trade off accuracy with recency. The more data you collect, the more accurate the location estimate will be. The longer you collect, the more out-dated might the model be already, e.g. due to slippage. Here you can read about it https://docs.pupil-labs.com/developer/core/pye3d/#pye3d-pupil-detection
@papr can I DM you or should I just ask my question here? It’s about going in-depth about how the hmd add-on does calibration
@papr I can confirm the left/right flip in the intrinsic matrix. I am working on understanding the compression. That's a strange one! FWIW, the intrinsic matrix saved to file seems equivalent to the intrinsic parameters associated with the "physical camera" game object Unity, and shown in the inspector.
Actually, maybe not.
Here's a snapshot of the screencast cam's properties in inspector
For a sensor size of 320 by 240, and vertFOV of 60, the focal length is 207.8
Here is your matrix:
array([[415.69219971, 0. , 320. ], [ 0. , 415.69219971, 240. ], [ 0. , 0. , 1. ]])
Either the focal length should be 207.8, or the principal point should be 160 x 120. Right?
No, that doesn't make sense. The size of the offset is not nearly the magnitude I would expect from 2x the focal distance.
I believe that camera sensor size in unity is not in pixels but in mm so 320 240 seems to be nonsense
not sure where is the camera matrix from but given the values I would expect that it is valid for resolution 640x480 and those are just dummy values where only focal length is manually set to some reasonable value bcs pupil software uses only that (fx+fy)/2 => that's why fx == fy and there is no way that some real world camera matrix would have values like that
I'm also suspicious, and looking into it now. I don't think the focal length is set correctly.
those are set approximately for their camera modules, if you want precise values just calibrate the camera yourself
This is about the camera intrinsics that we are reading from unity's main camera
oh sorry, seemed like camera_modules.py related
No worries, it is. We are trying to find out if hmd-eyes is using the correct intrinsics
@papr An issue with my notes - I can't find where the eye translations are set for the hmd gazer. Can you please point me to a line number?
Found the issue - it was the wrong fork / branch.
@papr do u know if you play around with any camera projection matrix (matrices) in unity or use it in calibration calculations ?
As far as I can see in the code I don’t think you do …(?)
I would guess so, but I am not sure
I feel like the code assumes whatever standard projection matrix unity uses or something like that, but in our setup we actually have two different projection matrices per eye, so we wanted to try taking the dot product between the reference point and those projection matrices
I’d do it thru unity but we can only send one reference point for both eyes , and I see where I’d make changes in the Python scripts but I’m not sure how difficult it would be to do that and make sure pupil capture still runs
Correct, you would need a custom hmd calibration choreography that is able to differentiate the ref data for each eye and then creates a dual monocular gazer.
But what if I differentiate within the Python code instead ? There’s already different variables and results per eye so if I just apply the transformations in the right place it should still work right ? (Maybe?)
I was only talking about the Python code. The unity code would need adjustments as well
How so?
Well, you seem to want to send different reference locations for each eye, correct? You will need to adjust the unity code to do so. But thinking about it, hmd-eyes only sends the 3d location of the reference location. There is no need to project it into a plane.
That’s what I was thinking too, but I’m not sure what else could be causing that radial offset we’re seeing in our data
Also, I’m trying to transform 3D coordinates to a different 3D coordinate…like, the camera center is offset per eye, and the offset per eye is equal and opposite
This
Can you please remind me: Do you use Realtime or Post-hoc calibration? And what data are you comparing?
Real-time, also uh….comparing data collected from what we’re doing now and then data from (ideally) after applying those transformations
But are you comparing the 3d reference data to the gaze_point_3d values?
Oh, yes yes
@papr if I’m trying to plot the gaze for one eye, should I do (for example) eye0_normal - eye0_center ?
@papr I recently had a data collection failure, and the log file is interesting, and full of issues, but uninformative error messages. I wonder if you could have a quick look and tell me if you recognize the underlying issue?
https://drive.google.com/file/d/16hR_kvx0uJXV4cBkCKsDCFxUkMpThzpG/view?usp=sharing
Interestingly, this recording was not stopped due to this issue. No case of "Non-monotonic timestamps"
Interesting logs include:
2021-11-18 10:38:32,490 - world - [ERROR] recorder: Notification without timestamp will not be saved.
Are you sending annotations to Capture? If so, please include a timestamp.
2021-11-18 10:40:56,680 - eye0 - [DEBUG] glfw: (65544) b'Win32: Failed to open clipboard: Access is denied. '
This one happens from time to time. Not sure what the underlying issue is. They should not affect the recording.
and LOTS of 2021-11-18 10:45:00,575 - world - [DEBUG] network_api.controller.pupil_remote_controller: Request: 't', Response: '650859.914927'
I can dig up some timestamps to tell you when recording failed...
Shortly after 672476.7406.
No, I'm not sending annotations.
ok, just remembered that annotations != notifications
It looks like these are the logs returned at the time of failure:
2021-11-18 16:45:17,401 - world - [DEBUG] network_api.controller.pupil_remote_controller: Request: 't', Response: '672476.740555' 2021-11-18 16:45:17,412 - world - [DEBUG] network_api.controller.pupil_remote_controller: Request: 't', Response: '672476.751806' 2021-11-18 16:45:26,890 - eye1 - [DEBUG] glfw: (65544) b'Win32: Failed to open clipboard: Access is denied. ' 2021-11-18 16:45:34,307 - eye1 - [DEBUG] glfw: (65544) b'Win32: Failed to open clipboard: Access is denied. ' 2021-11-18 16:45:34,806 - eye1 - [DEBUG] glfw: (65544) b'Win32: Failed to open clipboard: Access is denied.
The t request should not have an affect either. On the other side, you are sending a lot of them. I wonder if they are too many and somehow affect the recording process.
What kind of c# code produced the 't' request?
For context, I'm using the UXF "Unity Experimental Framework" to automate data logging.
Time sync related stuff
UXF is not setup to interface iwth Pupil Labs - I designed that part, and perhaps something I did is problematic
I recommend calling UpdateTimeSync() from time to time instead, and then using ConvertToPupilTime(Time.realtimeSinceStartup) for high frequency timestamp measurements
Yes, I was calling: "gameObject.GetComponent<TimeSync>().GetPupilTimestamp().ToString()"
Probably VERY frequently.
(every frame)
That should have generated noticable lag on your end
That's good to know 🙂
It didn't, but was probably running as a thread or subprocess
That the only red flag you see?
Yes.
Ok, thanks very much for the insight!
Another issue: 2021-11-22 15:06:13,139 - world - [DEBUG] recorder: Non-monotonic timestamps!Last timestamp: -832549.9458979922. Given timestamp: -832549.9522260039
This killed the recording - lost another participant. (We're pilot testing)
I did impelement the changes you suggested earlier today.
It stops the recording yes, but the recording should be intact.
Do you have the time sync plugin activated?
Yes, but let's just say that this kind of unpredictable behavior will make analysis very difficult.
Hurmn. I don't know that I do.
This is on the capture side - a plugin to be enabled via gui?
Yes. But it is unlikely you have it enabled if you do not need it.
Ok. Let me check...
You want it enabled, I imagine.
The opposite.
I want anything disabled that plays with the Pupil clock.
btw using 3.4.0
It was disabled
Can you check if you use the T remote command anywhere? https://github.com/pupil-labs/hmd-eyes/blob/master/plugin/Scripts/TimeSync.cs#L84 (unlikely based on the shared timestamps)
nope
I don't. My code is very minimal.
Yes, this is a new recording, and a new log
and, apparently, a new issue 😛
I can share the new log. one second!
When it failed, I manually restarted recording by clicking R
No indication of what caused the non-monotonic timestamp. If we are unlucky this is caused by ~~something internally in the camera~~. The monotonicity test was implemented in case someone explicitly changes the Pupil clock during a recording. But this does not seem the case here.
The issue can only be caused by two things: Either the Pupil clock offset changed or the monotonic lock was not monotonic. I will go sure and replace the custom clock implementation with time.monotonic in our next release.
This is the implementation of the software clock timestamping frames on Windows https://github.com/pupil-labs/pyuvc/blob/master/windows_time.pxi
Maybe we should replace it with https://docs.python.org/3/library/time.html#time.monotonic
This is in Capture, not HMD-Eyes, correct?
Correct. (although the former can be triggered via the T command)
Ok. I wish I could test this out for you, but I don't have the source setup on this machine yet.
The issue above is hard to reproduce. I am suspecting a floating rounding issue. That would be hard to reproduce. If you get the issue repeatedly it might not be related to the windows lock after all but to the Pupil clock offset.
...and it's unlikely I have time to setup before I leave for the thanksgiving holiday.
(for me, that's tomorrow afternoon)
if it happens again, we will let you know and will save and share the log
...but, I may have to shelve this experiment for a while if it does.
It's thanksgiving break, then two weeks of classes, then winter break.
So, my window for rounding up potential participants is small.
papr, how much error woudl you expect this to introduce if we removed the halting message?
if we allowed recording to continue
Do you have an intuition if this would cause only a one frame error, once in a while? Would allowingthe non-monotonic time stamp cause the entire pupil system to halt?
If at all, I would skip the frame. Otherwise you need to hallucinate a new timestamp. But if the clock is wrong then it is likely that the next frame might be affected as well. And yes, that could cause that no scene frames would be recorded anymore.
I am willing to accept a few occasional dropped frames if it allows the recording to continue.
Then I suggest writing a plugin that replaces the recorder plugin. But I am not 100% sure if the current architecture allows that. You might need to start this plugin via the network api which should be ok in your case
Ehrf. Ok, thanks.
Any estimate on how long it will be before the next capture update?
I want to publish it this week
/release?
Oh, then I'll just wait! Thanks 🙂
You are referring to the clock update, correct?
Yes
I think I want to give this change more time for testing. I can get you a pre-release including the change, though, if you want.
I noticed a VR environment, where a user is gazing towards objects. Is that environment or app available in the Oculus VR app store?
Check out https://github.com/pupil-labs/hmd-eyes/ 🙂
Hello. can i ask Hardware of Pupil VR add-on?
You can find general information about it here https://pupil-labs.com/products/vr-ar/ or you can contact sales@pupil-labs.com if you have specific questions.
Hey @papr , @user-3cff0d . Gabe here again using my professional account. @user-b14f98 is my other account) I would like to share my understanding of the offline calibration of HMD-Eyes data so that you can correct me where I'm wrong. Here's what I think I know...