I have a few questions about the HTC Vive add-on.

1: What is the delay between pupil movement and its position measurement. Like when the pupil is moving, how much time the add-on needs to detect the position? What's the process time of it?

2: How can we know if a new measurement is available for us to read? In our case, we need to use our own code to read the measurement, to avoid repeat reading the data, we need to check if there is a new data generated.

3: Since our case is time sensitive, we need the real-time. Is there any way we can transform the timestamp to the real-time?

This is the code I'm creating. I will understand if your response is, "Create a simplified version." It's clean, but a lot. https://github.com/gabrielDiaz-performlab/RAFT/blob/71d8f40d7ad3bf2bfab4cdb1e1b8a3db5bce215d/flow_source.py#L146

Thank you for the reply, just wondering if there is also sample code for reading the pupil measurement from the add-on?

Note the difference between pts (presentation timestamps) and dts (decoding timestamps). In the h264 format, frames are not necessarily decoded in presentation order. When creating new frames, ensure to maintain dts order, too.

Oh, no. That sounds very icky.

I was hoping to crawl through the vid in a for loop and build the new vid frame by frame. However, if there's an issue with order, then ... I have to buffer and re-order as I go?

Oh, that's what you mean!

And maybe setting the packet dts/pts too

...and don't set pts?

or, set both?

reading this tutorial (http://dranger.com/ffmpeg/tutorial05.html) now to try and figure this out myself. I have already read it, but ... apparently missed something

Ok thanks. I'll give that a shot. I thought this is what led to the eerror that I posed above about non-monotonic stamps, but trust yah

I think that you may be right.

"AttributeError: attribute 'dts' of 'av.frame.Frame' objects is not writable"

Looks like I can't set DTS directly.

I could set new_frame.pts = old_frame.dts ... ?

K

Running it. Let's see if it holds up. Thanks!

Any other critical settings related to timing? I'm also settings the: * video stream.time_base * video stream.codec_context.time_base * frame_out.time_base * packet.dts * packets.pts

Ok, great. I did this last night without setting pts / dts, and although the vid loads into player, scrobbling through doesn't seem to change anything

I assume it uses dts/pts for that

Yes. Changing the location on the player timeline at the bottom of the screen is not changing frame loc.

I do not see an error loading world.mp4

Oh, let me try that. I forgot that you suggested it.

No luck

Same behavior. In both cases, overread errors if I scroll too far in.

I suggest building a small dedicated script that extracts the following for a given video file: - frame.pts - frame.time_base - relative time (float(frame.pts * frame.time_base)) - frame.dts - packet.pts - packet.dts

Plot those on y-axis vs frame/packet indices on x-axis. Do this for the original and the newly generated one. The goal is that the latter produces the same values as the former.

for the delay time of the setup, we directly connect the add-on with the computer, in this case, what's the delay time for it?

I was able to find that the precision of the pupil cams are 0.08 - 0.1 degrees and the accuracy is around 1 degree. Just wondering what are the resolution and sensitivity of the tracker?

Hi, I need to use passthrough api and want to see only one real object, how can I achieved this? anyone please

Hi I’m just curious I just found out about this company is there a way to purchase the eye tracking modules

Hi! I was wondering how feasible it would be to disassemble (and possibly reassemble) the HTC Vive VR Add-on. We are looking to use the hardware for a custom purpose, but we're unsure whether it would be better for our application to have the casing or just have the parts, and we would like to be able to have both options available.

Thank you!

Ok, I have improved things quite a bit.

I was encoding with H.264, you were encoding with MPEG4

Apparently the transcoding introduced the timing issues.

The blue dotted lines are the video output. The pts and dts plotted here are multiplied by the video stream's time_base.

Here's a link to the notebook. https://github.com/PerForm-Lab-RIT/retinal_flow_toolkit/blob/main/sandbox/frame_pts-dts.ipynb

There are still some issues, but it's much better than before.

```

np.array(pts[:5]) * float(time_base) array([0.01499962, 0.02633707, 0.05130083, 0.05398642, 0.08163577])

np.array(pts_out[:5]) * float(time_base) array([0.01399252, 0.02532998, 0.05029374, 0.05297932, 0.08062867])

Time base A 1/65535 Time base B 1/65535 Num frames A 4408 Num frames B 4408 Avg rate A 144439140/2415979 Avg rate B 96292760/1611563 Encoded frames A: 0 Encoded frames B: 0 Start time A: 15000 Start time B: 13993 Duration A: 73731000 Duration B: 73788000

but, oh god, the quality of encoding into mpeg4 is horrible!

I switched the codec to hevc_nvenc because I could not improve the mpeg4 quality. It's possible there's a flag I'm unaware of.

The timestamps were very, very close to the original. I erased the world_lookup.npy. ....

Errrrr

FWIW, I loaded in the original movie file and scrobbled to the same frame index. It looks like the new one is only index off.

fyi, stream.frames and stream.average_rate are not 100% reliable in my experience

Dm'd. Sorry, it's 9.9 gig!

I can record a smaller one. I don't think there is anything special about the video file.

Not sure how these values are generated, to be honest. Wasn't aware that one could set them manually. Just wanted to point out, that one might not want to rely any calculations based on them.

I was surprised to se that the start times of the videos were different. I don't believe that you can set those values directly.

Just the world video, only 2 gig: https://drive.google.com/file/d/1H_MYIlrbX_KNW7hlBCGDdI0_hxJwsL3s/view?usp=share_link

3.2.2

Oh, I'm outdated!

Upgrading to 3.5...

(using a machine I don't usually use for this. Oops!)

This explains the skipped frame warning. Please use a newer version and regenerate the lookup table.

FWIW, I'm going to put this on google collab and see if someone on the pyav forum can help. Unforutnately, it's a fairly quiet forum, where questions far outnumber answers.

apologies. given

When I use this script to convert the video into a Player recording, I do not have any issues. https://gist.github.com/papr/bae0910a162edfd99d8ababaf09c643a Could you maybe also share the world timestamp file and info.player.json?

Latest release works well, too.

Interesting.

Just as sanity check: You did delete the lookup after switching to the new version, correct?

Yes

Let me double check

for a very sane sanity check

And another sanity check: You did upload the re-encoded video, and not the original, correct?

Oh, no. I did upload the original, because I thought that was what you were asking for.

Good question.

I will upload the re-encoded vid now. 🙂

Apologies for the misunderstanding. Could have been more explicit

I can't remember what codec I used for encoding this version. I've been trying a few, because it seems to affect timing.

https://drive.google.com/file/d/16bgITvrBGbVh4u1kS45AeUKzSYhAGfVj/view?usp=share_link

Yep. Named "new_world.mp4"

done

Huh. I just ran without setting any pts / dts, and now I get the *exact * same pts dts in my new video

same as the old

I'll carry this test through - gimme a few.

I should say, this is where I started.

Success!

Funny - there is an interaction with the codec. I get errors when using hevc_nvenc

Sorry, this is on encoding.

Transcoding.

when I add the video stream to my container object, I set a codec

This approach (not setting pts / dts) works when using "libx264" but not "h264_nvenc"

video encoding is zany, man.

Just check the options on this thing: https://superuser.com/questions/1296374/best-settings-for-ffmpeg-with-nvenc

Well, perfect is the enemy of the good, and all that. I'll stick with libx264

Thanks for working with me, here!

Happy to help! Although, you did 99% of the work here 😉

Isn't that how it should be? You were helpful in pointing me to the issues.

I learned quite a bit about video encoding that will actually be helpful as faculty in an imaging department!

Feel free to share this code and github repo as a ref if anyone else wants to transcode the world vid while doing opencv stuff to the outgoing frames

I still need to carry over what we've learned from the notebook to the main repo file (flow_source.py), but I should do that later today.

I'm working with the VR/AR add-on and trying to understand the data from pupil_positions.csv. Does the model_confidence refer to the confidence score of the entire 3D pupil detection data?

Meaning, does a model_confidence of 0.0 indicate that the entirety of the 3D pupil detection data should be disregarded?

In the tests that I conducted, I see that the confidence of 2D detection is 1 for a large sample of data but the model_confidence is 0.1 on the same data sample. Why does that happen / how can I improve the 3D detection?

https://docs.pupil-labs.com/core/best-practices/#pye3d-model

I see, thank you!

Hi everyone! I recorded eye movements in a virtual scene and then exported the fixation information from Pupil Player. As I attached images of the scene, there is a green wall, and I am looking for fixations on the green wall, and I need to remove other fixations from the analysis. So, I am wondering if there is a way to define the area of interest, in this case, green wall, and export fixation information associated with this area from Pupil Player, and post analysis since I did not define a surface during the recording.

Hi Use unity3D. Is there a suitable version recommended? Thank you

What does it mean when ... "[WARNING] video_capture.file_backend: Advancing frame iterator went past the target frame!" ?

Is this clearly an issue with the number of frames in the video?

Lookup tables use packet pts because one can read them out significantly faster than frame pts (which requires decoding the whole frame)

What is the wavelength of the IR LEDs?