Hello. I'm building a multimodal rig with Neon + EEG + gloves + webcams, everything going into LSL on one PC. Trying to get all streams aligned to within a few ms.

We're pulling the scene camera over RTSP and using Time Echo for the offset. I think we've got a sign inversion somewhere, we're seeing a consistent ~332 ms error against our other streams.

I wanted to ask:

1. Can you confirm the exact formula? Is it device_time + offset = client_time, or the other way round?

2. Is the timestamp on an RTSP scene frame the sensor capture time, or when it was encoded/sent? And is it the same timebase as the phone's native recording?

3. For send_event, is the event stamped on the device or on the client? Can I pass my own timestamp in the device timebase?

4. Does the offset drift over a session? Should we re-run Time Echo periodically rather than once at the start?

We're on wired Ethernet to the phone already. Let me know if you need any more information, our main aim is to sync up each of these modalities as much as we can based on the frame rate.

Hi @user-a67b22 👋 ! That sounds great. If you're using LSL, the easiest option is to enable Stream to LSL in the Companion App:

https://docs.pupil-labs.com/neon/data-collection/lab-streaming-layer/

This gives you the data directly in your LabRecorder session, including gaze data and event timestamps.

If you're using RTSP to pull the scene camera stream, each frame already contains its capture timestamp in Unix epoch nanoseconds. You can use the clock offset to convert between the Companion Device clock and your PC host clock.

As, for your specific questions:

1. Can you confirm the exact formula? Is it device_time + offset = client_time, or the other way around?

Yes. To convert a timestamp from the Companion Device to the PC timebase, you take the RTSP timestamp and add the offset:

device_time + offset = client_time

For the opposite direction, when converting from PC time to Neon time, you subtract the offset.

See the manual clock offset correction example here

1. Is the timestamp on an RTSP scene frame the sensor capture time, or when it was encoded/sent? And is it the same timebase as the phone's native recording?

It's the capture time, using the Companion Device's clock in UNIX epoch nanoseconds, same as recorded one.

1. For send_event, is the event stamped on the device or on the client? Can I pass my own timestamp in the device timebase?

You can choose either approach, let the system timestamp the event on arrival, or provide your own timestamp.

See: https://pupil-labs.github.io/pl-realtime-api/dev/methods/simple/remote-control/#with-manual-clock-offset-correction

1. Does the offset drift over a session? Should we re-run Time Echo periodically rather than once at the start?

Yes, the offset can drift, so you should estimate it regularly for the best timing accuracy.

If you stay within LSL, though, LSL already handles this clock synchronization for you.

Thank you Miguel. Will give these a shot in my current code base : )

Hi, I’m using Neon with Companion 2.9.47_prod on a OnePlus 8T / Android 11, through the Python Real-Time API.

The gaze stream returns EyestateEyelidDualMonoGazeData, including the worn field.

However, worn remains True continuously, even when the glasses are completely removed and left off for more than 30 seconds. During that time, timestamps continue to update and eye-state / pupil values are still produced.

I found older Pupil Labs Discord discussions mentioning that the worn detector was not yet implemented for Neon.

Could you confirm whether worn is supposed to work with Neon in Companion 2.9.47_prod, or whether it is still not implemented / not reliable?

Thanks.

Hi @user-4a5aa2 , the Worn detector is implemented for Neon. Note that the device continues to estimate & produce gaze and eye state timestamps & data, even when not worn. Rather, if you need to filter such values from your pipeline, then you would typically also monitor the Worn signal and exclude any gaze/eye-state values when Worn is 0.

However, since you took the glasses off, do you know what the eye cameras were "looking at", when they sat on the table? Or, where were they sitting? It is possible that the eye cameras were aimed at something that was a false positive.

Thanks Rob. I just tested this again with the glasses placed inside their closed case, so the eye cameras were not looking at a face or anything resembling eyes.

worn still remains True continuously.

Interestingly, in this condition both eyelid apertures are essentially zero:

Worn=True | Pup G=2.47 D=2.86 | Apert G=0.00 D=0.00

The timestamps/data continue to update, but worn never switches to False.

So it does not seem to be caused simply by the eye cameras seeing a false-positive eye-like object.

Would you be able to make a recording where you just let them sit on a table, as you would normally place them, and share that with [email removed] You can use any file sharing service you prefer. Then, we can provide better feedback.

Hi Rob,

I made the recording you requested with this sequence:

• about 5 seconds with the Neon glasses worn
• about 15 seconds with the glasses removed and sitting normally on a table
• about 5 seconds with the glasses worn again

I also checked the native worn ps1.raw file from that recording.

It contains 3394 samples, and every single sample is 255:

unique values: [255] count: 3394

So worn remained continuously at 255 throughout all three phases, including the 15 seconds when the glasses were not being worn.

This confirms that the behavior is not only visible through the Real-Time API; the recorded raw data itself also contains worn = 255 continuously.

I’ll send you the full recording as requested.

Thanks. We can inspect the recording and pass the logs onto the relevant team.

Hi Rob,