Unable to sync less than 3 Ms at best

[ajavamind]

I'm have two Samsung S8 phones mounted for stereo photography and capture photos with SoftwareSync running in debug mode. I am not able to sync these cameras to less than 3 Milliseconds. A sample size of 10 capture syncs ranged from 3 to 13 Ms (7, 6, 10, 4, 6, 10, 13, 7, 3, 9). I'm using a 10 LED linear array display counter, updated at 1 Ms intervals and measure the difference of the LED binary display values between the two camera photos. I verified the timing of the LED count up sequence with an oscilloscope - exactly 1 Ms. The SoftwareSync shutter speed was set to 1/2000 second.

The phase configuration json file used:
{
"periodNs": 33323892,
"goalPhaseNs": 15000000,
"alignThresholdNs": 200000,
"overheadNs": 200000,
"minExposureNs": 33170000
}

The periodNs value was selected based on the frameDurationMs value from debug output.
com.googleresearch.capturesync V/CameraController: onCaptureCompleted: timestampMs = 1,595,063,382,777.953, frameDurationMs = 33.323892, phase = 30.534, sequence id = 0
otherwise these are same values as Pixel 2 phase configuration.

I do not yet understand the algorithm well enough to change these values. Could you advise how to determine the constants for a phone and how to derive these numbers. Note that the Samsung S8 shares the same cpu/gpu as the Pixel 2 but have different cameras.

When I first began to use the SoftwareSync app I was not seeing any captures saved at all.
The shouldSaveFrame(long synchronizedTimestampNs) method in CameraController always returned false.
When I powered down and up both phones nearly simultaneously, only then would the photos be saved.
When any phone was turned off for a while, and capture resumed, the photos would not be saved.
I'm guessing the SystemClock.elapsedRealtimeNanos() clocks in the phones become too far out of range to trigger the save.

My goal is to get less than 1 Ms sync at least. Any help would be appreciated, thanks.

[Elucidation]

Hello, thank you for the debug information, lets see, the accuracy is dependent on a couple things (have a look at the abstract and further for more information https://arxiv.org/abs/1812.09366 ): - Network ping/latency (primary) When running the app, you should see a "x.xx ms" next to each client on the leader device, this shows the best round trip ping/latency between that client and leader. One should expect the accuracy to be <= that value, and likely less than 1/2 to 1/3 of that number. Is this number showing 3 - 13 Ms or more ? If so, then networking is likely the issue. More info: https://github.com/google-research/libsoftwaresync#how-networking-and-communications-work To reduce that, if you're using one of the phones as a hotspot, try closing other apps to reduce load on the phones to improve networking. Or, temporarily try a standalone router to confirm if that was the issue. - Phase alignment config accuracy (secondary) When running the app, phase alignment accuracy in ms should show on each phone, and ideally should be < 1ms for each, preferably closer to 0.2ms (alignThesholdNs in the config, try changing this higher if it's failing to converge). The periodNs is the measured duration between frames coming into the phone, you can manually measure this by looking at timestamps of onCaptureCompleted and calculate the average period and place in periodNs, this value (with some tuning to account for overhead via overheadNs) is critical for keeping phase alignment from drifting. You can tell if phase alignment is drifting if the phase accuracy value showing on the phones is incrementing/decrementing with time. Try tuning periodNs if this is the case. - Phone hardware limitations Trying different phones we saw that our ability to interact with the phase of frames on the phones were limited, the Samsung S8 phone had some limitations in the minimum accuracy possible with phase, I only had one S8 phone on hand so used it only as a client, it's possible it may not work well as a good leader device. You can confirm if that's the case via the networking comments above. Regarding not seeing captures saved at all, I've not seen this before, it may be a permissions issue or a folder creation permission issue, if you're able to get debug error logs please open a new issue linking those and I'll see if I can help. Regarding phone restart required; you shouldn't need to restart the phones however, you should be able to get the same effect by force stopping the apps and reopening. Regarding starting/stopping phone/apps simultaneously; you shouldn't need to start up the phones anywhere near the same time, nor the apps, as the network sync will correct for the differences in elapsedRealtimeNanos. Feel free to start a separate issue with this, please make note of the phone network setups, the ms values shown next to each client for the leader , and phase accuracy values. Regarding apps turning on/off for a while, it sounds like some intracacy of apps being paused in the background. You can start an issue on this, however I'm not sure I'll be able to help with this, to keep things simple I suggest force stopping all apps and starting them up roughly within the same minute or two for your capture sessions.

…

On Sun, Jul 19, 2020 at 7:49 AM Andy Modla ***@***.***> wrote: I'm have two Samsung S8 phones mounted for stereo photography and capture photos with SoftwareSync running in debug mode. I am not able to sync these cameras to less than 3 Milliseconds. A sample size of 10 capture syncs ranged from 3 to 13 Ms (7, 6, 10, 4, 6, 10, 13, 7, 3, 9). I'm using a 10 LED linear array display counter, updated at 1 Ms intervals and measure the difference of the LED binary display values between the two camera photos. I verified the timing of the LED count up sequence with an oscilloscope - exactly 1 Ms. The SoftwareSync shutter speed was set to 1/2000 second. The phase configuration json file used: { "periodNs": 33323892, "goalPhaseNs": 15000000, "alignThresholdNs": 200000, "overheadNs": 200000, "minExposureNs": 33170000 } The periodNs value was selected based on the frameDurationMs value from debug output. com.googleresearch.capturesync V/CameraController: onCaptureCompleted: timestampMs = 1,595,063,382,777.953, frameDurationMs = 33.323892, phase = 30.534, sequence id = 0 otherwise these are same values as Pixel 2 phase configuration. I do not yet understand the algorithm well enough to change these values. Could you advise how to determine the constants for a phone and how to derive these numbers. Note that the Samsung S8 shares the same cpu/gpu as the Pixel 2 but have different cameras. When I first began to use the SoftwareSync app I was not seeing any captures saved at all. The shouldSaveFrame(long synchronizedTimestampNs) method in CameraController always returned false. When I powered down and up both phones nearly simultaneously, only then would the photos be saved. When any phone was turned off for a while, and capture resumed, the photos would not be saved. I'm guessing the SystemClock.elapsedRealtimeNanos() clocks in the phones become too far out of range to trigger the save. My goal is to get less than 1 Ms sync at least. Any help would be appreciated, thanks. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADKF3IO7F3U5FIAOWLBAYDR4MBXJANCNFSM4PBL3ANA> .

[ajavamind]

Thanks for your response and suggestions.

Trying to improve synchronization, I chose not to use the mobile hotspot method to connect because the S8 mobile hotspot will only work when a connection to the Internet goes through my Telecom provider with the phone radio. Instead I connected the two phones through a small Mobile router (TP-Link290A 5G channel). One phone is assigned as leader using a static IP address. (Airplane mode with only WIFI turned on). All other apps are closed (I am not able to shutdown hidden apps that are running in the background).

The client sync time shown in leader display varies from 3.4, 3.61, 3.83 to 4.33 ms after starting the app and stays fixed until the next app close and restart. The phase error is less than 0.05 ms (after pressing Align button) when using a frameDurationMs 33323892. I tried other (periodNs) frameDurationMs values and this value gave the minimum client sync time.

Debug output:
2020-07-19 21:44:24.157 9306-9446/com.googleresearch.capturesync V/CameraController: onCaptureCompleted: timestampMs = 1,595,209,456,117.579, frameDurationMs = 33.323892, phase = 21.764, sequence id = 0

After taking a couple of photos the synchronization becomes progressively worse. In one series, it went from 6 ms to 42 ms after a number of captures. I suspect autofocus may be causing a synchronization problem too, because under low light conditions, I saw instances of focus hunting.

I turned off auto focus, sync improved to less than 1 ms. for a few captures. However it was inconsistent over a group of captures, varying up to 120 ms worst case. This variation occurs with or without auto focus.

I am beginning to believe the S8 has too much network latency and/or background app overhead for good synchronization and that auto focus also contributes to the poor synchronization.

---

I have to restart each phone after a couple of hours before launching the app for it to save any image files. Shutting down and starting the app does not resolve this problem, only Android restart.

The ISO sensitivity range assigned by the app (0-800) does not match camera ISO limits. It seemed like too low ISO (0) setting caused noisy photos over the range. When I fixed this to match the camera ISO range (56-7237), photos looked better.