Skip to content

Commit 786ca0c

Browse files
vincent-dfinityadamspofford-dfinity
vincent-dfinity
and
adamspofford-dfinity
authored
chore: Improve canister snapshot download/upload (#4406)
* Stream snapshot downloads to disk directly. * Add progress bar for snapshot downloading. * Stream snapshot uploading from disk directly. * Add progress bar for snapshot uploading. * Implement new_progress stub in env. * Change the log level. * Support snapshot downloading with resuming. * Add snapshot download concurrency. * Support snapshot uploading with resuming. * Add snapshot upload concurrency * Revert the testing chunk size. * Update changelog and document. * Addressed review comments. * Added a test for download/upload with latency, disabled it for now. * Make the test passed locally with toxiproxy used. * Update CI to install toxiproxy for canister_extra tests. * Added two e2e tests for canister snapshot. * Use toxiproxy-cli directly for debugging... * Add more debugging info... * Make the snapshot tests be able to run in parallel. * Add canister_extra as serial... * Moved toxiproxy installation into provision script. * Make the canister snapshot network drop more robust... * Use limit_data instead. * fix hanging test (and mac `find -printf`) * remove focus tag * Add mention of `--resume` to error message --------- Co-authored-by: Adam Spofford <adam.spofford@dfinity.org>
1 parent b371d90 commit 786ca0c

File tree

9 files changed

+824
-204
lines changed

9 files changed

+824
-204
lines changed

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,14 @@
22

33
# UNRELEASED
44

5+
### feat: improved the canister snapshot download/upload feature
6+
7+
Improved the canister snapshot download/upload feature by
8+
- adding progress bars to snapshot download/upload
9+
- streaming snapshot download/upload directly to/from disk.
10+
- supporting download/upload with resuming.
11+
- supporting download/upload with concurrency, default to 3 tasks in parallel.
12+
513
# 0.30.0
614

715
### feat: `dfx start --system-canisters` for bootstrapping system canisters

‎docs/cli-reference/dfx-canister.mdx‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1028,11 +1028,13 @@ dfx canister snapshot download <canister> <snapshot> --dir <DIR>
10281028

10291029
You can use the following arguments with the `dfx canister snapshot download` command.
10301030

1031-
| Argument | Description |
1032-
|---------------|----------------------------------------------------------------------------|
1033-
| `<canister>` | The canister to download the snapshot from. |
1034-
| `<snapshot>` | The ID of the snapshot to download. |
1035-
| --dir `<dir>` | The directory to download the snapshot to. It should be created and empty. |
1031+
| Argument | Description |
1032+
|-------------------------------|--------------------------------------------------------------------------------------------|
1033+
| `<canister>` | The canister to download the snapshot from. |
1034+
| `<snapshot>` | The ID of the snapshot to download. |
1035+
| --dir `<dir>` | The directory to download the snapshot to. It should be created and empty if not resuming. |
1036+
| --resume | Whether to resume the download if the previous snapshot download failed. |
1037+
| --concurrency `<concurrency>` | The number of concurrent downloads to perform [default: 3]. |
10361038

10371039
### Examples
10381040

@@ -1056,11 +1058,13 @@ dfx canister snapshot upload <canister> --dir <DIR>
10561058

10571059
You can use the following arguments with the `dfx canister snapshot upload` command.
10581060

1059-
| Argument | Description |
1060-
|-----------------------|--------------------------------------------------------------------------------|
1061-
| `<canister>` | The canister to upload the snapshot to. |
1062-
| --dir `<dir>` | The directory to upload the snapshot from. |
1063-
| --replace `<replace>` | If a snapshot ID is specified, the snapshot identified by this ID will be deleted and a snapshot with a new ID will be returned. |
1061+
| Argument | Description |
1062+
|-------------------------------|--------------------------------------------------------------------------------|
1063+
| `<canister>` | The canister to upload the snapshot to. |
1064+
| --dir `<dir>` | The directory to upload the snapshot from. |
1065+
| --replace `<replace>` | If a snapshot ID is specified, the snapshot identified by this ID will be deleted and a snapshot with a new ID will be returned. |
1066+
| --resume `<resume>` | The snapshot ID to resume uploading to. |
1067+
| --concurrency `<concurrency>` | The number of concurrent uploads to perform [default: 3]. |
10641068

10651069
### Examples
10661070

‎e2e/tests-dfx/canister_extra.bash‎

Lines changed: 140 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,17 @@
11
#!/usr/bin/env bats
22

33
load ../utils/_
4+
load ../utils/toxiproxy
45

56
setup() {
67
standard_setup
78
dfx_new hello
9+
toxiproxy_start
810
}
911

1012
teardown() {
1113
dfx_stop
14+
toxiproxy_stop || true
1215
standard_teardown
1316
}
1417

@@ -105,6 +108,142 @@ teardown() {
105108
assert_contains '(1 : nat)'
106109
}
107110

111+
@test "canister snapshots download and upload via toxiproxy with high latency" {
112+
# Start the dfx server on a random port.
113+
dfx_port=$(get_ephemeral_port)
114+
dfx_start --host "127.0.0.1:$dfx_port"
115+
116+
# Start toxiproxy and create a proxy.
117+
proxy_port=$(get_ephemeral_port)
118+
toxiproxy_create_proxy "127.0.0.1:$proxy_port" "127.0.0.1:$dfx_port" proxy_high_latency
119+
120+
install_asset counter
121+
dfx deploy --no-wallet --network "http://127.0.0.1:$proxy_port"
122+
123+
assert_command dfx canister call hello_backend inc_read --network "http://127.0.0.1:$proxy_port"
124+
assert_contains '(1 : nat)'
125+
126+
# Create a snapshot to download.
127+
dfx canister stop hello_backend --network "http://127.0.0.1:$proxy_port"
128+
assert_command dfx canister snapshot create hello_backend --network "http://127.0.0.1:$proxy_port"
129+
assert_match 'Snapshot ID: ([0-9a-f]+)'
130+
snapshot=${BASH_REMATCH[1]}
131+
132+
# Add latency to the proxy.
133+
toxiproxy_add_latency 1500 300 proxy_high_latency
134+
135+
# Download through the proxy with latency.
136+
OUTPUT_DIR="output"
137+
mkdir -p "$OUTPUT_DIR"
138+
assert_command dfx canister snapshot download hello_backend "$snapshot" --dir "$OUTPUT_DIR" --network "http://127.0.0.1:$proxy_port"
139+
assert_contains "saved to '$OUTPUT_DIR'"
140+
141+
# Start the canister again.
142+
dfx canister start hello_backend --network "http://127.0.0.1:$proxy_port"
143+
assert_command dfx canister call hello_backend inc_read --network "http://127.0.0.1:$proxy_port"
144+
assert_contains '(2 : nat)'
145+
146+
# Upload the snapshot to create a new snapshot.
147+
assert_command dfx canister snapshot upload hello_backend --dir "$OUTPUT_DIR" --network "http://127.0.0.1:$proxy_port"
148+
assert_match 'Snapshot ID: ([0-9a-f]+)'
149+
snapshot_1=${BASH_REMATCH[1]}
150+
151+
# Stop the canister and load the new snapshot.
152+
dfx canister stop hello_backend --network "http://127.0.0.1:$proxy_port"
153+
assert_command dfx canister snapshot load hello_backend "$snapshot_1" --network "http://127.0.0.1:$proxy_port"
154+
155+
# Start the canister again and verify the loaded snapshot.
156+
dfx canister start hello_backend --network "http://127.0.0.1:$proxy_port"
157+
assert_command dfx canister call hello_backend read --network "http://127.0.0.1:$proxy_port"
158+
assert_contains '(1 : nat)'
159+
160+
toxiproxy_delete_proxy proxy_high_latency
161+
}
162+
163+
@test "canister snapshots download and upload via toxiproxy with network drop" {
164+
# Start the dfx server on a random port.
165+
dfx_port=$(get_ephemeral_port)
166+
dfx_start --host "127.0.0.1:$dfx_port"
167+
168+
# Start toxiproxy and create a proxy.
169+
proxy_port=$(get_ephemeral_port)
170+
toxiproxy_create_proxy "127.0.0.1:$proxy_port" "127.0.0.1:$dfx_port" proxy_network_drop
171+
172+
install_asset counter
173+
dfx deploy --no-wallet --network "http://127.0.0.1:$proxy_port"
174+
175+
assert_command dfx canister call hello_backend inc_read --network "http://127.0.0.1:$proxy_port"
176+
assert_contains '(1 : nat)'
177+
178+
# Create a snapshot to download.
179+
dfx canister stop hello_backend --network "http://127.0.0.1:$proxy_port"
180+
assert_command dfx canister snapshot create hello_backend --network "http://127.0.0.1:$proxy_port"
181+
assert_match 'Snapshot ID: ([0-9a-f]+)'
182+
snapshot=${BASH_REMATCH[1]}
183+
184+
# Add a 1MB limit_data toxic to force the snapshot download to fail.
185+
toxiproxy_add_limit_data limit_download 1000000 proxy_network_drop
186+
187+
# Download the snapshot should fail.
188+
OUTPUT_DIR="output"
189+
mkdir -p "$OUTPUT_DIR"
190+
assert_command_fail timeout -s9 10s dfx canister snapshot download hello_backend "$snapshot" --dir "$OUTPUT_DIR" --network "http://127.0.0.1:$proxy_port"
191+
192+
# For debugging.
193+
echo "OUTPUT_DIR contents:" >&2
194+
find "$OUTPUT_DIR" -maxdepth 1 -mindepth 1 -type f -exec du -h {} \+ >&2
195+
196+
# Remove the toxic.
197+
toxiproxy_remove_toxic limit_download proxy_network_drop
198+
199+
# Resume the download through the proxy.
200+
assert_command dfx -v canister snapshot download hello_backend "$snapshot" --dir "$OUTPUT_DIR" -r --network "http://127.0.0.1:$proxy_port"
201+
assert_contains "saved to '$OUTPUT_DIR'"
202+
203+
# Start the canister again.
204+
dfx canister start hello_backend --network "http://127.0.0.1:$proxy_port"
205+
assert_command dfx canister call hello_backend inc_read --network "http://127.0.0.1:$proxy_port"
206+
assert_contains '(2 : nat)'
207+
208+
# Add a 1MB limit_data toxic to force the snapshot upload to fail.
209+
toxiproxy_add_limit_data limit_upload 1000000 proxy_network_drop -u
210+
211+
# Upload the snapshot should fail.
212+
assert_command_fail timeout -s9 10s dfx canister snapshot upload hello_backend --dir "$OUTPUT_DIR" --network "http://127.0.0.1:$proxy_port"
213+
214+
# Loop to get the snapshot id.
215+
snapshot_1=""
216+
while IFS= read -r json_file; do
217+
[ -z "$json_file" ] && continue
218+
if [[ "$json_file" =~ ^[0-9a-f]+\.json$ ]]; then
219+
snapshot_1="${json_file%.json}"
220+
break
221+
fi
222+
done < <(find "$OUTPUT_DIR" -maxdepth 1 -type f -name '*.json' -exec basename {} \;)
223+
if [ -z "$snapshot_1" ]; then
224+
echo "No matching .json filename ([0-9a-f]+.json) found in $OUTPUT_DIR" >&2
225+
false
226+
fi
227+
228+
# Remove the toxic.
229+
toxiproxy_remove_toxic limit_upload proxy_network_drop
230+
231+
# Resume the upload through the proxy.
232+
assert_command dfx canister snapshot upload hello_backend --dir "$OUTPUT_DIR" -r "$snapshot_1" --network "http://127.0.0.1:$proxy_port"
233+
assert_contains "$snapshot_1"
234+
235+
# Stop the canister and load the new snapshot.
236+
dfx canister stop hello_backend --network "http://127.0.0.1:$proxy_port"
237+
assert_command dfx canister snapshot load hello_backend "$snapshot_1" --network "http://127.0.0.1:$proxy_port"
238+
239+
# Start the canister again and verify the loaded snapshot.
240+
dfx canister start hello_backend --network "http://127.0.0.1:$proxy_port"
241+
assert_command dfx canister call hello_backend read --network "http://127.0.0.1:$proxy_port"
242+
assert_contains '(1 : nat)'
243+
244+
toxiproxy_delete_proxy proxy_network_drop
245+
}
246+
108247
@test "can query a website" {
109248
dfx_start
110249

@@ -116,4 +255,4 @@ teardown() {
116255
assert_command dfx canister call e2e_project_backend get_url '("www.githubstatus.com:443","https://www.githubstatus.com:443")'
117256
assert_contains "Git Operations"
118257
assert_contains "API Requests"
119-
}
258+
}

‎e2e/utils/toxiproxy.bash‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
# Helpers for toxiproxy to use toxiproxy-server and toxiproxy-cli
5+
6+
: "${TOXIPROXY_HOST:=127.0.0.1}"
7+
: "${TOXIPROXY_PORT:=8474}"
8+
9+
# Check if toxiproxy server is running
10+
toxiproxy_is_running() {
11+
curl --silent --fail "http://${TOXIPROXY_HOST}:${TOXIPROXY_PORT}/version" >/dev/null 2>&1
12+
}
13+
14+
# Start toxiproxy server
15+
toxiproxy_start() {
16+
if toxiproxy_is_running; then
17+
return 0
18+
fi
19+
20+
if ! command -v toxiproxy-server >/dev/null 2>&1; then
21+
echo "toxiproxy-server not found in PATH" >&2
22+
return 1
23+
fi
24+
25+
toxiproxy-server -host "$TOXIPROXY_HOST" -port "$TOXIPROXY_PORT" >/dev/null 2>&1 3>&- &
26+
export E2E_TOXIPROXY_PID=$!
27+
28+
for _ in $(seq 1 50); do
29+
if toxiproxy_is_running; then
30+
return 0
31+
fi
32+
sleep 0.1
33+
done
34+
35+
echo "Toxiproxy server did not become available on ${TOXIPROXY_HOST}:${TOXIPROXY_PORT}" >&2
36+
return 1
37+
}
38+
39+
# Stop toxiproxy server
40+
toxiproxy_stop() {
41+
if [ -n "${E2E_TOXIPROXY_PID:-}" ]; then
42+
kill "$E2E_TOXIPROXY_PID" >/dev/null 2>&1 || true
43+
unset E2E_TOXIPROXY_PID
44+
fi
45+
}
46+
47+
# Create or replace a proxy
48+
toxiproxy_create_proxy() {
49+
local listen=$1 upstream=$2 name=$3
50+
51+
# Ensure toxiproxy-cli is available
52+
if ! command -v toxiproxy-cli >/dev/null 2>&1; then
53+
echo "toxiproxy-cli not found in PATH" >&2
54+
return 1
55+
fi
56+
57+
toxiproxy-cli delete "$name" >/dev/null 2>&1 || true
58+
toxiproxy-cli create --listen "$listen" --upstream "$upstream" "$name" >/dev/null 2>&1
59+
}
60+
61+
# Delete a proxy
62+
toxiproxy_delete_proxy() {
63+
local name=$1
64+
toxiproxy-cli delete "$name" >/dev/null 2>&1 || true
65+
}
66+
67+
# Set a proxy to enabled or disabled
68+
toxiproxy_toggle_proxy() {
69+
local name=$1
70+
toxiproxy-cli toggle "$name" >/dev/null 2>&1
71+
}
72+
73+
# Add latency toxic (downstream)
74+
toxiproxy_add_latency() {
75+
local latency=$1 jitter=$2 name=$3
76+
toxiproxy-cli toxic add -t latency -a latency="$latency" -a jitter="$jitter" -d "$name" >/dev/null
77+
}
78+
79+
# Add limit_data toxic (downstream)
80+
toxiproxy_add_limit_data() {
81+
local toxic_name=$1 bytes=$2 proxy_name=$3 direction=${4:-"-d"}
82+
toxiproxy-cli toxic add -n "$toxic_name" -t limit_data -a bytes="$bytes" ${direction:+"$direction"} "$proxy_name" >/dev/null
83+
}
84+
85+
# Remove a toxic
86+
toxiproxy_remove_toxic() {
87+
local toxic_name=$1 proxy_name=$2
88+
toxiproxy-cli toxic remove -n "$toxic_name" "$proxy_name" >/dev/null 2>&1 || true
89+
}

‎scripts/workflows/e2e-matrix.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
SELECTED_TESTS = ["dfx/bitcoin", "dfx/canister_http_adapter", "dfx/start"]
88

99
# Run these tests in serial
10-
SERIAL_TESTS = ["dfx/start", "dfx/bitcoin", "dfx/cycles-ledger", "dfx/ledger", "dfx/serial_misc"]
10+
SERIAL_TESTS = ["dfx/start", "dfx/bitcoin", "dfx/cycles-ledger", "dfx/ledger", "dfx/serial_misc", "dfx/canister_extra"]
1111

1212
def test_scripts(prefix):
1313
all_files = os.listdir(f"e2e/tests-{prefix}")

‎scripts/workflows/provision-linux.sh‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,22 @@ fi
5454
if [ "$E2E_TEST" = "tests-dfx/info.bash" ]; then
5555
sudo apt-get install --yes libarchive-zip-perl
5656
fi
57+
if [ "$E2E_TEST" = "tests-dfx/canister_extra.bash" ]; then
58+
VERSION=v2.10.0
59+
ARCH=$(uname -m)
60+
case "$ARCH" in
61+
x86_64|amd64) ARCH_DL="amd64" ;;
62+
arm64|aarch64) ARCH_DL="arm64" ;;
63+
*) echo "Unsupported ARCH: $ARCH" >&2; exit 1 ;;
64+
esac
65+
66+
BASE_URL="https://github.com/Shopify/toxiproxy/releases/download/$VERSION"
67+
curl -fsSL "$BASE_URL/toxiproxy-server-linux-${ARCH_DL}" -o toxiproxy-server
68+
curl -fsSL "$BASE_URL/toxiproxy-cli-linux-${ARCH_DL}" -o toxiproxy-cli
69+
chmod +x toxiproxy-server toxiproxy-cli
70+
sudo mv toxiproxy-server /usr/local/bin/
71+
sudo mv toxiproxy-cli /usr/local/bin/
72+
fi
5773

5874
# Set environment variables.
5975
echo "$HOME/bin" >> "$GITHUB_PATH"

0 commit comments

Comments
 (0)