feat(storage-proofs): store MTs on disk by default (#836)

BREAKING CHANGE: We now store MTs on disk by default, this
shouldn't have an impact on replication time but will increase disk
usage considerably (roughly 20x the sector size).

To store them on memory instead use the (new) `mem-trees` feature
(that replaces the old, now removed, `disk-trees`).

Author: schomatis

.circleci/config.yml

@@ -80,7 +80,7 @@ jobs:
             cargo +stable test --verbose --release --all
             RUSTFLAGS="-D warnings" cargo +stable build --examples --release --all
 
-  test_disk_trees:
+  test_mem_trees:
     docker:
       - image: filecoin/rust:latest
     working_directory: /mnt/crate
@@ -94,9 +94,9 @@ jobs:
             - cargo-v12-{{ checksum "rust-toolchain" }}-{{ checksum "Cargo.toml" }}-{{ checksum "Cargo.lock" }}-{{ arch }}
             - parameter-cache-{{ .Revision }}
       - run:
-          name: Test (nightly) disk-trees feature
+          name: Test (nightly) mem-trees feature
           command: |
-            cargo +$(cat rust-toolchain) test --verbose --release -p storage-proofs --features disk-trees -Z package-features
+            cargo +$(cat rust-toolchain) test --verbose --release -p storage-proofs --features mem-trees -Z package-features
           environment:
             FIL_PROOFS_REPLICATED_TREES_DIR: replicated-disk-trees
 
@@ -329,7 +329,7 @@ workflows:
       - test_release:
           requires:
             - cargo_fetch
-      - test_disk_trees:
+      - test_mem_trees:
           requires:
             - cargo_fetch
       - test_ignored_release:

README.md

@@ -159,24 +159,24 @@ To check that it's working you can inspect the replication log to find `using pa
 
 (You can also verify if the cache is working by inspecting the time each layer takes to encode, `encoding, layer:` in the log, where the first two layers, forward and reverse, will take more time than the rest to populate the cache while the remaining 8 should see a considerable time drop.)
 
-### Memory
-
-We try to generate the MTs in parallel to speed up the process but that takes 2 sector sizes per each layer (e.g., a 1 GiB sector may require, in the worst case scenario, up to 20 GiB of memory to hold the MTs alone). To reduce that (at the cost of speed) we have the (experimental) `disk-trees` feature to offload the MTs to disk when we don't use them. For example, to run the `zigzag` example with this feature you'd need to indicate so to `cargo`,
+In the most extreme case, to reduce time at the cost of *a lot* of memory consumption you can turn on the feature that stores MTs on memory (`mem-trees`) instead of on disk (the default) to generate them all on RAM and avoid disk I/O (if the HW doesn't have enough RAM to handle the MTs, roughly 20x the sector size, this won't have the desired effect as the OS will start backing them on disk anyway). For example, to run the `zigzag` example with this feature turned on you'd need to indicate so to `cargo`,
 
 ```
-# From inside the `storage-proofs` directory, where this feature
-# needs to be activated:
-cargo build                                                                   \
+# NEEDS TO BE RUN INSIDE `storage-proofs\` directory
+# for the `--features` to take effect.
+cargo run                                                                     \
   -p filecoin-proofs                                                          \
   --release                                                                   \
   --example zigzag                                                            \
   --features                                                                  \
-    disk-trees                                                               &&
-../target/release/examples/zigzag                                             \
+    mem-trees                                                                 \
+    --                                                                        \
     --size 1048576
 ```
 
-This optimization should reduce the maximum RSS, in the `zigzag` example, to 1-2 times the sector size used (so in the above command that tested ZigZag with a 1 GiB sector the maximum RSS reported by commands like `/usr/bin/time -v` should not exceed 2 GiB, please submit an issue if you observe otherwise).
+### Memory
+
+At the moment the default configuration is set to reduce memory consumption as much as possible so there's not much to do from the user side. (We are now storing MTs on disk, which were the main source of memory consumption.) You should expect a maximum RSS between 1-2 sector sizes, if you experience peaks beyond that range please report an issue (you can check the max RSS with the `/usr/bin/time -v` command).
 
 ## Generate Documentation
 

storage-proofs/Cargo.toml

@@ -51,7 +51,7 @@ pretty_env_logger = "0.3.0"
 default = []
 simd = []
 asm = ["sha2/sha2-asm"]
-disk-trees = []
+mem-trees = []
 big-sector-sizes-bench = []
 unchecked-degrees = []
 

storage-proofs/src/merkle.rs

@@ -11,16 +11,20 @@ use paired::bls12_381::Fr;
 use crate::hasher::{Domain, Hasher};
 
 pub use merkletree::merkle::{next_pow2, populate_leaves, Store};
-pub type MmapStore<E> = merkletree::merkle::MmapStore<E>;
-pub type DiskStore<E> = merkletree::merkle::DiskStore<E>;
-#[cfg(feature = "disk-trees")]
+
+#[cfg(not(feature = "mem-trees"))]
+type DiskStore<E> = merkletree::merkle::DiskStore<E>;
+#[cfg(not(feature = "mem-trees"))]
 pub type MerkleTree<T, A> = merkle::MerkleTree<T, A, DiskStore<T>>;
-#[cfg(not(feature = "disk-trees"))]
-pub type MerkleTree<T, A> = merkle::MerkleTree<T, A, MmapStore<T>>;
-#[cfg(feature = "disk-trees")]
+#[cfg(not(feature = "mem-trees"))]
 pub type MerkleStore<T> = DiskStore<T>;
-#[cfg(not(feature = "disk-trees"))]
-pub type MerkleStore<T> = MmapStore<T>;
+
+#[cfg(feature = "mem-trees")]
+type VecStore<E> = merkletree::merkle::VecStore<E>;
+#[cfg(feature = "mem-trees")]
+pub type MerkleTree<T, A> = merkle::MerkleTree<T, A, VecStore<T>>;
+#[cfg(feature = "mem-trees")]
+pub type MerkleStore<T> = VecStore<T>;
 
 /// Representation of a merkle proof.
 /// Each element in the `path` vector consists of a tuple `(hash, is_right)`, with `hash` being the the hash of the node at the current level and `is_right` a boolean indicating if the path is taking the right path.