Skip to content

Commit e829356

Browse files
jribbinkjribbink
authored
Remote Network Forking Improvements (#883)
Renames the flags to `--fork-host` & `--fork-height`, improves the remote store's reliability/performance for production use in CI (lru cache + rate limit management).
1 parent 1062b33 commit e829356

File tree

8 files changed

+573
-67
lines changed

8 files changed

+573
-67
lines changed

‎README.md‎

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,8 @@ values.
7373
| `--host` | `FLOW_HOST` | ` ` | Host to listen on for emulator GRPC/REST/Admin servers (default: All Interfaces) |
7474
| `--chain-id` | `FLOW_CHAINID` | `emulator` | Chain to simulate, if 'mainnet' or 'testnet' values are used, you will be able to run transactions against that network and a local fork will be created. Valid values are: 'emulator', 'testnet', 'mainnet' |
7575
| `--redis-url` | `FLOW_REDIS_URL` | '' | Redis-server URL for persisting redis storage backend ( `redis://[[username:]password@]host[:port][/database]` ) |
76-
| `--start-block-height` | `FLOW_STARTBLOCKHEIGHT` | `0` | Start block height to use when starting the network using 'testnet' or 'mainnet' as the chain-id |
77-
| `--rpc-host` | `FLOW_RPCHOST` | '' | RPC host (access node) to query for previous state when starting the network using 'testnet' or 'mainnet' as the chain-id |
76+
| `--fork-host` | `FLOW_FORK_HOST` | '' | gRPC access node address (`host:port`) to fork from |
77+
| `--fork-height` | `FLOW_FORK_HEIGHT` | `0` | Block height to pin the fork (defaults to latest sealed) |
7878
| `--legacy-upgrade` | `FLOW_LEGACYUPGRADE` | `false` | Enable upgrading of legacy contracts |
7979
| `--computation-reporting` | `FLOW_COMPUTATIONREPORTING` | `false` | Enable computation reporting for Cadence scripts & transactions |
8080
| `--checkpoint-dir` | `FLOW_CHECKPOINTDIR` | '' | Checkpoint directory to load the emulator state from, if starting the emulator from a checkpoint |
@@ -155,8 +155,7 @@ Post Data: height={block height}
155155
```
156156
157157
Note: it is only possible to roll back state to a height that was previously executed by the emulator.
158-
To roll back to a past block height when using a forked Mainnet or Testnet network, use the
159-
`--start-block-height` flag.
158+
To pin the starting block height when using a fork, use the `--fork-height` flag.
160159
161160
## Managing emulator state
162161
It's possible to manage emulator state by using the admin API. You can at any point
@@ -269,15 +268,14 @@ you must specify the network name for the chain ID flag and the RPC host
269268
to connect to.
270269

271270
```
272-
flow emulator --chain-id mainnet --rpc-host access.mainnet.nodes.onflow.org:9000
273-
flow emulator --chain-id mainnet --rpc-host access.devnet.nodes.onflow.org:9000
271+
flow emulator --fork-host access.mainnet.nodes.onflow.org:9000
272+
flow emulator --fork-host access.mainnet.nodes.onflow.org:9000 --fork-height 12345
274273
```
275274

276275
Please note, that the actual execution on the real network may differ depending on the exact state when the transaction is
277276
executed.
278277

279-
By default, the forked network will start from the latest sealed block when the emulator
280-
is started. You can specify a different starting block height by using the `--start-block-height` flag.
278+
By default, the forked network will start from the latest sealed block when the emulator is started. You can specify a different starting block height by using the `--fork-height` flag.
281279

282280
You can also store all of your changes and cached registers to a persistent db by using the `--persist` flag,
283281
along with the other SQLite settings.

‎cmd/emulator/start/start.go‎

Lines changed: 57 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -75,14 +75,18 @@ type Config struct {
7575
SqliteURL string `default:"" flag:"sqlite-url" info:"sqlite db URL for persisting sqlite storage backend "`
7676
CoverageReportingEnabled bool `default:"false" flag:"coverage-reporting" info:"enable Cadence code coverage reporting"`
7777
LegacyContractUpgradeEnabled bool `default:"false" flag:"legacy-upgrade" info:"enable Cadence legacy contract upgrade"`
78-
StartBlockHeight uint64 `default:"0" flag:"start-block-height" info:"block height to start the emulator at. only valid when forking Mainnet or Testnet"`
79-
RPCHost string `default:"" flag:"rpc-host" info:"rpc host to query when forking Mainnet or Testnet"`
78+
ForkHost string `default:"" flag:"fork-host" info:"gRPC access node address (host:port) to fork from"`
79+
ForkHeight uint64 `default:"0" flag:"fork-height" info:"height to pin fork; defaults to latest sealed"`
8080
CheckpointPath string `default:"" flag:"checkpoint-dir" info:"checkpoint directory to load the emulator state from"`
8181
StateHash string `default:"" flag:"state-hash" info:"state hash of the checkpoint to load the emulator state from"`
8282
ComputationReportingEnabled bool `default:"false" flag:"computation-reporting" info:"enable Cadence computation reporting"`
8383
ScheduledTransactionsEnabled bool `default:"true" flag:"scheduled-transactions" info:"enable Cadence scheduled transactions"`
8484
SetupEVMEnabled bool `default:"true" flag:"setup-evm" info:"enable EVM setup for the emulator, this will deploy the EVM contracts"`
8585
SetupVMBridgeEnabled bool `default:"true" flag:"setup-vm-bridge" info:"enable VM Bridge setup for the emulator, this will deploy the VM Bridge contracts"`
86+
87+
// Deprecated hidden aliases
88+
StartBlockHeight uint64 `default:"0" flag:"start-block-height" info:"(deprecated) use --fork-height"`
89+
RPCHost string `default:"" flag:"rpc-host" info:"(deprecated) use --fork-host"`
8690
}
8791

8892
const EnvPrefix = "FLOW"
@@ -147,31 +151,23 @@ func Cmd(config StartConfig) *cobra.Command {
147151
Exit(1, err.Error())
148152
}
149153

150-
if conf.StartBlockHeight > 0 && flowChainID != flowgo.Mainnet && flowChainID != flowgo.Testnet {
151-
Exit(1, "❗ --start-block-height is only valid when forking Mainnet or Testnet")
154+
// Deprecation shims: map old flags to new and warn
155+
if conf.RPCHost != "" && conf.ForkHost == "" {
156+
logger.Warn().Msg("❗ --rpc-host is deprecated; use --fork-host")
157+
conf.ForkHost = conf.RPCHost
152158
}
153-
154-
if (flowChainID == flowgo.Mainnet || flowChainID == flowgo.Testnet) && conf.RPCHost == "" {
155-
Exit(1, "❗ --rpc-host must be provided when forking Mainnet or Testnet")
159+
if conf.StartBlockHeight > 0 && conf.ForkHeight == 0 {
160+
logger.Warn().Msg("❗ --start-block-height is deprecated; use --fork-height")
161+
conf.ForkHeight = conf.StartBlockHeight
156162
}
157163

158-
serviceAddress := flowsdk.ServiceAddress(flowsdk.ChainID(flowChainID))
159-
if conf.SimpleAddresses {
160-
serviceAddress = flowsdk.HexToAddress("0x1")
164+
// In non-fork mode, fork-only flags are invalid
165+
if conf.ForkHost == "" && (conf.StartBlockHeight > 0 || conf.ForkHeight > 0) {
166+
Exit(1, "❗ --fork-height requires --fork-host")
161167
}
162168

163-
serviceFields := map[string]any{
164-
"serviceAddress": serviceAddress.Hex(),
165-
"servicePubKey": hex.EncodeToString(servicePublicKey.Encode()),
166-
"serviceSigAlgo": serviceKeySigAlgo.String(),
167-
"serviceHashAlgo": serviceKeyHashAlgo.String(),
168-
}
169-
170-
if servicePrivateKey != nil {
171-
serviceFields["servicePrivKey"] = hex.EncodeToString(servicePrivateKey.Encode())
172-
}
173-
174-
logger.Info().Fields(serviceFields).Msgf("⚙️ Using service account 0x%s", serviceAddress.Hex())
169+
// Service account logging is deferred until after server configuration to allow
170+
// higher-level wrappers to customize fork settings via ConfigureServer.
175171

176172
minimumStorageReservation := fvm.DefaultMinimumStorageReservation
177173
if conf.MinimumAccountBalance != "" {
@@ -183,6 +179,18 @@ func Cmd(config StartConfig) *cobra.Command {
183179
storageMBPerFLOW = parseCadenceUFix64(conf.StorageMBPerFLOW, "storage-per-flow")
184180
}
185181

182+
// Recompute chain ID and service address accurately for fork mode by querying the node.
183+
forkHost := conf.ForkHost
184+
resolvedChainID := flowChainID
185+
forkMode := forkHost != ""
186+
if forkMode {
187+
parsed, err := server.DetectRemoteChainID(forkHost)
188+
if err != nil {
189+
Exit(1, fmt.Sprintf("failed to detect remote chain id from %s: %v", forkHost, err))
190+
}
191+
resolvedChainID = parsed
192+
}
193+
186194
serverConf := &server.Config{
187195
GRPCPort: conf.Port,
188196
GRPCDebug: conf.GRPCDebug,
@@ -212,13 +220,13 @@ func Cmd(config StartConfig) *cobra.Command {
212220
SkipTransactionValidation: conf.SkipTxValidation,
213221
SimpleAddressesEnabled: conf.SimpleAddresses,
214222
Host: conf.Host,
215-
ChainID: flowChainID,
223+
ChainID: resolvedChainID,
216224
RedisURL: conf.RedisURL,
217225
ContractRemovalEnabled: conf.ContractRemovalEnabled,
218226
SqliteURL: conf.SqliteURL,
219227
CoverageReportingEnabled: conf.CoverageReportingEnabled,
220-
StartBlockHeight: conf.StartBlockHeight,
221-
RPCHost: conf.RPCHost,
228+
ForkHost: conf.ForkHost,
229+
ForkHeight: conf.ForkHeight,
222230
CheckpointPath: conf.CheckpointPath,
223231
StateHash: conf.StateHash,
224232
ComputationReportingEnabled: conf.ComputationReportingEnabled,
@@ -227,6 +235,24 @@ func Cmd(config StartConfig) *cobra.Command {
227235
SetupVMBridgeEnabled: conf.SetupVMBridgeEnabled,
228236
}
229237

238+
serviceAddress := flowsdk.ServiceAddress(flowsdk.ChainID(resolvedChainID))
239+
if conf.SimpleAddresses {
240+
serviceAddress = flowsdk.HexToAddress("0x1")
241+
}
242+
243+
serviceFields := map[string]any{
244+
"serviceAddress": serviceAddress.Hex(),
245+
"servicePubKey": hex.EncodeToString(servicePublicKey.Encode()),
246+
"serviceSigAlgo": serviceKeySigAlgo.String(),
247+
"serviceHashAlgo": serviceKeyHashAlgo.String(),
248+
}
249+
250+
if servicePrivateKey != nil {
251+
serviceFields["servicePrivKey"] = hex.EncodeToString(servicePrivateKey.Encode())
252+
}
253+
254+
logger.Info().Fields(serviceFields).Msgf("⚙️ Using service account 0x%s", serviceAddress.Hex())
255+
230256
emu := server.NewEmulatorServer(logger, serverConf)
231257
if emu != nil {
232258
for _, middleware := range config.RestMiddlewares {
@@ -241,6 +267,12 @@ func Cmd(config StartConfig) *cobra.Command {
241267

242268
initConfig(cmd)
243269

270+
// Hide and deprecate legacy flags while keeping them functional
271+
_ = cmd.PersistentFlags().MarkHidden("rpc-host")
272+
_ = cmd.PersistentFlags().MarkDeprecated("rpc-host", "use --fork-host")
273+
_ = cmd.PersistentFlags().MarkHidden("start-block-height")
274+
_ = cmd.PersistentFlags().MarkDeprecated("start-block-height", "use --fork-height")
275+
244276
return cmd
245277
}
246278

‎docs/overview.md‎

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,8 @@ values.
7373
| `--host` | `FLOW_HOST` | ` ` | Host to listen on for emulator GRPC/REST/Admin servers (default: all interfaces) |
7474
| `--chain-id` | `FLOW_CHAINID` | `emulator` | Chain to emulate for address generation. Valid values are: 'emulator', 'testnet', 'mainnet' |
7575
| `--redis-url` | `FLOW_REDIS_URL` | '' | Redis-server URL for persisting redis storage backend ( `redis://[[username:]password@]host[:port][/database]` ) |
76-
| `--start-block-height` | `FLOW_STARTBLOCKHEIGHT` | `0` | Start block height to use when starting the network using 'testnet' or 'mainnet' as the chain-id |
77-
| `--rpc-host` | `FLOW_RPCHOST` | '' | RPC host (access node) to query for previous state when starting the network using 'testnet' or 'mainnet' as the chain-id |
76+
| `--fork-host` | `FLOW_FORK_HOST` | '' | gRPC access node address (`host:port`) to fork from |
77+
| `--fork-height` | `FLOW_FORK_HEIGHT` | `0` | Block height to pin the fork (defaults to latest sealed) |
7878

7979
## Running the emulator with the Flow CLI
8080

@@ -149,8 +149,7 @@ Post Data: height={block height}
149149
```
150150

151151
Note: it is only possible to roll back state to a height that was previously executed by the emulator.
152-
To roll back to a past block height when using a forked Mainnet or Testnet network, use the
153-
`--start-block-height` flag.
152+
To pin the starting block height when using a fork, use the `--fork-height` flag.
154153

155154
## Managing emulator state
156155

@@ -246,14 +245,13 @@ you must specify the network name for the chain ID flag as well as the RPC host
246245
to connect to.
247246

248247
```
249-
flow emulator --chain-id mainnet --rpc-host access-008.mainnet24.nodes.onflow.org:9000
250-
flow emulator --chain-id mainnet --rpc-host access-002.devnet49.nodes.onflow.org:9000
248+
flow emulator --fork-host access.mainnet.nodes.onflow.org:9000
249+
flow emulator --fork-host access.mainnet.nodes.onflow.org:9000 --fork-height 12345
251250
```
252251

253252
Please note, the actual execution on the real network may differ depending on the exact state when the transaction is executed.
254253

255-
By default, the forked network will start from the latest sealed block when the emulator
256-
is started. You can specify a different starting block height by using the `--start-block-height` flag.
254+
By default, the forked network will start from the latest sealed block when the emulator is started. You can specify a different starting block height by using the `--fork-height` flag.
257255

258256
You can also store all of your changes and cached registers to a persistent db by using the `--persist` flag,
259257
along with the other sqlite settings.

‎go.mod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ require (
1010
github.com/google/go-dap v0.11.0
1111
github.com/gorilla/mux v1.8.1
1212
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
13+
github.com/hashicorp/golang-lru/v2 v2.0.7
1314
github.com/improbable-eng/grpc-web v0.15.0
1415
github.com/logrusorgru/aurora v2.0.3+incompatible
1516
github.com/onflow/cadence v1.8.1
@@ -94,7 +95,6 @@ require (
9495
github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 // indirect
9596
github.com/hashicorp/errwrap v1.1.0 // indirect
9697
github.com/hashicorp/go-multierror v1.1.1 // indirect
97-
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
9898
github.com/hashicorp/hcl v1.0.0 // indirect
9999
github.com/holiman/bloomfilter/v2 v2.0.3 // indirect
100100
github.com/holiman/uint256 v1.3.2 // indirect

0 commit comments

Comments
 (0)