Skip to main content

Mainnet

AWS:

  • m6g.xlarge or any equivalent instance type (ARM based instance)

Bare Metal:

  • 16GB RAM
  • 4 vCPUs
  • At least 1 TB SSD of storage - make sure it's extendable

Assumptions​

We're going to assume you are already logged into your Virtual Machine as a privileged user or as the root user.

Setup​

Installing a geth node via building the source is the easiest way to spin up your Ethereum Mainnet or Testnet node. It will take care of installing all the required system dependencies, and set all the configuration files in place.

Building geth requires both a Go (version 1.16 or later) and a C compiler. You can install them using your favourite package manager. Once the dependencies are installed, run:

make geth

or, to build the full suite of utilities run:

make all

Below you will find the geth flags that you should use to make sure your node will pass onboarding and integrity checks:

  • --config=path/to/config.toml TOML configuration file
  • --pprof Enable the pprof HTTP server(default: false)
  • --pprof.addr pprof HTTP server listening interface(default: 127.0.0.1)
  • --metrics Enable metrics collection and reporting(default: false)
  • --metrics.addr Enable stand-alone metrics HTTP server listening interface(default: 127.0.0.1)
  • --datadir=path/to/database Data directory for the databases and keystore(default: /root/.ethereum)
  • --syncmode Blockchain sync mode ("snap", "full" or "light")
  • --http Enable the HTTP-RPC server
  • --http.addr HTTP-RPC server listening interface (default: localhost)
  • --http.port HTTP-RPC server listening port (default: 8545)
  • --http.api API's offered over the HTTP-RPC interface (default: eth,net,web3)
  • --http.corsdomain Comma separated list of domains from which to accept cross origin requests (browser enforced)
  • --ws Enable the WS-RPC server
  • --ws.addr WS-RPC server listening interface (default: localhost)
  • --ws.port WS-RPC server listening port (default: 8546)
  • --ws.api API's offered over the WS-RPC interface (default: eth,net,web3)
  • --ws.origins Origins from which to accept websockets requests
  • --authrpc.jwtsecret=path/to/jwt.hex
/usr/local/bin/geth --config=/path/to/config.toml --metrics --metrics.addr=VALUE --datadir /path/to/database

Config file should look like this:

[Eth]

SyncMode = "snap"
NetworkId=1
EthDiscoveryURLs = ["enrtree://AKA3AM6LPBYEUDMVNU3BSVQJ5AD45Y7YPOHJLEF6W26QOE4VTUDPE@all.mainnet.ethdisco.net"]
SnapDiscoveryURLs = ["enrtree://AKA3AM6LPBYEUDMVNU3BSVQJ5AD45Y7YPOHJLEF6W26QOE4VTUDPE@all.mainnet.ethdisco.net"]

NoPruning = false
NoPrefetch = false
TxLookupLimit = 2350000
LightPeers = 100
UltraLightFraction = 75
DatabaseCache = 2048
DatabaseFreezer = ""
TrieCleanCache = 614
TrieCleanCacheJournal = "triecache"
TrieCleanCacheRejournal = 3600000000000
TrieDirtyCache = 1024
TrieTimeout = 3600000000000
SnapshotCache = 409
Preimages = false
EnablePreimageRecording = false
RPCGasCap = 50000000
RPCTxFeeCap = 1e+00
[Eth.Miner]
GasFloor = 0
GasCeil = 8000000
GasPrice = 1000000000
Recommit = 3000000000
Noverify = false

[Eth.Ethash]
CacheDir = "ethash"
CachesInMem = 2
CachesOnDisk = 3
CachesLockMmap = false
DatasetDir = "/root/.ethash"
DatasetsInMem = 1
DatasetsOnDisk = 2
DatasetsLockMmap = false
PowMode = 0
NotifyFull = false

[Eth.TxPool]
Locals = []
NoLocals = false
Journal = "transactions.rlp"
Rejournal = 3600000000000
PriceLimit = 1
PriceBump = 10
AccountSlots = 16
GlobalSlots = 5120
AccountQueue = 64
GlobalQueue = 1024
Lifetime = 10800000000000

[Eth.GPO]
Blocks = 20
Percentile = 60
MaxHeaderHistory = 1024
MaxBlockHistory = 1024
MaxPrice = 500000000000
IgnorePrice = 2

[Node]
IPCPath = "geth.ipc"
HTTPHost = "0.0.0.0"
InsecureUnlockAllowed = false
HTTPPort = 8545
HTTPVirtualHosts = ["*"]
HTTPModules = ["eth", "net", "web3", "txpool", "personal","debug"]
WSPort = 8546
WSModules = ["eth", "net", "web3","txpool", "personal","debug"]
WSHost = "0.0.0.0"
DataDir = "/node/data"
HTTPCors = ["*"]
WSOrigins = ["*"]

[Node.P2P]
MaxPeers = 60
NoDiscovery = false

StaticNodes = []
TrustedNodes = []
ListenAddr = ":30303"
EnableMsgEvents = false

[Node.HTTPTimeouts]
ReadTimeout = 30000000000
WriteTimeout = 30000000000
IdleTimeout = 120000000000

[Metrics]
Enabled = true
HTTP = "0.0.0.0"
Port = 6060

You now have a running Ethereum Mainnet RPC node. All you need to do now is wait for it to sync. You can check if the node is synced by running the API Call listed below from inside your environment. You are going to need to have the curl and jq packages installed for this, so make sure to install them beforehand.

curl -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "eth_syncing","params": []}' localhost:8545

If the result is false, it means that your node is fully synced.

Another way to check which block the node is at would be running:

curl -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "eth_blockNumber","params": []}' localhost:8545

The result should be a hex number (i.e 0x10c5815). If you convert it to a decimal number, you can compare it to the latest block listed on the Ethereum Mainnet explorer: https://etherscan.com/

The Ethereum node exports both RPC on port 8545 and WS on port 8546.

In order to test the WS endpoint, we will need to install a package called node-ws.

An example WS call would look like this:

wscat --connect ws://localhost:8546
> {"id":1, "jsonrpc":"2.0", "method": "eth_blockNumber","params": []}
info

Please make sure to also check the Official Documentation and the Github Repository posted above in order to make sure you are keeping your node up to date.

The Merge​

As you may know Ethereum has been moving to proof-of-stake!. The transition, known as The Merge, must first be activated on the Beacon Chain with the Bellatrix upgrade. After this, the proof-of-work chain will migrate to proof-of-stake upon hitting a specific Total Difficulty value.

The Terminal Total Difficulty value that triggered The Merge is 58750000000000000000000.

So an Ethereum node will consist in two components:

  • Execution Layer which will be referred as EL, and is basically the geth that is explained in the first part of this documentation page;
  • Consensus Layer which will be referred as CL and could be started via multiple alternatives like Lighthouse, Prysm, Lodestar, Teku. Our decision was to go with Prysm as they have very nice documentation about The Merge, which you can find here.

Running prysm node​

Below you will find the prysm flags that you should use to make sure your node will pass onboarding and integrity checks:

  • --accept-terms-of-use
  • --http-web3provider defaults to http://localhost:8551
  • --jwt-secret=/path/to/jwt.hex
  • --datadir=/path/to/datastore

In order to check if EL communicates with CL , you can have a look into the logs that should state the blocks are being generated and that there are no [ERROR] logs. Also, you can check the chain functionality by calling a JSON-RPC method like eth_chainId.

Sync from a checkpoint​

Official Docs: https://docs.prylabs.network/docs/prysm-usage/checkpoint-sync

Checkpoint sync is a feature that significantly speeds up the initial sync between your beacon node and the Beacon Chain. With checkpoint sync configured, your beacon node will begin syncing from a recently finalized checkpoint instead of syncing from genesis. This can make installations, validator migrations, recoveries, and testnet deployments way faster.

To sync from a checkpoint, your Prysm beacon node needs three pieces of information: the latest finalized BeaconState, the SignedBeaconBlock, and (if you're on a testnet) the genesis state for the network you're using. Together, the BeaconState and SignedBeaconBlock represent a single checkpoint state.

The following added flag starts a beacon node with checkpoint sync configured to pull checkpoint state from another local beacon node's RPC endpoint using port 3500:

--checkpoint-sync-url=http://localhost:3500

You can also refer to https://eth-clients.github.io/checkpoint-sync-endpoints/ in order to find public Ethereum Beacon Chain checkpoint sync endpoints.

Note that this is entirely optional. The beacon node requesting the checkpoint state from a prysm node started from scratch doesn't need this flag.

Running an archive node​

If you want to run an Ethereum Mainnet archive node you can do it by using erigon client. We prefer this option over others for infrastructure cost reasons.

Bare Metal:

  • RAM: >= 16GB RAM
  • CPU: >= 4 vCPUs
  • At least 3TB SSD of storage - make sure it's extendable: 2.5TB state (as of August 2022)

Installing a erigon node via building the source is the easiest way to spin up your Ethereum Mainnet Archive node. It will take care of installing all the required system dependencies, and set all the configuration files in place.

Building erigon requires Go version 1.20 or later. You can install it using your favourite package manager. Once the dependencies are installed, run:

make erigon
make rpcdaemon

Basically you will have two components: the node started using erigon and the rpc started using rpcdaemon.

Below you will find the erigon flags that you should use to make sure your node will pass onboarding and integrity checks:

  • --rpc.gascap=150000000 rpc gascap parameter
  • --http=false http rpc server parameter
  • --externalcl enables external consensus
  • --private.api.addr components will connect to erigon by this internal grpc API
  • --pprof Enable the pprof HTTP server(default: false)
  • --pprof.addr pprof HTTP server listening interface(default: 127.0.0.1)
  • --metrics Enable metrics collection and reporting(default: false)
  • --metrics.addr Enable stand-alone metrics HTTP server listening interface(default: 127.0.0.1)
  • --datadir=path/to/database Data directory for the databases and keystore(default: /root/.ethereum)
  • --authrpc.jwtsecret=path/to/jwt.hex

Below you will find the rpdaemon flags that you should use to make sure your node will pass onboarding and integrity checks:

  • --rpc.gascap=150000000 rpc gascap parameter
  • --datadir=path/to/database Data directory for the databases and keystore(default: /root/.ethereum)
  • --http.api=eth,erigon,web3,net,debug,trace,txpool
  • --http.addr=0.0.0.0
  • --ws
  • --http.vhosts=*
  • --http.corsdomain=*

Monitoring Guidelines​

In order to maintain a healthy node that passes the Integrity Protocol's checks, you should have a monitoring system in place. Blockchain nodes usually offer metrics regarding the node's behaviour and health - a popular way to offer these metrics is Prometheus-like metrics. The most popular monitoring stack, which is also open source, consists of:

  • Prometheus - scrapes and stores metrics as time series data (blockchain nodes cand send the metrics to it);
  • Grafana - allows querying, visualization and alerting based on metrics (can use Prometheus as a data source);
  • Alertmanager - handles alerting (can use Prometheus metrics as data for creating alerts);
  • Node Exporter - exposes hardware and kernel-related metrics (can send the metrics to Prometheus).

We will assume that Prometheus/Grafana/Alertmanager are already installed (we will provide a detailed guide of how to set up monitoring and alerting with the Prometheus + Grafana stack at a later time; for now, if you do not have the stack already installed, please follow this official basic guide here).

We recommend installing the Node Exporter utility since it offers valuable information regarding CPU, RAM & storage. This way, you will be able to monitor possible hardware bottlenecks, or to check if your node is underutilized - you could use these valuable insights to make decisions regarding scaling up/down the allocated hardware resources.

Below, you can find a script that installs Node Exporter as a system service.

#!/bin/bash

# set the latest version
VERSION=1.3.1

# download and untar the binary
wget https://github.com/prometheus/node_exporter/releases/download/v${VERSION}/node_exporter-${VERSION}.linux-amd64.tar.gz
tar xvf node_exporter-*.tar.gz
sudo cp ./node_exporter-${VERSION}.linux-amd64/node_exporter /usr/local/bin/

# create system user
sudo useradd --no-create-home --shell /usr/sbin/nologin node_exporter

# change ownership of node exporter binary
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

# remove temporary files
rm -rf ./node_exporter*

# create systemd service file
cat > /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
EOF

# enable the node exporter service and start it
sudo systemctl daemon-reload
sudo systemctl enable node_exporter.service
sudo systemctl start node_exporter.service

As a reminder, Node Exporter uses port 9100 by default, so be sure to expose this port to the machine which holds the Prometheus server. The same should be done for the metrics port(s) of the blockchain node (in this case, we should expose port 6060 - for monitoring the ethereum node).

Having installed Node Exporter and having already exposed the node's metrics, these should be added as targets under the scrape_configs section in your Prometheus configuration file (i.e. /etc/prometheus/prometheus.yml), before reloading the new config (either by restarting or reloading the config - please check the official documentation). This should look similar to this:

scrape_configs:
  - job_name: 'eth-node'
    scrape_interval: 10s
    metrics_path: /debug/metrics/prometheus
    static_configs:
      - targets:
        - '<NODE0_IP>:6060'
        - '<NODE1_IP>:6060' # you can add any number of nodes as targets
  - job_name: 'eth-node-exporter'
    scrape_interval: 10s
    metrics_path: /metrics
    static_configs:
      - targets:
        - '<NODE0_IP>:9100'
        - '<NODE1_IP>:9100' # you can add any number of nodes as targets

In the configuration file above, please replace:

  • <NODE0_IP> - node 0's IP
  • <NODE1_IP> - node 1's IP (you can add any number of nodes as targets)
  • ...
  • <NODEN_IP> - node N's IP (you can add any number of nodes as targets)

That being said, the most important metrics that should be checked are:

  • node_cpu_seconds_total - CPU metrics exposed by Node Exporter - for monitoring purposes, you could use the following expression:
    • 100 - (avg by (instance) (rate(node_cpu_seconds_total{job="eth-node-exporter",mode="idle"}[5m])) * 100), which means the average percentage of CPU usage over the last 5 minutes;
  • node_memory_MemTotal_bytes/node_memory_MemAvailable_bytes - RAM metrics exposed by Node Exporter - for monitoring purposes, you could use the following expression:
    • (node_memory_MemTotal_bytes{job="eth-node-exporter"} - node_memory_MemAvailable_bytes{job="eth-node-exporter"}) / 1073741824, which means the amount of RAM (in GB) used, excluding cache/buffers;
  • node_network_receive_bytes_total - network traffic metrics exposed by Node Exporter - for monitoring purposes, you could use the following expression:
    • rate(node_network_receive_bytes_total{job="eth-node-exporter"}[1m]), which means the average network traffic received, per second, over the last minute (in bytes);
  • node_filesystem_avail_bytes - FS metrics exposed by Node Exporter - for monitoring purposes, you could use the following expression:
    • node_filesystem_avail_bytes{job="eth-node-exporter",device="<DEVICE>"} / 1073741824, which means the filesystem space available to non-root users (in GB) for a certain device <DEVICE> (i.e. /dev/sda or wherever the blockchain data is stored) - this can be used to get an alert whenever the available space left is below a certain threshold (please be careful how you choose this threshold: if you have storage that can easily be increased - for example, EBS storage from AWS, you can set a lower threshold, but if you run your node on a bare metal machine which is not easily upgradable, you should set a higher threshold just to be sure you are able to find a solution before it fills up);
  • up - Prometheus automatically generated metrics - for monitoring purposes, you could use the following expressions:
    • up{job="eth-node"}, which has 2 possible values: 1, if the node is up, or 0, if the node is down - this can be used to get an alert whenever the node goes down (i.e. it can be triggered at each restart of the node);
  • chain_head_block - metrics exposed by the ethereum node - for monitoring purposes, you could use the following expressions:
    • increase(chain_head_block{job="eth-node"}[1m]), which means how many blocks ethereum has been producing in the last 1 minute - this can be used to get an alert whenever the node has fallen behind by comparing with a certain threshold (you should start worrying if the difference is greater than 2-3 for the last 5 minutes);
  • p2p_peers - metrics exposed by the ethereum node - for monitoring purposes, you could use the following expressions:
    • p2p_peers{job="eth-node"}, which means the number of peers connected to the node on the ethereum side - this can be used to get an alert whenever there are less peers than a certain threshold for a certain period of time (i.e. less than 3 peers for 5 minutes);

You can use the above metrics to create both Grafana dashboards and Alertmanager alerts.

info

Please make sure to also check the Official Documentation and the Github Repository posted above in order to make sure you are keeping your node up to date.