Shuffle algorithm of Ethereum 2.0

Introduction If you want to learn shuffle dance, you are in the wrong place. But trust me, shuffle in Eth2 is just as exciting. Shuffle list is a basic operation in Ethereum 2.0. It is mainly used to pseudo-randomly select verifiers to form a committee in every 12-second slot, and to select the proposer of the beacon chain block in each slot. Shuffle seems fairly straightforward. Although it has some pitfalls to be aware of, these pitfalls are very well understood in computer science. The gold standard is probably the Fisher-Yeats shuffle. So why don't we use it in Eth2? I will explain in detail at the end of the article, but in short - light client. The shuffling algorithm we use is swap-or-not, not Fisher-Yates. This choice is based on the paper that was originally used to build the encryption scheme. I recently rewrote our implementation in the Eth2 client Teku, so I wanted to write it while it was hot. Swap-or-Not Shuffle Algorithm One-round operation process Shuffle is performed in rounds. The process of each round is the same, so I will only demonstrate the process of one round below, it is much simpler than it looks. Choose a pivot point and find the first mirror index First, we choose a pivot index p, which is selected pseudo-randomly based on rounds and some other seed data. After the axis is selected, it is fixed in the round. Data: The TVL of the Ethereum bridge reached 24.89 billion US dollars: Jinse Finance reported that DuneAnalytics data showed that the current TVL of the Ethereum bridge reached 23.11 billion US dollars. Among them, the five bridges with the highest lockup volume are AvalancheBridge ($5.958 billion), PolygonBridges ($4.979 billion), FantomAnyswapBridge ($4.516 billion), ArbitrumBridges ($3.013 billion), and RoninBridge ($2.963 billion). [2022/3/2 13:31:07] Based on this pivot point, we select a mirror index m1 at the midpoint between p and 0, that is, m1=p/2. (For the sake of explanation, we will ignore the problem of troublesome rounding errors by one.) Pivot point and first mirror index from first mirror index to pivot point, replacement or not For mirror index m1 and pivot index p For each index between , we randomly decide whether to perform replacement on those elements. For example, for index i1, if we choose not to replace, then we continue to select the next index. If we decide to replace, then we replace the list element on i1 with the one on i1', i.e. its image on the mirror index. That is, i1 is replaced with i1'=m1-(i1-m1), so that the distances from i1 and i1' to m1 are equal. volmex.finance v1 has launched the Ethereum expansion solution Polygon: Jinse Finance reported that according to official news, the token volatility protocol volmex.finance announced that volmex.finance v1 has launched the Ethereum expansion solution Polygon. [2021/7/20 1:03:27] We make the same swap-or-not decision for each index between m1 and p. The swap-or-not decision from the first mirror index to the pivot computes the second mirror index After making all the index decisions from m1 to p, we now find the second mirror index with m2 as the midpoint, i.e. the point with equal distance to p and the end of the list. That is m2=m1+n/2. Second mirror index from pivot point to second mirror, replace or not Finally, we repeat the swap-or-not process, considering all point to pivot point p replacement decisions, that is, p to second mirror m2 Decide. If we choose not to replace, we continue to the next one. If we choose to replace, then we replace the element on j1 with its mirror image on j1' at mirror index m2. Goldman Sachs: Blockchain platforms such as Ethereum may become a huge market for reliable information providers: Goldman Sachs said that blockchain platforms like Ethereum may become a huge market for reliable information providers, just like Amazon is today same as the consumer goods market. (Golden Ten) [2021/5/23 22:33:15] The swap-or-not decision from the axis to the second mirror index is combined. At the end of the round, we have considered the value between m1 and m2 All indexes, that is, half of all indexes, and each index has a specific index on the other half, whether replaced or not. Therefore, all indexes have been considered once for replacement or not. The next round starts with incrementing (or decrementing) rounds so that we have a new pivot index, and then begins to cycle through the process described above. The process of moving from one mirror to another in the same round What's interesting What's smart When deciding whether to replace, the algorithm smartly chooses the higher of the candidate index or its mirror. It means that when it is below the axis, i_1 is selected instead of i_1'; when it is above the axis, i_k' is selected instead of i_k. This means, that we can iterate over the indices in the list flexibly: we can separate 0 to m1 and p to m2 into two separate loops, or combine both in the same loop from m1 to m2, as I did above described (and implemented) in this paper. The result is the same both ways: it doesn't matter whether I consider i_1 or mirror i_1'; the substitution or not yields the same result. Data: The income of Ethereum miners in a single week exceeds that of Bitcoin miners again: The average daily income of Ethereum miners last week was 77 million US dollars, surpassing that of Bitcoin miners (67 million US dollars). Since the beginning of this year, the income of Ethereum miners has been higher than that of Bitcoin. The main reason is that the price of Ethereum has risen sharply. Another key reason is the soaring transaction costs of the network. Currently, 40% of the income of Ethereum miners comes from Gas fees. But with Ethereum Improvement Proposal (EIP) 1559 going into effect this July, a portion of the fees will be sent to the network itself rather than miners. (The Block) [2021/5/13 21:57:15] Rounds In Eth2, the above process will be carried out 90 times. The original paper mentions that it takes 6lgN rounds to "start to show a good security margin on selective cryptographic attacks (CCA)", where N is the length of the list. In Vitalik's commented specification, he says that "experts in cryptography suggest that we can provide sufficient security with 4log2N rounds". The absolute maximum number of validators in Eth2, which is the maximum number of times we need to shuffle the list, is about 222 (4.2 million). The estimated value given by Vitalik is 88 rounds, and the estimated value in the paper is 92 rounds (assuming lg is the natural logarithm). So we're now in a roughly correct range, especially since we very likely won't end up with this many active-zce validators. Adjusting rounds based on list length could have interesting results, but we wouldn't do it, it might be an unnecessary optimization. Ethereum core developer: The London fork is expected in mid-July, including EIP-1559 and other upgrades: Tim Beiko, the core developer of Ethereum, tweeted yesterday that the London fork is expected to be carried out in mid-July, and the fork will include EIP -1559, EIP-3198, EIP-3238 and other upgrades. [2021/4/24 20:54:04] Interestingly, when the Least Authority audited the Beacon Chain specification, they initially found that there was a bias in the shuffling that selects block proposers (see Issue F) . But it turned out that they mistakenly used a shuffle configuration with only 10 rounds. When they increased the shuffle configuration to 90 rounds (the number we use on mainnet), the bias disappeared. The (pseudo)random shuffle algorithm requires us to randomly choose a pivot point in each round, and randomly choose whether to replace each element in each round. In Eth2, we definitely generate randomness from a seed value, whereby the same seed will always produce the same shuffling result. The axis index is generated by performing an 8-byte SHA2 hash on the seed concatenated with the round, and the axis index is generated by the eight bytes of the SHA2 hash of the seed value, which is concatenated with the round, so it Usually changes every round. The decisive bits used to decide whether to replace an element are extracted from the following elements: the SHA256 hash of the seed, the round, the index of the element on the list. Efficiency This shuffle algorithm is much slower than the Fisher-Yates algorithm. If the Fisher-Yates algorithm needs N times of shuffling, our algorithm needs 90N/4 times on average. We also have to consider the generation of pseudo-randomness, which is the most expensive part of the algorithm. Fisher-Yates needs randomness close to Nlog2N digits, and we need 90(log2N+N/2) digits, according to the N value range we need in Eth2, the excess digits are quite a lot (when N is one million , Eth2 needs about twice N). Why choose the swap-or-not algorithm? If the efficiency is not high, why choose this implementation? Shuffling Single Elements The brilliance of this algorithm is that if we only care about a few indices, we don't need to compute the shuffling of the entire list. In fact, we can use this algorithm on a single index to find out which index will be replaced. So, if we want to know where the element at index 217 has been shuffled, we can run the algorithm for that index only, without shuffling the entire list. Also, conversely, if we wanted to know what element was shuffled to index 217, we could run the algorithm in reverse to find element 217 (reverse means running the rounds from high to low, not low to high ). In summary, we can calculate where element i was shuffled to, and where element i came from (using the reverse operation) in constant time, and the calculation time does not depend on the length of the list. Fisher-Yates shuffle does not have this property, and cannot shuffle a single index, they often need to repeatedly shuffle the entire list. What is written in the Eth2 specification is how the algorithm is applied to shuffle a single index. In fact, shuffling the entire list at once is just an optimization of it! If we want, we can shuffle only one element of the list in turn: (reverse) run the shuffle to find which element ends up at index 0, run another shuffle to find which element ends up at index 1, and so on. The reason we don't do that is simply because the decision to swap-or-not requires generating a 256-bit hash at once, and throwing away 255 bits like that would be wasteful. If we use a 1-bit hash or oracle, shuffling one element of the list is about as efficient as shuffling the entire list. Be a real "light" client The reason why this feature makes sense is all because of the light client. Light clients are equivalent to observers of the Eth2 beacon chain and shard chain. They do not store the entire state, but hope to safely access the data on the chain. A necessary step in verifying that their data is correct, i.e. that no fraud has occurred, is the calculation of the committee attesting to the data. That is to use the shuffle algorithm, and we don't want the light client to have to store or shuffle the entire validator list. By swap-or-not shuffling, they can only perform calculations on a small subset of committee members they need, which will greatly improve overall efficiency. History If you like the archaeological nature of GitHub as much as I do, you can check out the discussion of the original shuffle algorithm search-zce for Eth2 here, and the final winner is announced here. If you want to look at the swap-or-not shuffling algorithm from another angle, you can take a look at a more visual explanation published by Protolambda. The last picture is when I listened to Justin Drake talk about swap-or-not shuffling on EthCC in 2019, while implementing the first version of swap-or-not shuffling in the Teku client (it was still called Artemis at the time)..{ }

Tags：

Huobi App