Why research reports cannot achieve the same scalability as EOS mainnet

Question

Lately there have been articles circulating claiming to refute EOS scalability claims

https://cdn0.tnwcdn.com/wp-content/blogs.dir/1/files/2018/11/EOS_Report.pdf

https://blog.bitmex.com/eos-report/

The latter claims effective 25 tx/s.

However we have seen 3000 tx/s mark in EOS Network monitor:

https://coincodex.com/article/2052/eos-achieves-3000-transactions-per-second/

Why these research reports cannot achieve the same throughput as we withness in the mainnet? Are there false claims or some difference in the way of measuring throughput?

Answer 1

There is only one report as far as I can tell, not multiple reports. The Bitmex Research article was removed both from their blog and from Google cache. The version cached on the Internet Archive shows just a mirror for the WhiteBlock report, so I'm not sure where to find more about the benchmark produced by Bitmex Research if it exists.

The reason why WhiteBlock couldn't achieve the same throughput as other EOSIO networks is largely unknown, as they haven't provided enough details about their environment and research methodology. Further, their paper omits any discussion regarding the differences of results between previous benchmarks and theirs.

If you want to read some thoughts from the community, the Telegram channel of EOS Mechanics has some discussion about this. I'll paste some quotes below without mentioning the authors, but their Telegram channel is open for anyone to join:

• it's very unclear what was happening during their tests, they saw forks, pending tx completely go missing or dropped, didn't know how to check pending tx, couldn't find how resource limits or validation work, or couldn't reach consensus with just 1 malicious bp - I was very confused so trying to figure out out. to me it sounded like an issue with configuration or maybe hardware/network caps. (some of this might be remembered wrong) they clearly needed or need help.

• They refused peer review. That's all that needs to be said.

• Meanwhile over on Jungle testnet they are successfully putting 4075 txs in a single block https://jungle.eosq.app/block/000fa5663cee34e05dd4b83d3f22b244f8c8ac9759daae87cef5dd5848639684 Including token transfers, so these aren't "empty". And these are BP nodes all over the world connected via public networks. It says something when WB can't reproduce similar results in a lab.

• I agree the documentation and tools for optimizing and testing performance could be a lot better. We intended to write more about that but it got deprioritized a bit. There's been various helpful articles and discussions in this chat, but it could be surfaced better for sure. For example Attic Lab's notes here https://github.com/atticlab/eos-bp-performance

• I think some of those questions are very typical and it helps to have responses if not only to educate those already using eos if they don't know the answer. they asked me for peer review, all I know. I'm ignoring consensys aspect of it and only talking to wb team who wants to specialize in benchmarking so far what I have: https://docs.google.com/document/d/e/2PACX-1vQcYW67LkkDqycGZGSK--L3WaohCVRPhC7PpjF25s1Te-SiIQA39ZXKo-iEMnmaUt45KqKw2ulr7RxT/pub

Also, here is an example of post detailing a benchmarking process from the community achieving about 9,000 TPS on an EOSIO testnet: https://medium.com/@cryptolions/new-maximum-eosio-tps-demonstrated-in-jungle-testnet-9179-15a485f2e79

Answer 2

The main factor is depend upon answers to these questions.

How much throughput of one bp was on the testnet of bitmex?

Because if one bp throughput is low that cause decrement in overall network CPU and users will have less available CPU.

For more information :- https://support.dexeos.io/hc/en-us/articles/360009054093-EOSIO-Why-CPU-bandwidth-varies-

How many bp were there for load balancing?

If thousand txs validated by one bp then bp will suffer all stress by validating all transaction. if there are as many as bp then stress can be distrubuted among these block producer by parallel transaction execution.

Were there any light node to validate only block header to reduce the stress for bps?

In "light" mode all incoming blocks headers will be fully validated; transactions in those validated blocks will be trusted.

From my point of view, if EOS mainnet can acheive some throughput that doesn't mean all EOS sister chain will produce the same throughput.