Lately there have been articles circulating claiming to refute EOS scalability claims

https://cdn0.tnwcdn.com/wp-content/blogs.dir/1/files/2018/11/EOS_Report.pdf

https://blog.bitmex.com/eos-report/

The latter claims effective 25 tx/s.

However we have seen 3000 tx/s mark in EOS Network monitor:

https://coincodex.com/article/2052/eos-achieves-3000-transactions-per-second/

Why these research reports cannot achieve the same throughput as we withness in the mainnet? Are there false claims or some difference in the way of measuring throughput?

  • The report from bitmex seems removed. Could you double check the link. Or maybe they found the error in the report and just removed it? – twksos Dec 2 at 4:34
  • Here is a cached version webcache.googleusercontent.com/… – Mikko Ohtamaa Dec 3 at 9:49
  • it appears to have been deleted – confused00 Dec 5 at 14:06
up vote 3 down vote accepted
+50

There is only one report as far as I can tell, not multiple reports. The Bitmex Research article was removed both from their blog and from Google cache. The version cached on the Internet Archive shows just a mirror for the WhiteBlock report, so I'm not sure where to find more about the benchmark produced by Bitmex Research if it exists.

The reason why WhiteBlock couldn't achieve the same throughput as other EOSIO networks is largely unknown, as they haven't provided enough details about their environment and research methodology. Further, their paper omits any discussion regarding the differences of results between previous benchmarks and theirs.

If you want to read some thoughts from the community, the Telegram channel of EOS Mechanics has some discussion about this. I'll paste some quotes below without mentioning the authors, but their Telegram channel is open for anyone to join:

  • it's very unclear what was happening during their tests, they saw forks, pending tx completely go missing or dropped, didn't know how to check pending tx, couldn't find how resource limits or validation work, or couldn't reach consensus with just 1 malicious bp - I was very confused so trying to figure out out. to me it sounded like an issue with configuration or maybe hardware/network caps. (some of this might be remembered wrong) they clearly needed or need help.

  • They refused peer review. That's all that needs to be said.

  • Meanwhile over on Jungle testnet they are successfully putting 4075 txs in a single block https://jungle.eosq.app/block/000fa5663cee34e05dd4b83d3f22b244f8c8ac9759daae87cef5dd5848639684 Including token transfers, so these aren't "empty". And these are BP nodes all over the world connected via public networks. It says something when WB can't reproduce similar results in a lab.

  • I agree the documentation and tools for optimizing and testing performance could be a lot better. We intended to write more about that but it got deprioritized a bit. There's been various helpful articles and discussions in this chat, but it could be surfaced better for sure. For example Attic Lab's notes here https://github.com/atticlab/eos-bp-performance

  • I think some of those questions are very typical and it helps to have responses if not only to educate those already using eos if they don't know the answer. they asked me for peer review, all I know. I'm ignoring consensys aspect of it and only talking to wb team who wants to specialize in benchmarking so far what I have: https://docs.google.com/document/d/e/2PACX-1vQcYW67LkkDqycGZGSK--L3WaohCVRPhC7PpjF25s1Te-SiIQA39ZXKo-iEMnmaUt45KqKw2ulr7RxT/pub

Also, here is an example of post detailing a benchmarking process from the community achieving about 9,000 TPS on an EOSIO testnet: https://medium.com/@cryptolions/new-maximum-eosio-tps-demonstrated-in-jungle-testnet-9179-15a485f2e79

  • 1
    Hopefully media will understand that as well. Small, controversial reports and unclear citing is very dangerous for a community like ours. Thanks for working this out. – TeeAttack42 Dec 5 at 13:43
  • Excellent answer! Keep up the good work. – Mikko Ohtamaa Dec 5 at 17:10
  • Thanks @confused00 for such a good insights. There were many false claims in the report indicates that Test performed by Ethereum developers which lacks knowledge about bitcoin i.e bitcoin also don't include state in merkle tree and also EOS but ethereum does, through that WB claims EOS is a distributed database for not including state in merkle tree. – Farhan-TroonTechnologies Dec 6 at 6:34

The main factor is depend upon answers to these questions.

How much throughput of one bp was on the testnet of bitmex?

Because if one bp throughput is low that cause decrement in overall network CPU and users will have less available CPU.

For more information :- https://support.dexeos.io/hc/en-us/articles/360009054093-EOSIO-Why-CPU-bandwidth-varies-

How many bp were there for load balancing?

If thousand txs validated by one bp then bp will suffer all stress by validating all transaction. if there are as many as bp then stress can be distrubuted among these block producer by parallel transaction execution.

Were there any light node to validate only block header to reduce the stress for bps?

In "light" mode all incoming blocks headers will be fully validated; transactions in those validated blocks will be trusted.

From my point of view, if EOS mainnet can acheive some throughput that doesn't mean all EOS sister chain will produce the same throughput.

  • 1
    Very good answer as well! I marked bounty for confused00 as he provided additional insight about the structure of studies – Mikko Ohtamaa Dec 5 at 17:12
  • 1
    Thanks, I like his insights and planning to test theory to checkout myself. – Farhan-TroonTechnologies Dec 6 at 6:40

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.