There is only one report as far as I can tell, not multiple reports. The Bitmex Research article was removed both from their blog and from Google cache. The version cached on the Internet Archive shows just a mirror for the WhiteBlock report, so I'm not sure where to find more about the benchmark produced by Bitmex Research if it exists.
The reason why WhiteBlock couldn't achieve the same throughout as other EOSIO networks is largely unknown, as they haven't provided enough details about their environment and research methodology. Further, their paper omits any discussion regarding the differences of results between previous benchmarks and theirs.
If you want to read some thoughts from the community, the Telegram channel of EOS Mechanics has some discussion about this. I'll paste some quotes below without mentioning the authors, but their Telegram channel is open for anyone to join:
-
it's very unclear what was happening during their tests, they saw forks, pending tx completely go missing or dropped, didn't know how to check pending tx, couldn't find how resource limits or validation work, or couldn't reach consensus with just 1 malicious bp - I was very confused so trying to figure out out. to me it sounded like an issue with configuration or maybe hardware/network caps. (some of this might be remembered wrong) they clearly needed or need help.
-
They refused peer review. That's all that needs to be said.
-
Meanwhile over on Jungle testnet they are successfully putting 4075 txs in a single block https://jungle.eosq.app/block/000fa5663cee34e05dd4b83d3f22b244f8c8ac9759daae87cef5dd5848639684 Including token transfers, so these aren't "empty". And these are BP nodes all over the world connected via public networks. It says something when WB can't reproduce similar results in a lab.
-
I agree the documentation and tools for optimizing and testing performance could be a lot better. We intended to write more about that but it got deprioritized a bit. There's been various helpful articles and discussions in this chat, but it could be surfaced better for sure. For example Attic Lab's notes here https://github.com/atticlab/eos-bp-performance
-
I think some of those questions are very typical and it helps to have responses if not only to educate those already using eos if they don't know the answer. they asked me for peer review, all I know. I'm ignoring consensys aspect of it and only talking to wb team who wants to specialize in benchmarking so far what I have: https://docs.google.com/document/d/e/2PACX-1vQcYW67LkkDqycGZGSK--L3WaohCVRPhC7PpjF25s1Te-SiIQA39ZXKo-iEMnmaUt45KqKw2ulr7RxT/pub