I am going to send out products to user when they pay me EOS. I have a service monitor the blockchain transaction. Should I send the product to the user once I see the transaction in the blockchain? Or should I only send the product if the transaction in the block becomes irreversible (with ~2.5 minute delay)?
According to the whitepaper:
Typical DPOS blockchains have 100% block producer participation. A transaction can be considered confirmed with 99.9% certainty after an average of 0.25 seconds from time of broadcast.
In addition to DPOS, EOS.IO adds asynchronous Byzantine Fault Tolerance (aBFT) for faster achievement of irreversibility. The aBFT algorithm provides 100% confirmation of irreversibility within 1 second.
However, you are right that these numbers don't apply in practice on the EOS mainnet, as the BFT commitments communication is not yet implemented. Thus, similar to other blockchains, it becomes a balance between UX and safety.
You can consider it confirmed before being irreversible, but you incur the risk of having the transaction ending up on an orphaned chain. I wish I could quantify that risk in some way, but few months of empirical data is too limited to really draw any reliable results from. For some reference, you could check what exchanges expect for deposits.
Theoretically, DPoS BFT is designed to offer guaranteed finality for irreversible blocks if 2/3+1 BPs are honest. If at least 1/3 BPs are byzantine, then there is no way to obtain 100% guaranteed confirmation of irreversibility under all network conditions.
Edit: Found this relevant quote from Dan:
the LIB algorithm was an optional addition designed to identify the number of confirmations required before an exchange has sufficient guarantees that a block will not get orphaned.
Note, however, that the quote refers to an older version of DPoS. In the current version, a LIB should provide 100% finality if 2/3+1 of the BPs are honest, not only sufficient guarantees.