TL;DR
- You need to migrate all rows.
- Secondary indices are stored in RAM.
Each row of multi_index with secondary index is stored by multiple types of chainbase objects like key_value_object (for storing data by primary key) and indexTYPE_object (for storing primary key by secondary key).
This secondary index object (indexTYPE_object) only has primary key and can be found by secondary key. When you search for a row with secondary key, this secondary index object is found first, and eosio::multi_index makes you get an exact row by primary key which was stored in secondary index object.
This secondary index object is created or updated when you call eosio::multi_index::emplace/modify, so if you add secondary index to an existing multi_index table, existing rows don't have these secondary index objects, but new rows will have. This will break normal multi_index operations.