We recently had a bug caused by a deltalake table’s schema being updated to add a new field without the writer being properly sync’ed up. This caused deltalake’s schema evolution to silently add a null value for that field, as designed, but the downstream consumer failed because that field was null.

Besides better communication when something like this occurs, is it possible to programmatically load and validate that the schema of the table we’ll be writing to is as expected, to prevent such an error from occurring again?