In data engineering, "slow" and "expensive" almost always stem from the same cause: reading unnecessarily large amounts of data. Athena, Redshift Spectrum, EMR, and Glue all spend time and money proportional to S3 data read. Thus optimization's essence is simple: minimize data read. Today we cover four weapons — file format, compression, partitioning, file size — among the most frequently tested in DEA-C01.
File formats fall into two classes.