Data lake tables

A Delta, Iceberg or Hudi table is a folder, and the folder itself says which of the parquet files inside it the table is currently made of. So the path is the whole SQL surface — name the folder and it answers:

SELECT region, sum(amount) FROM 'warehouse/sales' GROUP BY region

There is no delta_scan() or iceberg_scan() to remember. The engine looks at a quoted folder — a _delta_log beside the files, a metadata/ of metadata.json, a .hoodie — and reads the files that are the table now, not every file in the folder.

As of now, not a pile of files

	What it takes to be right
Delta	the adds and removes are replayed in version order; partition columns live in the log rather than in the parquet, column mapping names the physical columns, and a deletion vector marks rows inside a file that is otherwise live
Iceberg	the newest metadata.json names the current snapshot, whose manifest list and manifests name the live data files; position deletes, equality deletes and deletion vectors take their rows out, and a renamed column is read by its field id
Hudi	a merge-on-read slice stacks its base file under the log files written over it, and the last write of each record key wins

A table with none of that — the common one — comes out as a single scan over a file list, which is all it should ever have been.

A folder laid out as column=value partitions reads as one table the same way, with the partition columns taken from the folder names.

A lake table is read from a folder on this machine, on the server you are connected to, or in a bucket (s3://, gs://, az://). Click one in the tree and it opens as a notebook over its rows now, with its history as the next view. A bucket's table is copied to this machine first, file for file, and read again only where it changed; a table of more than 2 GB is refused.

The metadata is readable too

_delta_log opens as the commit history, every version with the files it added and removed underneath it. .hoodie opens as the timeline, including the requested and inflight instants a writer that died left behind. An Iceberg metadata/ opens as the chain it really is — metadata.json, snapshot, manifest list, manifest, and the data and delete files at the end of it.

Next: Cloud Logging.