Something broke at midnight, and the only witness was a log file of four gigabytes sitting in a bucket. I wanted the last fifty lines. I ended up moving all four gigabytes to get them, twice.

Today

The console showed me the file's name, its size and a download button. That is all a console can do with a file. So I downloaded it, which took long enough to make coffee. Then I opened it in my editor, and the editor tried to read the whole thing into memory and stopped answering.

The second attempt was smarter, or felt smarter. I piped the cloud's copy command into a pager and paged to the end. The pager was fine. But to reach the end it had to read everything before it, so the four gigabytes crossed the network again, this time while I watched.

1. Cloud consolefind the log
2. Browserdownload 4 GB
3. Editoropen it, hang
4. Terminalstream it to a pager
5. Pagerwait to reach the end

4 GB moved to read 50 lines

None of those tools did anything wrong. Each did exactly what it was built to do. The problem is the shape of the walk: every step assumed the whole file had to be here before any of it could be looked at.

Ideal

So I tried to start from what I actually wanted. Not the file. Fifty lines. A screen holds about fifty lines of log, and I can only read one screen at a time.

Then the question is whether I can get fifty lines without getting the rest. I think the answer has been yes for a long time. S3, Cloud Storage and Azure Blob all accept an HTTP Range request, which asks for bytes 3,900,000,000 to 3,900,065,535 of an object and gets exactly those. A file in a bucket is not a sealed box. It is a long row of bytes, and you are allowed to point at any part of it.

What makes it feel impossible is line numbers. Line nine million is not at a byte offset you can compute, because lines have different lengths. But you can find it cheaply if you read the file once, forward, a few megabytes at a time, and write down where every hundredth line starts. That list of checkpoints is small. After it exists, any line is one seek to the nearest checkpoint and one short read.

This is how I want it to look, and for a file on my own disk it is how it looks now:

A 2 GB csv from a local folder in the text view, scrolled to its last lines. The page holds a few thousand lines; the rest stays on disk.

That file is 2 GB. The view asks for the lines on screen in blocks of two thousand and keeps eight blocks at most, so the page never holds more than a small window of the file. Scrolling to the end asks for different bytes. It does not ask for more of them.

Where it stops

I have to be honest about the bucket, because it is the case the story starts with. The view asks for ranges, and every cloud's API would answer them. But between the two sits a local cache, on by default, and today it fetches the whole object into the cache before it answers the first range. So a 4 GB log in a bucket still crosses the network once, in full. It crosses only once, and the editor never has to hold it, but it is not yet the fifty lines I asked for. Passing a range straight through when the object is not cached yet is the next step, and I haven't built it.

The other cost is the one pass. To know line numbers, the file has to be read once from the front. The view draws from the first chunk while the pass runs, and hex needs no pass at all, because a byte offset is already an address. But if you jump straight to line nine million, you wait for the checkpoints to reach it.

Line nine million, as fast as line one.