Rachel Kroll
Okay, wow, my post about an empty file has generated a lot of feedback, mostly for the throwaway line at the bottom about an empty file being the smallest version of /bin/true possible.
First up, Rob Pike wrote in with a link to a tweet of his. Thanks, Rob! That's exactly the kind of feedback which blows my mind: when someone who Definitely Knows shows up and provides the authoritative answer.
A bunch of other folks wrote in to either mention the tweet, link to the tweet, or pin it on the lawyers (and, in one case, GNU). It all works out to be right, so everybody wins.
Garrett wrote in with a link to a great story about another zero-byte file from the world of CP/M called The Infinitely Profitable Program. (Look for the part about "go.com" if you are impatient.)
I also heard from the author of this piece about POSIX true(1) who pointed out some of the references, including one by John Chambers called The /bin/true Command and Copyright. So, it seems people have put some actual work into this figuring out just how it all came to be.
...
One person mentioned that there have been disagreements in the Linux realm about filesystem operations, atomicity, and dropping zero byte files all over the place. I definitely saw some of this in the job in question. I've had this story on the back burner for a while, and now seems like the perfect opportunity to share it, so here we go.
Besides zero-byte files, another problem we saw was that machines would *mix up file contents* when they got rebooted during write operations. This would usually manifest as RPM crap being written to files that had no business containing RPM information.
I found out about it from a different angle. One of the teams I supported owned this "proxy" (really a cache, but the name stuck) that ran on all of the boxes. You'd ask it to get config items for you, and it would connect to the config backend store and would set up "watches" on them. Then, whenever it changed, it would feed you the update.
At some point, someone decided to make the "proxy" remember what had been explicitly requested on the box, and so they had it dump out a list of the items requested, one per line, in a *plain text file*. This worked okay, but then it crossed paths with the filesystem corruption monster. Their file picked up all kinds of binary crap that had no connection to anything in the config store.
The next time the machine came up, it started their "proxy", and it tried to do the "warm up" mode. It read through that file, and every time it saw a newline in that binary crap, it treated it as a path. Then it created a thread to go and load it from the backend. Now, since this was absolute garbage, there's no way that fetch would ever succeed. They assumed that only valid paths would show up in here, so they had these things retry until it did succeed. And, of course, there's no way a bunch of gibberish will succeed, so the threads didn't go away.
Now for the final rub: every one of these threads made the process that much bigger. This is the VSZ you see in tools like ps - not physical memory, but rather the overall *virtual* size of it. It can be many times bigger than the actual physical allocation (RSS) of a process.
In this case, for some reason, some well-intentioned individual had tried setting a limit on how big the program could get. They used ulimit (aka setrlimit) to do this. The problem is that RSS is *NOT* one of the things you can control with u/rlimits. There's a setting which looks a lot like it would work for such things, but it limits *VSZ* only.
So, when all of this finally got smooshed together, you had a "proxy" that would come up, try to start tons of threads that never went away, would balloon its VSZ, and then the limit would kick in and it would start failing memory allocations. That is, C++ would throw a "bad alloc" exception for things like "new", and the program would die because nobody ever expects *that* to fail.
I never found out just how much cross-pollination was happening between files on that company's machines. For all we know, it may have resulted in cat picture #1's contents being written to cat picture #2's file, or vice-versa. That would be a pretty epic privacy failure, wouldn't it?
As best I can recall, it was something crazy involving the kernel's attempts to flush its buffers to the disk when shutting down. Somehow, buffer A wouldn't necessarily flush to file A, and so on down the line. There are undoubtedly people who remember this and know exactly what happened, but I am not one of them.
Fixing all of these warts involved making the kernel not mix up its buffers, and actually removing the well-intentioned but ultimately useless "ulimit -v" that someone had placed in the startup script. As for the plain text file, well, I tried pitching them on making it into something strongly (and not "stringly") typed with framing or some other way to detect corruption, but the devs didn't go for it. Sigh.