Third Law of Sane Personal Computing
Volatile storage devices (i.e. RAM) shall serve exclusively as read/write cache for non-volatile storage devices. From the perspective of all software except for the operating system, the machine must present a single address space which can be considered non-volatile. No computer system obeys this law which takes longer to fully recover its state from a disruption of its power source than an electric lamp would.
Personal computer operating systems known to obey this law:
1 [*]
Notes and observations:
When you make a deliberate change to the system, the change stays made: this is the basic reliability we expect of paper and pencil. And of almost every other tool, save for our computers!
To be continued.
[*] Corrections are welcome.
I'm not sure that PalmOS actually qualifies (indeed, having been bitten by dead batteries and losing all my data more than once, I'm closer to being sure it doesn't). Apple's now defunct Newton, however, does. It also does the "single address space" thing better than any system I've ever come across.
Shame Apple are busy making devices designed for conspicuous consumption rather than actual usage, really.
Dear Mr. Foo,
>...Apple are busy making devices designed for conspicuous consumption rather than actual usage...
I keep hearing this, and there is definitely the implication that someone is making devices for "actual usage." Pray tell, who?
And with nothing Microsoft in them, and no braindead half-written drivers (i.e. Linux on ThinkPad)? Apple has an ironclad monopoly here. I sincerely wish that it were otherwise.
Which laptop maker will sell a machine where every single hardware component is guaranteed to work perfectly under Linux? Without endless driver-hunting, Dependency Hell, crashes, and "small" but intolerable imperfections.
My Centrino-based Thinkpad (circa 2007) failed to properly suspend the video chip under a perfectly-configured Linux, draining a freshly charged battery in only twelve hours. Note that I do not care if this (documented) bug has since been fixed. Neither do I care just who was at fault (the chip vendor, for failing to release docs; the slacking developers; the devil - I simply don't care.) Such cases are rampant, the culprits are unrepentant, and there is no vendor which will offer a money-back guarantee of nothing of this kind happening.
Apple has a total monopoly on portable computers that I (and quite a few other people) have any desire to use.
Yours,
-Stanislav
Pray tell, who?
For mobile devices, nobody, as far as I can tell. That's the problem. iOS appears to be aimed fairly and squarely at "digital consumption", and the horde of followers (Android, WebOS, MinMo and the like) are aimed fairly and squarely at "being successful by being like iOS". I am certain that there's a market for mobile devices aimed at digital creation, but nobody seems to have the balls to go out on a limb and create one.
Apple had a wondrous, paradigm shattering device with the Newton. It "simply worked", unobtrusively and efficiently. It was, in the way that Windows is a reflection of a way that C++ dictates OS design (ISTR you writing something rather close to that), a reflection of one way a reflective, dynamic language dictates OS design.
NewtonOS should be where OSX is today. An OS without "file systems", without the archaic "metadata via file extensions" kludge, etc etc. Instead, we have the rotting guts of NeXTStep with layer and layer of veneer to make it usable. It might be better than anything else out there, but that doesn't make it good.
Instead, NewtonOS languishes in a "protected IP" vault in Cupertino somewhere, its later (and best) HWR tech is now part of MS (who are, of course, doing nothing particular with it and won't until someone else makes a success of HWR in the marketplace). Apple, meanwhile take some some minor concepts that were commonplace in NewtonOS, botch and butcher them onto the UNIX™ atrocity that is OSX, and we all have to make do with what we can.
I totally agree with your statements on the iPad - indeed, I'd go further. It's not just portable computers that Apple has a monopoly on. They may have a tiny market share, but they have the monopoly on hassle-free desktop computing as well. Like I said, OSX is better than anything else currently on the market. It's the least smelly turd.
Of course, this makes me sound like a Newton (if you'll excuse the language) "fanboi". I am, of course, but only in that it's one of the few systems that actively help you do things.
I'm currently working on a lisp-based OS myself (which is how and where I came across your blog); I truly believe that there is room for something new and innovative in the OS space, something that reflects your laws of sane computing and desires for LoperOS. And we're not the only ones with that mad vision, I'm certain.
I made a post on how I'm going about doing persistent memory (it's pretty low level) on my own Lisp based OS, here :
http://stm8sdiscovery.blogspot.com/2011/10/garbage-collection-for-sdmmc-based.html
Dear Mr. Foo,
Why suffer with erase blocks and other EEPROM minutiae when you can buy Flash arrays with built-in (hardware) wear-leveling? They are common as dirt. Even a $1 USB keychain has wear leveling and perfectly disk-like block addressing as far as the outside world is concerned.
Memory management and many other kinds of low-level functionality are best done purely in hardware, for reasons described here.
Use "generational" GC to avoid thrashing the slow/persistent storage.
An encyclopedic reference for modern GC algorithms is "Garbage Collection: Algorithms for Automatic Dynamic Memory Management" by R. Jones and R. D. Lins.
Or, in a simplified version, use old-fashioned semi-space GC and persist only the freshly GC'd half. Use a supercapacitor as an uninterruptible power source that never needs replacement and can be soldered straight to the board. Trigger instant migration of all volatile RAM contents to Flash upon failure of conventional power.
Yours,
-Stanislav
Hi Stanislav
You my have misunderstood why I'm dealing at an erase group level. It's nothing to do with wear levelling, which is handled quite capably by even the cheapest SD card.
It's simply that writes to SD, especially those with associated erases, are prohibitively expensive.
A class 4 SD card has a sustained write speed of 4 MB/s. Just erasing a group may stall the device for up to half a second, and that's without writing back the data that was already there.
As such, a goal has to be minimising the number of writes and erases, whilst maintaining consistency as best we can (and yes, a supercap is gonna be one of the first things added to the board, if possible).
So copying is out. A "normal" semi-space GC (generational or not) will require a dump and erase of the entire heap, every GC cycle. Remember, flash is "write once", so moving a single cell in a block means rewriting the entire block, and rewriting a block means erasing and rewriting the entire erase group. As such, my GC has to be "in place" (to remove changes to the storage structure by the garbage collector) and it has to be block-local for performance reasons. I'd like it to be concurrent, too. And, of course, I want a pony.
Anyway, thinking about this has made me spot a big(gish) flaw in my algorithm. It doesn't totally avoid changes to the storage structure, as it's storing the free cell list with the data, and needs to "zero" all free structures after updating any extra-group refcounts. In reality, a cell being freed shouldn't require a writeback, only allocation or modification should. I've modified to suit.
The Jones / Lins book is invaluable, particularly the section on distributed GC, but I haven't actually found anything that is directly relevant to what I'm trying to do. Maybe that's simply an indication that what I'm trying to do is wrong, though. About the closest I've found is Hertz, Feng & Berger's "Bookmarking Collector" - http://www.cs.umass.edu/~emery/pubs/f034-hertz.pdf - a good read.
As for hardware that implements garbage collection - hell, that would be a major step forward. But if you're going there, why not go the whole hog and expose only a transactional, persistent memory interface? I'd love to see something like that.
Simon
Dear Simon,
> A class 4 SD card has a sustained write speed of 4 MB/s. Just erasing a group may stall the device for up to half a second
Have you considered a (striped) RAID array of SD cards?
Or a heroically large (> 16GB) battery-backed RAM cache?
> why not go the whole hog and expose only a transactional, persistent memory interface?
That's my goal.
Yours,
-Stanislav
Sadly, the hardware design is out of my hands. It's possible I could hack something together to put multiple SD cards in place, I guess looks at micro-SD adaptor on desk, then at soldering iron.
Also, yay. Go for it!