First ps0t, or The Illusion of Progress, or Why This Page is Inflicted Upon You.
The road to wisdom/Well, it's plain/And simple to express:
Err/And err/And err again/But less/and less/and less. -- Piet Hein
As a child, I obsessively devoured books on the history of science and technology. They all lied to me. The lie, of course, was a subtle and almost certainly unintentional one. These books painted a picture of a 'wheel of progress' which could be counted on to turn strictly forward, however slowly. Except for a token mention of the Dark Ages, none of them let on just how wobbly the ratchet on that wheel really is. I formed the impression that knowledge could never be truly lost, except in cataclysmic upheavals like the ancient barbarian victories or a future thermonuclear war.
I was not fully cured of this illusion until I became interested in programming computers. Like many others, I spent many years immersed in the Microsoft universe and eventually broke free. My escape, like that of others, led me to Linux and a fascination with the 'Unix Way.' For several years, I believed that personal computing has languished in a decades-long Dark Age, and that the work of Torvalds and the GNU project were the beginning of a new enlightenment. Then I discovered that software could be capable of a lot more than simply not being Microsoft. And so my quest began.
I am not fascinated with the past for its own sake. It is of interest solely because disturbingly good ideas are buried within it. I have gathered them, mixed in some of my own, and am now ready to cook something delicious.
The Loper OS Project aims to remake computing. I am aware that others have tried and failed. Laughing? The job is not nearly as difficult as it sounds. Much of the complexity found in modern systems software is accidental, rather than inherent. A large portion of my thinking has gone into figuring out what parts of an operating system are non-axiomatic - vestiges from a computational Stone Age which could safely be swept away.
The project has now consumed a nearly year-long stretch of my free time, and I intend to see it through to victory. I have no funding, other than the income from my day job. I have no helpers. And yet I think that I can produce a working demo, in the very near future.
Right now I have very little working code. This is because I have spent much of my time so far developing an understanding of the Lisp Machine systems which inspired Loper, and of the X86-64 architecture - the only consumer-market CPU sufficiently capable for my purposes.
This blog exists mostly for my own eyes. It is intended as a diary, and happens to be public solely because there is no good reason to hide it. Progress on the project has been slow but steady. However, most of it has been confined to the inside of my head. Periodically I peek at my empty repository and suffer a spell of pessimism. The (hopefully at least daily) updates will remind me that the work is inching forward.
If Loper eventually gets off the ground, these notes will serve as a chronicle of my work. If it dies like every other similar effort has, they will sit patiently and wait for someone to pick up the torch.
Well thought ! I totally agree with you. I also would like to shift some unsuccessful paradigms bloating my computer experience.
I read a lot about such experiences like the Lisp Machines or even the SmallTalk-native machines of the Xerox Parc. Was a fun it must have been !
I don't know if it is worth reinventing the booting process, making the project more difficult I guess. Maybe it is necessary... You don't give much technical details (or I missed them): if you skip C, how do you program your bootloader: does your JIT Lisp compiler already produce the needed assembly ? Whaaa !
I wish you much luck in your effort. ... and I wish me, I'll be able to use it in a few years (-;
Keep on doing this good job.
Signed:
A-supportive-wannabee-Lisp-Newbie
Unfortunately, I am young enough to never have experienced working on a Lisp Machine. Though having read various accounts and specs I see the fall of something that could have been radically good.
This is a noble effort and I hope it comes to fruition.
~S
I recommend you target a hardware virtualization layer (e.g. VMware, hypervisor), at least to start. This would shrink the edit-test cycle, make easier the testing of alternative hardware configurations, and greatly reduce the pain involved. Much modern OS development occurs using virtualization.
I believe Lisp would prove a terrible language for a modern OS.
Lisp was not designed for hardware with large amounts of SIMD, MIMD, and CMP parallelism. Even if you don't like Clojure due to its suckling upon the JVM, you might consider stealing its CMP and concurrency features.
Lisp also is not a 'secure' language; that is, Lisp offers excessive authority to the functions and expressions executed. Absent a secure language at the base, you'll need rely upon hardware security mechanisms (rings and such). Those, in turn, force a lot of inefficiencies and inconveniences upon application developers - serialization, parsing, context switching, out-of-language signaling, poor callback support, etc.
Dear dmbarbour,
Please read D. Hillis's doctoral thesis: "The Connection Machine":
http://dspace.mit.edu/handle/1721.1/14719
And then come back and tell me about how a massively parallel architecture cannot be elegantly programmed in a Lisp.
And pray tell, what is a "secure language"? And have you actually programmed in Common Lisp? In any modern Lisp? Just what subversive action can one perform from inside a REPL enclosed in a "let" shadowing the whole global environment?
Hardware security mechanisms like the x86 "rings" are only necessary when you allow code brought in from the outside world to directly address the RAM. And why should we?
Yours,
-Stanislav
A "secure language" is one that supports security on the path-of-least-resistance. See secure interaction design; a language is a form of HCI. In particular, it means that there is little or no security risk for running untrusted code; developers need be explicit about the security risk by, for example, passing arguments to the untrusted function. E language is a relatively secure language, using object capability principles, whose only default security risk is resource consumption (non-termination or memory allocation). Caja is an attempt to secure JavaScript and thus reduce need for the same-origin policy (which greatly hinders high-performance mashups in regular JavaScript).
Lisp is not a secure language. It is a memory-safe language. Memory-safety is a precondition for a secure language, but is insufficient by itself. As one example of insecurity in Lisp: if an object in Lisp passes me a list, I can mutate that list and it will affect the internal properties of that object. This is true even if the 'object' in question is a closure of a REPL enclosed in a “let” shadowing the whole global environment. To achieve security, developers of the object would be forced to perform an explicit deep-copy of the list every time they share it. Doing so would be both inconvenient and inefficient - two obstacles keeping security from the path-of-least-resistance. Lisp similarly allows deep mutation of arguments passed to closures/functions, mutation of the global environment, communication between different blocks of untrusted code via the global environment, and mutation of the dynamic environment (special vars). Each of these introduces default security risk. Additionally, Lisp suffers the similar issues as E: the ambient authority to CPU time and allocation allow untrusted code to exhaust compute resources.
Regarding parallelism: I didn't say that you cannot write parallel programs in Lisp. But Lisp sets the bar excessively high for typical application developers. A language designed for concurrency and parallelism should simplify it... allow developers to leverage SIMD and/or CMP without too much effort, without making the code non-portable to other architectures, without a PhD. Immutable data is a big win, since developers don't need to worry about arguments changing as they read them. Separating evaluation from IO is also useful: confinement simplifies reordering code or pushing it off to a simplified processor (such as a GPGPU) that cannot perform regular IO tasks.
Exactly. We shouldn't run those millions of lines of Linux kernel created by randoms off the internet and give them direct access to our "RAM". But we do, and a single bad line in it will lead to privilege escalation or worse, remote code execution. A computer that cannot execute untrusted code is useless, as all it can do is boot and power cycle at best. It is not possible to use modern mainstream such as Windows, unixes, etc, because code requires ambient authority.
[...] As a child, I obsessively devoured books on the history of science and technology. They all lied to me. The lie, of course, was a subtle and almost certainly unintentional one. These books painted a picture of a ‘wheel of progress’ which could be counted on to turn strictly forward, however slowly. Except for a token mention of the Dark Ages, none of them let on just how wobbly the ratchet on that wheel really is. I formed the impression that knowledge could never be truly lost, except in cataclysmic upheavals like the ancient barbarian victories or a future thermonuclear war. http://www.loper-os.org/?p=4 [...]
Stanislav,
Regardless of if you ever hit your goal (I'm not holding my breath though), I do enjoy your writings and find them interesting and relevant.
I'd like to see something usable though, in some form, someday.
Dear Lucas,
Don't hold your breath. But if I live long enough, I will reach my goal.
In the meantime, you can find plenty of "usable" things at your local computer store.
Yours,
-Stanislav