Before VMS there was the family of RSX-11 operating systems which also had ASTs (now called APCs in NT parlance), IRPs, etc. Dave Cutler led the RSX-11M variant which significantly influenced VMS. The various concepts and design styles of the DEC family of operating systems that culminated in NT goes back to the 1960s.
It's sad that the article didn't mention VMS or MICA since NT didn't magically appear out of the void two years after Microsoft hired the NT team. MICA was being designed for years at DEC West as part of the PRISM project.
In many ways NT was a new, ground up implementation of “VMS NT”.
It started elegant, but all the backwards compatibility, technical debt, bad ideas, and dozens of versions later, with an endless list of perpetual features driven by whoever had a bigger wand at Microsoft at the time of their inception, takes a toll. Windows now is much more complicated than it could be.
It shocks me some apps get Windows NT4 style buttons even on Windows 11.
>In many ways NT was a new, ground up implementation of “VMS NT”.
Most definitely. There was a lot of design cleanup from VMS (e.g. fork processes -> DPCs, removing global PTEs and balance slots, etc), optimizations (converging VMS's parallel array structure of the PFN database into one), and simplification (NT's Io subsystem with the "bring your own thread" model, removing P1 space, and much more). SMP was also designed into NT from the beginning. You can start seeing the start of these ideas in the MICA design documents but their implementation in C instead of Pillar (variant of Pascal designed for Mica) in NT was definitely the right thing at the time.
>It shocks me some apps get Windows NT4 style buttons even on Windows 11.
This is good, though. The alternative is that the app won't run at all, right? Windows NT is good because of that background compatibility, both for business apps and games.
Under Windows it's very rare to have trouble to running software. When you have trouble it's usually due to some security considerations or because you're using something which has roots in other operating systems.
MacOS & Linux are nothing like this. You can run most software, as most of the basis for modern software on those stacks is available in source form and can be maintained. Software which isn't breaks.
Apple/Google with their mobile OSes take that a step further, most older software is broken on those platforms.
The way they've kept compatibility within Windows is something I really love about the platform.. but it I keep wondering if there's a way to get the best of both worlds. Can you keep the compatibility layer as an adhoc thing, running under emulation, so that the core OS can be rationalised?
In fairness, closed source software is a very very tiny minority of the software available on Linux, which is why ABI backwards-compatibility hasn't been much of a concern. In that respect, it's essentially the polar opposite of Windows and even MacOS.
However, it'd be very nice if it did become more of a focus (especially in the glibc/audio/glx areas), especially now that gaming has become very popular on Linux.
Trying to get old, closed-source games like Unreal Tournament to work on Linux can be a real chore.
I'm not so sure, I like the Linux model of 99.999% of the code you'll run being available in source form. The result is that we have that code running everywhere.
The GDI libraries/APIs that provide that are all still there, you just need to find a framework that lets you see them, are kick through the abstraction walls of [insert chosen app framework] to access them more manually. Be prepared for a bit of extra work on what more modern UI libraries make more automatic, and having to discuss everything rather than just what you want to differ from default.
I think you can get back to Win9x/2k style controls by instructing the system to not add any theming. If you're finding a panel that is using 3.x controls, they're likely in the resources of the app/dll. Although the 3.x file picker can still be found in a couple of rare corners of the OS.
Vaxes also had hardware support for ASTs in VMS (unlike NT) - they were essentially software interrupts that only triggered when the CPU was in a process context and no enabled interrupts were pending - so you could set a bit in a mask in another thread's context that would get loaded automatically on context switch and triggered once the thread was runnable in user mode .... device drivers could trigger a similar mechanism in kernel mode (and the 2 intermediate hardware modes/rings). There were also atomic queue instructions that would dispatch waiting ASTs
Months ago I found this presentation on youtube, "Re-architecting SWIS for X86-64"[0], about how VMS was ported from VAX to Alpha to Itanium to x86 that did not have the same AST behaviour.
Didnt it end brilliantly for MS? Settlement involved MS supporting Alpha while DEC trained its enormous sales/engineering arm to sell and support NT thus killing any incentives to buy DEC hw in the first place. DEC moved upstream the value chain and Microsoft moved tons of NT to all existing DEC corporate customers.
>I don’t think anyone has ever bothered to make FS code async, it’s already quite horrible as it is. Not in a mainstream kernel anyway. [...] Same for Windows disk IO with IOCPs.
IOCP merely allows IO completion to happen on worker threads in a threadpool rather than the particular thread that initiated the IO request. Many concepts in the NT kernel come from a long line of production kernels in mainstream DEC operating systems going back to RSX-11M and VMS. That entire lineage of operating systems (culminating in NT) all have true asynchronous IO using IO Request Packets (IRPs) within the kernel to initiate/queue IO requests and immediately return. There are individual cases within NTFS where blocking will occur but those are special cases rather than the rule as the kernel IO system in general is entirely async.
"Embedded software" and "embedded system" also have similar issues with being nebulously defined in various situations. Some things will never perfect and at least firmware is kind of cute.
Usually this is still host firmware and not a secondary controller, at least on x86 platforms. To detect/use the USB controller you still need to configure the chipset/root complex and do an initial PCIe bus scan to set up PCIe BAR apertures. After that occurs you need a (primitive) USB stack that is able to talk to the USB controller to enumerate the USB devices as well as block storage and filesystem layers. All of this code is implemented as a collection of DXE drivers that implement various UEFI protocols which are initialized in the UEFI DXE phase that runs after SEC and PEI phases. On Intel platforms PEI does things like training the DRAM and PCIe links so memory is always available to DXE. Unfortunately, there's still a lot of code that needs to run to get this to work.
After FW update binaries are located it's not uncommon to write them to a scratch flash and then reset the system. On reset somewhere in the flow the scratch flash is checked for an update and then the hardware sequenced flashing registers in the chipset are utilized to actually flash the firmware. Another reset is performed to boot from the freshly flashed firmware.
There are variations on this flow depending on the firmware implementation and platform/vendor which can simplify it but that is usually the basic idea. Various microcontrollers are definitely employed for other platforms (even on x86, such as the embedded controller though these perform auxiliary tasks to the host firmware rather than the whole thing).
Updating without a CPU installed usually is on the embedded controller itself but that's not a normal update flow IIRC.
Except that it can't actually do that, because x86 DRTM doesn't remove SMM handlers installed by the system firmware, ACPI tables also remain resident and could be changed to contain malicious code, etc.
DRTM does not remove any malicious firmware provided code or data. Traditionally it is merely a measurement mechanism that happens after ExitBootServices which measures platform state in an unforgable manner. Practically the DRTM event can also cause certain chipset registers to get locked or SMM supervisors to get launched depending on the platform. SMM and ACPI tables (on some x86 platforms certain tables are rebuilt) are measured into a PCR by the secure loader or security processor during the DRTM event. The idea is that if malicious code or data was present then the PCR values wouldn't match the previous boot session and TPM secrets wouldn't unseal.
While what you said is technically correct, it is by design and any compromised firmware can do as it pleases before the DRTM event at the cost of getting caught and having the device fail attestation or not be able to access encrypted data (depending on what policy is layered on top of DRTM itself as it is just a security primitive). By having PCRs get reset during the DRTM event secrets are much more reliably able to be sealed to specific PCR values.
The claim that SMM is measured by (Intel) DRTM is interesting. Do you have any details on that? To my knowledge Intel was trying to solve this issue using the concept of an 'SMM Transfer Monitor (STM)' not simply by measuring the SMM environment [1]. But it's been 8 years since [1] was written so if you have links to more current information, it'd be welcome.
Unfortunately, I don't believe there is any up to date and detailed public documentation on the modern DRTM flows that exist on both Intel and AMD platforms. Maybe documentation has been recently updated but I’m not sure if I’m able to share more beyond what I already have.
These are good examples and this might be slightly off topic but I wish that there were more resources on how modern x86 platforms actually work instead of using these legacy interfaces that are either emulated in SMM with IO port trapping or relegated to requiring chipset hacks to keep them working (such as VGA device hacks in the PCI spec or well known chipset legacy IO ports decoding to the LPC bridge).
While it is essentially impossible to truly understand every aspect of todays platforms I think that one can build a lot of intuition about how things work by studying how transaction routing is handled in the platform, e.g. how the uncore and chipset routes TLPs (whether it's raw PCIe or packets essentially wrapped by other interconnect protocols like IF/HyperTransport/DMI/QPI/etc) through the system with various base address registers and decoding windows. This gives way to understanding how modern MSI interrupts and DMA works as well. While I understand that truly going into things like PCI configuration, ACPI, UEFI, chipset interfaces on bus 0, modern booting, DMA and interrupt remapping, DRAM, register locking and security, security processors, power management, various types of runtime firmware, etc is a a many-years-long exercise I think that the core ideas don't need to be spread around all over the place for one to get familiarity with the ideas and patterns that underpin our PCs.
I think that a general conceptual overview on which one can hang more detailed knowledge acquired by reading specs is needed with the amount of complexity that exists today. There are some really good resources that I've collected over the years but there is still something missing. Maybe I or someone else will get around to it some day.
I'm not sure why you have legacy in quotes, that is what it is. Modern platforms do not act like a PC AT and there is a ton of cruft to keep those interfaces working which is set up and facilitated by firmware (not so open, at least TianoCore exists regardless of your thoughts on UEFI). Modern x86 systems programming is so much more than these interfaces. I think that it would be beneficial for others to understand how things are actually working on modern platforms.
Regarding your linked post:
For corporate users legacy methods of booting /are/ a security threat. Firmware attacks are a real threat for enterprises. Having IBVs all hand roll things like the S3 boot script or communication with SMM (not using standardized com buffers) is not wise. Secure boot combined with modern security features like using DRTM to remove firmware from the trusted computing base are important and should not be discounted because it's not an 40 year old interface. The PC platform is historically extremely insecure and there is a ton of work still to be done to effectively provide the primitives needed to secure the platform against determined attackers.
If you want to have a USB stack in SMM owning your USB controller to emulate port 60h with IO SMI traps that's fine but it's not a reason to hold the platform back.
A lot of critical system configuration is stored in the SYSTEM hive which isn't explicitly backed by a file mapping and is loaded at boot via firmware services so this will be fast as it is non-paged and mapped in kernel space for the entire boot session. On newer builds of Windows 10 other hives are memory mapped into the usermode address space of the minimal Registry process. Whenever you do a registry read the kernel will temporarily attach your thread to the Registry process' address space and the read to the UM mapped section will occur which will naturally fault in the data from disk. The requesting process' thread will then be unattached and the information will be returned. Since non-SYSTEM/ELAM hives are memory mapped the kernel's cache manager and memory manager subsystems are the ones that "own" and control the mapped memory. The file cache is tuned based on the particular system's hardware characteristics to be as performant as possible. There are registry-specific caches in between to reduce the need to attach to the Registry process but this isn't going to be a disk IO speed bottleneck.
The program you're thinking of is procmon which is a part of the Sysinternals suite of tools.
It's sad that the article didn't mention VMS or MICA since NT didn't magically appear out of the void two years after Microsoft hired the NT team. MICA was being designed for years at DEC West as part of the PRISM project.