A myth busted and an FAQ: ESXi is *not* based on Linux! But what is it?

I have often heard or read the statement "ESXi is based on Linux" in this or some other variation when technical people discuss about how certain things work (or are supposed to work) in ESXi. This statement is not only plain wrong, but just doesn't make any sense! In fact it is like saying that horses developed from cows, because both have a head, a tail and four legs ...

In this post I will explain why, and I also answer some other questions that are closely related to this discussion. The first one - surprisingly - is ...

What is Linux?

"Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution." (Source: Wikipedia). It is important to understand that there is not one Linux (like there is e.g. one Microsoft Windows), but a multitude of it, each assembled and distributed by different parties: There are community supported free distributions like Debian, Ubuntu and Fedora and commercially licensed and supported distributions by e.g. Redhat and Novell/SuSE.

All Linux distributions have one thing in common: The Linux kernel. The Wikipedia article quoted above calls it the defining component of Linux. Almost every operating system is built upon a kernel which is a software layer that manages all hardware of a computer - it provides e.g. CPU scheduling, memory management and device drivers on the one hand, and defined interfaces to the application processes that run on top of it on the other hand.

The first version of the Linux kernel was authored by Linus Torvalds in 1991. He is commonly referred to as the creator of Linux.

There is also a wide range of free and Open Source software that you will usually find in any Linux distribution: The X window system for graphical applications, desktop environments like Gnome and KDE, the web browser Firefox, and server services like the Apache http server, etc.

So, ESXi is just another Linux?!

The answer to this question is a clear No, because ESXi is not built upon the Linux kernel, but uses an own VMware proprietary kernel (the VMkernel) and software, and it misses most of the applications and components that are commonly found in all Linux distributions.

What do ESXi and Linux have in common!?

So ESXi is not based on Linux, but why do many people believe this? I think the main reason for this is that you can remotely open a terminal session on an ESXi host via the ssh protocol (if it has been enabled). And if you are in such a session you can use lots of commands (like cd, ls, df, grep, ps etc.) that are also available in a Linux (or other Unix-) shell.

This is possible, because a software called busybox - a lightweight shell that is able to interpret and execute these commands - runs as a userworld of the VMkernel. But busybox and all other Unix-like tool that are available in an ESXi shell are not Linux things. They are available in any Unix-like operating system and can even run on Windows (e.g. in a Cygwin runtime environment)!

If you have ever taken a closer look at the ESXi internals that the shell exposes then you might have noticed more similarities to Linux (and other Unix-like operating systems), e.g.:
  • The typical directory structure with e.g. /usr, /bin, /sbin/var, /etc, /dev, /lib and /tmp
  • A SysV-style init process that uses e.g. the configuration file /etc/inittab and scripts in a directory /etc/init.d to start services at boot time
  • Shared Libraries of the GNU C compiler in /lib
  • In ESXi 5.1 there is even the Xorg X server installed! I believe that it is used to implement the new virtualized GPU feature in vSphere 5.1 (see number 5 in my vSphere 5.1 top new features post)
ESXi even uses the same binary format for executables (ELF) than Linux does, so it is really not a big surprise anymore that you can run some Linux binaries in an ESXi shell - provided that they are statically linked or only use libraries that are also available in ESXi! (I exploited this "feature" when describing how to run HP's hpacucli tool in ESXi and when building the ProFTPD package for ESXi).

After reading all this you might wonder "How can I tell that ESXi is not based on Linux then?". It is because all these concepts and software exists independently from Linux (and the Linux kernel!) and are used in many operating systems, resp. are implemented, maintained and distributed as OS-independent, "autonomous" packages. VMware just re-used all these mature concepts and software and this was in fact way smarter than re-inventing all these wheels! That means: ESXi and Linux are not based on each other, but share a common fundament.

But you can even use Linux device drivers in ESXi!

Not directly ... you cannot use binary Linux driver modules in ESXi. Lots of Linux device drivers can be adapted to ESXi though by modifying their source code and compiling them specifically for ESXi. That means that the VMkernel of ESXi implements a sub-set of the Linux kernel's driver interfaces, but also extends and adapts them to its own hypervisor-specific needs.

In my opinion this was another very clever move of the VMware ESXi architects and developers, because it makes it relatively easy to port an already existing Linux driver of a hardware device to ESXi. So the partners that produce such devices do not need to develop ESXi drivers from scratch. And it also enables non-commercial community developers to write device drivers for devices that are not supported by ESXi out-of-the-box!

However, you should be aware that the VMkernel only implements those driver interfaces that are necessary for acting as a hypervisor: that is mainly the network and storage stacks. You will e.g. not be able to easily port a driver for a scanner device or non-standard serial hardware to ESXi. The same applies to drivers for wireless LAN adapters, because they depend on kernel subsystems, other low-level drivers or management tools that are not available in ESXi and the VMkernel.

But ESXi is a kind of Unix, right?

Another clear No here! An operating system may only be called a "UNIX" if it fully implements the Single UNIX specification of the Open Group and has passed the associated certification process. ESXi does not (and was probably never meant to) implement all the necessary functionality and APIs that make up the UNIX standard.

You might call VMware ESXi Unix-like, but even this is disputed by official sources: Mike Foley - a Senior Technical Marketing Manager at VMware - recently explained in an interesting blog post titled "It's a Unix system, I know this" that "A shell does not an OS make". His main point is that you do (or at least should) not manage an ESXi host through the shell, but via the management tools and scriptable APIs that are available for it, e.g. through PowerCLI. Or in other words: ESXi is not designed as a "Unix", but as an embedded system that is managed through defined interfaces (of which most are not really Unix-like).

Okay, but what is ESXi then?

I used many words here to describe what ESXi is not. But how do explain what it actually is? Here is my try: ESXi is
  • a bare metal (or type-1) hypervisor
  • that is purpose-built,
  • designed as an embedded system,
  • incorporates free Open Source software components, but also uses
  • a unique proprietary OS kernel - the VMkernel -
  • which implements unique features that are not available in any other operating system (e.g. the VMFS and VisorFS file systems.

Wait ... classic ESX is different - That is based on Linux, right?

Those of you who have ever touched or even installed a classic ESX installation might know that it uses a Redhat Linux installation to boot itself and to provide management functionality. This Linux is the so-called Service Console OS (or short COS) of ESX. Its Linux kernel loads the VMkernel as a module and then passes complete control over the system to it. The VMkernel hypervisor in turn will then run the COS as a privileged VM.

So classic ESX really incorporates a Linux and depends on it for getting started and managed. According to this particular meaning you can really say that classic ESX is based on Linux, but that does not mean that the VMkernel and the VMware specific OS components developed from Linux. It just uses it as a piggyback.

The classic ESX architecture has probably very much boosted the perception that "ESX(i) is based on Linux". But with the rise of the ESXi architecture and the associated elimination of the COS the classic ESX architecture is obsolete now. The VMkernel of ESXi boots directly on the hardware and does not need a Linux OS anymore.

However, the classic ESX architecture specifically caused a lot of concerns and very controversial discussions about licensing topics and legal issues, and that brings us to the last question:

Does VMware ESX(i) violate the GPL?

As explained above ESX and ESXi incorporate quite a few Open Source software that is licensed under the GNU Public License (GPL). The most important terms and conditions of the GPL are the following:
  1. The GPL grants the right (or "guarantees the freedom") to "use, study, share (copy) and modify the software" (source: Wikipedia).
  2. Binary distributions of the software must be accompanied by its source code (to enable 1.).
  3. In contrast to general believe GPL licensed software does not need to be provided for free (i.e. at no cost), but it can be sold.
  4. Software that was derived from GPL licensed software must be itself licensed under the GPL.
VMware fulfills condition (2) by providing the source code of all GPL licensed software that is used in ESX(i) (and other VMware products). The Open Source code of the products are usually made available together with their binary downloads, see e.g. the download page of vSphere 5.1.

Back in 2008 there were voices claiming that the VMkernel is a derived work of the Linux kernel (and thus violates condition 4.), because of the fact that the VMkernel is loaded and initialized as a Linux kernel module in classic ESX. A post at Venturecake.com suggested this and got a lot of media coverage and lead to very lively discussions involving prominent people like the Linux kernel hacker Alan Cox and VMware's Zachary Amsden.
Venturecake.com has vanished from the Internet a long time ago, but those of you who want to travel back in time and enjoy history lessons will find this article and the attached discussions archived at the Wayback machine. Enjoy!

The accusation that VMware violates the GPL became obsolete with the architecture change towards ESXi, and - up to now - it was never decided in front of a court, because no one ever sued VMware for this. That means there is no proof as of today that VMware ever violated the GPL.


This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.


34 comments:

  1. "The answer to this question is a clear No, because ESXi is not built upon the Linux kernel," --

    Ahhh, but leaked source code about a year ago does indicate it was built on the Linux kernel.

    ReplyDelete
    Replies
    1. Can you point us to a source for this claim?

      Delete
    2. The leaked source code indicates no such thing. Andreas's description of the relationship between Linux and ESX is very accurate.

      Delete
  2. There have been various reports of this ESX source code leakage back then. That caused security concerns, but there were no reports of Linux source code findings in there.

    ReplyDelete
  3. Well, I tend to believe that VMkernel is anyway a customized Linux kernel, probably heavily patched. But I have no proof of this, just a personal opinion =)

    ReplyDelete
  4. I had ESXi v5.0 crash on me (hardware issue) and have it display a line with 'CENTOS' in it!
    I don't believe for a minute it's not *based* on some ported form of Linux, that been heavily customized, and stripped down. Ok, maybe it shouldn't be considered Linux then...

    ReplyDelete
    Replies
    1. And this could not have been caused by a CentOS running on ESX?

      Delete
  5. Funny, as SSH'ing into one of my 5.1u1 boxes and running 'uname -a' reports:
    VMkernel **** 5.1.0 #1 SMP Release build-1312873 Sep 6 2013 21:52:18 x86_64 GNU/Linux

    Not only that, the HP ESXi ISO image dump shows: [ 0.006999] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.11.0-14-generic #21-Ubuntu
    (logged from a failed attempt at installing onto a HP Supermicro Gen8)

    ReplyDelete
    Replies
    1. You see this, because the development environment of the VMkernel and ESXi software is Linux - that means a Linux system and the GNU C compiler is used to compile the VMkernel. It also makes use of some of the Linux kernel include files to implement the Linux compatible driver interface.

      Delete
  6. Thanks for this post.. hit the nail right on the head as I also thought VMware was using a Linux based kernel

    ReplyDelete
  7. Hi Andrea,

    In this article : http://en.wikipedia.org/wiki/VMware_ESX#Linux_dependencies

    They say that it was initially based on Linux but then modified.

    Could you enlight us regarding this topic.

    Regards

    ReplyDelete
    Replies
    1. Hi Anonymous,

      No, you do not find this sentence (or a similar statement) there!

      This Wikipedia article describes the classic ESX architecture in pretty much the same way than I do in my post. It also clearly distinguishes between the Linux kernel of the service console (that is also used to boot classic ESX) and the "VMkernel" that is then loaded and takes over control.

      Andreas

      Delete
    2. Ok, I probably made a shortcut in my reading ;-)

      Rereading your article I think wikipedia's article may need some clarifications !!

      As I do not have the knowledge, I suggest you help them out a bit ;-)

      Regards

      Delete
    3. Running on ESXi 5.5 I ran "uname -a" form ssh

      This is console output :
      VMkernel x.x.x.x 5.5.0 #1 SMP Release build-2068190 Aug 22 2014 19:00:51 x86_64 GNU/Linux

      Why does GNU/Linux appear here ?

      Delete
    4. Hi Anonymous

      uname is a link to the busybox shell that ESXi uses. The term "GNU/Linux" that it outputs here is hard coded in the busybox source code. VMware should have changed the busybox source code so that it displays something else, but they didn't.
      However, this is in no way a proof that the VMkernel is based on the Linux kernel.

      Andreas

      Delete
  8. Sorry. I am still not convinced, that the ESX / ESXi / "VMkernel" is not based on Linux.
    I still somehow got the impression, that VMware took a Linux Kernel and stripped it down to the bare minimum they needed.

    And Andreas Peetz, you never clearly stated, that VMware has indeed written the Kernel from scratch. ANd your wording sounds like just dancing around a clear statement:

    Regarding Linux:
    "...not built upon the Linux kernel"
    ...because it is not using the full kernel.

    Regarding UNIX:
    "An operating system may only be called a "UNIX" if it fully implements the Single UNIX specification of the Open Group and has passed the associated certification process."
    ...about the same! It does NOT implement ALL UNIX requirements and therefore may not be called UNIX.

    So all your explanations just did the opposite for me. They just proved my impression, that VMware just took Linux and stripped it down. So not using the complete Kernel, ESX(i) is not built UPON Linux and not a CERTIFIABLE UNIX neither.

    That is the impression I've got!

    ReplyDelete
    Replies
    1. Thanks for sharing your opinion!

      Unfortunately you do not add anything new to the discussion, but just state your personal impressions.
      The fact that you are doing this anonymously doesn't make it better.

      I welcome you back if you have any reliable source or proof of your claims to share.

      Andreas

      Delete
  9. I really wanted to argue with you when I first read your post... but the more I thought about it, the more I couldn't. You are correct in stating that what defines an OS as being "linux" is the kernel. Everything else (at least for the linux OS - definitely doesn't hold true for windows or OS X) is additional programs that do not define the OS. This is how android can still be linux even though the user interface is totally different - it uses the linux kernel.

    As I thought about what defined the linux kernel, I came to one conclusion that it is the GPL opensource code within it. VMWare has not released or acknowledged using any of the linux code. Unless VMWare is doing something illegal, the VMKernel must have been re-written from the ground up. Just because it uses the same specs as the linux kernel does not make it linux. That said, it wouldn't be the first time a company used GPL linux code within a proprietary system (remember linksys/cisco?).

    All in all, good article. Thanks!

    ReplyDelete
  10. Finally we will probably know more about whether VMKernel is based on Linux source code or not!

    Software Freedom Conservacy is funding a lawsuit in this matter against VMware. See:
    https://sfconservancy.org/news/2015/mar/05/vmware-lawsuit/
    https://sfconservancy.org/linux-compliance/vmware-lawsuit-faq.html

    ReplyDelete
    Replies
    1. Hi pabouk,

      thank you for your comment!

      It is a fact (and VMware does not deny) that the VMkernel uses GPL'ed driver code from the Linux kernel.

      What is argued in front of a court now is whether this makes the VMkernel itself (and ESXi as a whole) subject to the GPL.

      Andreas

      Delete
  11. What about this?

    https://git.sfconservancy.org/?p=vmkdrivers;a=summary

    ReplyDelete
    Replies
    1. This is the Open Source Code used in the VMkernel and published by VMware. So what?

      Delete
  12. Well, I think this is more about philosophy, or even logic.
    If you call operating system a means of transportation, then you have linux (truck), windows (motorbike), OSX (bike), UNIX (passenger car)...etc...
    Then in trucks you have brands like Mercendes - Ubuntu, CentOS - Volvo, etc...
    So if someone will build a truck using same philosophy and rules to transport other trucks then what would you call it ? I say truck.
    Especially when it uses some common parts (GPL etc) as other trucks to make it work. And those parts are not used in other means of transportation like motorbikes (Windows).

    So simply put for me linux is a category and vmware is in it.

    If you build OS that is communicating with firmware/hardware in a new way that make common parts used in other linux systems unusable in it. That is totally different you can call it a new category - and not linux.

    But that's just me.

    ReplyDelete
    Replies
    1. I like your "vehicle analogy" and would say that the ESXi truck does not share the engine (= kernel) with the Linux truck, but only some of the engine's interfaces / control mechanisms. Indeed it does share some parts that almost every truck has, but it also has some very unique parts that only the ESXi truck has (like this gadget to transfer goods from one ESXi truck to another while both are driving - VMotion :-))

      While you might say that both Linux and ESXi are trucks I'd still stay that the ESXi truck is not derived from (or based on) the Linux truck.

      Delete
  13. You can directly program in c code and access most lower level functions, then compile and run the code directly in ESXi. From my point of view, if it has a duck beak and hair, it is a platypus.

    ReplyDelete
    Replies
    1. Solaris allows running native Linux binaries. So does FreeBSD. Heck, even Windows 10 allows this now!

      Are they all derived from Linux?

      Or do you - as a zoologist - still have a lot to learn about Operating Systems?

      Delete
  14. The author seems to make the case that VMware didn't base this on Linux, and thus VMware built something from the ground up.

    Due to lack of info, you're able to semantically argue your point; however, ESXi is very obviously and so heavily based on a Unix-style OS, that you'd be hard-pressed to prove that there isn't direct bit-for-bit code all over the VMkernel.

    This code originated somewhere *outside* of VMware's developers' minds, and therefore should be considered borrowed.

    ReplyDelete
    Replies
    1. Hi Tony,

      VMware does not keep it secret that it has "borrowed" (I'd say re-used) a lot of GPL'ed Open Source Code, and - to comply with the GPL terms - they have also published all this code.

      You seem to mistakenly believe that all Open Source software is somehow Linux based or belongs to Linux, only because it is *also* used in Linux. This is plain false. Almost all of the Open Source applications used in ESXi exist independently of Linux and are also used in other Unix-like derivates (like FreeBSD and even commercial Unix OSs).

      A well documented exception to this is ESXi's compatibility layer to Linux kernel device drivers, but I also explained that already in my post.

      So, honestly, I just don't get your point.

      Andreas

      Delete
    2. I wasn't at all insinuating that all open source software was based on Linux. I was simply stripping down the argument to borrowed code (open source) vs. non-borrowed code.

      I'm not saying it's illegal, or that you shouldn't trust them. I was simply stating they didn't build anything from the "ground up."

      That being said, I'm not real sure why I was making that point, and apologize if I came off as combative.

      Delete
  15. Few points here: Unless we can see the full source code of the VMKernel then we cannot say for sure if it is based on the Linux kernal or not. Dogmatically stating "it isn't" without having seen the source code and comparing it not "proof" that it isn't.

    Personally I do suspect that it is a rebadged stripped down Linux or BSD kernel for one reason: Both BSD and Linux are very stable efficient kernels with open (do one degree or another) code. Why go to the effort of reinventing the wheel when the world and his wife embeds particularly the Linux kernel into many devices. It just seems logistically easier to do so. Of course, I haven't seen the code either so I cannot state it IS, but my suspicion still lies in it being derived from the Linux kernel or the BSD kernel.

    The only issue of course is the closed nature of the VMKernel... if it IS based on Linux then the code should be released too, I am not so sure about BSD however because that is not released under the GPL, but the BSD License.

    Personally, it doesn't matter one way or another to me but it would be interesting to KNOW with proof one way or another.

    ReplyDelete
    Replies
    1. Hi Tim,

      the Linux kernel is a pretty universal multi-purpose Operating System kernel, whereas the VMkernel was developed as a very lightweight specialized Hypervisor-only kernel that has unique features like the VMM (Virtual Machine Monitor), the VMFS file system, VMotion (Live Migration) code etc.

      This is why it would have made more sense to develop the VMkernel from scratch instead of stripping down a Linux kernel.

      But maybe you are right. In the end it's about trust or distrust. I personally do not have any reason to distrust VMware. And although you and me are not able to inspect the full VMkernel source code I'm sure that other third parties have been able to do this, e.g. in the ongoing GPL lawsuit against VMware.

      - Andreas

      Delete
  16. What a marvellous, well put together article, thank you! (The comments section is interesting to read as well.) Aside from debating any finer nuances, this overall understanding of some of the core architecture and even how it differs from ESX/COS has been extremely helpful in approaching ESXi and VMWare for the first time.

    ReplyDelete
  17. Till date, it explains pretty much. Right to basics.

    ReplyDelete
  18. My mate made a kit car, it had front brakes from a Porsche 914 in it... does that mean it is actually a Porsche after all?

    ReplyDelete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!