kaustubh yerkade

Posted on Aug 1 • Updated on Oct 17

Exploring Hypervisors: How They Power Virtual Machines and Modern Cloud Infra

#hypervisor #linux #virtualmachine #cloud

Part 1. What is a Hypervisor ?

PS - This is still in progress............

Hypervisors are the backbone of modern virtualization, Its a software/firmware that allows multiple operating systems to run on a single physical hardware simultaneously. It does this by creating and managing virtual machines (VMs), which are isolated environments that mimic the hardware of a real computer.

Our dear friend Viru along with Jay is on a journey to understand the hypervisor. Let's join him & see what he discovers....

Feature	Type 1 Hypervisor (Bare-Metal)	Type 2 Hypervisor (Hosted)
where	Installed directly on the physical hardware.	Installed on top of a host operating system.
Perf	Generally higher performance due to direct hardware access.	Slightly lower performance due to the additional layer of the host OS.
Use case	Preferred in enterprise and data center environments.	Commonly used for development, and testing environments.
Security	Generally more secure due to reduced attack surface.	Security can be more vulnerable due to the host OS layer.
Examples	VMware ESXi, Microsoft Hyper-V, Xen, KVM.	VMware Workstation, Oracle VM VirtualBox.

So, as far we know,

Virtualization is the process of creating a software-based or virtual version of something, whether that be compute, storage, networking, servers, or applications. Its simulated computing environment, or virtual environment, instead of a physical one.

And what makes virtualization possible? Its the hypervisor

And what is a hypervisor ? is it's a software or a firmaware that runs on a bare metal or a host operating system which creates & runs VMs. A hypervisor is simpler than an operating system because it does only one thing i.e Emulate multiple copies of the bare metal (Intel x86 architecture). Noncritical instructions run on the hardware directly while critical instructions are discovered and replaced with traps into the VMM virtual machine manager to be emulated by software.

We cannot really allow the guest operating system to disable interrupts for the entire machine or modify the page-table mappings. The trick is to make the operating system on top of the hypervisor think that it has disabled interrupts, or changed the machine’s page mappings.

Hypervisors are the backbone of virtualization, It enables multiple instances of operating systems to run on a single physical machine efficiently. The OS on VM thinks its running on real hardware , but its not , It is the task of the hypervisor to provide this illusion and to do it efficiently ! Thanks to engineers at IBM , Intel & AMD 🙏....

When you create a VM, the hypervisor allocates a portion of the hardware resources (CPU, memory, storage) to the VM. This process includes setting up following virtual components

virtual CPUs vCPU
virtual memory
virtual storage
virtual network interfaces.

VMs allow you to more easily scale your apps by adding more physical or virtual servers to distribute the workload across multiple VMs. As a result you can increase the availability and performance of your apps.

and By abstracting and managing the underlying hardware resources, It provide the foundation for cloud computing, server consolidation, and other advanced computing paradigms.

Hypervisors in cloud

GCP -GCP uses KVM based hypervisor. Nested virtualization is used where the physical host and its hypervisor are the level 0 (L0) environment. The L0 environment can host multiple level 1 (L1) VMs. On each L1 VM is another hypervisor, which is used to install the level 2 (L2) VMs -

AWS -AWS uses a customised version of hypervisor blended with multiple acrhitectures of XEN , KVM & ESXi. EC2 instaces uses Nitro, a lightweight hypervisor.

Azure -Azure uses hyper-v as their main hypervisor It was devloped my microsoft with several security, performance, and networking features.

Check out XCP-ng - https://xcp-ng.org/
An open source type1 hypervisor for bare metal, It can be integrated with Xen Orchestra, for advanced management & web console.

How a Virtual Machine Boots Up ?
Once the orchestration of system resources(user defined) is done by the hypervisor,It is ready for the boot process.
It also ensures to start up VM correctly & run isolated instance on the host system.

RAM allocation
The host allocates the amount of RAM required by guest VM, the same amount is deducted from the host. But in reality its just upper limit & thin provisioning without page faults & host crash.

CPU allocation
How vCPUs Are Mapped to Physical CPUs

Direct Mapping: In some cases, a vCPU might be mapped directly to a single physical CPU core. This is often done in high-performance environments where the goal is to provide as much dedicated processing power to the VM as possible.
Time-Sharing: More commonly, a vCPU represents a timeslice on one or more physical CPUs. The hypervisor schedules time on the physical CPUs for the vCPUs of different VMs, effectively sharing the physical CPU resources among multiple VMs.
Context Switching: The hypervisor performs context switching between different vCPUs by saving the state of the currently running vCPU and loading the state of the next vCPU to be executed.
Overcommitment: The hypervisor can assign more vCPUs to VMs than there are physical CPU cores. This is called overcommitment and is possible because not all vCPUs will need to use their assigned CPU time simultaneously.

Hypervisors provide administrators with tools to define resource allocation policies, such as setting CPU shares, limits, and reservations for each VM. These policies influence how CPU resources are distributed, especially during contention.

1. Initialization of the Hypervisor
Hypervisor Startup: The hypervisor must be running on the host system. For Type 1 hypervisors (bare-metal), the hypervisor is the first layer of software that runs directly on the hardware. For Type 2 hypervisors (hosted), it runs on top of the host operating system.

2. Creation of the Virtual Machine
VM Configuration: The hypervisor reads the configuration for the VM, which includes specifications for virtual CPUs, memory, storage, network interfaces, and other virtual hardware. And allocates the resources.

3. Boot Process
Virtual BIOS/UEFI: The hypervisor initializes a virtual BIOS or UEFI for the VM. This acts like the BIOS/UEFI in a physical machine, performing initial hardware checks and loading the bootloader.

Bootloader Execution: The virtual BIOS/UEFI loads the bootloader from the VM’s virtual disk. The bootloader is responsible for loading the operating system kernel.

4. Operating System Kernel
Kernel Loading: The bootloader loads the OS kernel into memory. The kernel is the core part of the operating system that manages the system resources and hardware interaction.

Kernel Initialization: The OS kernel initializes its components, such as memory management, process management, and device drivers. At this stage, the VM’s virtual hardware interfaces with the hypervisor, which translates these interactions to the physical hardware.

5. Initialization of Operating System
System Services and Daemons: The operating system starts system services and background processes (daemons). This includes networking services, system logging, and other essential functions.

User Environment: Finally, the OS loads the user environment, such as a graphical user interface (GUI) or command-line interface (CLI), making the VM ready for user interaction.

6. Guest OS Running
Normal Operation: The VM is now fully booted and runs just like a physical machine. The hypervisor continues to manage the VM’s access to physical resources, ensuring isolation and efficient resource utilization.

As of now x86 CPUs are manufactured by 2 key companies- Intel & AMD.
& Their on-chip features like the following, helps make better use of and improve the performance of virtualization resources.

Both Intel and AMD have integrated virtualization support into their modern CPUs.
Intel introduced VT-x technology, previously codenamed Vanderpool , on November 13, 2005, within the Pentium 4 series. The CPU flag indicating VT-x capability is "VMX," which stands for Virtual Machine Extension.

AMD developed its initial generation of virtualization extensions under the codename Pacifica These extensions were initially released as AMD Secure Virtual Machine(SVM) and later rebranded under the trademark AMD Virtualization, abbreviated as AMD-V.

The CPU flag for AMD-V is svm

SVM enables following instruction set -

1. VMRUN- Transitions the CPU from the host mode to the guest mode
2. VMLOAD- Loads the VMCB from memory into the processor
3. VMSAVE- Saves the current VMCB state from the processor to memory.
4. CLGI- Prevent interrupts from being delivered to the processor.
5. STGI- Sets the Global Interrupt Flag, allowing interrupts to be delivered to the processor.
6. VMMCALL- Used by guest operating systems to invoke services or functions provided by the hypervisor.
7. INVLPGA- Invalidates TLB(Translation lookaside buffer) entries associated with a specific guest physical address and an ASID (Address Space Identifier).
8. SKINIT- Initializes the secure execution environment by setting up necessary states and configurations for secure execution.

Check out repo for detailed explanation & code demos- https://github.com/kaustubhyerkade/hypervisor-1

The CPU flag for Intel VT-x is vmx

Activating VT-x from BIOS enables following instruction set -

1.VMPTRLD- Loads the Virtual Machine Control Structure (VMCS) from memory into the processor.
2.VMPTRST- Saves the current VMCS pointer to a specified memory location.
3.VMCLEAR- Clears a VMCS structure from memory.
4.VMREAD- Retrieves data from the VMCS and stores it in a register.
5.VMWRITE- Writes a value to a field in the VMCS.
6.VMCALL- to make a call to the hypervisor
7.VMLAUNCH- Starts a VM for the first time using the VMCS.
8.VMRESUME- Continues execution of a VM that was previously launched but paused or interrupted.
9.VMXOFF- Ends VMX mode and returns the processor to normal operation.
10.VMXON- Activates VMX operations and initializes VMX mode.
11.INVEPT- invalidates specific entries in the EPT(Extended Page tables) to ensure that stale or outdated address translations are not used.
12.INVVPID- invalidates entries in the VPID(Virtual Processor Identifier) cache.
13.VMFUNC- VMFUNC is used to invoke functions that are typically managed by the hypervisor.

VMCS LifeCycle :

set up ––––> VMXON ––––––> VMLAUNCH/–––––––> VM has –––––– ––––––––> VMOFF
rule set                   VMRESUME         control       |
                          ^                               |
                          |                               |
                          |                         Tries to exec.
                          |                         root operation
                          |                               |
                      VMM exec.  <–––– VM–exit <––––––––––
                     instructions

Check out repo for detailed explanation & code demos- https://github.com/kaustubhyerkade/hypervisor-1

The main difference between Intel's VT-x and AMD's AMD-V is how complete their virtualization features are. AMD-V is more comprehensive and doesn't have certain limitations. Where as VT-x needs the virtual machine's non-root code to run with paging enabled , which means it can't virtualize older real-mode code and some protected-mode software like firmware and OS loaders. This makes VT-x more complicated to use. AMD-V doesn't have this requirement, making it simpler to implement.

Hypervisors were first conceptualized and implemented by IBM in the 1960s. IBM developed the CP-40 and CP-67 systems, which introduced the concept of virtualization by allowing a single mainframe computer to run multiple operating systems concurrently.

CP-67 model & floor plan-

https://bitsavers.org/pdf/ibm/360/functional_characteristics/GA27-2719-2_360-67_funcChar.pdf

These early hypervisors laid the groundwork for modern virtualization technologies and enabled more efficient use of computing resources.

Here are some examples of popular hypervisors:

VMware ESXi: A widely-used enterprise-level hypervisor that provides robust virtualization capabilities for data centers.
https://www.vmware.com/products/cloud-infrastructure/esxi-and-esx
Microsoft Hyper-V: Included with Windows Server, this hypervisor is used for running multiple virtual machines on Windows systems.
https://learn.microsoft.com/en-us/virtualization/
KVM (Kernel-based Virtual Machine): A Linux-based open-source hypervisor that's built into the Linux kernel, making it highly efficient for running virtual machines on Linux hosts.
https://linux-kvm.org
Xen: An open-source hypervisor that provides a robust and flexible platform for virtualization, used in various cloud services like AWS.
https://xenproject.org/
Citrix Hypervisor (formerly XenServer): Based on the Xen hypervisor, it's designed for server virtualization and is popular in enterprise environments.
https://docs.xenserver.com/en-us/citrix-hypervisor/
QEMU: A generic and open-source machine emulator and virtualizer that, when used with KVM, can provide a robust virtualization solution.
https://www.qemu.org/
Oracle VM VirtualBox: A free and open-source hypervisor that allows you to run multiple operating systems on a single physical machine. It's popular for desktop virtualization.
Parallels Desktop: A hypervisor for macOS that allows users to run Windows, Linux, and other operating systems on their Mac computers.

hyperconvergence -

Hypervisor offers various features-

Resource Management -
----->1.Dynamic resource Allocation
----->2.Resource Pooling
----->3.Fault Isolation
----->4.Security Isolation
Scalability
----->1.Vertical and Horizontal Scaling
----->2.Support for Multiple VMs
----->3.VM Mobility
----->4.VM Snapshots
----->5.VM Cloning
High Availability and Fault Tolerance
----->1.Automatic Failover
----->2.Fault Tolerance
Virtual Networking
Storage Management
----->1.Virtual Disks
----->2.Storage Integration
----->3.Multi-OS Support
----->4.Centralized Management
----->5.Resource Overcommitment
Performance Monitoring

1.Resource Management:

1. Dynamic Resource Allocation: Efficiently distributes CPU, memory, storage, and network resources among virtual machines (VMs) based on demand.
2. Resource Pooling: Aggregates physical resources to create a centralized resource pool for flexible and efficient allocation.
Isolation:

3. Fault Isolation: Ensures that failures or issues in one VM do not affect others.

4. Security Isolation: Provides strong security boundaries between VMs to protect data and applications.

2.Scalability:

2.1 Vertical and Horizontal Scaling: Allows VMs to scale up (more resources) or scale out (more VMs) to meet workload demands.

2.2 Support for Multiple VMs: Capable of running multiple VMs on a single physical host.

2.3 VM Mobility: Enables VMs to be moved between physical hosts with minimal downtime, useful for maintenance and load balancing.
Snapshots and Cloning:

2.4 VM Snapshots: Allows capturing the state of a VM at a specific point in time for backup or testing purposes.

2.5 VM Cloning: Enables creating exact copies of VMs quickly for deployment or testing.

3.High Availability and Fault Tolerance:

Automatic Failover: Provides mechanisms to automatically restart VMs on another host in case of hardware failure.
Fault Tolerance: Ensures continuous operation of critical VMs by replicating them on a standby host.

4.Virtual Networking:

Virtual networking provides VLANs and other network segmentation techniques to isolate VM traffic. Itabstracts and manages network resources through software, enabling flexible, scalable, and efficient network operations within VMs. And encompasses components such as virtual NICs, switches, routers, VLANs, and VPNs, and leverages technologies like SDN to centralize and automate network management across VMs.

Virtual networking offers numerous benefits, including scalability, flexibility, cost efficiency, simplified management, and enhanced security, making it a critical technology in modern data centers, cloud environments, and enterprise networks.

Bridging: VMs are directly bridged to the physical network, which means each VM gets its own IP from the same network as the host. Bridged networking provides direct access between the VM and the external network.

NAT: In NAT mode, the hypervisor acts as a router, allowing VMs to share the host’s IP address. Traffic from the VMs appears as coming from the hypervisor’s IP.

Software-Defined Networking (SDN): Some hypervisors integrate SDN concepts, allowing for more flexible, programmable network configurations(Network policies). SDN controllers can dynamically configure vSwitches and manage traffic flow across VMs and networks.

Overlay networks (VXLAN): are used in large-scale environments to extend Layer 2 networks over Layer 3 infrastructures, allowing VMs to communicate across different physical hosts in data centers.
VXLAN encapsulates Ethernet frames into UDP packets, which are then routed across the underlying network, allowing for network scalability and isolation.

Network Function Virtualization (NFV): Hypervisors can host virtualized network appliances (like routers, firewalls) to manage traffic flow dynamically.

DPDK (Data Plane Development Kit): Used in some hypervisors for high-performance packet processing by bypassing the kernel's networking stack.

SR-IOV(Single Root I/O Virtualization): Directly assigns physical NIC resources to VMs for better network performance and lower latency.

NIC Teaming/Bonding: Aggregating multiple NICs for redundancy and increased throughput, ensuring the hypervisor can handle larger amounts of traffic.

5.Storage Management:

5.1 Virtual Disks: Manages virtual disk images for each VM, allowing flexible storage allocation.
Storage Integration: Supports integration with various storage technologies like SAN, NAS, and local disks.
Compatibility and Support:

5.2 Multi-OS Support: Can run different operating systems concurrently on a single physical machine.
Hardware Compatibility: Works with various hardware configurations and supports advanced hardware features like VT-x and AMD-V.
Management and Automation:

5.3 Centralized Management: Provides management consoles and APIs for centralized control of the virtualization environment.
Automation Tools: Includes tools for automating VM deployment, configuration, and management tasks.
Performance Optimization:

5.4 Resource Overcommitment: Allows more VMs to be run than the actual physical resources available by efficiently managing idle resources.

6.Performance Monitoring: Provides tools to monitor and optimize the performance of VMs and the hypervisor itself.

Containers Vs Virtual Machines-

Containers are best suited for lightweight, fast, and scalable deployment of applications, especially in microservices architecture where applications are broken down into smaller, independent services.

Virtual Machines are ideal for scenarios requiring strong isolation , the ability to run different operating systems, and supporting legacy applications that may not be container-friendly.

Choosing between containers and VMs depends on the specific needs of the application, the existing infrastructure, and the desired level of isolation and resource efficiency.

PART 2. Creating a hypervisor

Types of virtualization -
1. Full Virtualization
A hypervisor completely emulates the underlying hardware, allowing an unmodified guest operating system to run as if it were on a physical machine. E.g- VMware Workstation, Microsoft Hyper-V.
Use Case:- Running multiple, different operating systems on a single physical machine.

2. Paravirtualization
The guest OS is aware of the virtual environment and interacts with the hypervisor directly, requiring modifications to the OS to achieve better performance. E.g- Xen with paravirtualized Linux kernels.
Use Case:- Environments where performance is critical and the OS can be modified to support the hypervisor.

3. Hardware-Assisted Virtualization
Uses CPU extensions (like Intel VT-x or AMD-V) to enhance virtualization performance, allowing full virtualization with minimal overhead.
E.g- VMware ESXi, KVM.
Use Case: Modern server environments where maximizing performance and security is key.

4. OS-Level Virtualization (Containerization)
The host OS kernel allows multiple isolated user-space instances, called containers, which share the same OS kernel but operate independently.
Examples: Docker, LXC (Linux Containers).
Use Case: scalable applications, microservices, and rapid deployment environments.

How to code a hypervisor-

1.Check virtualization status -
Windows - open systeminfo.exe -

2.Linux - type hostnamectl

So Viru, we will try to create a hypervisor from scratch.
Since my laptop is on amd ryzen , I will follow AMD-v guidelines.

Also , Ravi told us about Pacifica. SVM = AMD-v

Architecture - AMD-v
VMCB: The Virtual Machine Control Block (VMCB) is crucial in managing the state of virtual machines (VMs). It holds the state of the guest CPU, control fields, and the interrupt handling setup.