A Method for Comparative Analysis of Trusted Execution Environments

The problem of secure remote computation has become a serious concern of hardware manufacturers and software developers alike. Trusted Execution Environments (TEEs) are a solution to the problem of secure remote computation in applications ranging from “chip and pin” financial transactions [40] to intellectual property protection in modern gaming systems [17]. While extensive literature has been published about many of these technologies, there exists no current model for comparing TEEs. This thesis provides hardware architects and designers with a set of tools for comparing TEEs. I do so by examining several properties of a TEE and comparing their implementations in several technologies. I found that several features can be detailed out into multiple sub-feature sets, which can be used in comparisons. The intent is that choosing between different technologies can be done in a rigorous way, taking into account the current features available to TEEs.


Glossary attestation
The process of assuring the validity and security of a system. The system can only be valid and secure if the code and data stored on the system have not been altered in any way that either causes the system to operate outside specifications or compromises the trust model. 4, 11, 14, 17, 21-25, 34, 39, 55-60, 64 axiomatic A semantic of logic which formalizes the definition of a concept using a set of criteria or axioms, all of which must be true for the given definition to be met. 43 chain of trust A series of entities that engage in secure transactions in order to provide a service. The first entity in the chain is referred to as the root of trust, while the last entity in the chain is often the end user or application requiring a secure transaction. 9, 32, 37, 38 cryptographic hash A cryptographic hash is the result of a mathematical function whose input is data of variable length and whose output is data of a deterministic value and fixed length. If this hash represents a piece of software, we can consider this hash the software's "identity" [9] for purposes of attestation. Many systems of attestation refer to this hash as a "measurement". 19,22,24,34,57 hart In a RISC-V system, a core contains an independent instruction fetch unit and each core can have multiple hardware threads. These hardware threads are referred to as harts. 42, 43 viii operational semantic A semantic of logic which formalizes the definition of a concept by generating a golden output model. Any system meeting the definition must produce the same output as that defined in the model. 43 privilege ring Also known as a "protection ring" or "protection domain" [33], these modes of operation allow a processor to restrict access to memory or special instructions. Switching between rings is the function of low-level software or firmware. Before the problem of secure remote computation described in Chapter 1, these rings provided adequate protection for software applications. 1,5,7,8,19,[21][22][23]30 Root of Trust "A computing engine, code, and possibly data, all co-located on the same platform which provides security services. No ancestor entity is able to provide trustworthy attestation for the initial code and data state of the Root of Trust" [25]. For system security purposes one might say, "the buck stops here". xi, 11,17,35,53,57 Trusted Compute Base When referenced generally, the code which must be trusted in order for the system to be considered secure. Historically, computer architecture security relied on processor modes or privilege modes where code was allowed to execute. In these modes, separation of privileges is achieved and often referred to as "rings" with "ring 0" being the most privileged machine mode where OS kernel code runs and privilege ring 3 being the least privileged user mode where application code runs. Figure 1.1 shows this concept and we will briefly discuss how it is applied to different architectures in the coming chapters.
As applications became more complex, specifically with the advent of largescale virtualization and the internet, this simple security model broke down as executed code could no longer be trusted, nor its origin verified. The problem of "secure remote computation" arises where the data owner must trust not only the software provider, but also the remote computer and infrastructure on which that software is executed. Homomorphic encryption solves the problem of secure remote computation to some extent, however the performance overhead of this transaction limits its application [22].
In an attempt to address the problem of secure remote computation, microprocessor designers have implemented different types of Trusted Execution Environments (TEEs), first defined by the Open Mobile Terminal Platform and ratified in 2009 [45]. These TEEs are intended to allow for code and data to reside in a specially provisioned area of memory where the processor can reserve access rights using a given set of rules. The processor will then guarantee the confidentiality Hierarchical protection domains or protection rings. Processors will define the method for switching between modes in such a way that code running in "ring 3" should not have access to code or data running in the rings below it. Attempts to access memory or to run special privileged instructions should throw an exception. Device drivers may run at a higher privilege level to gain access to special instructions. and integrity of that region of memory based on the configuration of the system.
In choosing a TEE, hardware architects have many complex features to consider before picking a platform. Currently, many resources exist that cover the features and design details of specific TEEs, however there are few resources that combine this information into one source for comparison. In this thesis, we will examine Intel Software Guard Extensions (SGX), Arm TrustZone, and RISC-V Physical Memory Protection (PMP) in order to provide a methodology for comparing and evaluating TEEs by considering these technologies' respective strengths and weaknesses. The goal of this thesis is not to characterize one technology as overall superior to another, nor is the goal to expose fatal flaws that make one technology 2 inherently insecure. Rather, we will describe the properties of a hardware TEE and illustrate how each technology implements those properties. A comparison of these different implementation details yields a method of evaluating TEEs for those looking to choose the TEE that best fits their needs.
These three technologies were chosen for specific reasons. Intel SGX is available on most consumer desktop/laptop products, and the technology is now becoming popular on their Xeon line of server/workstation processors. Arm is the de facto architecture used in cell phones and embedded devices. The popularity of these two architectures makes them a natural choice for comparison. The third technology, RISC-V PMP, is a newcomer to the stage of TEE options. We will examine how the more mature technologies have informed choices made by the architects of RISC-V, as they are clearly addressing specific limitations with current technologies.
While the method used to examine these technologies will be restricted to those aforementioned three TEEs, this method can be applied to any TEE.
The goal of this thesis is to provide hardware implementers with a method for comparative analysis and not to act as a definitive guide to any one of these three technologies. Any hardware implementation details are given only as examples and are not intended to be used in production systems. Furthermore, this thesis is an examination of the hardware and not the many possible software implementations available. As such, while we may discuss firmware and software throughout the thesis, it is not meant to be a guide to properly implementing a complete solution.
This thesis will begin by giving a brief history of TEEs as well as some of the cryptographic underpinnings of the technology (chapter 2). In considering these topics, we will provide definitions for many of the technologies used in a TEE in order to constrain ourselves to hardware implementations only. We will then 3 give a detailed analysis of Intel SGX (chapter 3), Arm TrustZone (chapter 4), and RISC-V PMP (chapter 5). We will examine how these technologies provide the fundamental properties of a hardware TEE: code integrity, data integrity, and data confidentiality. Furthermore, we will describe how some of these solutions can provide more advanced features like code confidentiality, authenticated launch, programmability, attestation, and recoverability. Lastly we will provide a comparison of the three technologies and their methods for providing the fundamental and advanced properties of a hardware TEE (chapter 6).

Overview
In order to understand Trusted Execution Environments (TEEs) we must first look at how computer architecture security worked before the problem of secure remote computation. The processor modes discussed in the introduction contained four privilege rings with the most privileged level, privilege ring 0, executing operating system kernel code. However, in most modern x86 client systems there are 3 levels of even greater privilege than privilege ring 0. We will examine each of these briefly and describe why they do not provide the same type of protections that a TEE provides.

The Predecessors of the TEE
In 1990, Intel integrated a new privilege mode into their i386SL processor called System Management Mode (SMM) [67], now commonly referred to as "privilege ring -2" (privilege ring negative two). OS code does not have access to this privilege level and only specific firmware, usually provided by the platform manufacturer, is allowed to execute code in SMM. This processor mode is entered by using a special System Management Interrupt (SMI). This interrupt has the highest priority of any interrupt as of the writing of this thesis, and is even higher priority than a non-maskable interrupt (NMI). Common tasks one might perform in SMM include thermal management, power management, or even something as simple as altering the volume output in response to buttons on a laptop keyboard [66]. SMM would remain the most privileged level of code execution until the release of Intel Management Engine (Intel ME) in 2008 [19].  [18]. This firmware code is an implementation of the Unified Extensible Firmware Interface (UEFI) specification, which is the successor to the Extensible Firmware Interface (EFI) specification [68].
There are several reasons why SMM is not a suitable replacement for a TEE.
Firstly, using SMM for any tasks which consume a significant amount of time would cause the system to hang while SMIs are handled. As such, SMM tasks are confined to small workloads which can return quickly and do not need to happen in rapid succession. However, tasks which take significant amounts of time and may occur in rapid succession are exactly the types of tasks we wish to run in a TEE.
Tasks like processing a credit card payment, or verifying the identity of the user with biometrics would all take significant time to run and would need to happen in quick succession.
Secondly, SMM is very restrictive in terms of how code is developed for the 6 platform. All code executed in SMM must be located inside the system firmware, loaded into SMRAM, and locked before the OS is loaded into memory. As such, firmware code does not have easy access to communicating with the host OS but must use special EFI SMM variables [60].
Lastly, while updating firmware in the field is possible [69], it almost always requires a reboot of the system. Updating the code in a TEE is a critical function that will be required much more often than a firmware update and should not require system reboot each time. These are just a few reasons why SMM is not a suitable solution to the problem of secure remote computation. Perhaps the overarching reason is that SMM was not designed to solve the problem of secure remote computation. Regardless, we have shown why it should not be used where a TEE would be better suited.
In 2005 [61], Intel released their first processor to allow the next most privileged mode of their x86 processors, Intel Virtualization Technology (VT-x) 1 . Commonly referred to as "privilege ring -1" (ring negative one), VT-x added new instructions called Virtual Machine Extensions (VMSs). These instructions are used to create a layer of hardware isolation between the host OS from the guest OS. It is tempting to see a TEE as a type of virtualization. Indeed, one early form of a trusted computing environment used a "dedicated closed virtual machine" [21] to achieve many of the goals of a modern TEE. However, as others have shown [54], this type of virtualization fails to cover several key properties of a TEE. For our purposes, we will consider these properties attestability and code confidentiality, and will discuss these in depth in coming chapters.
In 2008 [19], Intel developed what is today the most privileged processor level on x86 hardware, the Intel Management Engine (Intel ME). This technology, commonly referred to as "privilege ring -3" (privilege ring negative three) is not actually a processor mode at all. It is instead a feature of some Intel chipsets. Most of the properties of Intel ME are obscured by the proprietary firmware that it runs. The majority of information available regarding Intel ME is found either in Intel's publications or from the plethora of reverse engineering done by security researchers.
As such, we will not consider Intel ME a suitable replacement for a TEE for two key reasons: most of the information about this technology is gained by indirect means like reverse engineering and the firmware is not intended to be modified from the signed binary provided by Intel.
These three x86 "negative privilege rings" provide hardware protection of various kinds and each has its own restricted area of memory. However, none of these technologies serve as a viable solution to the problem of secure remote computation. It is no surprise that Intel eventually developed Software Guard Extensions (SGX), their own implementation of a TEE in order to address this problem. Before we address SGX specifically, we will continue to track the development of Trusted Execution Environments (TEEs) by starting with their use in handsets, the precursor to today's smartphones.

Birth of a TEE
The concept of securing computation is not a new idea. As we have seen with Intel ME, neither is the technique of using a processor other than the main, gen-  [48]. The OMTP standard defines an "execution environment" as a combination of five elements: a processing unit, a bus, physical memory, a boot process, and the contents of the execution environment [45]. These contents include the code, data, and keys required for computation. The OMTP document goes on to describe two sets of security requirements which meet their definition of a TEE, called "Profile 1 and 2". Profile 2 provides greater security than Profile 1, however both meet their definition of a TEE.
Both profiles provide protection against unauthorized memory access given a list of vulnerable attack surfaces summarized in Table 2.1. The core requirements for an OMTP TEE cover these "threat groups" to different degrees. There are 27 core requirements in total and it is out of the scope of this thesis to list or go into details on them. Sufficed to say that they address the attack surfaces described in Table 2.1. Also, it is clear from the threat groups described that these threat models are only applicable to handsets and do not cover typical embedded devices.
Another standards body called GlobalPlatform took up the task of standardizing the use of TEEs across multiple embedded platforms.
GlobalPlatform Incorporated is a nonprofit industry organization that began in 1999 with the mission of providing standardization around mobile payment devices and software. In 2011 they created a TEE model [25] that met the needs of embedded use cases. They define a TEE as a system providing "isolated execution,

Group
Attack Surface

Group 1
Hardware modules used for accessing memories, only including attacks mounted via modules built into the "mobile equipment" design (e.g. DMA module) Group 2 Colour LCD controllers or "graphics chip of the mobile device" designed to be pointed at memory blocks and could dump that information to the screen   whereas the TEE has only one purpose: to "provide a safe area of the device to protect assets and execute trusted code" [25]. These standards were first put into use by complex SoCs with the release of processors implementing the ARM Cortex A5 core in 2012. These cores included Arm TrustZone as well as a reference firmware implementation for building secure applications. Arm TrustZone has since become the de facto standard for embedded devices as well as as Androidbased mobile phones requiring a TEE.
Up to now, we have only covered relatively simple use cases like handsets, smartphones, and embedded devices. Bridging the gap between these simple devices and desktop or server processors will require we cover the current dominant architecture in that space, namely the x86 family implemented by Intel and AMD.

From Handsets to the Cloud
In 2013 at a workshop titled Hardware and Architectural Support for Security and Privacy (HASP), Intel introduced Software Guard Extensions (SGX) for Intel architecture. SGX is not an extension to the x86 architecture, but rather to the Intel architecture. As such, AMD developed their own solution called Secure Encrypted Virtualization (SEV). As mentioned previously, we will not discuss AMD SEV in this thesis as its architecture is quite different from Intel SGX. While there are several papers published in the proceedings from the HASP 2013 workshop [1,30,42], as well as two patents from Intel's original filings [32,41], we will use a more recent paper by Costan [15] as well as a recent whitepaper from Intel [31] in exploring 13 SGX.
Intel's introduction of SGX opened up the desktop and laptop markets to a new type of software application capable of hardware-backed secure "enclaves". It also gave application developers on Intel platforms the chance to utilize both local and remote attestation, providing assurance from the vendor regarding code integrity and confidentiality. It wasn't until 2021 that Intel would make this technology available in multi-socket Xeon CPUs, opening the door for cloud providers to take advantage of TEEs on Intel platforms [13].

Initial SGX Enclave Setup
Configuration settings for SGX exists as part of the platform firmware, and most firmware vendors provide simple tools for enabling SGX. If SGX is enabled, the firmware is responsible for setting aside a memory region called the Processor  This section of specially allocated memory is used to store the Enclave Page Cache (EPC), which are the 4kb pages holding both the enclave data and code.
The exact layout of the PRM and EPC are model-specific, and depend on firmware settings. While untrusted system software both assigns these EPCs to an enclave and loads them with data, it is the CPU which keeps track of all the EPCs ensuring that they only belong to one enclave. Once the system software loads data into the enclave it asks the CPU to mark that enclave as initialized, after which no other data may be loaded into the enclave as this setup process is disabled for that enclave. After initialization, a measurement of the enclave is taken by means of a cryptographic hash. Checking that measurement later ensures that any operations performed on the enclave are done so in a secure environment.  It is important to note that the firmware which sets up the PRM and creates the EPC is vendor specific. As of the writing of this thesis, it is highly probable that the firmware is based on the open source project TianoCore, an implementation of UEFI spearheaded by Intel in 2006 and now led by engineers from companies like Intel, Arm, Apple, and Red Hat.

Executing SGX Enclave Code
Execution flow can only move into an enclave via a special CPU instruction EEN-TER, much like switching from user mode to kernel mode. The actual execution happens in "user mode" also known as "privilege ring 3" and takes advantage of address translation from the operating system or hypervisor. The benefit to this model is that the OS or hypervisor will quickly provide address translation lowering overhead. As Costan notes, "the operating system and hypervisor are still in full control of the page tables and extended page tables (EPT), and each enclave's code uses the same address translation process and page tables as its host application. This minimizes the amount of changes required to add SGX support to existing system software." [15]. The downside is that code executing inside the enclave does not have access to system calls (syscall) or any other high privilege operations. An inability to make system calls limits the types of operations that can be preformed inside an enclave. The code executing inside the enclave does have access to its entire address space which includes the "host application" that caused the creation of the enclave.
The CPU executing the enclave code will perform an Asynchronous Enclave Exit (AEX) whenever execution moves outside the enclave such as servicing an interrupt or during a page fault. The CPU state is saved inside the enclave memory metadata before exiting, ensuring that the CPU can securely restore the state of enclave execution. There are special machine mode CPU instructions that are used both in allocating EPC pages to the enclave as well as evicting those pages into untrusted DRAM. Page management instructions allow code outside the enclave to operate on code within the enclave. SGX uses cryptographic protections to assure the confidentiality, integrity, and "freshness" [15] of the evicted EPC pages while they are stored in untrusted memory.
As mentioned the application that makes calls into an enclave or "host application" lives in the same address space as the enclave code and data. Restricting host and enclave to the same address space has benefits in terms of application size variability, but may open the host application up to attack should the enclave code become compromised [55]. SGX allows enclaves to execute multiple threads through a Thread Control Structure (TCS) which allows multiple logical cores to execute enclave code. Within the EPC metadata, reserved secure memory called the State Save Area (SSA) allows the threads to save their state when a context switch happens, like servicing an interrupt. In this way, Intel SGX is able to allow a specific amount of code and data to remain protected while still allowing access to that data by code outside the trust boundary. The enclave's memory is protected in states shaded blue. Note that in the enclave is not secure while in the uninitialized state as pages are being added or extended. Until a measurement is loaded into the MRENCLAVE register, code and data integrity can not be assured. The CPU will call AEX in order to service interrupts, and the state of the enclave will be saved in the SSA. Reprinted as a simplified version from Costan's Figure 63 [15].

Life Cycle of an SGX Enclave
In order to understand the life cycle of an enclave, we must consider the specific x86 instructions used to create and manage these enclaves. Many of these instructions which create enclaves, extend pages, and remove enclaves operate in "privilege ring 0", one of the most privileged modes. Whereas attestation, entering, and exiting the enclave can be done from "privilege ring 3" the least privileged 21 mode. The first of the privileged instructions in our life cycle is ECREATE which fills a protected data structure located inside the EPC with the size and measurement hash of the enclave. This data structure, called the SGX Enclave Control Structure (SECS), is used by the hardware and is not directly accessible to software. The system will then add pages to the EPC with the EADD instruction, and extend the EPC page measurement with the EEXTEND instruction. When executing EADD and EEXTEND the system will ensure that the page's address range is within the Enclave Linear Range (ELRANGE). The EEXTEND instruction allows for the accumulation of a hash of all the pages in the EPC. This measurement can be used later for attestation that the enclave has not been tampered with or changed in some way.
It is important to note that in its uninitialized state, none of the enclave code or data is encrypted. For example, any privileged code running at "privilege ring 0" can gain access to these data and structures. Enclaves must be initially built on a system that is known to be secure, such that the measurements taken are considered a "gold standard" with which to preform attestation on a local or remote machine at some later time. When the EINIT instruction is called, the enclave is considered fully built, the measurement is locked down, and "privilege ring 3" (user) applications can now enter the enclave and attest that it contains the code and data that the developer intended.
EBLOCK, ETRACK, EWB, and ELOAD 1 are paging instructions run with "privilege ring 0" privileges. The goal is to allow the paging of secure pages into and out of main memory while ensuring the confidentiality and integrity of those pages. Information stored inside the EPC called the Paging Crypto MetaData (PCMD) keeps track of the identity of the enclave the page belongs to and a pointer to an access rights structure. There is also a Version Array (VA) which is used to store the version numbers of pages evicted from the EPC. These versioned and access controlled pages are therefore hardware protected, and any change to the versioning, access rights, or origins of the page will result in a page fault. It is possible to have 2 instances of the same enclave, however pages cannot be swapped between them, and the hashes of these pages will not be the same.
Once an application has requested that "privilege ring 0" components build the enclave and EENTER is called, the enclave may begin execution. The hardware is responsible for saving (AEX) and restoring (ERESUME) the architectural state of execution should any external events like interrupts or exceptions cause execution to leave the enclave. The EGETKEY and EREPORT instructions operate in user mode ("privilege ring 3") and seal data based on the key the developer provides.
Using these two instructions SGX applications operating in "privilege ring 3" are able to preform local attestation of the enclave, perhaps the most vital function of any Trusted Execution Environment (TEE). Finally, once the enclave is no longer needed, the system will call EEXIT, and a synchronous enclave exit will occur. As Costan notes, "EEXIT does not modify most registers, so enclave authors must make sure to clear any secrets stored in the processor's registers before returning control to the host process." [15].

Attestation with Intel SGX
Software attestation of enclaves is required to ensure the integrity of the enclave.
This attestation can happen locally between two enclaves on the same platform or remotely between two different platforms. As previously noted, the measurement of the enclave includes a SHA-256 hash of the enclave's attributes as well as the content, position, and access rights of its pages. This measurement is stored in a register called MRENCLAVE which represents the enclave's TCB. The EREPORT instruction is used to generate a signed report of this TCB and the EGETKEY instruction then retrieves the key used to validate said report. Local attestation of enclaves can be done using symmetric encryption as the hardware can ensure the integrity of the single key being used to verify the MRENCLAVE value. Remote attestation must be done using asymmetric encryption (both a public and private key) and requires the remote SGX enabled platform to query an Intel attestation server.
Local attestation can be used by enclaves running on the same platform and will provide assurance as to the identity of the enclaves. The local enclave which attests to the reporting enclave is called the "Quoting Enclave" [31]. alone is not enough to assure that the platform and enclave are genuine SGX enclaves [34]. Remote attestation requires asymmetric keys as well as a remote server with which to verify the quote of an enclave.
Intel platforms include two statistically unique "root keys" which provide assurance that the system is indeed a genuine Intel SGX platform. These include the Root Provisioning Key and the Root Seal Key [31] which are stored in hardware fuses. Intel also provides a more advanced form of asymmetric encryption called Enhanced Privacy ID (EPID). EPID is essentially a group signature where the system can attest to a message being signed by a member of the group without A remote challenger may request that an application provide attestation. The application must then request a report from the enclave and locally verify that against a quoting enclave. The quoting enclave will use the remote along with asymmetric keys to produce a remote attestation "quote" to be returned to the challenger. The challenger may then use some verification service to check the validity of the quote. This figure was reproduced from Figure 1 [31].
divulging the identify of the signer. The signature algorithm is nothing new [11] and has been included in the ISO/IEC 20008 and 20009 standards. EPID includes several revocation schemes which allow keys to be revoked based on checks performed by the verifier and/or issuer. The Intel Attestation Service (IAS) will take submission of quotes provided, verify the validity of the signatures, and verify that they have not been revoked.
In this chapter, we have briefly described Intel Software Guard Extensions (SGX) from the initial setup of the enclave through to the remote attestation of that enclave's contents. This system is by far the most complex of the three we will examine in this thesis. We have only covered the properties that are necessary to 25 understand if we are to appropriately apply our method of comparison. Depending on the use case, a much more in depth knowledge of SGX may be required.
However, this thesis will show that even with this rudimentary understanding, one can still apply rigorous method to analysing features of a TEE. Next, we will cover Arm's framework for building TEEs called TrustZone, as well as the reference im- Arm SoC processors create a more absolute separation between the concepts of "secure" and "normal or insecure" operation than Intel Software Guard Extensions (SGX). At its highest level this is accomplished using the Secure Configuration 1 Arm Limited produces three ISA profiles [3]: A-Profile for general purpose compute, R-Profile for real-time, and M-Profile for microcontrollers. These profiles can be seen in the architecture version naming scheme, such that ARMv8-A is "version 8" of the ARM instruction set, with a focus on general purpose compute. The cores that Arm Limited licenses to customers are labeled "Cortex", with ARM Cortex-A55 referencing a specific microarchitecture implementation of the ARMv8-A ISA.
2 AArch64 is the 64-bit state of any ARMv8-A core which is capable of also running in the 32-bit state called AArch32 [3].
Register (SCR) "Non-Secure bit" (NS) with 1 meaning non-secure and 0 meaning secure. This is perhaps the most fundamental element that separates Arm's two security worlds. Digging a bit deeper, this separation of worlds is accomplished using four separate primitives: one on the bus, one on the SoC core, one as part of the memory infrastructure, and finally one as part of the debug infrastructure.

ROM
Internal peripherals Processing core(s)

SoC
External memory

External peripherals
One Possible TrustZone Platform in this "TrustZone-enabled AMBA3 AXI bus fabric" [3] ensures that no "Secure World" resources can be accessed by "Normal World" components. These bits include AWPROT for write transactions and ARPROT for read transactions where like the NS bit low is Secure and high is Non-secure.
Secondly, SoCs which implement the ARMv8-A instruction set must also implement extensions which enable a single physical processor core to safely and efficiently execute code from both the Normal World and the Secure World in a time-sliced fashion [44]. The value of the Non-Secure bit is read from the SCR and passed along down the bus to the memory controller and peripherals. A new instruction, the Secure Monitor Call (SMC), is added which allows the core to switch between the secure and normal modes. We will discuss the secure monitor firmware which is responsible for handling these interrupts in the next section.
Thirdly, the memory infrastructure includes security features like TrustZone Address Space Controller (TZASC) and TrustZone Memory Adapter (TZMA) [44]. Lastly, the security-aware debug infrastructure controls debug access to the Secure World. This includes "secure privileged invasive (JTAG) debug, secure privileged non-invasive (trace) debug, secure user invasive debug, and secure user non-invasive debug" [3]. By using two different signals into the core along with two different bits in a secure register, the core can report either invasive or non-invasive debug info. In this way, the core is able to debug either the normal world only, or it can debug both the secure and normal worlds together. These four primitives provide a framework or scaffolding on which to build a platform capable of secure computation.
In the next section we discuss the firmware that will implement a TEE using TrustZone's features. First, we will point out a potentially confusing difference between how Intel and Arm create privilege levels. Unlike Intel platforms which refer to their privilege levels as privilege rings, Arm uses "Exception Levels" EL0 through EL3 [4]. Here EL3 is the highest, most privileged level where as EL0 is the lowest and least privileged level. Much like with the x86 architecture, exceptions like data aborts, prefetch aborts, and other interrupts can be taken from the level at which they occur to the same or any higher privileged level, but not a level which has less privileges. So, for example, an interrupt occurring in the OS kernel (EL1) can be handled in the kernel or in the secure monitor (EL3), but not in the lower privileged application level (EL0). Practically speaking this means that the user applications running on a system which has not been compromised will not have access to kernel or lower exceptions. Another common confusion point between Arm and Intel is that Intel's "privilege ring 0" is the highest privilege level while Arm's "EL0" is the lowest privilege level.
See Table 4.1 for a complete list of privilege levels, their description, and how they might be implemented in an ARMv8-A system. Each exception level manages its own page tables and control registers with the exception of EL0 which is managed by EL1 [4]. This is a common practice across architectures where the kernel level mode controls the page table for the applications running on top of it.      Figure 4.3: Secure World implementation using ARM TrustZone. The SoC boots into the Secure World and a monitor is registered which acts as the interface between the "secure" and "normal" worlds. All EL3 exceptions are caught by the secure monitor, and there is a special EL3 exception called a "secure monitor call" which is used to switch the processor between the two worlds. Figure recreated and modified from the original [59]. and the user application's concerns like displaying a web page. At no point in this interaction should the user application code have access to the network card's buffers, however the kernel is able to read and write these buffers and mediate the flow of data to/from the client application. This "guarded" flow of data is similar to how memory in the Secure World is kept separate from the Normal World using an instruction which generates an exception called a Secure Monitor Call (SMC).

TrustZone Attestation
Local attestation using TrustZone is dependent on several measurements taken during the boot process. If the boot process is not secured then "all bets are off", and we cannot assure the integrity or confidentiality of the code or data inside the TEE. We will follow this boot flow step by step, and each step number is illustrated 34 in Figure 4 and verifies the BL3 x content certificate using the BL3 x public key (Step 7). BL2 can now extract and save the BL3 x hash used for BL3 x image verification (Step 8).
BL2 verifies the BL3 3 key certificate using the NW public key (Step 9) and verifies the BL3 3 content certificate using the BL3 3 pub key(Step 10). BL2 extracts and saves the BL3 3 hash used for BL3 3 image verification (Step 11). Finally, execution is transferred to verified BL3 x (Step 12) and BL3 3 images (Step 13).
We have now securely booted, and our secure monitor provided by TF-A should be loaded into BL3 1 running in the highest privilege level of EL3. The trusted OS, Open-source Portable TEE (OP-TEE) is loaded into BL3 2 and runs at EL1. Any trusted applications will run in EL0 of the "secure world" built on top of this secure OS. A "normal world" bootloader like U-Boot or some UEFI implementation is loaded into BL3 3 and runs in the privilege level of EL2 3 . An operating system Since loading a TA from the REE file system creates an inherently larger attack surface, there are two ways for OP-TEE to load applications from a more secure location. The first method is known as "early TA" and allows applications to be linked into the data section of the TEE core blob itself. The "early TA" method has two benefits: applications can be loaded from a known secure source and applications can be loaded before the normal world or its file system have been initialized. The other, more robust option is to load the application from secured storage, an OP-TEE implementation of the GlobalPlatform specification for Trusted Storage [25].
OP-TEE secure storage follows the GlobalPlatform TEE Core API document booted.  TEE only supports one private key per trusted OS. As such, each TA will be signed with the same key. This step completes the chain of trust and we are able to assure that each step in loading the application, from PoR to loading the application into memory, has occurred in a way which preserves the integrity and confidentiality of 38 the data as well as the integrity of the code.
The complexity of remote attestation on mobile platforms, specifically cell phones, has been well studied and understood for over a decade [43]. There have been many recent efforts towards a more formally proven type of remote attestation [12,20] which can be used in IoT and embedded applications. However, Arm TrustZone as a framework leaves the implementation of remote attestation as an exercise for the platform vendor or manufacturer. We will return to this point in "According to Linus Torvalds, Linux has succeeded at least in part because it followed good design principles, which allowed it to be extended in ways that he didn't envision when he started work on the kernel. Similarly, Larry Wall explains how he created Perl in such a way that its feature-set could evolve naturally, as human languages evolve, in response to the needs of its users." [46]. Key to RISC-V's success will be the proliferation of open source extensions, tools, and designs that anyone can use as a platform on which to produce and launch innovative hardware. Sanctum [16], TIMBER-V [65], Mi6 [10], and Keystone Enclave [14,36,37]. This thesis will concentrate on Keystone Enclave as it is open source and still in active development.

The RISC-V Memory Model
An understanding of how RISC-V handles memory is required in order to understand how RISC-V PMP protects said memory. There are two current memory models available as part of the RISC-V specification. The first memory model presented in the specification is the RISC-V Weak Memory Order (RVWMO) model, "which is designed to provide flexibility for architects to build high-performance scalable designs while simultaneously supporting a tractable programming model" [62]. There is also the "Ztso" extension which gives us the RISC-V Total Store Ordering (RVTSO) memory consistency model. One can think of RVTSO as the "less flexible" model when compared to RVWMO. We will only cover RVWMO briefly here as knowledge of RVTSO is not required to understand PMP, nor is a deep understanding of the RVWMO required.
The memory model of any instruction set architecture defines the values returned by a load. As such, we must first understand what we are loading into memory. RISC-V hardware threads, which are referred to in the specification as "harts" [62], have a byte-addressable address space of 2 XLEN bytes for all memory accesses. Here, XLEN refers to the width of a register in either 32 or 64 bits.
Words are defined as having 32 bits. Halfwords are 16 bits, doublewords are 64 bits (8 bytes), and so on. The address space can be thought of as a ring, so that the last memory address is adjacent to the first. Memory instructions will simply wrap around the space ignoring that they have effectively "walked off the end".
The specification leaves room for virtual memory by allowing for both explicit and implicit stores and loads. Chapter 3 of the RISC-V Unprivileged Specification [62] introduces the "Zifencei" extension, which defines a fence instruction that explicitly synchronizes writes to instruction memory and instruction fetches on the same hart. The order in which these loads and stores happen is up to the implementation, and the rules for consistency are defined in the memory model. Any implementation which follows the RVWMO memory model must conform to the thirteen rules and satisfy the three axioms.

RISC-V Physical Memory Attributes
RISC-V system's physical memory map has various properties and capabilities that are described in detail in the Privileged Specification [63] and referred to as Physical Memory Attributes (PMA). These attributes determine things like read, write, and execute permissions of a specific region of physical memory. Most systems built on the RISC-V architecture will require that the PMAs are checked later in the pipeline and that this check is done in hardware. This is in contrast to other architectures where these types of checks are done in virtual page tables where the Translation Lookaside Buffer (TLB) contains the information the pipeline needs regarding these attributes. RISC-V systems call the mechanism for making these checks the "PMA checker" and many physical attributes will be hardwired into the checker when designing the chip. For those attributes that are not known at design time there are special platform specific control registers that can be configured at runtime.
Memory regions are given attributes based on if they are part of main memory or part of I/O. The access width of a region can be anything from 8-bit byte to long multi-word bursts. As the specification states, "Complex atomic memory operations on a single memory word or doubleword are performed with the load-reserved (LR) and store-conditional (SC) instructions." [63]. RISC-V also allows for special atomic memory operations (AMOs) which are instructions that 44 perform operations for multiprocessor synchronization. A given PMA will contain information regarding which atomic operations are allowed for a specific region of memory. Table 5.1 lists the AMO available to I/O as of this writing.

AMO Class Supported Operations
AMONone None

AMOSwap amoswap
AMOLogical above + amoand, amoor, amoxor AMOArithmetic above + amoadd, amomin, amomax, amominu, amomaxu As mentioned earlier, regions of memory will be classified either as main memory or as I/O, and this must be considered by the FENCE instruction when ordering memory. Regardless of whether the memory model used is RVWMO or RVTSO, memory regions may be classified as either having relaxed or strong ordering. Strongly ordered memory regions use a "channel" mechanism to guarantee ordering. Using PMAs, systems can decide to set the type of ordering dynamically or not. The specification requires that all regions of memory be coherent, such that any change made by one agent to a memory region must eventually be visible other agents of the system. Cacheability is left up to the platforms, however three types are called out: "master-private, shared, and slave-private" [63]. PMAs will describe the specific cache features of each region as well as if the region is idempotent.

RISC-V Physical Memory Protection
Unlike PMAs, RISC-V Physical Memory Protection (PMP) consists of a set of configurations that can be changed dynamically during runtime. The privileged specification describes a "PMP unit" as a set of "per-hart machine-mode control registers to allow physical memory access privileges (read, write, execute) to be specified for each physical memory region." [63]. These registers must be checked in parallel with the PMA attributes described in the previous section.
As we discussed in the previous two chapters, processor privilege levels or "modes" are a critical foundation to our security model. RISC-V PMP allows for specific registers that are only available to the highest privileged machine-mode or m-mode. The different processor modes for RISC-V are described briefly in Table   5.2. Note that H Mode is still only in the draft state, and is not applicable when discussing PMP. While PMP is available in both 64 bit and 32 bit versions, we will only describe 32 bit configurations in this thesis. It is enough for our comparison that we understand a 64 bit implementation to be possible and to hold to the same or standards as the 32 bit version.
(a) 24 Figure 5.1a describes the RV32 PMP configuration CSR layout. Figure 5.1b is the PMP address register format for RV32. Lastly, Figure 5.1c shows the PMP configuration register format. Figures reproduced from the RISC-V privileged specification [63].
There are currently up to 16 PMP configuration registers available in both 32 and 64 bit modes and their layout for 32 bit is shown in Figure 5.1a. Next, in Figure 5.1c, we see a description of the layout for those configuration registers where WARL is Write-Any Read-Legal and WIRI is Write-Ignored Read-Ignored. This is to say that the two bits 5 and 6 are simply ignored in this register as they are reserved. All the other bits will follow the RISC-V common WARL field rules as defined in the CSR section of the specification. There is also an address register format defined in Figure 5.1b, which is the starting address of the PMP region and encodes bits 33-2 of the 34-bit physical address length 1 . No "stop address" is required, and if only one PMP region is defined, it will encompass the entire memory space. Indeed when many systems boot they may choose to create a single PMP region with read/write/execute enabled for all modes as a default setting.
The "L" bit of the configuration register defines if that region of memory is locked and cannot be read, written, or executed from, regardless of the processor mode. A reset is required for m-mode to once again control that CSR and allow that region's permissions to change. As mentioned, bits 6 and 5 are reserved. Bits 4 and 3 define the type of address matching scheme that will be used to match the store or load address range against the PMP address register. The options for this two bit value are OFF, Top of Range (TOR), Naturally Aligned 4-byte region (NA4), or Naturally Aligned Power of Two (NAPOT). If set to NAPOT the granularity can be set by the system to any NAPOT value greater than or equal to 8. The last three bits are set to 1 or 0 for read, write, and execute, with 1 meaning "enabled" and 0 meaning "disabled". These bits currently apply to both S Mode and U Mode, though we will discuss the future of S Mode PMP in a following section.
Using these configuration registers and address registers, RISC-V systems are able to create regions of memory with simple read, write, and execute privileges enforced by hardware. This allows systems to build secured areas of memory similar to that of both Intel Software Guard Extensions (SGX) and Arm TrustZone.
However, without a framework on which to build these secured areas of memory, the specification only enables hardware engineers to create a foundation on which to build a Trusted Execution Environment (TEE). We will now examine Keystone Enclave, an open source framework for building hardware TEEs using RISC-V PMP as the foundation.

Keystone Enclave
One of the first TEE solutions for RISC-V PMP came out of the MIT Computer Science & Artificial Intelligence Laboratory (CSAIL) and a project called Sanctum [16]. This work was done just before the RISC-V PMP task group ratified the first  RISC-V is highly customize and we can imagine many different possible solutions when building a platform. Shaded areas imply that some logic is present which allows PMP to secure the TEE. Reproduced as a simplified version of Figure 2-3 of the GlobalPlatform specification [25] as well as Teschke's rendering [59].
As detailed in Figure 5.3, Keystone is a full software stack containing a firmware base which launches both trusted and untrusted "runtimes". These "runtimes" then launch both trusted, isolated applications into the enclave as well as untrusted applications into the user space of a Linux-based operating system. Keystone defines a set of TEE primitives [36]: Secure Boot, a "Secure Source of Randomness", and Remote Attestation. Keystone builds its framework off these primitives and allows for customization of their firmware which they call a secure monitor (SM).
The SM is used to "enforce TEE guarantees on the platform" [37], and does so by taking advantage of key properties of RISC-V machine mode. These properties include machine mode's programmability, its ability to delegate interrupts, and its ability to manage memory access using PMP. There are two key features of RISC-V PMP which Keystone takes advantage of in building their framework. The first is memory isolation which allows the secure monitor (SM) running in machine mode to restrict access to the enclave's memory.
The second feature is the ability for PMP to allow Keystone Enclave secure applications to use multiple hardware threads in executing code. As Lee notes, "RISC-V provides per-hardware-thread views of physical memory via machine-mode and PMP registers. Using RISC-V thus allows multiple concurrent and potentially multi-threaded enclaves to access disjoint memory partitions while also opening up supervisor-mode and the MMU for enclave use." [36]. The memory management of these enclaves is controlled by the Keystone Enclave runtime called "Eyrie" which can be seen in the high level diagram in Figure   5.3 as RT1 and RT2. Eyrie runs in Supervisor Mode and not only performs page table operations, but also allows for dynamically resizing the enclave. The security monitor has a plugin which allows for pages which must be evicted from secure memory to be encrypted such that the confidentiality of the data can be assured as it leaves the trusted region. This plugable interface of the secure monitor and Eyrie runtime, along with the open source nature of the project as a whole, allows for great flexibility. For example, the secure monitor has an "on-chip memory plugin" [37] that allows for use of a scratchpad memory onto which the enclave can be loaded. The Eyrie runtime has a plugin which allows an enclave to communicate with host applications in a secure manner using a shared buffer. While syscalls are not allowed inside the enclave itself, the secure monitor can either proxy those syscalls to the untrusted application, or run them in a trusted context if appropriate (e.g. mmap, brk, getrandom) [37].
While Keystone Enclave provides many features that we require in our definition of a TEE, some features are notably left to the user. Remote attestation is possible, however as the Keystone authors note, "Key distribution, revocation, attestation services, and anonymous attestation are orthogonal challenges" [37].
There are projects currently working on the problem of remote attestation on 52 RISC-V hardware [56], however these efforts are still in their infancy. Keystone Enclaves provide a method of local attestation which makes use of provisioned keys.
Enclaves can, during runtime, request signed attestation from the secure monitor in a kind of attestation report similar to that found in Sanctum [35]. This local attestation is built off of a Secure Boot process that is very similar to TrustZone's model, and includes the use of hardware key management and cryptographic engines. Efforts to continue work on secure boot RISC-V systems is the focus of research [29], but no dominant technology currently exists as a de facto standard.

Future Extensions of RISC-V Memory Protection
Three additional features of RISC-V PMP will doubtless play a role in future work done on Keystone Enclave or any RISC-V based TEE framework. The first is a draft of an Extended Physical Memory Protection (ePMP) specification which details how PMP will be able to secure user mode code from being executed by machine mode. Attacking user mode from a more privileged mode is a well known security vulnerability first address in Intel Ivy Bridge processors about a decade ago [53]. Seagate recently announced their completion of a chip which takes advantage of this draft ePMP features. The draft ePMP specification is currently supported inside the Spike simulator, which is the de facto functional simulator for RISC-V [7]. ePMP has also been implemented in open source hardware cores like the Ibex core from lowRISC [39] as well as the open source Root of Trust (RoT) OpenTitan [28].
The second additional feature, S-Mode Physical Memory Protection (sPMP), is also in draft status as of this writing. sPMP will provide a way to ensure that systems without virtual memory can take full advantage of PMP. Currently the 53 separation between S Mode and U Mode is done in virtual memory. This extension will allow for PMP configuration registers specifically reserved for S Mode, moving that protection from virtual memory into hardware registers. While sPMP is not implemented yet in any simulators or publicly available hardware, sPMP will likely see more development this year as the specification nears ratification.
The final additional feature, the draft specification for I/O Physical Memory Protection (IOPMP), aims to provide memory protection for additional memory masters such as Direct Memory Access (DMA). These memory masters will all share one IOPMP unit that will allow for both single core and multi-cores systems, support error reporting, and support a scalable number of IOPMP entries.
The current draft specification defines an address table and IOPMP configuration registers which provide address matching for each of the IOPMP addresses.
All three of these extensions, ePMP, sPMP, and IOPMP are in various stages of ratification by RISC-V International. As such, their features are still in flux to different degrees. However, ePMP, sPMP, and IOPMP show the direction that RISC-V is heading in terms of TEE development as an attempt to provide as many if not more features than currently available TEE technologies.
We have now covered three different implementations of a Trusted Execution Environment (TEE). These high level descriptions will act as a general reference in the next chapter where we will provide a method for comparing these technologies.
The method we present can be applied to any set of TEEs regardless of their complexity or their intended target hardware. The three examples of a TEE have provided a foundation for the systematic and rigorous comparison that will be detailed in the following chapter.

Chapter 6
Trusted Execution Environment Comparisons

A Method for Comparing TEEs
When we compare Intel Software Guard Extensions (SGX), Arm TrustZone, RISC-V Physical Memory Protection (PMP), or any group of Trusted Execution Environments (TEEs) we must first consider what we are comparing. As such we define any given TEE based on a set of properties that the TEE must implement or optionally might implement. We will make the assumption that the end goal of the comparison is to choose a TEE that best meets the needs of the system architect.
As such, we must take the system architect's needs into consideration in choosing which properties that we will compare. An architect designing an embedded system with no access to networking will certainly not require traditional remote attestation, as an example.
As mentioned in Chapter 1, we will use the properties of a TEE as defined by the Confidential Computing Consortium (CCC) [58]. The CCC is an open source project that brings together hardware vendors, cloud providers, and software developers to accelerate the adoption of TEE technologies and standards. They define code integrity, data integrity, and data confidentiality as three required properties of any hardware TEE. All three of the TEE technologies which we have summarized in the preceding chapters provide these three required features in different ways. The CCC also discusses more advanced features like code confidentiality, authenticated launch, programmability, attestation, and recoverability, which can be considered optional properties of a TEE. As a final option of a TEE, we will include extensibility, which will be discussed in detail shortly.
Notably absent from our list of required properties is code confidentiality. There are several reasons why one might want code to be confidential. Code may contain proprietary algorithms for machine learning or other intellectual property. Likewise code may contain information about health or safety systems which must remain confidential to assure functional safety. On the other hand, the code that runs inside the TEE may be open source, making the feature of code confidentiality nonessential. As such, code confidentiality is considered an optional property by our definition.
By authenticated launch, we mean a mechanism by which the TEE can prevent a host application from loading, or limit its functionality based on a set of criteria or security model. One can think of this in terms of a banking application on a cell phone. If any stage of the attestation process fails, we want to recognize that code integrity can no longer be assured. We may want to fail to load the banking application with appropriate errors if code integrity is compromised. However, we may want some of the features of the banking application available, while restricting others. Authenticated launch may restrict withdrawing funds from an account while allowing the application to provide text messaging with technical support in order to resolve the problem.
By programmability, we mean the ability of a TEE to be programmed with code that is unrestricted and able to be changed by the end user. This is apposed to a TEE with a limited set of available functions or one that is pre-programmed by the manufacturer or system vendor. Programmability will be limited by the type of instructions which are allowed to be executed from inside the TEE. The size of the region of code that is allowed to be used by the TEE may also be restricted.
Lastly, the way in which code running inside the TEE is allowed to communicate with code running outside the TEE is also a consideration of programmability. We will consider a TEE programmable to different degrees based on these limitations.
By attestation, we will use the definition by the CCC provided in Section 6 of their paper [58]. The attestation workflow starts with a secure connection between the two parties, identified as the verifier and the attester. The verifier must first challenge the attester to initiate the check. The challenge from the verifier signals to the attester that it must communicate with trusted hardware to request a measurement of the TEE. The trusted hardware is responsible for the measurement value as well as any certification of origin. The attester can then return these values to the verifier. The verifier will check any signatures provided based on the security model. Attestation may be performed locally on the host machine. If the verifier is on a remote machine to that of the attester we would consider that to be remote attestation. Attestation should take advantage of a Root of Trust (RoT), however that root of trust may be either hardware or software based. If a given system contains a hardware RoT we can consider that system more secure by the same definition we gave earlier of a hardware TEE.
By recoverability, we mean a way for the TEE to "roll back" to some known good state should a failure occur while preforming validation on the integrity or confidentiality of the system. The goal of recoverability is to re-establish trust in the Trusted Compute Base (TCB). Part of the TCB may not be easily recoverable.
For example, if the platform firmware loaded onto an Intel SGX system is found to be untrustworthy, a simple firmware update may not be capable of removing malicious code. However, if the code running inside the SGX enclave is found to be untrustworthy, simply reloading that code from its original source may be enough to reliably restore trust in the application.
Finally, we add extensibility as the final optional property of a TEE. Extensibility was not defined in the CCC document, however extensibility can be observed to some extent in all three TEE technologies we have covered. We define extensibility as the ability to customize the deployment and management of the TEE environment, including hardware, firmware, and software. This property comes at the cost of supporting those customizations. As Linus's Law states, "Given enough eyeballs, all bugs are shallow" [52]. Regardless, the burden of support for customization must be a consideration, and a healthy open source project is no small feat.
While these properties are a good starting point for analysis and comparison, we must break these properties down further in order to engage in a more rigorous comparison. Authenticated launch can be broken up into two parts: the ability to prevent application launch, or the ability to launch applications with limited functionality. Programmability will be broken into three types: the size available to the TCB, the number of unique trusted applications one can run, and the number trusted OS's one can run to manage these applications. Attestation can be divided into three parts, with remote attestation, local attestation, and if a hardware root of trust is available. Recoverability will be divided into three parts describing system which allow: revocation of keys or other components, ability to recover the TCB using a hardware RoT, and finally if re-issuing keys is possible without the need for re-provisioning. Extensibility will be broken up into three parts: hardware, firmware, and software. We will consider the levels of extensibility as either "highly extensible", "somewhat extensible", or "not extensible".

Mapping Data Points
This method uses tables to chart the comparison of these properties and subproperties we have discussed. In the following Tables 6.1, 6.2, and 6.3, we give example tables for the three technologies discussed in the previous chapters. The first Table 6.1 covers all six of the optional properties of a TEE that we have discussed in this chapter. Rather than grading each implementation of the property in a simple "good, better, best" fashion, we list if the property is possible in the given solution, and if so, the dependency of that property. Next, in Tables 6.2  Noting that the optional property of a TEE is available and listing required dependencies is helpful, however we also make a point to parenthetically note an example dependency where applicable. At the highest level, as in Table 6.1, this can be quite broad indeed. For example, attestation is possible with TrustZone and is provided locally by firmware or remotely by SoC manufacturers like Qualcomm. This is further illustrated in Table 6.2, where we show that local attestation can be achieved merely by using the appropriate firmware, however to achieve remote attestation, a system like Qualcomm's MSM8974 Snapdragon-based platform is 59 required [57]. Again, note that we are using quite an old example with Qualcomm's MSM8974, circa 2016. Readers wishing to implement this method of comparison would be wise to chose current security platforms. Likewise, we list Intel as the only current solution to providing remote attestation for SGX solutions, however it is highly likely that other solutions are available.
The final Table 6 the Keystone project allows developers to extend its feature set with a set of "plugable interfaces" [37]. While Intel and others have worked to provide open source and extensible firmware for their platforms, the goal of those firmware projects is not to provide systems with a way to extend the features provided by Intel SGX. Therefore, while firmware available for SGX is indeed open source, it is not specifically designed with extensibility for SGX in mind.
As mentioned earlier, extensible systems as described in Table 6.3 may lack vendor-supplied support options. While Intel systems may seem like the more ridged of the three options, Intel aims to provide a high level of support for their large cloud and data center customers. Providing adequate support to those customers is a delicate balance between the ability to customize Intel systems and the need to provide uniformly high quality support to those customers. One can certainly build a RISC-V system which is completely customized from the HDL to the software SDK, but the responsibility of supporting that system will lie with the entity which customized it or the open source community built up around that system.

Intel SGX
Arm TrustZone RISC-V PMP    Proprietary solutions are extensible only if modifications are supported by the vendor, as is the case with Arm and "extensible HDL". Certainly few if any Arm vendors will allow you to modify their HDL directly, however Arm's IP model does allow for some inherent modification by allowing one to pick and choose which IP they wish to use. Note that with Intel SGX, the concept of open source firmware will only allow limited extensibility to the functionality of the TEE. TEE Technologies are colored in green while properties of the TEE are colored in yellow . These data points are examples to the process of comparison and should not be considered as a valid or current source of information.

Considerations and Limitations
In this comparison we have not distinguished between platform manufacturers and platforms vendors. As an example, Supermicro is a platform manufacturer which might produce a platform based on an Intel Xeon Processor currently code named "Ice Lake-SP" which will support SGX on multi-socket systems. They might then sell those platforms to a vendor like Hewlett Packard or Dell. How Intel SGX will function on these platforms becomes a combination of the instructions Intel has added to the architecture, the workflow inherent in the microarchitecture, how the chipset is assembled, how Supermicro assembled that chipset into a platform, and how HP or Dell supports that platform with firmware, software, and possibly remote attestation services. The same is true of Arm systems and RISC-V systems.
One quickly sees the complexity of securing systems in the real world. Our method for comparison is meant only as a high level tool for assessing the appropriateness of a TEE technology to a use case.
The method we have described will also be limited by the scope of the use case.
The complexity of the comparison will grow in a linear manner to the complexity of the use case. One can quickly postulate that a multi-party system based on heterogeneous TEEs will reach levels of complexity such that this method begins to break down, or at the very least requires much more rigor than the simple comparisons shown here. This method is submitted not as a golden standard for comparison, but rather as a baseline on which further research may provide avenues for comparison not imagined in this thesis.

Conclusion
This thesis has described a method that allows for a comprehensive and rigorous comparison of Trusted Execution Environments (TEEs). This method involves identifying the properties of interest for hardware architects, firmware authors, and TEE software designers alike. Key to our comparison of the implementations of those properties is breaking each property down into relevant constituent parts.
The relevancy of each property will depend on the use case of the designer, and should be taken into account during comparison. This method does not provide us with which technology is the "best," but rather gives us insight into which TEE might best fit a specific set of needs.
There are several use cases for systems requiring a TEE that this thesis does not take into account. These use cases could be the subject of future work in improving this methodology. Multi-socket processors [34] add another layer of complexity to TEEs and should be considered for cloud compute or data center use cases. Likewise systems might require multiple TEEs or heterogeneous TEE environments, perhaps for enhanced security use cases. As an example, a system running both TrustZone alongside RISC-V Physical Memory Protection (PMP) may require communication between these different TEEs to assure secure computation. Examining the details of how these systems are able to interact and communicate allows for new avenues of comparison. Future work covering these use cases would add to the rigorous nature of this method.
This proposed method will save hardware architects the frustration that comes with a wide range of technology choices. The method provides an organizational framework for characterizing properties of a TEE and provides value to product designers who must focus on possible use cases ranging from mobile devices to cloud services. It is possible that this method may be insufficient for some future version of a TEE, as evidenced by how new technologies like RISC-V Physical Memory Protection (PMP) are both mimicking older technologies as well as developing new and innovative features. However, as the use of TEEs becomes more common and these technologies take on new unique properties, it is likely that these new properties will be easily integrated into this method of comparison.