Enhanced Instruction Set Randomization Design Space Exploration M. Tarek Ibn Ziad Columbia University mtarek@cs.columbia.edu Simha Sethumadhavan, Columbia University, simha@cs.columbia.edu Instruction Set Randomization: Instruction set randomization (ISR) was proposed early in the last decade as a countermeasure against code injection attacks. It provides illusion of a secret instruction set. However, prior ISR schemes are ineffective against code-reuse attacks. In our previous work, Polyglot [1], we presented the design of a hardware-based ISR scheme, which is effective against code-reuse attacks, and even counter state-of-the-art variants, such as “just-in-time” ROP (JIT-ROP). Polyglot creates an “ISRized” binary by symmetrically encrypting (with AES) a diversified version of it, at page granularity, with randomly generated keys. These key-to-address mappings are then asymmetrically encrypted (with ECC) using the target processor’s public key and packaged into the binary itself. Since code is encrypted at a page granularity, the executable, and its required shared libraries, possibly encrypted by different sources, are able to interoperate. Lastly, asymmetric encryption ties the binaries to their respective hosts. To accommodate per-page encryption, Polyglot introduces a new type of page table entry for randomized (i.e., ISR- encrypted) pages. On an instruction page fault, the page walk mechanism procures encrypted entry, decrypts it to obtain the page key and translation, which are then deposited into a modified ITLB. ECC-163 and SHA-256 accelerators are added to the MMU to carry out the decryption according to the Elliptic Curve Integrated Encrypted Scheme. On an I-cache miss, as instructions are fetched from memory, they are decrypted using the page’s key and stored, in plaintext, in the I-cache. Henceforth, as long as an instruction is not evicted, execution uses its decrypted form. Moreover, Polyglot employs code randomization to prevent predictable code layout and hence code reuse attacks with low performance overheads. In this project, we analyzed how Polyglot can be extended for Heterogeneous System Architectures (HSA). 2. Heterogeneous System Architectures (HSA): HSA may include (1) multicore processors with same ISA and microarchitecture, (2) multicore processors with same ISA and different microarchitecture (e.g., arm big.LITTLE SoC), (3) multicore processors with different ISA and shared virtual memory (e.g., CPU with GPU), or (4) multicore processors with different ISA and disjoint virtual memory (e.g., CPU with discrete GPU). The above categories can have one of the memory sharing schemes shown below.