This is a more technical blob post about a specific aspect of the ESP32 Wi-Fi hardware; see previous blog posts for a more general overview.
In one of the previous blog posts, we talked about implementing station mode on the ESP32. We then only implemented connecting to open networks, since there is a bunch of extra work that goes into connecting to WPA protected networks. In this blog post, we’ll go into what needs to be implemented to connect WPA2 networks, and how the ESP32 hardware plays into that.
Protected networks: general structure Link to heading
To use a WPA2 protected network, you basically have to do three things: basically consists of 3 parts:
- key derivation / 4 way handshake (when connecting)
- encrypting/decrypting packets (for every packet sent/received)
- keeping the keys correct at runtime (every n minu)
Key derivation Link to heading
- The Pairwise Master Key (PMK) is generated from the passphrase and SSID (
PMK = PBKDF2(HMAC−SHA1, passphrase, ssid, 4096, 256)) - The station and AP then do a 4-way handshake to derive and install a Pairwise Temporal Key between the AP and the station. This key encrypts all unicast traffic
- The 4 way handshake also provides the Group Temporal Key to the client. This key is used to encrypt multicast traffic. They are rekeyed every time a STA joins or leave, or every n minutes, depending on AP settings.
Hardware acceleration of encrypting/decrypting packets Link to heading
Every protected packet we send needs to be encrypted, and every protected packet we receive needs to be decrypted. If the ESP32 would need to do this in software, it would cost a lot of CPU time; this is why it is implemented in hardware. Instead of having to encrypt a packet before handing the encrypted result over to the radio hardware, we can just hand the plaintext packet to the hardware and tell it which “crypto key slot” index to use.
The ESP32 has 25 crypto key slots. Every crypto key slot contains the nescessary information to encrypt and decrypt packets:
- the cipher suite (CCMP, GCMP, …)
- the virtual interface index on which the packet will be decrypted (see also the previous blog post on the RX filter; which is basically the VIF)
- the key ID
- the MAC address of the peer who will send encrypted packets to us
- the key
- (some extra bits relating to protected management frames and Signaling and Payload Protected A-MSDU’s)
Encryption flow Link to heading
The encryption flow works as follows:
- Before sending any packets, set up the crypto slot with the correct information
- To have the hardware send an encrypted packet: prepare the packet as for unencrypted data, but with some exceptions:
- set the ‘protected’ bit in the frame control field
- after the 802.11 MAC header, but before the data, construct and insert the 8 byte CCMP header (contains the nonce and key id)
- add 8 bytes to the end of the packet: the hardware will overwrite these to insert the MIC (message integrity checksum)
- Send the packet as normally, but indicate which crypto key slot that the hardware should use in the PLCP1 register
It should be noted that there are a lot of algorithms that the hardware has support for:
enum wpa_alg {
WIFI_WPA_ALG_NONE = 0,
WIFI_WPA_ALG_WEP40 = 1,
WIFI_WPA_ALG_TKIP = 2,
WIFI_WPA_ALG_CCMP = 3,
WIFI_WAPI_ALG_SMS4 = 4,
WIFI_WPA_ALG_WEP104 = 5,
WIFI_WPA_ALG_WEP = 6,
WIFI_WPA_ALG_IGTK = 7,
WIFI_WPA_ALG_PMK = 8,
WIFI_WPA_ALG_GCMP = 9,
};
It’s neat to see that they have support for SMS4, a Chinese cipher that was suggested for 802.11i, but was ultimately rejected. It lives on in WAPI.
Decryption flow Link to heading
The decryption flow is transparent for the hardware: once you set up the key slot with the correct interface and MAC address, the hardware will automatically decrypt the packet. This is done based on the MAC address in the key slot and in the frame. There is a bit in the key slot that indicates whether the keyslot is for unicast or multicast frames.
Unicast: RA (addr 1) of the encrypted frame is the MAC address of module
For unicast slots, a packet will decrypt if the MAC address in the key slot matches the TA (address 2) of a frame.
Multicast: RA (addr 1) of the encrypted frame is a multicast address
Here, it does not really seem to matter what the address in the key slot is; decryption will happen as long as the RA is multicast and the packet gets through the filters
As a recap:
| hw key idx 0 | To DS | From DS | reception if addr 2 matches BSSID in RX filter | reception if addr 3 matches BSSID in RX filter | notes |
|---|---|---|---|---|---|
| 0 | 0 | no | yes | ||
| 0 | 1 | yes | no | ||
| 1 | 0 | yes | yes | decryption works even works when addr2 and addr3 both don’t match the addr in the keyslot! | |
| 1 | 1 | yes | yes | decryption even works when addr2, addr3 and addr4 both don’t match the addr in the keyslot! |
Testing this Link to heading
To reverse engineer this part of the hardware, I wrote a Python script that generates example plaintexts and ciphertexts, also called ’test vectors’ in the cryptography world. However, this had a bit of a chicken and egg problem: how do we know that our Python implementation to generate test vectors is correct? Luckily for us, there are some public test vectors: the FreeBSD net80211 regression tests contain 8 CCMP test vectors.
Unfortunately, none of the 8 test vectors contain a test case where there is a ‘4 address frame’ (my term for a packet that has both from-DS and to-DS set in its Frame Control field, and as such has 4 MAC addresses). To back up a bit and explain what a ‘4 address frame’ is: normally, Wi-Fi frames contain 3 MAC addresses. This is enough for a transmitter, receiver and an extra address for either the destination, source or BSSID. This is sufficient for the case where you have access points and stations. However, there is a special case (where a packet is both going to and coming from the distribution network; most often the case in mesh networks) where there are 4 addresses in a 802.11 frame.
This 4th address, if present, is used in the encryption algorithm (more specifically, in calculating the AAD); so is critical to handle this correctly to correctly encrypt/decrypt ‘4 address frames’. I was worried that the hardware might not do this correctly since:
- it is likely a tiny bit cheaper with regards to amount of gates used to not handle this special case
- Espressif does not send any ‘4 address frames’ as far as I know, let alone encrypted ‘4 address frames’
- ‘4 address frames’ seem to be pretty uncommon
Using the 802.11 standard, I implemented this special case in the Python test program. This was then validated by generating a packet and decrypting it with Wireshark. I then tested encryption on the ESP32, and to my relief, the hardware does handle this correctly. The decryption is also correctly implemented. Good job Espressif!
Appendix: demonstation code Link to heading
See https://github.com/esp32-open-mac/esp32-open-mac/pull/24. Note that this still uses the Espressif HAL (wDev_Insert_KeyEntry); implementing the HAL ourselves and doing the 4 way handshake will be done in another PR.
Appendix: Python test vector script: Link to heading
from scapy.layers.dot11 import Dot11, Dot11QoS
from scapy.all import hexdump, wrpcap, sendp, RadioTap
from Crypto.Cipher import AES
import struct
def bytes_to_c_arr(data, lowercase=True):
return [format(b, '#04x' if lowercase else '#04X') for b in data]
TESTCASE = 11
transmit_if = 'wlan2mon'
# These testcases come from freebsd/head/tools/regression/net80211/ccmp/test_ccmp.c
if TESTCASE == 1:
plain = bytes([
0x08, 0x48, 0xc3, 0x2c, 0x0f, 0xd2, 0xe1, 0x28, # /* 802.11 Header */
0xa5, 0x7c, 0x50, 0x30, 0xf1, 0x84, 0x44, 0x08,
0xab, 0xae, 0xa5, 0xb8, 0xfc, 0xba, 0x80, 0x33,
0xf8, 0xba, 0x1a, 0x55, 0xd0, 0x2f, 0x85, 0xae, # /* Plaintext Data */
0x96, 0x7b, 0xb6, 0x2f, 0xb6, 0xcd, 0xa8, 0xeb,
0x7e, 0x78, 0xa0, 0x50,
])
pn = 0xB5039776E70C
key = bytes.fromhex("c9 7c 1f 67 ce 37 11 85 51 4a 8a 19 f2 bd d5 2f")
key_id = 0
expected = bytes([
0x08, 0x48, 0xc3, 0x2c, 0x0f, 0xd2, 0xe1, 0x28,
0xa5, 0x7c, 0x50, 0x30, 0xf1, 0x84, 0x44, 0x08,
0xab, 0xae, 0xa5, 0xb8, 0xfc, 0xba, 0x80, 0x33,
0x0c, 0xe7, 0x00, 0x20, 0x76, 0x97, 0x03, 0xb5,
0xf3, 0xd0, 0xa2, 0xfe, 0x9a, 0x3d, 0xbf, 0x23,
0x42, 0xa6, 0x43, 0xe4, 0x32, 0x46, 0xe8, 0x0c,
0x3c, 0x04, 0xd0, 0x19, 0x78, 0x45, 0xce, 0x0b,
0x16, 0xf9, 0x76, 0x23
])
elif TESTCASE == 4:
plain = bytes([
0xa8, 0xca, 0x3a, 0x11, 0x71, 0x2a, 0x9d, 0xdf, 0x11, 0xdb,
0x8e, 0xf8, 0x22, 0x73, 0x47, 0x01, 0x59, 0x14, 0x0d, 0xd6,
0x46, 0xa2, 0xc0, 0x2f, 0x67, 0xa5,
0x4f, 0xad, 0x2b, 0x1c, 0x29, 0x0f, 0xa5, 0xeb, 0xd8, 0x72,
0xfb, 0xc3, 0xf3, 0xa0, 0x74, 0x89, 0x8f, 0x8b, 0x2f, 0xbb,
])
pn = 0xF670A55A0FE3
key = bytes.fromhex('8c 89 a2 eb c9 6c 76 02 70 7f cf 24 b3 2d 38 33')
key_id = 0
expected = bytes([
0xa8, 0xca, 0x3a, 0x11, 0x71, 0x2a, 0x9d, 0xdf, 0x11, 0xdb,
0x8e, 0xf8, 0x22, 0x73, 0x47, 0x01, 0x59, 0x14, 0x0d, 0xd6,
0x46, 0xa2, 0xc0, 0x2f, 0x67, 0xa5, 0xe3, 0x0f, 0x00, 0x20,
0x5a, 0xa5, 0x70, 0xf6, 0x9d, 0x59, 0xb1, 0x5f, 0x37, 0x14,
0x48, 0xc2, 0x30, 0xf4, 0xd7, 0x39, 0x05, 0x2e, 0x13, 0xab,
0x3b, 0x1a, 0x7b, 0x10, 0x31, 0xfc, 0x88, 0x00, 0x4f, 0x35,
0xee, 0x3d,
])
elif TESTCASE == 7:
plain = bytes([
0x18, 0x79, 0x81, 0x46, 0x9b, 0x50, 0xf4, 0xfd, 0x56, 0xf6,
0xef, 0xec, 0x95, 0x20, 0x16, 0x91, 0x83, 0x57, 0x0c, 0x4c,
0xcd, 0xee, 0x20, 0xa0,
0x98, 0xbe, 0xca, 0x86, 0xf4, 0xb3, 0x8d, 0xa2, 0x0c, 0xfd,
0xf2, 0x47, 0x24, 0xc5, 0x8e, 0xb8, 0x35, 0x66, 0x53, 0x39,
])
pn = 0x5EEC4073E723
key = bytes.fromhex('1b db 34 98 0e 03 81 24 a1 db 1a 89 2b ec 36 6a')
key_id = 3
expected = bytes([
0x18, 0x79, 0x81, 0x46, 0x9b, 0x50, 0xf4, 0xfd, 0x56, 0xf6,
0xef, 0xec, 0x95, 0x20, 0x16, 0x91, 0x83, 0x57, 0x0c, 0x4c,
0xcd, 0xee, 0x20, 0xa0, 0x23, 0xe7, 0x00, 0xe0, 0x73, 0x40,
0xec, 0x5e, 0x12, 0xc5, 0x37, 0xeb, 0xf3, 0xab, 0x58, 0x4e,
0xf1, 0xfe, 0xf9, 0xa1, 0xf3, 0x54, 0x7a, 0x8c, 0x13, 0xb3,
0x22, 0x5a, 0x2d, 0x09, 0x57, 0xec, 0xfa, 0xbe, 0x95, 0xb9,
])
elif TESTCASE == 10: # custom testcase for 4 address mode
plain = bytes([
0x08, 0x48, 0xc3, 0x2c, 0x0f, 0xd2, 0xe1, 0x28, # /* 802.11 Header */
0xa5, 0x7c, 0x50, 0x30, 0xf1, 0x84, 0x44, 0x08,
0xab, 0xae, 0xa5, 0xb8, 0xfc, 0xba, 0x80, 0x33,
0xf8, 0xba, 0x1a, 0x55, 0xd0, 0x2f, 0x85, 0xae, # /* Plaintext Data */
0x96, 0x7b, 0xb6, 0x2f, 0xb6, 0xcd, 0xa8, 0xeb,
0x7e, 0x78, 0xa0, 0x50,
])
pn = 0xB5039776E70C
key = bytes.fromhex("c9 7c 1f 67 ce 37 11 85 51 4a 8a 19 f2 bd d5 2f")
key_id = 0
parsed = Dot11(plain)
setattr(parsed.FCfield, 'to-DS', True)
setattr(parsed.FCfield, 'from-DS', True)
parsed.addr4 = '00:23:45:67:89:ab'
plain = parsed.build()
expected = None
elif TESTCASE == 11:
# encrypt with custom mac addr
plain = bytes([
0x08, 0x48, 0xc3, 0x2c, 0x0f, 0xd2, 0xe1, 0x28, # /* 802.11 Header */
0xa5, 0x7c, 0x50, 0x30, 0xf1, 0x84, 0x44, 0x08,
0xab, 0xae, 0xa5, 0xb8, 0xfc, 0xba, 0x80, 0x33,
0xf8, 0xba, 0x1a, 0x55, 0xd0, 0x2f, 0x85, 0xae, # /* Plaintext Data */
0x96, 0x7b, 0xb6, 0x2f, 0xb6, 0xcd, 0xa8, 0xeb,
0x7e, 0x78, 0xa0, 0x50,
])
pn = 0
key = bytes.fromhex("c9 7c 1f 67 ce 37 11 85 51 4a 8a 19 f2 bd d5 2f")
key_id = 0
parsed = Dot11(plain)
broadcast = 'ff:ff:ff:ff:ff:ff'
ra = '00:23:45:67:89:ab'
bssid = 'f0:ae:a5:b8:fc:ba'
unrelated = 'ae:25:aa:63:1c:8e'
setattr(parsed.FCfield, 'from-DS', False)
setattr(parsed.FCfield, 'to-DS', True)
parsed.addr1 = broadcast
parsed.addr2 = unrelated
parsed.addr3 = unrelated
# parsed.addr4 = unrelated
# parsed.addr4 = bssid
parsed.FCfield.retry = False
plain = parsed.build()
expected = None
else:
assert False, 'Testcase not found'
dot11 = Dot11(plain)
print(dot11)
# Sanity check on scapy library
assert (dot11.build() == plain)
assert dot11.proto == 0, "Only PV0 supported, no 802.11ah"
dot11_copy = dot11.copy()
mac_1 = bytes.fromhex(dot11.addr1.replace(':', ''))
mac_2 = bytes.fromhex(dot11.addr2.replace(':', ''))
mac_3 = bytes.fromhex(dot11.addr3.replace(':', ''))
mac_4 = None if dot11.addr4 is None else bytes.fromhex(dot11.addr4.replace(':', ''))
plaintext_data = dot11.payload.build()
print("Original packet:")
hexdump(dot11)
# mask out the bits that should be masked out
dot11.subtype &= 0b1000
dot11.FCfield.retry = False
setattr(dot11.FCfield, 'pw-mgt', False)
dot11.FCfield.MD = False
dot11.FCfield.protected = 1
assert Dot11QoS not in dot11, "QoS not implemented; see page 2493 (CCMP AAD) of 80211-2020.pdf"
frame = dot11.build()
# this bytes.fromhex("00 00") assumes the fragment number is 0
assert dot11.SC & 0b1111 == 0, "Fragment number != 0 not implemented"
aad = frame[:2] + mac_1 + mac_2 + mac_3 + bytes.fromhex("00 00") + (bytes() if mac_4 is None else mac_4)
print('AAD:')
hexdump(aad)
pn_packed = struct.pack("<Q", pn)[:6]
print("Packed PN:")
hexdump(pn_packed)
ccmp_header = bytes([
pn_packed[0],
pn_packed[1],
0, # reserved
(1<<5 | key_id << 6),
pn_packed[2],
pn_packed[3],
pn_packed[4],
pn_packed[5],
])
print("CCMP header:")
hexdump(ccmp_header)
# bytes.fromhex("00") assumes [Priority Management PV1 Zeros] are all zero
nonce = bytes.fromhex("00") + mac_2 + struct.pack(">Q", pn)[-6:]
print("CCM nonce")
hexdump(nonce)
cipher = AES.new(key, AES.MODE_CCM, nonce, mac_len=8)
cipher.update(aad)
msg = nonce, aad, cipher.encrypt(plaintext_data), cipher.digest()
calculated = dot11_copy.build()[:(24 if dot11.addr4 is None else 24+6)] + ccmp_header + msg[2] + msg[3]
print("on ESP32:")
print(', '.join(bytes_to_c_arr(dot11_copy.build()[:(24 if dot11.addr4 is None else 24+6)] + ccmp_header + plaintext_data + bytes([0]*8))))
if transmit_if is not None:
dot11 = Dot11(calculated)
dot11.FCfield.protected = True
sendp(RadioTap() / dot11, iface=transmit_if)
if expected is None:
# wrpcap("out.pcap", [Dot11(calculated)])
pass
else:
if calculated == expected:
print("Calculated frame matches expectation")
hexdump(calculated)
else:
print("Calculated does not match expected")
print("Calculated:")
hexdump(calculated)
print("Expected")
hexdump(expected)