TLS/SSL Handshake Explained

TL;DR: TLS encrypts communication between client and server. Understanding the handshake helps you identify weak configurations, test certificate validation, and exploit cryptographic vulnerabilities.

Table of Contents

Quick Reference

TLS/SSL Versions

Essential Commands

Common Ports Using TLS

Why TLS Matters for Pentesters

TLS is everywhere - and misconfigurations are common.

What TLS Protects

Real-World TLS Breaches

What Pentesters Test

1. Protocol versions - Deprecated SSL/TLS versions
2. Cipher suites - Weak or export ciphers
3. Certificate validation - Expiry, chain, hostname
4. Known vulnerabilities - BEAST, POODLE, Heartbleed
5. Configuration - HSTS, certificate pinning
6. Implementation bugs - Specific library issues

TLS vs SSL: What’s the Difference

TL;DR: SSL is the old, broken protocol. TLS is the modern replacement. People say “SSL” but usually mean TLS.

History

1995 ── SSL 2.0 ── Netscape (broken, deprecated)
     │
1996 ── SSL 3.0 ── Netscape (POODLE, deprecated)
     │
1999 ── TLS 1.0 ── IETF standardization (BEAST, deprecated)
     │
2006 ── TLS 1.1 ── Minor improvements (deprecated)
     │
2008 ── TLS 1.2 ── Major improvements (current)
     │
2018 ── TLS 1.3 ── Complete redesign (recommended)

Why It Matters

Finding deprecated protocols = finding for your pentest report.

The TLS Handshake

TL;DR: Client and server negotiate encryption, verify certificates, and establish a secure session in a carefully orchestrated exchange.

TLS 1.2 Handshake (Full)

┌─────────┐                                              ┌─────────┐
│ Client  │                                              │ Server  │
└────┬────┘                                              └────┬────┘
     │                                                        │
     │  ──────────── 1. ClientHello ─────────────────────►   │
     │     - TLS version                                      │
     │     - Random bytes (Client Random)                     │
     │     - Session ID                                       │
     │     - Cipher suites supported                          │
     │     - Compression methods                              │
     │     - Extensions (SNI, etc.)                           │
     │                                                        │
     │  ◄──────────── 2. ServerHello ────────────────────    │
     │     - TLS version selected                             │
     │     - Random bytes (Server Random)                     │
     │     - Session ID                                       │
     │     - Cipher suite selected                            │
     │     - Compression method                               │
     │                                                        │
     │  ◄──────────── 3. Certificate ────────────────────    │
     │     - Server's certificate chain                       │
     │                                                        │
     │  ◄──────────── 4. ServerKeyExchange ──────────────    │
     │     - Key exchange parameters (DHE/ECDHE)              │
     │     - Signature                                        │
     │                                                        │
     │  ◄──────────── 5. ServerHelloDone ────────────────    │
     │                                                        │
     │  ──────────── 6. ClientKeyExchange ───────────────►   │
     │     - Pre-master secret (encrypted)                    │
     │       or DH public value                               │
     │                                                        │
     │  ──────────── 7. ChangeCipherSpec ────────────────►   │
     │     - Switching to encrypted communication             │
     │                                                        │
     │  ──────────── 8. Finished ────────────────────────►   │
     │     - Encrypted verification                           │
     │                                                        │
     │  ◄──────────── 9. ChangeCipherSpec ───────────────    │
     │                                                        │
     │  ◄──────────── 10. Finished ──────────────────────    │
     │                                                        │
     │  ════════════ Encrypted Application Data ══════════   │
     │                                                        │

Key Generation Process

                    Client Random (32 bytes)
                           +
                    Server Random (32 bytes)
                           +
                    Pre-Master Secret
                           │
                           ▼
                   ┌───────────────┐
                   │  PRF (KDF)    │  Pseudo-Random Function
                   └───────┬───────┘
                           │
           ┌───────────────┼───────────────┐
           │               │               │
           ▼               ▼               ▼
    ┌────────────┐  ┌────────────┐  ┌────────────┐
    │ Client     │  │ Server     │  │ IV for     │
    │ Write Key  │  │ Write Key  │  │ CBC mode   │
    └────────────┘  └────────────┘  └────────────┘

Vulnerabilities in Each Step

TLS 1.3 Handshake

TL;DR: TLS 1.3 is faster (1 round trip), more secure (removed weak ciphers), and simpler (fewer options to misconfigure).

TLS 1.3 vs 1.2

TLS 1.3 Handshake

┌─────────┐                                              ┌─────────┐
│ Client  │                                              │ Server  │
└────┬────┘                                              └────┬────┘
     │                                                        │
     │  ──────────── ClientHello ────────────────────────►   │
     │     - Supported versions                               │
     │     - Key shares (ECDHE)                               │
     │     - Cipher suites                                    │
     │                                                        │
     │  ◄──────────── ServerHello ───────────────────────    │
     │     - Selected version                                 │
     │     - Key share                                        │
     │                                                        │
     │  ◄──────────── EncryptedExtensions ───────────────    │
     │  ◄──────────── Certificate ───────────────────────    │
     │  ◄──────────── CertificateVerify ─────────────────    │
     │  ◄──────────── Finished ──────────────────────────    │
     │                                                        │
     │  ──────────── Finished ───────────────────────────►   │
     │                                                        │
     │  ══════════════ Application Data ══════════════════   │

What TLS 1.3 Removed

Certificates and PKI

TL;DR: Certificates prove server identity. The chain of trust goes from server cert → intermediate CA → root CA (trusted by your browser).

Certificate Chain

┌─────────────────────────────────────────────────────────────────┐
│                        Root CA Certificate                       │
│              (Self-signed, in browser/OS trust store)            │
│                      DigiCert, Let's Encrypt, etc.               │
└───────────────────────────────┬─────────────────────────────────┘
                                │ Signs
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Intermediate CA Certificate                    │
│              (Signed by Root, signs end-entity certs)            │
└───────────────────────────────┬─────────────────────────────────┘
                                │ Signs
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                   End-Entity Certificate                         │
│                    (Server's certificate)                        │
│              Subject: CN=www.example.com                         │
└─────────────────────────────────────────────────────────────────┘

Certificate Fields

Certificate Validation Checks

# View certificate with openssl
openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | \
    openssl x509 -noout -text

# View certificate dates
openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | \
    openssl x509 -noout -dates

# View certificate chain
openssl s_client -connect example.com:443 -servername example.com -showcerts

# Check certificate SANs
openssl s_client -connect example.com:443 2>/dev/null | \
    openssl x509 -noout -ext subjectAltName

# View certificate with curl
curl -vI https://example.com 2>&1 | grep -A6 "Server certificate"

# Check certificate expiry
echo | openssl s_client -connect example.com:443 2>/dev/null | \
    openssl x509 -noout -enddate

Cipher Suites Explained

TL;DR: A cipher suite is a combination of algorithms for key exchange, authentication, encryption, and message integrity.

Cipher Suite Naming

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
│    │     │        │   │   │    │
│    │     │        │   │   │    └── Hash for PRF
│    │     │        │   │   └─────── Mode (GCM = AEAD)
│    │     │        │   └─────────── Key size
│    │     │        └─────────────── Encryption algorithm
│    │     └──────────────────────── Authentication (cert type)
│    └────────────────────────────── Key exchange
└─────────────────────────────────── Protocol

Cipher Suite Components

TLS 1.3 Cipher Suites (Only 5)

Weak Cipher Indicators

# List supported ciphers with openssl
openssl s_client -connect example.com:443 -cipher 'ALL:COMPLEMENTOFALL'

# Check specific cipher
openssl s_client -connect example.com:443 -cipher 'AES256-SHA'

# List cipher suites with nmap
nmap --script ssl-enum-ciphers -p 443 example.com

# Using sslscan
sslscan example.com

# Using testssl.sh
testssl.sh --cipher-per-proto example.com

Common TLS Vulnerabilities

1. Heartbleed (CVE-2014-0160)

OpenSSL memory leak exposing server memory, potentially including private keys.

┌─────────┐                                     ┌─────────┐
│ Client  │  Heartbeat: "Send back 64KB"       │ Server  │
│         │ ────────────────────────────────►  │         │
│         │  (But I only sent 1 byte)          │         │
│         │                                     │         │
│         │  ◄───────────────────────────────  │         │
│         │  Response: 64KB of server memory!  │         │
│         │  (May contain private keys,        │         │
│         │   passwords, session data)         │         │
└─────────┘                                     └─────────┘

# Using nmap
nmap --script ssl-heartbleed -p 443 target.com

# Using testssl.sh
testssl.sh --heartbleed target.com

# Using Metasploit
use auxiliary/scanner/ssl/openssl_heartbleed
set RHOSTS target.com
run

# Manual check with openssl
echo "QUIT" | openssl s_client -connect target.com:443 2>&1 | grep "server extension"
# If "heartbeat" appears, further testing needed

2. POODLE (CVE-2014-3566)

Padding Oracle On Downgraded Legacy Encryption - attacks SSL 3.0 CBC mode.

# Check for SSL 3.0 support
openssl s_client -connect target.com:443 -ssl3

# Using nmap
nmap --script ssl-poodle -p 443 target.com

# Using testssl.sh
testssl.sh --poodle target.com

# Check for TLS POODLE (CVE-2014-8730)
testssl.sh --fallback target.com

3. BEAST (CVE-2011-3389)

Browser Exploit Against SSL/TLS - attacks TLS 1.0 CBC mode.

# Check TLS 1.0 support
openssl s_client -connect target.com:443 -tls1

# Using testssl.sh
testssl.sh --beast target.com

# Check cipher preference
nmap --script ssl-enum-ciphers -p 443 target.com | grep -i "cbc"

4. CRIME & BREACH

Compression-based attacks that leak encrypted data.

CRIME - TLS compression leaks session tokens BREACH - HTTP compression leaks response body secrets

# Check for TLS compression (CRIME)
openssl s_client -connect target.com:443 2>/dev/null | grep "Compression"
# "Compression: NONE" = Safe
# "Compression: zlib" = Vulnerable

# Using testssl.sh
testssl.sh --crime target.com

# Check HTTP compression (BREACH)
curl -H "Accept-Encoding: gzip, deflate" -I https://target.com
# If Content-Encoding: gzip appears on sensitive pages = potentially vulnerable

5. DROWN (CVE-2016-0800)

SSLv2 enables cross-protocol attack to decrypt TLS sessions.

# Check for SSLv2
openssl s_client -connect target.com:443 -ssl2

# Using nmap
nmap --script sslv2-drown -p 443 target.com

# Using testssl.sh
testssl.sh --drown target.com

6. Logjam (CVE-2015-4000)

Weak Diffie-Hellman parameters allow MITM attacks.

# Check DH parameters
openssl s_client -connect target.com:443 -cipher "EDH" 2>/dev/null | grep "Server Temp Key"

# Using nmap
nmap --script ssl-dh-params -p 443 target.com

# Using testssl.sh
testssl.sh --logjam target.com

7. FREAK (CVE-2015-0204)

Factoring RSA Export Keys - downgrade to 512-bit RSA.

# Check for EXPORT ciphers
openssl s_client -connect target.com:443 -cipher EXPORT

# Using testssl.sh
testssl.sh --freak target.com

8. ROBOT (CVE-2017-13099)

Return Of Bleichenbacher’s Oracle Threat - RSA key exchange attack.

# Using testssl.sh
testssl.sh --robot target.com

# Using robot-detect tool
python robot-detect.py -h target.com -p 443

Vulnerability Summary

Testing TLS Configuration

Testing Checklist

• Check supported TLS versions (disable < TLS 1.2)
• Enumerate cipher suites (remove weak ciphers)
• Verify certificate validity (dates, chain, hostname)
• Test for known vulnerabilities (Heartbleed, POODLE, etc.)
• Check for certificate pinning bypass
• Verify forward secrecy (ECDHE/DHE)
• Test HSTS configuration
• Check OCSP stapling
• Verify certificate transparency

# === Protocol Version Testing ===

# Test each protocol version
openssl s_client -connect target.com:443 -ssl3    # Should fail
openssl s_client -connect target.com:443 -tls1    # Should fail
openssl s_client -connect target.com:443 -tls1_1  # Should fail
openssl s_client -connect target.com:443 -tls1_2  # Should work
openssl s_client -connect target.com:443 -tls1_3  # Should work

# === Cipher Suite Testing ===

# Enumerate all ciphers
nmap --script ssl-enum-ciphers -p 443 target.com

# Check for weak ciphers
openssl s_client -connect target.com:443 -cipher 'NULL:EXPORT:LOW:DES:RC4:MD5:ANON'
# Should fail if properly configured

# Check cipher preference
testssl.sh --server-preference target.com

# === Certificate Testing ===

# View full certificate details
echo | openssl s_client -connect target.com:443 2>/dev/null | \
    openssl x509 -text -noout

# Check certificate expiry
echo | openssl s_client -connect target.com:443 2>/dev/null | \
    openssl x509 -dates -noout

# Verify certificate chain
openssl s_client -connect target.com:443 -verify 5

# Check hostname match
openssl s_client -connect target.com:443 -verify_hostname target.com

# === Vulnerability Testing ===

# Comprehensive test with testssl.sh
testssl.sh target.com

# Quick vulnerability scan
testssl.sh --vulnerable target.com

# Specific checks
testssl.sh --heartbleed --poodle --beast --logjam --drown target.com

# === Header Testing ===

# Check HSTS
curl -sI https://target.com | grep -i "strict-transport"

# Check all security headers
curl -sI https://target.com | grep -iE "(strict-transport|x-frame|x-content|x-xss|content-security)"

Certificate Attacks

1. Self-Signed Certificate Acceptance

If client accepts self-signed certs, MITM is trivial.

# Generate self-signed certificate
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes \
    -subj "/CN=target.com"

# Test with mitmproxy using self-signed
mitmproxy --cert cert.pem

# Check if client accepts invalid cert
# If app doesn't error on self-signed = vulnerable

2. Certificate Pinning Bypass

Mobile apps often pin certificates. Bypassing enables MITM.

# Using Frida with objection
objection --gadget com.target.app explore
# Then: android sslpinning disable

# Using Frida script
frida -U -f com.target.app -l ssl-unpin.js

# Using apktool + modification
apktool d app.apk
# Modify network_security_config.xml
apktool b app -o app-modified.apk

3. Hostname Verification Bypass

If client doesn’t verify hostname matches certificate.

MITM and TLS Interception

TLS Interception Setup

┌─────────┐                    ┌──────────┐                    ┌─────────┐
│ Client  │ ◄─── TLS 1 ──────►│  Proxy   │◄───── TLS 2 ──────►│ Server  │
│         │   (Proxy's cert)   │  (MITM)  │   (Server's cert)  │         │
└─────────┘                    └──────────┘                    └─────────┘

Tools for TLS Interception

# === mitmproxy Setup ===

# Start mitmproxy
mitmproxy

# Install CA cert on client (from http://mitm.it when connected)

# Transparent mode
mitmproxy --mode transparent

# === Burp Suite Setup ===

# 1. Start Burp
# 2. Configure browser proxy: 127.0.0.1:8080
# 3. Export CA: Proxy > Options > CA Certificate > Export
# 4. Install CA in browser/system

# === sslsplit Setup ===

# Generate CA
openssl genrsa -out ca.key 2048
openssl req -new -x509 -key ca.key -out ca.crt -days 365

# Run sslsplit
sslsplit -D -l connections.log -j /tmp/sslsplit -S /tmp/sslsplit/logs \
    -k ca.key -c ca.crt ssl 0.0.0.0 8443

# Redirect traffic (iptables)
iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 8443

Tools for TLS Testing

Command Line Tools

Online Tools

Browser Extensions

# === testssl.sh ===

# Full test
./testssl.sh target.com

# Quick check
./testssl.sh --fast target.com

# Specific checks
./testssl.sh --protocols --ciphers target.com

# JSON output
./testssl.sh --jsonfile results.json target.com

# === sslscan ===

# Basic scan
sslscan target.com

# Show certificate
sslscan --show-certificate target.com

# Check specific port
sslscan --port=8443 target.com

# === sslyze ===

# Regular scan
sslyze target.com

# All checks
sslyze --regular target.com

# JSON output
sslyze --json_out=results.json target.com

# === nmap ===

# Enumerate ciphers
nmap --script ssl-enum-ciphers -p 443 target.com

# All SSL scripts
nmap --script "ssl-*" -p 443 target.com

# Specific vulnerability
nmap --script ssl-heartbleed -p 443 target.com

Practice Labs

Beginner

Intermediate

Advanced

Home Lab Setup

# Create vulnerable TLS server for testing

# 1. Generate certificates
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

# 2. Run server with weak configuration (for testing only!)
openssl s_server -accept 4433 \
    -cert server.crt -key server.key \
    -ssl3 \
    -cipher 'ALL:COMPLEMENTOFALL'

# 3. Test against it
testssl.sh localhost:4433

Glossary

What’s Next?

Now that you understand TLS, continue your learning path:

Summary

TLS/SSL security is critical for penetration testing:

1. Handshake - Understand the negotiation process and where attacks occur
2. Versions - Ensure only TLS 1.2+ is supported
3. Cipher Suites - Remove weak ciphers, prefer AEAD
4. Certificates - Validate chain, expiry, hostname, revocation
5. Vulnerabilities - Test for Heartbleed, POODLE, BEAST, DROWN, etc.
6. MITM - Understand interception for testing and attack scenarios

Always include TLS configuration testing in your engagements. Weak TLS is a common finding that demonstrates real security risk.

Found this guide helpful? Check out the other posts in the SecureKhan penetration testing series.