Encoding vs Encryption vs Hashing

TL;DR: Encoding transforms data format (reversible, no key). Encryption protects data confidentiality (reversible, requires key). Hashing creates a fingerprint (one-way, no reversal). Developers often confuse these - that’s where vulnerabilities live.

Table of Contents

Quick Reference

At a Glance

Common Examples

Quick Identification

Why This Matters for Pentesters

Common Vulnerabilities from Confusion

Real-World Breaches

The Key Differences

Visual Comparison

┌─────────────────────────────────────────────────────────────────────┐
│                           ENCODING                                  │
│                                                                     │
│   "Hello"  ───────►  "SGVsbG8="  ───────►  "Hello"                 │
│              encode               decode                            │
│                                                                     │
│   • Anyone can decode                                               │
│   • No secret/key needed                                            │
│   • NOT for security                                                │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│                          ENCRYPTION                                 │
│                                                                     │
│   "Hello"  ──────────►  "x#$%^&"  ──────────►  "Hello"             │
│           encrypt                  decrypt                          │
│           (with key)               (with key)                       │
│                                                                     │
│   • Only key holder can decrypt                                     │
│   • Key must be kept secret                                         │
│   • FOR confidentiality                                             │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│                            HASHING                                  │
│                                                                     │
│   "Hello"  ──────────►  "185f8db32271..."                          │
│              hash                                                   │
│                                                                     │
│   "Hello"  ──────────►  "185f8db32271..."  (same input = same hash)│
│                                                                     │
│   • Cannot reverse to get "Hello"                                   │
│   • Same input always gives same output                             │
│   • FOR verification/integrity                                      │
└─────────────────────────────────────────────────────────────────────┘

Decision Tree

Do you need to get the original data back?
│
├── NO ─────────► Use HASHING
│                 (passwords, integrity checks)
│
└── YES ──► Does it need to be kept secret?
            │
            ├── NO ─────────► Use ENCODING
            │                 (transport, format conversion)
            │
            └── YES ─────────► Use ENCRYPTION
                              (sensitive data storage/transmission)

Encoding Deep Dive

TL;DR: Encoding converts data to a different format for safe transport. It’s NOT security - anyone can decode it.

Why Encoding Exists

Base64 Encoding

Converts binary to text using 64 characters (A-Z, a-z, 0-9, +, /).

Original: Hello World
Binary:   01001000 01100101 01101100 01101100 01101111 ...
Base64:   SGVsbG8gV29ybGQ=

Characteristics:

• Output is ~33% larger than input
• Uses = padding at end (0-2 chars)
• Alphabet: A-Za-z0-9+/
• URL-safe variant uses -_ instead of +/

# Encode to Base64
echo -n "Hello World" | base64
# SGVsbG8gV29ybGQ=

# Decode from Base64
echo "SGVsbG8gV29ybGQ=" | base64 -d
# Hello World

# Python
python3 -c "import base64; print(base64.b64encode(b'Hello').decode())"
python3 -c "import base64; print(base64.b64decode('SGVsbG8=').decode())"

# URL-safe Base64
python3 -c "import base64; print(base64.urlsafe_b64encode(b'Hello').decode())"

# CyberChef (online)
# https://gchq.github.io/CyberChef/

URL Encoding (Percent Encoding)

Encodes special characters for use in URLs.

Original: Hello World!
Encoded:  Hello%20World%21

Common Encodings:

# URL encode
python3 -c "import urllib.parse; print(urllib.parse.quote('Hello World!'))"
# Hello%20World%21

# URL decode
python3 -c "import urllib.parse; print(urllib.parse.unquote('Hello%20World%21'))"
# Hello World!

# Double encoding
python3 -c "import urllib.parse; print(urllib.parse.quote(urllib.parse.quote('../etc/passwd')))"
# %252e%252e%252fetc%252fpasswd

# Using curl
curl -G --data-urlencode "param=Hello World" "http://example.com"

HTML Entity Encoding

Encodes special characters for safe display in HTML.

Original: <script>alert('XSS')</script>
Encoded:  &lt;script&gt;alert(&#39;XSS&#39;)&lt;/script&gt;

Common Entities:

# Python HTML encode
python3 -c "import html; print(html.escape('<script>alert(1)</script>'))"
# &lt;script&gt;alert(1)&lt;/script&gt;

# Python HTML decode
python3 -c "import html; print(html.unescape('&lt;script&gt;'))"
# <script>

# PHP
php -r "echo htmlentities('<script>');"
php -r "echo html_entity_decode('&lt;script&gt;');"

Hex Encoding

Represents bytes as hexadecimal values.

Original: Hello
Hex:      48656c6c6f

# Hex encode
echo -n "Hello" | xxd -p
# 48656c6c6f

# Hex decode
echo "48656c6c6f" | xxd -r -p
# Hello

# Python
python3 -c "print('Hello'.encode().hex())"
python3 -c "print(bytes.fromhex('48656c6c6f').decode())"

# SQL hex encoding (for SQLi)
# SELECT * FROM users WHERE name = 0x61646d696e (admin)

Unicode Encoding

Different representations of Unicode characters.

Character: é
UTF-8:     c3 a9 (2 bytes)
UTF-16:    00e9 (2 bytes)
UTF-32:    000000e9 (4 bytes)
URL:       %C3%A9
HTML:      é or é

# Check encoding
file -bi document.txt

# Convert encodings
iconv -f UTF-8 -t UTF-16 input.txt > output.txt

# Python Unicode
python3 -c "print('\u003cscript\u003e')"
# <script>

# Unicode normalization
python3 -c "import unicodedata; print(unicodedata.normalize('NFC', 'café'))"

Encryption Deep Dive

TL;DR: Encryption makes data unreadable without the key. Symmetric uses one key; asymmetric uses a key pair.

Symmetric vs Asymmetric

┌────────────────────────────────────────────────────────────────┐
│                    SYMMETRIC ENCRYPTION                        │
│                                                                │
│   Same key for encrypt and decrypt                             │
│                                                                │
│   Plaintext ──[Key]──► Ciphertext ──[Key]──► Plaintext        │
│                                                                │
│   Examples: AES, ChaCha20, 3DES                                │
│   Use: Bulk data encryption, disk encryption                   │
│   Speed: Fast                                                  │
│   Challenge: Key distribution                                  │
└────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────┐
│                   ASYMMETRIC ENCRYPTION                        │
│                                                                │
│   Different keys for encrypt and decrypt                       │
│                                                                │
│   Plaintext ──[Public Key]──► Ciphertext ──[Private Key]──► Plaintext
│                                                                │
│   Examples: RSA, ECC, ElGamal                                  │
│   Use: Key exchange, digital signatures                        │
│   Speed: Slow                                                  │
│   Advantage: No key distribution problem                       │
└────────────────────────────────────────────────────────────────┘

AES (Advanced Encryption Standard)

The gold standard for symmetric encryption.

Parameters:

• Key sizes: 128, 192, or 256 bits
• Block size: 128 bits (16 bytes)
• Modes: ECB, CBC, CTR, GCM, etc.

Modes Comparison:

ECB vs CBC (Visual):

ECB Mode (BAD):                    CBC Mode (Good):
┌───────┐  ┌───────┐              ┌───────┐  ┌───────┐
│Block 1│  │Block 1│              │Block 1│  │Block 2│
│ AAAA  │  │ AAAA  │              │ AAAA  │  │ AAAA  │
└───┬───┘  └───┬───┘              └───┬───┘  └───┬───┘
    │          │                      │          │
    ▼          ▼                      ▼          ▼
┌───────┐  ┌───────┐              ┌───────┐  ┌───────┐
│ Same  │  │ Same  │              │ Diff  │  │ Diff  │
│Output!│  │Output!│              │Output │  │Output │
└───────┘  └───────┘              └───────┘  └───────┘

Same input = Same ciphertext      Same input = Different ciphertext
(Information leaked!)             (Pattern hidden)

# === OpenSSL AES Encryption ===

# Encrypt file (AES-256-CBC)
openssl enc -aes-256-cbc -salt -in plaintext.txt -out encrypted.enc -k password

# Decrypt file
openssl enc -aes-256-cbc -d -in encrypted.enc -out decrypted.txt -k password

# AES-256-GCM (recommended)
openssl enc -aes-256-gcm -salt -in plaintext.txt -out encrypted.enc -k password

# === Python AES ===

# pip install pycryptodome
python3 << 'EOF'
from Crypto.Cipher import AES
from Crypto.Random import get_random_bytes
from Crypto.Util.Padding import pad, unpad

key = get_random_bytes(32)  # 256-bit key
cipher = AES.new(key, AES.MODE_GCM)
ciphertext, tag = cipher.encrypt_and_digest(b"Hello World")
print(f"Ciphertext: {ciphertext.hex()}")
print(f"Nonce: {cipher.nonce.hex()}")
print(f"Tag: {tag.hex()}")
EOF

RSA (Rivest-Shamir-Adleman)

Most common asymmetric encryption.

Parameters:

• Key sizes: 2048, 3072, 4096 bits
• Based on: Factoring large primes is hard

Common Uses:

• TLS key exchange
• Digital signatures
• Encrypting symmetric keys

# === Generate RSA Key Pair ===

# Generate private key
openssl genrsa -out private.pem 2048

# Extract public key
openssl rsa -in private.pem -pubout -out public.pem

# === Encrypt/Decrypt ===

# Encrypt with public key
openssl rsautl -encrypt -pubin -inkey public.pem -in plaintext.txt -out encrypted.bin

# Decrypt with private key
openssl rsautl -decrypt -inkey private.pem -in encrypted.bin -out decrypted.txt

# === Examine Keys ===

# View private key components
openssl rsa -in private.pem -text -noout

# View public key
openssl rsa -pubin -in public.pem -text -noout

Common Encryption Mistakes

Hashing Deep Dive

TL;DR: Hashing creates a fixed-size fingerprint of data. Same input = same hash. Used for passwords, integrity, and deduplication.

Hash Properties

Hash Comparison

MD5 (Message Digest 5)

Status: Broken - DO NOT use for security

Input:  Hello World
MD5:    b10a8db164e0754105b7a99be72e3fe5

Why It’s Broken:

• Collision attacks demonstrated (2004)
• Pre-image attacks feasible for some cases
• Fast = easy to brute force

# Generate MD5
echo -n "Hello World" | md5sum
# b10a8db164e0754105b7a99be72e3fe5

# Python
python3 -c "import hashlib; print(hashlib.md5(b'Hello World').hexdigest())"

# Crack with hashcat
hashcat -m 0 -a 0 hash.txt wordlist.txt

# Crack with john
john --format=raw-md5 --wordlist=wordlist.txt hash.txt

SHA Family

SHA-1: Weak (collision found 2017) SHA-256/512: Currently secure

Input:    Hello World
SHA-1:    0a4d55a8d778e5022fab701977c5d840bbc486d0
SHA-256:  a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9ad9f146e
SHA-512:  2c74fd17edafd80e8447b0d46741ee243b7eb74dd2149a0ab1b9246fb30382f27e853d
          8585719e0e67cbda0daa8f51671064615d645ae27acb15bfb1447f459b

# SHA-1
echo -n "Hello World" | sha1sum

# SHA-256
echo -n "Hello World" | sha256sum

# SHA-512
echo -n "Hello World" | sha512sum

# Python
python3 -c "import hashlib; print(hashlib.sha256(b'Hello World').hexdigest())"

# Hashcat modes
# SHA-1:   -m 100
# SHA-256: -m 1400
# SHA-512: -m 1700

Password Hashing (bcrypt, Argon2, scrypt)

Why Regular Hashes Are Bad for Passwords:

• Too fast = easy to brute force
• GPUs crack billions per second

Password Hashes Are:

• Intentionally slow (configurable)
• Include salt automatically
• Designed to resist GPU attacks

bcrypt Hash Format:

$2a$10$N9qo8uLOickgx2ZMRZoMye.IjqQBrkHBSgSst/w/cP/LLyZLxPJhO
│  │  │                      │                                 │
│  │  │                      └── 22 char salt ─────────────────┘
│  │  └── Cost factor (2^10 = 1024 iterations)
│  └── Version (2a, 2b, 2y)
└── Algorithm identifier

# Generate bcrypt hash (Python)
python3 -c "import bcrypt; print(bcrypt.hashpw(b'password', bcrypt.gensalt(rounds=12)))"

# Verify bcrypt
python3 << 'EOF'
import bcrypt
stored = b'$2b$12$...'  # stored hash
if bcrypt.checkpw(b'password', stored):
    print("Match!")
EOF

# Generate Argon2 hash
python3 -c "from argon2 import PasswordHasher; ph = PasswordHasher(); print(ph.hash('password'))"

# Hashcat bcrypt
hashcat -m 3200 -a 0 hash.txt wordlist.txt

# John the Ripper bcrypt
john --format=bcrypt hash.txt

HMAC (Hash-based Message Authentication Code)

Combines hash with secret key for authentication.

HMAC-SHA256(key, message) = hash(key ⊕ opad || hash(key ⊕ ipad || message))

Used For:

• API authentication
• Message integrity
• JWT signatures

# Generate HMAC-SHA256
echo -n "message" | openssl dgst -sha256 -hmac "secret_key"

# Python
python3 -c "import hmac; import hashlib; print(hmac.new(b'key', b'message', hashlib.sha256).hexdigest())"

# Verify HMAC
python3 << 'EOF'
import hmac
import hashlib

def verify_hmac(key, message, signature):
    expected = hmac.new(key, message, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)  # Timing-safe!

print(verify_hmac(b'key', b'message', 'expected_sig'))
EOF

Common Mistakes and Vulnerabilities

Testing Checklist

• Check for Base64 “encrypted” secrets
• Identify MD5/SHA1 password hashes
• Look for ECB mode encryption
• Find hardcoded encryption keys
• Check for IV/nonce reuse
• Test for padding oracle vulnerabilities
• Verify HMAC uses constant-time comparison
• Check password hash cost factor

Vulnerability Matrix

Identifying Data Types

By Length

By Pattern

Identification Tools

# hashid
hashid 'b10a8db164e0754105b7a99be72e3fe5'
# [+] MD5

# hash-identifier
hash-identifier
# Enter hash: ...

# hashcat --identify
hashcat --identify hash.txt

# Online: hashes.com/en/tools/hash_identifier

Cracking and Breaking

Hash Cracking Workflow

1. Identify hash type
       │
       ▼
2. Check online databases
   - crackstation.net
   - hashes.com
   - cmd5.org
       │
       ▼
3. Try common passwords
   - rockyou.txt
   - common wordlists
       │
       ▼
4. Rule-based attack
   - hashcat rules
   - john rules
       │
       ▼
5. Mask attack (patterns)
   - ?d?d?d?d?d?d (6 digits)
   - ?u?l?l?l?l?d?d (Ulllld)
       │
       ▼
6. Brute force (last resort)

Hashcat Cheat Sheet

# === Hashcat ===

# Dictionary attack
hashcat -m 0 -a 0 hashes.txt rockyou.txt

# With rules
hashcat -m 0 -a 0 hashes.txt rockyou.txt -r rules/best64.rule

# Mask attack (8 char, lowercase + digits)
hashcat -m 0 -a 3 hashes.txt ?l?l?l?l?l?d?d?d

# Combination attack
hashcat -m 0 -a 1 hashes.txt wordlist1.txt wordlist2.txt

# Show cracked
hashcat -m 0 hashes.txt --show

# === John the Ripper ===

# Auto-detect and crack
john hashes.txt

# With wordlist
john --wordlist=rockyou.txt hashes.txt

# With rules
john --wordlist=rockyou.txt --rules hashes.txt

# Show cracked
john --show hashes.txt

# === Online Databases ===

# CrackStation
curl -X POST -d "hash=b10a8db164e0754105b7a99be72e3fe5" https://crackstation.net/

# hashes.com API (requires account)

Tools Reference

Encoding/Decoding

Encryption

Hashing

Online Resources

Practice Labs

Beginner

Intermediate

Practice Exercises

1. Identify the encoding:

   • SGVsbG8gV29ybGQ=
   • 48656c6c6f
   • %48%65%6c%6c%6f

2. Identify the hash:

   • 5d41402abc4b2a76b9719d911017c592
   • $2a$10$N9qo8uLOickgx2ZMRZoMye...
   • 2c74fd17edafd80e8447b0d46741ee24...

3. Crack these hashes:

   • 5f4dcc3b5aa765d61d8327deb882cf99 (MD5)
   • 5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8 (SHA1)

Glossary

What’s Next?

Now that you understand encoding, encryption, and hashing:

Summary

Understanding encoding, encryption, and hashing is fundamental:

Remember:

• Encoding ≠ Security
• MD5/SHA1 passwords = easy win
• Look for hardcoded keys
• Check for proper salt/iterations
• ECB mode = visible patterns

Found this guide helpful? Check out the other posts in the SecureKhan penetration testing series.