kbarbel640-del Encoding & Formats
Encode, decode, and inspect data in common formats. Covers Base64, URL encoding, hex, Unicode, JWTs, hashing, checksums, and serialization formats.
When to Use
- Decoding a Base64 string from an API response or config
- URL-encoding parameters for HTTP requests
- Inspecting hex dumps of binary data
- Decoding JWT tokens to see claims
- Computing or verifying file checksums
- Converting between character encodings (UTF-8, Latin-1, etc.)
- Understanding wire formats (protobuf, MessagePack)
Base64
Encode and decode
# Encode string
echo -n "Hello, World!" | base64
# SGVsbG8sIFdvcmxkIQ==
# Decode string
echo "SGVsbG8sIFdvcmxkIQ==" | base64 -d
# Hello, World!
# Encode a file
base64 image.png > image.b64
cat file.bin | base64
# Decode a file
base64 -d image.b64 > image.png
# Base64url (URL-safe variant: + → -, / → _, no padding)
echo -n "Hello" | base64 | tr '+/' '-_' | tr -d '='
# Base64url decode
echo "SGVsbG8" | tr '-_' '+/' | base64 -d
In code
// JavaScript (browser + Node.js 16+)
btoa('Hello'); // "SGVsbG8="
atob('SGVsbG8='); // "Hello"
// Node.js Buffer
Buffer.from('Hello').toString('base64'); // "SGVsbG8="
Buffer.from('SGVsbG8=', 'base64').toString(); // "Hello"
// Binary data
Buffer.from(binaryData).toString('base64');
Buffer.from(b64String, 'base64');
# Python
import base64
base64.b64encode(b"Hello").decode() # "SGVsbG8="
base64.b64decode("SGVsbG8=") # b"Hello"
# URL-safe Base64
base64.urlsafe_b64encode(b"Hello").decode()
base64.urlsafe_b64decode("SGVsbG8=")
URL Encoding
Encode and decode
# Python one-liner
python3 -c "from urllib.parse import quote; print(quote('hello world & foo=bar'))"
# hello%20world%20%26%20foo%3Dbar
# Decode
python3 -c "from urllib.parse import unquote; print(unquote('hello%20world%20%26%20foo%3Dbar'))"
# hello world & foo=bar
# curl does it automatically for --data-urlencode
curl -G --data-urlencode "q=hello world & more" https://api.example.com/search
In code
// JavaScript
encodeURIComponent('hello world & foo=bar');
// "hello%20world%20%26%20foo%3Dbar"
decodeURIComponent('hello%20world%20%26%20foo%3Dbar');
// "hello world & foo=bar"
// encodeURI vs encodeURIComponent:
encodeURI('https://example.com/path?q=hello world');
// "https://example.com/path?q=hello%20world" (preserves URL structure)
encodeURIComponent('https://example.com/path?q=hello world');
// "https%3A%2F%2Fexample.com%2Fpath%3Fq%3Dhello%20world" (encodes everything)
from urllib.parse import quote, unquote, urlencode
quote('hello world') # 'hello%20world'
unquote('hello%20world') # 'hello world'
urlencode({'q': 'hello world', 'page': 1}) # 'q=hello+world&page=1'
Hex
View and convert
# File hex dump
xxd file.bin | head -20
xxd -l 64 file.bin # First 64 bytes only
# Hex dump (compact, no ASCII)
xxd -p file.bin
# Convert hex to binary
echo "48656c6c6f" | xxd -r -p
# Hello
# od (alternative)
od -A x -t x1z file.bin | head -20
# hexdump
hexdump -C file.bin | head -20
# Python
python3 -c "print(bytes.fromhex('48656c6c6f').decode())" # Hello
python3 -c "print('Hello'.encode().hex())" # 48656c6c6f
In code
// JavaScript
Buffer.from('Hello').toString('hex'); // "48656c6c6f"
Buffer.from('48656c6c6f', 'hex').toString(); // "Hello"
// Number to hex
(255).toString(16); // "ff"
parseInt('ff', 16); // 255
# Python
"Hello".encode().hex() # '48656c6c6f'
bytes.fromhex('48656c6c6f').decode() # 'Hello'
hex(255) # '0xff'
int('ff', 16) # 255
Unicode
Inspect characters
# Show Unicode code points
echo -n "Hello 世界" | python3 -c "
import sys
for char in sys.stdin.read():
print(f'U+{ord(char):04X} {char} {char.encode(\"utf-8\").hex()}')"
# U+0048 H 48
# U+0065 e 65
# ...
# U+4E16 世 e4b896
# U+754C 界 e7958c
# Convert Unicode escape to character
printf '\u0048\u0065\u006c\u006c\u006f' # Hello
echo -e '\xE4\xB8\x96\xE7\x95\x8C' # 世界
# File encoding detection
file -bi document.txt
# text/plain; charset=utf-8
Encoding conversion
# Convert between encodings
iconv -f ISO-8859-1 -t UTF-8 input.txt > output.txt
iconv -f UTF-16 -t UTF-8 input.txt > output.txt
# List available encodings
iconv -l
# Python
python3 -c "
with open('latin1.txt', 'r', encoding='iso-8859-1') as f:
content = f.read()
with open('utf8.txt', 'w', encoding='utf-8') as f:
f.write(content)
"
Common Unicode issues
BOM (Byte Order Mark):
UTF-8 BOM: EF BB BF at start of file
Remove: sed -i '1s/^\xEF\xBB\xBF//' file.txt
Normalization (NFC vs NFD):
"é" can be U+00E9 (one char) or U+0065 U+0301 (e + combining accent)
Python: import unicodedata; unicodedata.normalize('NFC', text)
Mojibake (wrong encoding):
"café" appears as "café" → file is UTF-8 but read as Latin-1
Fix: re-read with correct encoding
JWT (JSON Web Tokens)
Decode a JWT
# JWT has 3 parts separated by dots: header.payload.signature
# Each part is Base64url-encoded
# Decode header and payload
TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"
# Decode header
echo "$TOKEN" | cut -d. -f1 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
# {"alg":"HS256","typ":"JWT"}
# Decode payload
echo "$TOKEN" | cut -d. -f2 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
# {"sub":"1234567890","name":"John Doe","iat":1516239022}
# One-liner function
jwt_decode() {
echo "$1" | cut -d. -f2 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
}
jwt_decode "$TOKEN"
In code
// JavaScript (no library needed for decoding)
function decodeJWT(token) {
const [header, payload] = token.split('.').slice(0, 2)
.map(part => JSON.parse(atob(part.replace(/-/g, '+').replace(/_/g, '/'))));
return { header, payload };
}
// Check expiry
function isJWTExpired(token) {
const { payload } = decodeJWT(token);
return payload.exp && payload.exp < Math.floor(Date.now() / 1000);
}
# Python
import json, base64
def decode_jwt(token):
parts = token.split('.')
# Add padding
def pad(s): return s + '=' * (4 - len(s) % 4)
header = json.loads(base64.urlsafe_b64decode(pad(parts[0])))
payload = json.loads(base64.urlsafe_b64decode(pad(parts[1])))
return header, payload
header, payload = decode_jwt(token)
Hashing
Common hash functions
# MD5 (not for security — only for checksums/dedup)
echo -n "Hello" | md5sum # Linux
echo -n "Hello" | md5 # macOS
# SHA-256 (standard for integrity)
echo -n "Hello" | sha256sum
echo -n "Hello" | shasum -a 256
# SHA-1 (deprecated for security, still used in git)
echo -n "Hello" | sha1sum
# SHA-512
echo -n "Hello" | sha512sum
# Hash a file
sha256sum file.bin
md5sum file.bin
# openssl (works everywhere)
echo -n "Hello" | openssl dgst -sha256
openssl dgst -sha256 file.bin
In code
// Node.js
const crypto = require('crypto');
crypto.createHash('sha256').update('Hello').digest('hex');
// "185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969"
// File hash
const fs = require('fs');
const hash = crypto.createHash('sha256');
hash.update(fs.readFileSync('file.bin'));
console.log(hash.digest('hex'));
import hashlib
hashlib.sha256(b"Hello").hexdigest()
# "185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969"
# File hash
with open("file.bin", "rb") as f:
print(hashlib.sha256(f.read()).hexdigest())
Checksums for file integrity
# Generate checksum file
sha256sum *.tar.gz > checksums.sha256
# Verify checksums
sha256sum -c checksums.sha256
# Compare two files without reading content
sha256sum file1.bin file2.bin
# or
cmp file1.bin file2.bin && echo "Identical" || echo "Different"
Serialization Formats
JSON ↔ other formats
# JSON to YAML
python3 -c "import json, yaml, sys; yaml.dump(json.load(sys.stdin), sys.stdout)" < data.json
# YAML to JSON
python3 -c "import json, yaml, sys; json.dump(yaml.safe_load(sys.stdin), sys.stdout, indent=2)" < data.yaml
# JSON to CSV
jq -r '.[] | [.id, .name, .email] | @csv' data.json > data.csv
# CSV to JSON
python3 -c "
import csv, json, sys
reader = csv.DictReader(open(sys.argv[1]))
print(json.dumps(list(reader), indent=2))
" data.csv
# JSON to TOML
python3 -c "import json, tomli_w, sys; tomli_w.dump(json.load(sys.stdin), sys.stdout.buffer)" < data.json
# Pretty-print JSON
jq '.' data.json
python3 -m json.tool data.json
Binary formats (inspection)
# MessagePack → JSON
python3 -c "
import msgpack, json, sys
data = msgpack.unpackb(sys.stdin.buffer.read(), raw=False)
print(json.dumps(data, indent=2))
" < data.msgpack
# Protobuf (decode without schema — shows field numbers)
protoc --decode_raw < data.pb
# CBOR → JSON
python3 -c "
import cbor2, json, sys
data = cbor2.loads(sys.stdin.buffer.read())
print(json.dumps(data, indent=2, default=str))
" < data.cbor
Quick Decode Script
#!/bin/bash
# decode.sh — Auto-detect and decode common encoded strings
INPUT="${1:-$(cat)}"
# Try Base64
B64_DECODED=$(echo "$INPUT" | base64 -d 2>/dev/null)
if [[ $? -eq 0 && -n "$B64_DECODED" ]]; then
echo "Base64 → $B64_DECODED"
fi
# Try URL encoding
if echo "$INPUT" | grep -q '%[0-9A-Fa-f]\{2\}'; then
URL_DECODED=$(python3 -c "from urllib.parse import unquote; print(unquote('$INPUT'))" 2>/dev/null)
echo "URL → $URL_DECODED"
fi
# Try JWT
if echo "$INPUT" | grep -qP '^eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.'; then
echo "JWT header:"
echo "$INPUT" | cut -d. -f1 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
echo "JWT payload:"
echo "$INPUT" | cut -d. -f2 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
fi
# Try hex
if echo "$INPUT" | grep -qP '^[0-9a-fA-F]+$' && [[ $((${#INPUT} % 2)) -eq 0 ]]; then
HEX_DECODED=$(echo "$INPUT" | xxd -r -p 2>/dev/null)
if [[ -n "$HEX_DECODED" ]]; then
echo "Hex → $HEX_DECODED"
fi
fi
Tips
- Base64 increases data size by ~33%. Use it for embedding binary data in text formats (JSON, XML, email), not for compression or encryption.
- Base64url (RFC 4648) uses
-and_instead of+and/, and omits padding=. JWTs and URL parameters use this variant. - SHA-256 is the standard for integrity checks. MD5 is fine for dedup and non-security checksums but broken for cryptographic use.
- JWTs are signed, not encrypted. Anyone can decode the header and payload. Only the signature verifies authenticity. Never put secrets in JWT claims.
- When files display garbled text (mojibake), the problem is almost always wrong encoding assumption. Check with
file -biand re-read with the correct encoding. xxd -p(plain hex) andxxd -r -p(reverse) are the fastest way to convert between binary and hex on the command line.- URL-encode with
encodeURIComponent(JavaScript) orurllib.parse.quote(Python), not by hand. Manual encoding misses edge cases.