Wp Utf8 Text

WordPress UTF-8 Text Handling

WordPress 6.9 modernized UTF-8 handling. New code should prefer the explicit UTF-8 helpers instead of older heuristics and hand-written byte regexes.

This skill is for server-side plugin text processing. It is not about block editor text controls.

When to use this skill

Trigger when ANY of the following is true:

• Code uses seems_utf8(), mb_check_encoding(), iconv(), utf8_encode(), utf8_decode(), or byte-level regexes.
• A plugin imports or exports CSV, XML, feeds, JSON, email, AI prompt data, logs, filenames, or third-party API text.
• The task mentions invalid UTF-8, mojibake, replacement character �, XML generation, REST encoding errors, or noncharacters.

The core helpers

seems_utf8() is deprecated in WP 6.9. Use wp_is_valid_utf8() for validation.

Validate, scrub, or reject

Pick behavior based on the boundary:

Replacing invalid bytes is lossy. Once U+FFFD is inserted, you cannot know what the original byte sequence was.

Examples

Reject invalid imported text:

$name = (string) ( $row['name'] ?? '' );

if ( ! wp_is_valid_utf8( $name ) ) {
    return new WP_Error(
        'myplugin_invalid_utf8',
        __( 'The imported name contains invalid UTF-8 bytes.', 'myplugin' )
    );
}

Scrub before XML output:

$title = wp_scrub_utf8( (string) $title );

$xml .= '<title>' . esc_xml( $title ) . '</title>';

Preserve raw bytes as an encoded forensic value, scrub only a decoded copy for display. Do not write invalid bytes directly to normal text meta/options: the database charset layer may reject, strip, or alter them.

$encoded = base64_encode( $payload );
update_post_meta( $post_id, '_myplugin_raw_payload_b64', $encoded );

$raw = base64_decode(
    (string) get_post_meta( $post_id, '_myplugin_raw_payload_b64', true ),
    true
);
if ( false === $raw ) {
    return new WP_Error( 'invalid_raw_payload', __( 'Stored payload is invalid.', 'myplugin' ) );
}
$safe_for_screen = wp_scrub_utf8( $raw );
echo esc_html( $safe_for_screen );

wp_check_invalid_utf8()

wp_check_invalid_utf8( $text, false ) returns an empty string for invalid UTF-8 on UTF-8 sites. Since WP 6.9, wp_check_invalid_utf8( $text, true ) replaces invalid byte sequences with U+FFFD instead of silently removing them.

Use it when you are already in a WordPress escaping/sanitizing path that expects this helper. For new explicit validation logic, prefer wp_is_valid_utf8() and wp_scrub_utf8() because the intent is clearer.

Noncharacters

Unicode noncharacters can be valid UTF-8 while still being inappropriate for interchange formats. Use wp_has_noncharacters() when producing XML, strict external API payloads, or data that will be consumed outside WordPress.

if ( ! wp_is_valid_utf8( $text ) ) {
    return new WP_Error( 'invalid_utf8', __( 'Invalid text encoding.', 'myplugin' ) );
}

if ( wp_has_noncharacters( $text ) ) {
    return new WP_Error(
        'myplugin_noncharacter_text',
        __( 'The text contains Unicode noncharacters that cannot be exported.', 'myplugin' )
    );
}

WordPress 7.1 code-point compatibility

WordPress 7.1 supplies UTF-8-only compatibility implementations of mb_chr() and mb_ord() when the mbstring extension does not provide them. The optional encoding must be UTF-8 or null; this is not a replacement for the full mbstring extension or arbitrary charset conversion. Both functions can return false, so check the result for untrusted code points or bytes.

Because these functions are defined during WordPress bootstrap, standalone tools that load plugin files without Core cannot assume they exist. Either bootstrap WordPress, require mbstring in that tool, or feature-detect.

In 7.1, antispambot() obfuscates valid multibyte characters as whole Unicode code points instead of processing their bytes independently. Its output is intentionally randomized and may differ across calls. Invalid UTF-8 byte spans pass through without obfuscation. Never use antispambot() as validation, encryption, authorization, or deterministic cache-key generation; validate the email first and treat the function only as best-effort source obfuscation.

Critical rules

• Do not use seems_utf8() in new code. It is deprecated as of WP 6.9.
• Do not scrub identifiers. Reject invalid bytes for slugs, tokens, IDs, and security-sensitive values.
• Do not scrub too early. Replacement is lossy and may destroy useful import/debug information.
• Validate before serialization boundaries. XML, JSON, feed, sitemap, and external API payloads should not receive invalid bytes.
• Call Unicode code-point helpers only after UTF-8 validation/scrubbing. Noncharacter checks do not replace byte-sequence validation.
• Remember ASCII ambiguity. A string can be valid UTF-8 and still originate from a non-UTF-8 encoding if it contains only ASCII.
• Do not snapshot one exact antispambot() output. Assert decoding/equivalence or structural properties because the function is randomized.

Common mistakes

// WRONG - deprecated and less explicit.
if ( ! seems_utf8( $value ) ) {
    $value = '';
}

// RIGHT - clear validation.
if ( ! wp_is_valid_utf8( $value ) ) {
    return new WP_Error( 'invalid_utf8', __( 'Invalid text encoding.', 'myplugin' ) );
}

// WRONG - silently changes an identifier.
$slug = sanitize_key( wp_scrub_utf8( $raw_slug ) );

// RIGHT - reject bad bytes before deriving identifiers.
if ( ! wp_is_valid_utf8( $raw_slug ) ) {
    return new WP_Error( 'invalid_slug_encoding', __( 'Invalid slug encoding.', 'myplugin' ) );
}
$slug = sanitize_key( $raw_slug );

Cross-references

• Run wp-security-audit when invalid text comes from uploads, REST, AJAX, or third-party APIs.
• Run wp-rest-api when text is accepted or returned through REST schemas.
• Run wp-html-api when text is being inserted into HTML fragments.

What this skill does NOT cover

• Full charset conversion from legacy encodings such as Windows-1250 or ISO-8859-2.
• Browser-side text handling.
• Translation/i18n placeholder correctness.

References

• WordPress 6.9 UTF-8 dev note: https://make.wordpress.org/core/2025/11/18/modernizing-utf-8-support-in-wordpress-6-9/
• UTF-8 helpers: wp-includes/utf8.php
• wp_check_invalid_utf8() / seems_utf8(): wp-includes/formatting.php
• WordPress 7.1 compatibility functions: wp-includes/compat.php
• antispambot(): wp-includes/formatting.php