Codex Mojibake on Windows: Why It Happens and How to Stop It Permanently

You ask Codex to edit a file. The code works, but comments, HTML, or documentation suddenly contain strange character sequences. An em dash becomes —. A copyright sign becomes ©. An accented letter such as é becomes é.

This problem is called mojibake. The word comes from Japanese and refers to text that has been decoded using the wrong character encoding.

It can look like Codex is randomly damaging punctuation. Usually, however, the model is only one part of the problem. The actual failure is a mismatch between UTF-8 text and an older Windows tool that reads or writes the same bytes as Windows-1252, ANSI, or UTF-16.

The problem is especially common when Codex works through Windows PowerShell 5.1.

This guide explains the exact failure, shows how to prove whether a file is really corrupt, provides safe repair methods, and includes a ready-to-use AGENTS.md policy that prevents the problem without banning proper typography or multilingual text.

What Mojibake Actually Is

Text files store bytes. An encoding defines how those bytes map to characters.

For example, an em dash is Unicode code point U+2014. In UTF-8 it is stored as three bytes:

E2 80 94

If software mistakenly interprets those UTF-8 bytes using Windows-1252, it produces three unrelated characters instead of one em dash. If that mistaken text is then saved as UTF-8, the corruption becomes permanent.

That creates the familiar signatures:

The key point is that the original UTF-8 bytes were not inherently wrong. They were decoded incorrectly.

Why It Often Appears During Codex Work

Codex frequently edits code through command-line tools. On Windows, that may involve Windows PowerShell 5.1, whose encoding defaults are inconsistent.

Our controlled test produced these results:

This produces two distinct failure modes.

Failure Mode 1: The File Is Correct, but the Terminal Looks Wrong

A UTF-8-without-BOM file can be completely valid while this command displays it incorrectly:

Get-Content .\article.md

If someone sees the garbled terminal output and "repairs" the source file, they can corrupt a file that was never broken.

Read it explicitly as UTF-8 instead:

[IO.File]::ReadAllText(
    (Resolve-Path .\article.md),
    [Text.Encoding]::UTF8
)

Or:

Get-Content -Raw -Encoding UTF8 .\article.md

Failure Mode 2: Misread Text Is Saved Back to Disk

This is the destructive version:

1. A correct UTF-8 file is read as Windows-1252.
2. The resulting wrong characters exist in memory.
3. A command saves those wrong characters as UTF-8.
4. The file now contains genuine mojibake.

At that point, changing the editor's display encoding will not restore the original characters. The wrong characters are now the file's actual content.

How to Determine Whether the File Is Really Corrupt

Do not judge from terminal output alone.

Use a strict UTF-8 decoder:

$path = (Resolve-Path .\article.md).Path
$bytes = [IO.File]::ReadAllBytes($path)
$strictUtf8 = [Text.UTF8Encoding]::new($false, $true)

try {
    $text = $strictUtf8.GetString($bytes)
    'Valid UTF-8'
}
catch {
    'Invalid UTF-8 bytes'
}

Then inspect $text, not the default Get-Content rendering.

Also check the byte-order mark:

$bytes[0..3] | ForEach-Object { '{0:X2}' -f $_ }

Common prefixes are:

EF BB BF   UTF-8 with BOM
FF FE      UTF-16LE with BOM
2D 2D 20   Ordinary ASCII text beginning with "-- "

UTF-8 without BOM has no required prefix.

How to Repair Confirmed Mojibake Safely

First, preserve a copy and work on a small confirmed section. Never run a blind full-repository replacement.

For the classic case where UTF-8 was decoded as Windows-1252 and then resaved, this reversal may recover the original text:

$wrong = [IO.File]::ReadAllText($path, [Text.Encoding]::UTF8)
$windows1252 = [Text.Encoding]::GetEncoding(1252)
$originalBytes = $windows1252.GetBytes($wrong)
$candidate = [Text.Encoding]::UTF8.GetString($originalBytes)

Review $candidate carefully before writing it. This operation can damage legitimate Windows-1252 or multilingual text if the diagnosis is wrong.

For a few known punctuation errors, a targeted edit through apply_patch is safer than re-encoding the entire file.

When an automated write is unavoidable, use explicit UTF-8 without BOM:

$utf8NoBom = [Text.UTF8Encoding]::new($false)
[IO.File]::WriteAllText($path, $candidate, $utf8NoBom)

The Permanent Fix

The permanent fix is not "never use em dashes." It is to make encoding explicit and preserve Unicode according to the file type.

Code, Configuration, SQL, and Automation

Prefer ASCII when Unicode is merely decorative:

- instead of an em dash in a code comment
-> instead of a decorative arrow
straight quotes in scripts and configuration

This removes unnecessary encoding risk without changing program behavior.

Literal HTML

HTML entities preserve typography while keeping source bytes ASCII:

—
’
©
…
→

Do not put these entities into a C# or Razor value that will be HTML-encoded. It may display — literally because the ampersand becomes &.

Prose, Books, Names, and Translations

Keep real Unicode. A manuscript should retain proper curly quotes, em dashes, accented names, and non-English characters. The solution is an end-to-end UTF-8 workflow, not degraded typography.

PowerShell Scripts

The safest rule for Windows PowerShell 5.1 is to keep .ps1 automation ASCII-only.

If a PS1 genuinely requires Unicode literals, save it as UTF-8 with BOM and test it under Windows PowerShell 5.1. PowerShell 7 uses much safer UTF-8 defaults, but a project should not silently assume PowerShell 7 is installed.

Add an EditorConfig Rule

Place this .editorconfig in the project root:

root = true

[*.{cs,cshtml,razor,css,js,json,md,html,sql,xml,config,props,targets,yml,yaml,txt,ps1}]
charset = utf-8

In EditorConfig, utf-8 means UTF-8 without BOM. Use utf-8-bom only for files that specifically require a BOM.

What AGENTS.md Should Say

Codex reads AGENTS.md as persistent repository instructions. Put this in the project root:

# Workspace Instructions

## Text Encoding

This workspace is edited from Windows PowerShell 5.1. Its implicit text
encoding behavior is unsafe, so use these rules for every text-file change.

1. Preserve the file's intended content. Real Unicode is allowed and expected
   in customer-facing copy, names, translations, manuscripts, and other prose.
   Never strip or replace meaningful typography merely to make a file ASCII.

2. Default code, comments, scripts, SQL, JSON keys, configuration, and
   automation to ASCII when Unicode adds no functional value.

3. Use `apply_patch` for manual file edits. Do not rewrite source files through
   PowerShell redirection, `Out-File`, or `Set-Content`.

4. Never trust a default `Get-Content` rendering of a UTF-8 file. PowerShell
   5.1 can display correct UTF-8-without-BOM text as mojibake. Read explicitly:

   `[IO.File]::ReadAllText($path, [Text.Encoding]::UTF8)`

5. If automation must write text, specify UTF-8 explicitly. For UTF-8 without
   BOM use:

   `[IO.File]::WriteAllText($path, $text, [Text.UTF8Encoding]::new($false))`

6. Keep executable `.ps1` files ASCII-only. If a PowerShell script genuinely
   requires Unicode literals and must run under Windows PowerShell 5.1, save it
   as UTF-8 with BOM and test it under that exact shell.

7. In literal HTML markup, ASCII entities such as `—`, `’`, and
   `©` are safe. Do not place entities in dynamically HTML-encoded C# or
   Razor strings unless that rendering path is intended to emit literal HTML.

8. After changing text files, run the project's encoding verification command.
   Treat possible mojibake matches as review findings, not automatic
   replacements. Legitimate multilingual text must be preserved.

This policy does not prevent Codex from writing polished prose. It prevents fragile tools from silently changing the bytes.

The Exact Prompt to Give Codex

You can ask Codex to diagnose and permanently fix its own workflow with this prompt:

Test this repository for mojibake and unsafe text-encoding behavior. Do not
assume garbled terminal output means a file is corrupt.

First identify the exact shell and version. Create an isolated temporary test
directory inside the workspace. Construct a test string from Unicode code
points so the test command itself cannot corrupt it. Include an em dash, curly
quotes, ellipsis, arrow, bullet, copyright sign, accented letter, and
non-breaking space.

Test these paths separately:

1. apply_patch/editor writes
2. File.WriteAllText with explicit UTF-8 without BOM
3. default Get-Content versus an explicit UTF-8 read
4. default Set-Content
5. Set-Content -Encoding UTF8
6. default Out-File
7. PowerShell redirection with >
8. a UTF-8 PS1 containing Unicode with and without BOM
9. reading a correct UTF-8 file incorrectly and resaving it

Compare bytes and Unicode code points, not visual terminal output. Distinguish
display-only corruption from actual on-disk corruption.

Do not modify production files until the experiment proves the failure mode.
Once proven, add a balanced AGENTS.md policy that preserves real Unicode in
human-facing prose but defaults code and automation to ASCII. Add an
.editorconfig UTF-8 rule and an ASCII-only verification script that detects
invalid UTF-8, UTF-16 text, common mojibake signatures, and Unicode PS1 files
without BOM. The scanner must report possible problems but never automatically
rewrite legitimate multilingual content.

Run the scanner against both known-good Unicode files and deliberately broken
test files. It must pass the good files and fail the bad ones. Scan the active
repository using explicit UTF-8 decoding, repair only confirmed corruption,
verify dependent hashes or generated artifacts, and remove the temporary lab.
Report the exact methods tested and their results.

That prompt makes Codex establish evidence before changing files. It also requires the fix to persist beyond the current chat.

Common Mistakes to Avoid

Mistake 1: Converting Everything to ASCII

This destroys proper typography and multilingual content. ASCII is a sensible default for code and automation, not a universal content policy.

Mistake 2: Trusting the Terminal

PowerShell 5.1 can display a correct file incorrectly. Always decode the file explicitly before repairing it.

Mistake 3: Rewriting an Entire Large File

A full-file read-modify-write through the wrong encoding can corrupt every non-ASCII character. Prefer targeted patches.

Mistake 4: Adding a BOM Everywhere

A BOM solves the PowerShell 5.1 script-detection problem, but some tools and file formats expect UTF-8 without BOM. Use it deliberately, not universally.

Mistake 5: Treating Every Suspicious Sequence as Corruption

Automated scanners can produce false positives in multilingual text. They should identify review candidates, never perform automatic replacements.

Mistake 6: Repairing a Pinned Artifact Without Updating Its Hash

Even a comment-only change alters a file's SHA-256. If the file is deployment-pinned, verify that executable content is unchanged and update every recorded hash consistently.

A Practical Final Checklist

Before Codex finishes an edit involving text files:

1. Confirm the intended encoding.
2. Use apply_patch for manual edits.
3. Read UTF-8 explicitly during verification.
4. Do not trust default PowerShell display output.
5. Run a strict UTF-8 and mojibake scan.
6. Preserve legitimate Unicode.
7. Keep PowerShell 5.1 scripts ASCII-only or use a tested BOM.
8. Build and render-test affected pages.
9. Recalculate hashes for pinned artifacts.
10. Remove temporary test files.

Mojibake is not mysterious, and it is not fixed by forbidding good typography. It is fixed by making every read and write operation explicit, testing the actual bytes, and giving Codex persistent instructions that survive the current conversation.