Programming
What encodingcode page is cmdexe using
Ever typed a character into your Windows command prompt only to see a strange symbol appear? Or perhaps run a script that unexpectedly outputs garbled text? The culprit is often a mismatch between the encoding used by cmd.exe and the encoding of the text being input or displayed. Understanding how cmd.exe handles encoding is crucial for anyone working with text files, scripts, or international characters within the Windows command-line environment. This post dives into the intricacies of cmd.exe encoding, exploring how it works, why it matters, and how to troubleshoot common encoding issues.
Decoding the Default: Understanding Cmd.exe’s Encoding
By default, cmd.exe uses the system’s active code page. This code page is a legacy system inherited from the days of DOS and determines the character set available for use. Think of it as a map that links specific numerical codes to characters. In many Western systems, the active code page is typically set to 437 (US) or 850 (Multilingual Latin I). These code pages handle most standard English characters without a hitch. However, they can struggle with characters from other languages, especially those with accents or different alphabets.
You can check your system’s active code page by typing chcp into the command prompt. The command will return a number corresponding to the active code page. Knowing this number is crucial when dealing with text files or scripts that might be using a different encoding.
For instance, if you’re working with a UTF-8 encoded file, and your cmd.exe is using code page 437, you might encounter issues displaying or processing certain characters correctly.
The Importance of Code Page Matching
Matching the encoding of your input, output, and cmd.exe environment is crucial for preventing data corruption and ensuring your scripts run as expected. A mismatch can lead to several problems:
- Incorrect character display: Characters may appear as question marks, boxes, or other unexpected symbols.
- Script malfunction: Scripts relying on specific characters might fail to execute correctly or produce unexpected results.
- Data loss: In severe cases, encoding mismatches can lead to data corruption, especially when dealing with files containing non-ASCII characters.
Troubleshooting Encoding Issues in Cmd.exe
If you encounter encoding problems, here are a few troubleshooting steps:
- Identify the encoding of the problematic text: This might involve checking the file properties or using a text editor that can display encoding information.
- Change the active code page: Use the
chcpcommand followed by the desired code page number (e.g.,chcp 65001for UTF-8). - Use a more Unicode-friendly environment: Consider using PowerShell, which has better support for Unicode and different encodings.
For example, if you are working with a UTF-8 encoded file, you would use chcp 65001 before running any commands that interact with the file. This ensures that cmd.exe uses the correct character mapping, preventing garbled text or script errors.
Unicode and the Future of Cmd.exe
While cmd.exe is primarily based on legacy code pages, the increasing prevalence of Unicode has led to improvements in its handling of international characters. The introduction of chcp 65001 for UTF-8 support is a significant step forward. However, PowerShell offers a more robust and comprehensive solution for working with different encodings in the Windows environment. Its native support for Unicode and various cmdlets designed for encoding manipulation make it a powerful tool for anyone working with multilingual text or international character sets.
Consider this: A script designed to process user input from different regions might fail in cmd.exe if the input contains characters outside the active code page. However, the same script is likely to function flawlessly in PowerShell due to its robust Unicode support.
Frequently Asked Questions
Q: What is the difference between a code page and Unicode?
A: Code pages are limited mappings of characters to numerical values, whereas Unicode is a universal character set that aims to encompass all characters from all languages.
Understanding how encoding works in cmd.exe is essential for avoiding frustrating character display issues and ensuring your scripts function correctly. While the legacy system of code pages can be cumbersome, using the correct chcp command can help mitigate many problems. For a more modern and versatile approach, consider exploring PowerShell and its superior Unicode handling. Learn more about advanced scripting techniques here. By grasping these concepts, you’ll be better equipped to navigate the complexities of character encoding within the Windows command-line environment and leverage the full potential of your scripts and tools. Check out these resources for further reading on character encoding: Unicode Consortium, Microsoft’s CHCP documentation, and IANA Character Sets.
Question & Answer :
When I open cmd.exe on Windows, what encoding is it using?
How can I check which encoding it is currently using?
Does it depend on my regional setting or are there any environment variables to check?
What happens when you type a file with a certain encoding? Sometimes I get garbled characters (because of incorrect encoding) and sometimes it kind of works.
However, I don’t trust anything as long as I don’t know what’s going on. Can anyone explain?
Yes, it’s frustrating—sometimes type and other programs print gibberish, and sometimes they do not.
First of all, Unicode characters will only display if the current console font contains the characters. So use a TrueType font like Lucida Console instead of the default Raster Font.
But if the console font doesn’t contain the character you’re trying to display, you’ll see question marks instead of gibberish. When you get gibberish, there’s more going on than just font settings.
When programs use standard C-library I/O functions like printf, the program’s output encoding must match the console’s output encoding, or you will get gibberish. chcp shows and sets the current codepage. All output using standard C-library I/O functions is treated as if it is in the codepage displayed by chcp.
Matching the program’s output encoding with the console’s output encoding can be accomplished in two different ways:
- A program can get the console’s current codepage using
chcporGetConsoleOutputCP, and configure itself to output in that encoding, or - You or a program can set the console’s current codepage using
chcporSetConsoleOutputCPto match the default output encoding of the program.
However, programs that use Win32 APIs can write UTF-16LE strings directly to the console with WriteConsoleW. This is the only way to get correct output without setting codepages. And even when using that function, if a string is not in the UTF-16LE encoding to begin with, a Win32 program must pass the correct codepage to MultiByteToWideChar. Also, WriteConsoleW will not work if the program’s output is redirected; more fiddling is needed in that case.
type works some of the time because it checks the start of each file for a UTF-16LE Byte Order Mark (BOM), i.e. the bytes 0xFF 0xFE. If it finds such a mark, it displays the Unicode characters in the file using WriteConsoleW regardless of the current codepage. But when typeing any file without a UTF-16LE BOM, or for using non-ASCII characters with any command that doesn’t call WriteConsoleW—you will need to set the console codepage and program output encoding to match each other.
How can we find this out?
Here’s a test file containing Unicode characters:
ASCII abcde xyz German äöü ÄÖÜ ß Polish ąęźżńł Russian абвгдеж эюя CJK 你好
Here’s a Java program to print out the test file in a bunch of different Unicode encodings. It could be in any programming language; it only prints ASCII characters or encoded bytes to stdout.
import java.io.*; public class Foo { private static final String BOM = "\ufeff"; private static final String TEST_STRING = "ASCII abcde xyz\n" + "German äöü ÄÖÜ ß\n" + "Polish ąęźżńł\n" + "Russian абвгдеж эюя\n" + "CJK 你好\n"; public static void main(String[] args) throws Exception { String[] encodings = new String[] { "UTF-8", "UTF-16LE", "UTF-16BE", "UTF-32LE", "UTF-32BE" }; for (String encoding: encodings) { System.out.println("== " + encoding); for (boolean writeBom: new Boolean[] {false, true}) { System.out.println(writeBom ? "= bom" : "= no bom"); String output = (writeBom ? BOM : "") + TEST_STRING; byte[] bytes = output.getBytes(encoding); System.out.write(bytes); FileOutputStream out = new FileOutputStream("uc-test-" + encoding + (writeBom ? "-bom.txt" : "-nobom.txt")); out.write(bytes); out.close(); } } } }
The output in the default codepage? Total garbage!
Z:\andrew\projects\sx\1259084>chcp Active code page: 850 Z:\andrew\projects\sx\1259084>java Foo == UTF-8 = no bom ASCII abcde xyz German ├ñ├Â├╝ ├ä├û├£ ├ƒ Polish ─à─Ö┼║┼╝┼ä┼é Russian ð░ð▒ð▓ð│ð┤ðÁð ÐìÐÄÐÅ CJK õ¢áÕÑ¢ = bom ´╗┐ASCII abcde xyz German ├ñ├Â├╝ ├ä├û├£ ├ƒ Polish ─à─Ö┼║┼╝┼ä┼é Russian ð░ð▒ð▓ð│ð┤ðÁð ÐìÐÄÐÅ CJK õ¢áÕÑ¢ == UTF-16LE = no bom A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ♣☺↓☺z☺|☺D☺B☺ R u s s i a n 0♦1♦2♦3♦4♦5♦6♦ M♦N♦O♦ C J K `O}Y = bom ■A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ♣☺↓☺z☺|☺D☺B☺ R u s s i a n 0♦1♦2♦3♦4♦5♦6♦ M♦N♦O♦ C J K `O}Y == UTF-16BE = no bom A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ☺♣☺↓☺z☺|☺D☺B R u s s i a n ♦0♦1♦2♦3♦4♦5♦6 ♦M♦N♦O C J K O`Y} = bom ■ A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ☺♣☺↓☺z☺|☺D☺B R u s s i a n ♦0♦1♦2♦3♦4♦5♦6 ♦M♦N♦O C J K O`Y} == UTF-32LE = no bom A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ♣☺ ↓☺ z☺ |☺ D☺ B☺ R u s s i a n 0♦ 1♦ 2♦ 3♦ 4♦ 5♦ 6♦ M♦ N ♦ O♦ C J K `O }Y = bom ■ A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ♣☺ ↓☺ z☺ |☺ D☺ B☺ R u s s i a n 0♦ 1♦ 2♦ 3♦ 4♦ 5♦ 6♦ M♦ N ♦ O♦ C J K `O }Y == UTF-32BE = no bom A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ☺♣ ☺↓ ☺z ☺| ☺D ☺B R u s s i a n ♦0 ♦1 ♦2 ♦3 ♦4 ♦5 ♦6 ♦M ♦N ♦O C J K O` Y} = bom ■ A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ☺♣ ☺↓ ☺z ☺| ☺D ☺B R u s s i a n ♦0 ♦1 ♦2 ♦3 ♦4 ♦5 ♦6 ♦M ♦N ♦O C J K O` Y}
However, what if we type the files that got saved? They contain the exact same bytes that were printed to the console.
Z:\andrew\projects\sx\1259084>type *.txt uc-test-UTF-16BE-bom.txt ■ A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ☺♣☺↓☺z☺|☺D☺B R u s s i a n ♦0♦1♦2♦3♦4♦5♦6 ♦M♦N♦O C J K O`Y} uc-test-UTF-16BE-nobom.txt A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ☺♣☺↓☺z☺|☺D☺B R u s s i a n ♦0♦1♦2♦3♦4♦5♦6 ♦M♦N♦O C J K O`Y} uc-test-UTF-16LE-bom.txt ASCII abcde xyz German äöü ÄÖÜ ß Polish ąęźżńł Russian абвгдеж эюя CJK 你好 uc-test-UTF-16LE-nobom.txt A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ♣☺↓☺z☺|☺D☺B☺ R u s s i a n 0♦1♦2♦3♦4♦5♦6♦ M♦N♦O♦ C J K `O}Y uc-test-UTF-32BE-bom.txt ■ A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ☺♣ ☺↓ ☺z ☺| ☺D ☺B R u s s i a n ♦0 ♦1 ♦2 ♦3 ♦4 ♦5 ♦6 ♦M ♦N ♦O C J K O` Y} uc-test-UTF-32BE-nobom.txt A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ☺♣ ☺↓ ☺z ☺| ☺D ☺B R u s s i a n ♦0 ♦1 ♦2 ♦3 ♦4 ♦5 ♦6 ♦M ♦N ♦O C J K O` Y} uc-test-UTF-32LE-bom.txt A S C I I a b c d e x y z G e r m a n ä ö ü Ä Ö Ü ß P o l i s h ą ę ź ż ń ł R u s s i a n а б в г д е ж э ю я C J K 你 好 uc-test-UTF-32LE-nobom.txt A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ♣☺ ↓☺ z☺ |☺ D☺ B☺ R u s s i a n 0♦ 1♦ 2♦ 3♦ 4♦ 5♦ 6♦ M♦ N ♦ O♦ C J K `O }Y uc-test-UTF-8-bom.txt ´╗┐ASCII abcde xyz German ├ñ├Â├╝ ├ä├û├£ ├ƒ Polish ─à─Ö┼║┼╝┼ä┼é Russian ð░ð▒ð▓ð│ð┤ðÁð ÐìÐÄÐÅ CJK õ¢áÕÑ¢ uc-test-UTF-8-nobom.txt ASCII abcde xyz German ├ñ├Â├╝ ├ä├û├£ ├ƒ Polish ─à─Ö┼║┼╝┼ä┼é Russian ð░ð▒ð▓ð│ð┤ðÁð ÐìÐÄÐÅ CJK õ¢áÕÑ¢
The only thing that works is UTF-16LE file, with a BOM, printed to the console via type.
If we use anything other than type to print the file, we get garbage:
Z:\andrew\projects\sx\1259084>copy uc-test-UTF-16LE-bom.txt CON ■A S C I I a b c d e x y z G e r m a n õ ÷ ³ ─ Í ▄ ▀ P o l i s h ♣☺↓☺z☺|☺D☺B☺ R u s s i a n 0♦1♦2♦3♦4♦5♦6♦ M♦N♦O♦ C J K `O}Y 1 file(s) copied.
From the fact that copy CON does not display Unicode correctly, we can conclude that the type command has logic to detect a UTF-16LE BOM at the start of the file, and use special Windows APIs to print it.
We can see this by opening cmd.exe in a debugger when it goes to type out a file:

After type opens a file, it checks for a BOM of 0xFEFF—i.e., the bytes 0xFF 0xFE in little-endian—and if there is such a BOM, type sets an internal fOutputUnicode flag. This flag is checked later to decide whether to call WriteConsoleW.
But that’s the only way to get type to output Unicode, and only for files that have BOMs and are in UTF-16LE. For all other files, and for programs that don’t have special code to handle console output, your files will be interpreted according to the current codepage, and will likely show up as gibberish.
You can emulate how type outputs Unicode to the console in your own programs like so:
#include <stdio.h> #define UNICODE #include <windows.h> static LPCSTR lpcsTest = "ASCII abcde xyz\n" "German äöü ÄÖÜ ß\n" "Polish ąęźżńł\n" "Russian абвгдеж эюя\n" "CJK 你好\n"; int main() { int n; wchar_t buf[1024]; HANDLE hConsole = GetStdHandle(STD_OUTPUT_HANDLE); n = MultiByteToWideChar(CP_UTF8, 0, lpcsTest, strlen(lpcsTest), buf, sizeof(buf)); WriteConsole(hConsole, buf, n, &n, NULL); return 0; }
This program works for printing Unicode on the Windows console using the default codepage.
For the sample Java program, we can get a little bit of correct output by setting the codepage manually, though the output gets messed up in weird ways:
Z:\andrew\projects\sx\1259084>chcp 65001 Active code page: 65001 Z:\andrew\projects\sx\1259084>java Foo == UTF-8 = no bom ASCII abcde xyz German äöü ÄÖÜ ß Polish ąęźżńł Russian абвгдеж эюя CJK 你好 ж эюя CJK 你好 你好 好 � = bom ASCII abcde xyz German äöü ÄÖÜ ß Polish ąęźżńł Russian абвгдеж эюя CJK 你好 еж эюя CJK 你好 你好 好 � == UTF-16LE = no bom A S C I I a b c d e x y z …
However, a C program that sets a Unicode UTF-8 codepage:
#include <stdio.h> #include <windows.h> int main() { int c, n; UINT oldCodePage; char buf[1024]; oldCodePage = GetConsoleOutputCP(); if (!SetConsoleOutputCP(65001)) { printf("error\n"); } freopen("uc-test-UTF-8-nobom.txt", "rb", stdin); n = fread(buf, sizeof(buf[0]), sizeof(buf), stdin); fwrite(buf, sizeof(buf[0]), n, stdout); SetConsoleOutputCP(oldCodePage); return 0; }
does have correct output:
Z:\andrew\projects\sx\1259084>.\test ASCII abcde xyz German äöü ÄÖÜ ß Polish ąęźżńł Russian абвгдеж эюя CJK 你好
The moral of the story?
typecan print UTF-16LE files with a BOM regardless of your current codepage- Win32 programs can be programmed to output Unicode to the console, using
WriteConsoleW. - Other programs which set the codepage and adjust their output encoding accordingly can print Unicode on the console regardless of what the codepage was when the program started
- For everything else you will have to mess around with
chcp, and will probably still get weird output.