Why does é become ã©

Moreover, it is impossible to know whether a text uses an ISO table or Windows table because they both correspond just to a sequence of bytes. This freedom taken by Microsoft would be less of a problem if Windows applications used UTF-8 by default but this is not the case. As long as there is no special character outside the Windows table, most Windows applications do not encode texts using UTF Thus, sending a Windows text file to a linux server or a proprietary application can easilylead to confusion.

Because Unicode can encode all possible characters, it has become a nightmare for artists creating fonts because redrawing each character is an huge task. And they do not do it so that they can focus on the language which they are interested in. In addition, the Unicode standard can add new characters to the table and existing fonts become incomplete!

As a result, for exotic languages, it may be necessary to work with specific fonts. However, the fonts affect only the display for end users and in no way it would disrupt your the processing or the database storage of your strings. The Byte Order Mark is a sequence of unprintable Unicode bytes placed at the beginning of a Unicode text to facilitate its interpretation.

This Byte Order Mark is neither standard nor mandatory but it makes it easier for compatible applications to determine the subtype of Unicode format and to define the direction for reading the bytes. For non-compatible applications, this sequence of bytes is considered as some normal characters in extended ASCII. Another problem of the BOM is the confusion it can bring to a user. So, in a Unicode text editor, it is difficult to know if the BOM has been applied or not since it is invisible and also optional in a UTF-8 file.

Many users most likely do not know what the BOM is and how it can crash non-compatible applications. Since the BOM is invisible to the user, the confusion is obvious and inevitable.

However , in section below, we will provide you with standard tools so that you can quickly determine if your file is as you expect it to be. Regardless of the origin of a file, whether generated automatically, or sent by a data provider, or built manually, it may be useful to verify with absolute certainty its format and to show the possible tag BOM.

If you do not have access to advanced and usually paid text editors, you can easily do this with standard hexadecimal editors on Windows and linux. Above all, remember that the absence of the BOM tag does not mean that the file is not a Unicode file. Indeed, on the contrary, it may be necessary to remove it to increase compatibility with your applications.

When you create a Unicode file using VBA macros destined for format-sensitive applications, you will probably encounter some difficulties in mastering the BOM. For a start, you can use the commands suggested in the previous section to check your output files. To Kill a CodingBird ,. As I said before , encoding issues are quite common, and yet, they can be very tricky to debug: the reason is that any link in the long chain between the data storage sql or not and the client can be the culprit and has to be investigated.

I have recently experienced this first hand, and it was tricky enough to be the object of a future post. The reason lies in the UTF -8 representation.

Characters below or equal to are written on two bytes of the form yyyyy 10xxxxxx where the scalar representation of the character is: yyyyyxxxxxx see here for more details. Therefore its UTF -8 representation is Please help us improve Stack Overflow. Take our short survey. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Asked 8 years, 6 months ago.

Active 4 years, 9 months ago. Viewed 17k times. Improve this question. You could start off by posting the code you're using.

Chances are you're just reading using the default character encoding when it should probably be UTF-8, but we can't tell without seeing your code. Also note operating system and default locale set in your system.

I think for other languages there is another non-unicode encoding used in Windows. I'm accessing child folders by giving root folder path.

Root path name is in English. This looks like a UTF-8 byte sequence decoded using a legacy encoding e. Ensure the JRE's default encoding matches the system default encoding. What does System. Add a comment. Active Oldest Votes.