John O'Conner

NetBeans 6.1 uses UTF-8 encoding for source files

I'm always happy when a company or product adopts Unicode as its charset. I think it makes perfect sense to do so. There are lots of good reasons why standardizing on Unicode is the right thing:

• the data charset can represent all modern, useful -- and many not-so-useful -- scripts
• charset consistency helps prevent data loss across application boundaries, including those among application, business tiers, application servers, and databases
• a single, multilingual charset simplifies the localization process

I was pleased to see that NetBeans 6.0 and the 6.1 beta uses the UTF-8 encoding (a Unicode encoding) as its default for project configuration and source files. The following figure shows the default setting in the project's property sheet:

This makes it much easier to edit non-ASCII, non-English source and property files. You can type text in any supported Unicode script right into Java source code. A legitimate usage would be comments or even localizable text in ListResourceBundle files. You can do that because NetBeans will save your files using the UTF-8 encoding by default...and NetBeans even puts this setting into the project's Ant file settings so that it compiles the file properly using the correct javac arguments. For example, NetBeans would use the following for a UTF-8 source file:

javac -encoding UTF-8 YourSource.java

Despite the potential benefits of this, NetBeans 6.1 still doesn't support this correctly in my humble opinion. Why not? Well, the biggest reason is simple: file corruption and permanent data loss. Ouch!

Let's take a simple "Hello, world!" example in Japanese. This is simple for NetBeans because of the UTF-8 encoding. The NetBeans editor even displays it correctly as shown here:

Unfortunately, the joy of this discovery was short lived when I discovered how easy it is to corrupt this data. Feel like experimenting with the charset encoding? Surely someone will. I suspected what would happen, so I didn't do this with any substantial code base...but someone will. I sure hope they use version control software.

Reopen that project property sheet, select another encoding, say ISO-8859-1 or windows-1252 since so many American and Western-European programmers use that by default. What happens? It appears as if nothing has happened. In this NB 6.1 beta, I see the same editor screen as before. NetBeans apparently hasn't yet reloaded the file from disk, so I'll force the issue by closing and reloading this file. The result is here:

Some of you, the super careful, nit-picky ones will now argue with me, "But John, you haven't really lost anything yet. 8859-1 and CP 1252 don't have those characters, but the original byte values are still entact. You can get them back in this example." OK, I concede the point. But now I'll show you some serious data loss, no messing around this time. Instead of ISO-8859-1, pick US-ASCII as the target charset. Add a comment or another line of code to the file. Save. And there you have it as shown here:

Now that's just not good. Did NetBeans save the file correctly? Sure. However, NetBeans can do better than this. I would argue that if NetBeans knows that the target encoding does not support the source encoding, it should at least warn the user that the resulting file will contain garbage characters and that parts of the file will be lost--permanantly in many cases.

So, just in case anyone over there in the NetBeans developer group can hear me...you have to fix this. Yes, I know it's a silly mistake for someone to do this, but NetBeans can help them avoid the problem. Just provide a warning dialog, "Saving this file in the target encoding will cause data loss because the target encoding does not support all characters in this file or project." Keep the encoding feature, just perfect it by helping some users avoid this costly mistake. The fact is that most software developers still don't understand character sets and encodings, and this is just an accident waiting to happen.

On a personal note: I really love NetBeans. And I hope this blog qualifies me for the NetBeans 6.1 blogging contest! I could probably file this under the "suggestions on how to enhance NetBeans 6.1" category.

Posted at 02:43AM Mar 30, 2008 by John O'Conner in NetBeans  |  Comments[2]