John O'Conner

Another solution for non-UTF8 encoded source files

Recently I mentioned a potential problem when saving source files in a non-Unicode charset encoding. The potential data loss is significant for large projects. After thinking about the problem a little more, I have a potential solution, a solution that allows you to save to a non-Unicode encoding but also prevents data loss.

You are familiar with \u notation for non-ascii characters in property files? I think the same encoding can work for non-ascii characters in any Java source file. I'm not suggesting that should be the preferred representation. I think the \u notation is only tolerable, something to be avoided whenever possible. However, in this situation -- saving files that were once in UTF-8 -- this might be the only option for storing the files without data loss.

Here's how it would work. First, NetBeans 6.1 uses UTF-8 for a project's default source code and configuration file encoding, an excellent choice by the way. So, now imagine that your source code has the Euro currency symbol in it. That's Unicode code point U+20AC. And the character itself is this: ?. If you can't see the actual character (maybe you don't have a capable font?), here's the image instead:

Now, let's imagine that you need to change your project encoding for some reason. So, maybe you choose ISO-8859-1, which doesn't contain the Euro symbol. You can still represent the Euro character, but you'll have to encode it with the \u. Wouldn't it be nice if NetBeans did this for you, creating \u20AC in your file instead of converting the character to a meaningless ? question mark. I think that would be better. And it's entirely possible. It doesn't prevent NetBeans from converting the file to the target encoding as requested by the user, and it allows NetBeans to prevent data loss by using the \u encoding for characters that are not in the target charset.

So, what do you think? Maybe the NetBeans team can get this into the 6.1 product before final release?

Have you seen NetBeans 6.1? Give it a try, and blog about it. Who knows, you might win $500!

Posted at 09:53PM Apr 05, 2008 by John O'Conner in NetBeans  | 

NetBeans 6.1 Blog Contest

Get involved with NetBeans 6.1 and win cool stuff. The NetBeans 6.1 blog contest is active now. Don't miss your chance to win $500 or a t-shirt -- you always need an extra one of those, right?

Posted at 02:28PM Mar 30, 2008 by John O'Conner in NetBeans  | 

NetBeans 6.1 uses UTF-8 encoding for source files

I'm always happy when a company or product adopts Unicode as its charset. I think it makes perfect sense to do so. There are lots of good reasons why standardizing on Unicode is the right thing:

• the data charset can represent all modern, useful -- and many not-so-useful -- scripts
• charset consistency helps prevent data loss across application boundaries, including those among application, business tiers, application servers, and databases
• a single, multilingual charset simplifies the localization process

I was pleased to see that NetBeans 6.0 and the 6.1 beta uses the UTF-8 encoding (a Unicode encoding) as its default for project configuration and source files. The following figure shows the default setting in the project's property sheet:

This makes it much easier to edit non-ASCII, non-English source and property files. You can type text in any supported Unicode script right into Java source code. A legitimate usage would be comments or even localizable text in ListResourceBundle files. You can do that because NetBeans will save your files using the UTF-8 encoding by default...and NetBeans even puts this setting into the project's Ant file settings so that it compiles the file properly using the correct javac arguments. For example, NetBeans would use the following for a UTF-8 source file:

javac -encoding UTF-8 YourSource.java

Despite the potential benefits of this, NetBeans 6.1 still doesn't support this correctly in my humble opinion. Why not? Well, the biggest reason is simple: file corruption and permanent data loss. Ouch!

Let's take a simple "Hello, world!" example in Japanese. This is simple for NetBeans because of the UTF-8 encoding. The NetBeans editor even displays it correctly as shown here:

Unfortunately, the joy of this discovery was short lived when I discovered how easy it is to corrupt this data. Feel like experimenting with the charset encoding? Surely someone will. I suspected what would happen, so I didn't do this with any substantial code base...but someone will. I sure hope they use version control software.

Reopen that project property sheet, select another encoding, say ISO-8859-1 or windows-1252 since so many American and Western-European programmers use that by default. What happens? It appears as if nothing has happened. In this NB 6.1 beta, I see the same editor screen as before. NetBeans apparently hasn't yet reloaded the file from disk, so I'll force the issue by closing and reloading this file. The result is here:

Some of you, the super careful, nit-picky ones will now argue with me, "But John, you haven't really lost anything yet. 8859-1 and CP 1252 don't have those characters, but the original byte values are still entact. You can get them back in this example." OK, I concede the point. But now I'll show you some serious data loss, no messing around this time. Instead of ISO-8859-1, pick US-ASCII as the target charset. Add a comment or another line of code to the file. Save. And there you have it as shown here:

Now that's just not good. Did NetBeans save the file correctly? Sure. However, NetBeans can do better than this. I would argue that if NetBeans knows that the target encoding does not support the source encoding, it should at least warn the user that the resulting file will contain garbage characters and that parts of the file will be lost--permanantly in many cases.

So, just in case anyone over there in the NetBeans developer group can hear me...you have to fix this. Yes, I know it's a silly mistake for someone to do this, but NetBeans can help them avoid the problem. Just provide a warning dialog, "Saving this file in the target encoding will cause data loss because the target encoding does not support all characters in this file or project." Keep the encoding feature, just perfect it by helping some users avoid this costly mistake. The fact is that most software developers still don't understand character sets and encodings, and this is just an accident waiting to happen.

On a personal note: I really love NetBeans. And I hope this blog qualifies me for the NetBeans 6.1 blogging contest! I could probably file this under the "suggestions on how to enhance NetBeans 6.1" category.

Posted at 02:43AM Mar 30, 2008 by John O'Conner in NetBeans  |  Comments[2]