Migrating from Latin-1 to UTF-8
You'd think this sort of problem would be resolved by now, but it's not. It's still almost impossible to quickly and easily migrate an application from the too common default Latin-1 to UTF-8 character set encoding. The problem isn't that UTF-8 can't handle the conversion. No, that's definitely not it. UTF-8 can represent any Latin-1 character and much, much more. The problem is that the Latin-1 charset is so deeply ingrained as the default in every software interface that you just have so many faulty conversion points. A conversion point is a handoff point between one software component and another, a place where character encodings matter and where faulty conversions are way too common.
Here's an example: a simple web application that stores names and addresses in a database. Chances are, if you haven't done anything explicit to change this, the web page itself will have no charset encoding associated with it. And neither will your application server. And neither will your database. And without explicit settings, many applications use Latin-1 as the default character set. So, you'll be able to enter, store, retrieve, and display common Western European names, but you won't be able to handle Russian or Japanese or Chinese or, well, you get the idea.
So let's imagine you decide to convert from Latin-1 to UTF-8 so that you open up your application to the rest of the world's languages and scripts. What does that mean? What must you do? How do you start?
Here are some of the charset conversion points you'll need to resolve as you migrate through this problem:
- database tables
- database connections
- application and/or web server frameworks
- web page
- form encodings
- JavaScript or other browser scripts
To help you get started, I've discussed the first 4 conversion points in the article Character Conversions from Browser to Database. Go ahead, take a look. But come back here to let me know what you think.
I'll talk about some of the JavaScript issues in an upcoming blog.
Posted at 10:26PM Apr 12, 2008 by John O'Conner in Unicode |
Basic Definitions for a Unicode Discussion
If you want to communicate, defining confusing terms right up front is always a good first step. So I'll try to define some Unicode terms:
| Character | The smallest unit of meaning in a written language. This unit typically has a common shape and meaning, although specific shapes can vary quite dramatically. Specific shapes are more commonly called glyphs. |
| Character Set | An unordered collection of characters. |
| Coded Character Set | An ordered character set in which each character has an assigned integer value. |
| Code Point | The integer value of a character within a coded character set. |
| Character Encoding | A mapping of code points to a series of bytes. |
| Code Unit | A single octet or byte of an encoded character. |
| Charset | Often used as a synonym for Coded Character Set. |
You can always see more terms by visiting the Unicode Glossary.
Posted at 11:40PM Apr 08, 2008 by John O'Conner in Unicode |
Unicode 5.1 released this week
The Unicode Consortium released Unicode 5.1 this week. With more than 100,000 characters, Indic and South East Asian script additions, and more, this release enables ideographic variation sequences used in Japanese, Chinese, and Korean text. Learn more about these additions and more by reading the Unicode 5.1 Press Release.
Posted at 12:33AM Apr 08, 2008 by John O'Conner in Unicode |
Call for participation: 32nd Internationalization & Unicode Conference
The Internationalization & Unicode Conference is the technical conference for software and web internationalization engineers. If you have a product that implements the Unicode standard or an idea that will help others work with this standard, share your knowledge.
The "call for participation" is an invitation for Unicode gurus, clients, and vendors to submit papers for this upcoming conference. Learn more at the Conference web site.
Posted at 12:02AM Apr 03, 2008 by John O'Conner in Unicode |
| Archives | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Links | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Referrers | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
