Encoding URLs for non-ASCII query params

JavaScript Unicode Web

Are you a web service API developer? The web truly is a world-wide web. Unfortunately, a great number of globally unaware developers are on the global web. This creates an odd situation in which web services are globally accessible but only locally or regionally aware.

There are a few important things to remember when creating a global web service. Let’s just cover ONE today: non-ASCII query parameters are valid, useful, and often necessary for a decent, global web service.

It seems so obvious to me, and it probably does to you. Sometimes a service needs to exchange or process non-ASCII data. The world is a big place, and although English is an important part of the global web, more people speak a different language. English is a big percent, but lots of people use Chinese or an Indic language too. Let’s make sure your web service can process all those non-ASCII characters in English or any other language!

Let’s look at some examples of non-ASCII query params:

  • http://example.com?name=田中&city=東京
  • http://example.com?名前=田中&市=東京

In these examples, you must perform two steps to get the query params (both keys and values) into the correct form:

  1. Convert the keys and their values to UTF-8 if they are not already.
  2. Perform the “percent encoding” on each UTF-8 code unit

To do #1, you’ll need to use whatever character conversion utility you have in your developer’s library: the Java charset encoding converters, whatever.

The #2 step is the important one for this blog. For each hexadecimal code unit in the UTF-8 query portion, you must “percent encode” the code unit. Let’s look at the first example query params:


The JavaScript function encodeURI actually does a good job of doing this for us:

encodeURI("name=田中&city=東京") produces the string:


Notice that you should also include this encoding for the keys in the param list. In the next example, I’ve used Japanese values for both keys and values.

encodeURI(“名前=田中&市=東京”) produces this string:


Note that both the keys and vaues have been “percent encoded”.

On the server side, your server will understand how to decode these values into their correct UTF-8 string values if you have configured it correctly. Correct configuration of a server usually involves a charset conversion filter for a servlet container and sometimes just a config setting for Apache.

More on this at a later time.


Leave a Reply

Your email address will not be published. Required fields are marked *