Author Archives: joconner

Setting JVM Proxy Variables

Setting JVM Proxy Variables

This would not typically warrant a blog post. However, I just spent days resolving it, and I wish I had found the solution earlier. I’m putting this out there for you, just in case you’re being whipped around by this problem.

The JDK responds to several system environment variables to control how it works for proxies. Here are the main ones:

  • http.proxyHost
  • http.proxyPort
  • http.nonProxyHosts

You need to set these so that the JRE libraries work properly for network connections, particularly library classes in the java.net package.

Setting System Properties

Set these properties in your JAVA_OPTS exported environment variable or on the command line with the -D option like this:

java -Dhttp.proxyHost=proxy.example.com

Watching for Trouble

And here’s the real reason I’m posting about this. I am working with the AWS Java SDK, deploying apps to EC2 instances. These are Linux instances, which are not a problem except in this situation.

Not only did I need to set a proxy host but also a set of non proxy hosts to correctly access EC2 Meta-Data. I set my http.nonProxyHosts like this:

java -Dhttp.nonProxyHosts=’localhost,*.compute.internal,169.254.169.254′ MyApp.jar

However, I consistently received runtime errors from the AWS Java SDK libraries telling me that the metadata service couldn’t be reached, HTTP 403 errors for miles and miles.

The problem: the http.nonProxyHosts string is in the wrong format!

Using the Correct Format

The correct format is to use an OR symbol (“|”) between those hosts instead of a comma (“,”). Additionally, a simple OR is not sufficient. The problem is that the “|” needs to be escaped on Linux systems. So the correct way to set the variable is like this:

java -Dhttp.nonProxyHosts=’localhost\|*.compute.internal\|169.254.169.254′ MyApp.jar

If your app refuses to connect with an external service and you’re diddling with proxies, check your proxy environment variables again:

  1. Use “|” instead of a comma between non-proxy hosts.
  2. Use “*” instead of just leaving subdomains empty. Use this (“*.compute.internal”), not this (“.compute.internal”)

Enabling CLDR Locale Data in Java and Groovy

Enabling CLDR Locale Data in Java and Groovy

Duke jdk8 1 Today a colleague asked whether I could provide a list of localized country names for Portuguese. I know that the CLDR database contains all that great localization goodness, but I didn’t have the time to dig through the raw data. Still, I wanted to help, and I know that the Java platform has included CLDR data since 1.8. I decided to create a short Groovy language script to retrieve and display the information. Because Groovy runs in the JVM, it also has access to that CLDR information.

The Groovy Script

Here’s the script I came up with. I named it “countries”.

#!/usr/bin/env groovy 
/** 
 * Created by joconner on 6/29/16.
 */

supportedLangs = Locale.getISOLanguages();

if (!(this.args.length > 0 && supportedLangs.contains(this.args[0].toLowerCase()))) {
    printf("Usage: countries <lang_code>\n")
    printf("\t<lang_code> must be an ISO lang code. Examples: en, pt, fr, es, ja.\n")
    System.exit(0)
}
currentLocale = new Locale.Builder().setLanguageTag(this.args[0].toLowerCase()).build()
locales = Locale.getISOCountries()
for(c in locales) {
    targetLocale = new Locale.Builder().setRegion(c).build()
    printf("%s, %s\n", targetLocale.getCountry(), targetLocale.getDisplayCountry(currentLocale))
}

I won’t go into the script in detail. If you are familiar with Java, you’ve probably seen these classes and methods before. The interesting thing is that the CLDR data is not enabled by default.

Enabling CLDR

For backward compatibility, CLDR locale data is not enabled by default. However, you can enable it by setting a system property on the Java command line. The system property is java.locale.providers. If you set this to the value CLDR, the JRE will use CLDR as the locale data source.

Run the above script like this to get Portuguese (pt) names for all supported ISO countries in the JRE: groovy -Djava.locale.providers=CLDR countries pt

Although this example is written in Groovy, the same system property is available with the Java command line as well. Just run your Java application like this: java -Djava.locale.providers=CLDR

Analyzing Differences

I ran the above script twice, once using CLDR data and once using the default JRE data. For the pt language, I discovered a couple differences in JDK 1.8.0_92:

CLDR JRE
BQ, Bonaire, Sint Eustatius, and Saba BQ, Bonaire, Sint Eustatius and Saba
SX, Sint Maarten SX, Sint Maarten (Dutch part)

I suspect you’ll find other and perhaps more significant differences for other languages.

Downloading Code

You can git the code for this article from my github account. Enjoy!

Iterating Java Characters

Logo60s2 A char is not always a full Unicode character. I touched lightly on this before, but the point should be repeated. Unicode has evolved, and the Java char definition has too. If you need to iterate through a string of characters, you have lots of ways to interpret that functionality. You could mean any of these and maybe other options too:

  • maybe you mean UTF-16 code units
  • maybe you mean glyphs on the screen
  • maybe you mean Unicode code points

For this blog, let’s assume the latter; you need to iterate through Unicode code points.

Mistaking a Char for a Code Point

I recently ran across some code that confirmed that business names contain valid characters. The code looked something like this:

public boolean isValidBusinessName(String businessName) {
    boolean isValidName = true;
    for(int i = 0; i < businessName.length() && isValidName; i++) {
        char ch = businessName.charAt(i);
        isValidName = isValidCharacter(ch);
    }
    return isValidName;
}

The code authors were surprised when isValidBusinessName failed to accept all printable Unicode characters. The problem is that the method parses the string one code unit at a time. It doesn’t always test full code points. Instead, when the business name contains characters outside the basic multilingual plane, the algorithm incorrectly tests surrogate values.

Correctly Iterating Code Points

A more-correct way to parse the business name is by code point. That code looks similar, but is slightly different:

public boolean isValidBusinessName(String businessName) {
    boolean isValidName = true;
    for(int i = 0, codePoint = 0; i < businessName.length() && isValidName; i += Character.charCount(codePoint)) {
        codePoint = text.codePointAt(i);
        isValidName = isValidCharacter(codePoint);
    }
    return isValidName;
}

This algorithm grabs code points from the text string and doesn’t incorrectly work on orphaned surrogate values. Surrogate pairs represent a single character, and you should typically not separate the two halves.

Checking Twice

Chances are that you have code like this in your product. With all the new emoji characters and so many interesting and valid characters above the basic multilingual plane, you may want to reconsider how you parse text. If your application should allow those additional characters, you’ll need to revisit areas that parse strings or validate individual characters.

Good luck in your work. You’re not alone in this. Lots of great help already exists in the excellent Javadocs. Make sure you look at both the Character and String classes again if you haven’t looked at their documentation in a while. They both have new methods that will help you work with code points instead of code units.

Adding a Server to Your AWS VPC

Vpc In a previous blog, I described how to start with AWS Cloudformation by creating a simple network with a vpc and a subnet. This time I’ll finish up basic Cloudformation by showing how to add a couple security groups and a server.

Creating Security Groups

Security groups define what ports and addresses that your servers will respond to. They also define what addresses that your servers can send information to. In an very common design pattern, you would create security groups and subnets for every tier in your stack, typically web, business logic, and database. For this example, I’ll forgo that for something much simpler: a single business tier with a couple security groups to define allowed incoming and outgoing traffic.

The first security group will allow SSH access. Here’s how I represent it in a Cloudformation template:

"SECURITYGROUPSSH": {
    "Type": "AWS::EC2::SecurityGroup",
    "Properties": {
        "GroupDescription": "Enable SSH access to port 22",
        "VpcId": {
        "Ref": "VPC01"
    },
    "SecurityGroupIngress": [{
        "CidrIp" : "0.0.0.0/0",
        "IpProtocol" : "tcp",
        "ToPort" : 22,
        "FromPort": 22
    }]
}

This group basically says that communication on port 22 is allowed into servers within this group from any IP address.

The second security group will allow both HTTP and HTTPS traffic:

"SECURITYGROUPAPPSERVER": {
    "Type": "AWS::EC2::SecurityGroup",
    "Properties": {
        "GroupDescription": "Enable HTTP/S access on port 80/443",
        "VpcId": {
            "Ref": "VPC01"
        },
        "SecurityGroupIngress": [
            {
                "CidrIp": "0.0.0.0/0",
                "IpProtocol": "tcp",
                "ToPort":80,
                "FromPort": 80
            },
            {
                "CidrIp": "0.0.0.0/0",
                "IpProtocol": "tcp",
                "ToPort":443,
                "FromPort": 443
            }
        ]
    }
}

Creating the App Server

The last step of our network is to launch a server instance within our subnet and groups. The following shows the Cloudformation resource:

"SERVER01": {
    "Type": "AWS::EC2::Instance",
    "Properties": {
        "ImageId": "ami-d2c924b2",
        "SecurityGroupIds": [ 
            {"Ref": "SECURITYGROUPAPPSERVER"}, 
            {"Ref": "SECURITYGROUPSSH"}],
        "InstanceType": "t2.micro",
        "KeyName": "charprop",
        "SubnetId": {
            "Ref": "SUBNET01"
        }
    }
}

AWS resources always have a Type, and in this case it is AWS::EC2::Instance. Other important attributes are defined within the Properties object. Note a few of the important properties: ImageId, SecurityGroupIds, InstanceType, KeyName, and SubnetId.

Servers are created with OS and application images. Images are predefined and have identifiers like ami-d2c924b2, which happens to be a minimal CentOS image. You can create your own image or pick from hundreds, maybe thousands, at the AWS Marketplace.

The SecurityGroupIds assigns security groups. In this case, I’ve assigned 2. The “Ref” keyword means that these security group ids are defined within this same Cloudformation template by their logical id.

The InstanceType defines the server CPU and basic RAM sizes. The t2.micro has 1 GB RAM and 1 virtual CPU. Of course, your application needs will much likely be different. For this example, I will eventually deploy a simple RESTful service, and I think this size will suffice.

Finally, KeyName refers to a keypair defined in my AWS account. If you want to SSH into any instance, you’ll need to associate a keypair and references its name in the Cloudformation template. You can also create key pairs within the template, which I have not done here.

Finishing the Deployment

The last step just deploys a server, a do-nothing server that just sits there. Of course, the point of this blog series is more significant. I wanted to show how to use Cloudformation to set up a network and deploy an application within it. So there is one last piece to this: deploying an application onto the server.

Stay tuned for the last blog in this particular series. Next step: deploy an application.

Getting Source Code

You can see the source for this example on my Github charprops project. The final project will deploy a character properties service that provides Unicode data on characters. Check it out and help out too!

A Little Java Character History

Logo60s2

The Java language has supported Unicode from its beginning. In those early days, the Unicode character set defined characters with integer values in the range 0x0000 through 0xFFFF. That’s 65,536 possible character values in the full Unicode set. Java’s char type was defined to represent a single character in that range.

However, Unicode changed. It grew bigger. It can now define character values all the way up to 0x10FFFF. The range grew by 16x. As a result of that growth, Java’s char type simply cannot represent every possible Unicode character anymore. A char still has its original range definition, so it can only have an unsigned integer value up to 0xFFFF.

Fortunately, the Unicode consortium considered how its growth might affect existing systems. It created a clever encoding form that allowed systems to use two 16-bit values as an alias for character values above 0xFFFF. That encoding form is called UTF-16. The consortium quickly partitioned a couple special ranges within the original 65,536 values that could be used in this encoding form. Those special values are called surrogates. A pair of surrogates, in the UTF-16 encoding form, can represent any defined character above 0xFFFF.

To keep up with the expanded Unicode range, Java’s char type has changed its definition a little bit. It is now a Unicode code unit in the UTF-16 encoding form. It’s still a 16-bit value, but you can’t really think of it as just a character anymore. It’s a code unit. Some 16-bit code unit values are complete characters, but some are only part of a surrogate pair. Remember surrogate values are not complete characters. A valid surrogate pair represents a single Unicode character somewhere above 0xFFFF.

So, let’s get right to the point. Sometimes a char is a complete character, and sometimes its only part of a surrogate pair. This makes text processing tricky.

In a future post, I’ll describe how to correctly iterate through a Java string. Because a char isn’t what is used to be, parsing a string isn’t as simple as it once was.

Using Cloudformation to Create a Virtual Private Cloud in AWS

Cloudformation

Creating your network infrastructure in AWS is simplified by a service called Cloudformation. Cloudformation allows you to specify your network subnets, groups, and other resources in a JSON file. When you submit that JSON file to AWS, the service will create the resources in your AWS account. This article demonstrates Cloudformation by creating a basic virtual private cloud (VPC) with an accompanying subnet and gateway. You will need an AWS account to test this out. Preferably you would also have read about using the command line AWS client.

A Cloudformation template is a JSON file that describes each resource in your network. Every resource has a specific set of attributes that you can define within this template. Amazon documents all resources and their attributes on its on site as well, so take a look at it for more complete details.

The general structure of a template is a basic map of resources within a “Resources” id. Each resource has a Type and various Properties. A VPC resource that defines a set of IP addresses in a 10.0.0.0/28 block, looks like this:

{
  "Resources": {
    "VPC01": {
      "Type": "AWS::EC2::VPC",
      "Properties": {
        "CidrBlock": "10.0.0.0/28",
        "Tags": [ 
          {
            "Key": "Name",
            "Value": "vpc-charprop"
          }]
      }
    },
    ...
}

Within a Cloudformation template, you can refer to other resources using their logical id. For example, you can associate a subnet with “VPC01” with the following declaration:

  "SUBNET01": {
    "Type": "AWS::EC2::Subnet",
    "Properties": {
      "VpcId": {
        "Ref": "VPC01"
      },
      "CidrBlock": "10.0.0.0/28",
      "Tags": [
        {
          "Key": "Name",
          "Value": "subnet-charprop-public"
        }
      ]
    }
  },

I’ve placed these and other resources into a larger template called network.json. Using this file, you submit your resource creation request with the AWS CLI with this simple command:

aws cloudformation create-stack --stack-name charprop-network --template-body file://./network.json

Assuming network.json is in your current working directory, the command should return immediately, showing a JSON description something like this:

{
  "StackId": "arn:aws:cloudformation:us-west-2:446581796491:stack/charprop-network/c3aa1530-0848-11e6-a533-50a68a2012ba"
}

You’ve successfully created a VPC and subnet using Cloudformation!

In the next article, I’ll add security groups and a Linux machine instance.

Preparing to Use the Amazon Web Services Command Line Client

General AWScloud

I’m so impressed with the AWS services that I’m going to use them to create my own set of services for my personal domains. The plan is to introduce a few proof-of-concept services to show others both how to use AWS and how to have some fun with a couple i18n/g11n services as well.

Let’s get started. First thing, you’ll need an AWS account. It’s not difficult at all, and you can get a 12-month free trial. Yes, 12 months F-R-E-E! If you’re a hands-on engineer or dev ops person, you owe it to yourself to investigate this if you haven’t already.

If you’re new to AWS, you’ll spend your first hours browsing the console, creating server instances, etc. However, for anything beyond casual browsing, the AWS command-line client (AWS CLI) is critical. The AWS CLI helps you create resource templates that you can reuse over and over again in scripts. And we all know that creating a well-defined script is critical for creating anything in a reproducible, reliable, standard way. This article describes how to setup and use the AWS CLI. Your steps are these:

  1. Create a ‘deploy’ user.
  2. Create an access key for the ‘deploy’ user.
  3. Install and configure the AWS CLI.
  4. Test it out.

Creating a User

General user

Whether you use the CLI or an AWS SDK for Java, Ruby, or other language, you’ll run those scripts/programs using a set of secret access keys. You get those keys when you create users for your account. In the console, click on the Identity and Access Management (IAM) links. Create yourself a new user called “deploy”. You should use this user when creating or managing resources. Place that user in a administrative group or provide a policy that allows for creation of the resources you’ll need.

Creating an Access Key

Security Identity AWSIAM long termsecuritycredential

Once you have a “deploy” user, you’ll need to create an “Access Key” for it. Download it immediately at creation time. This is the only time you’ll be able to download it. This “key” has two parts: a key id, and a key secret. The command line client will need these, so store it away somewhere safe.

Installing the AWS CLI

SDKs AWSCLI

You can install the CLI using a variety of options. Two easy options are the pip and Homebrew installers.

Use this to install with the Python pip installer:

pip install awscli

Or use Homebrew on a Mac:

brew install awscli

Once installed, you can use the following the create a default aws profile to use with your account. Remember that access key you created earlier? You’ll need that now. Run the following:

aws configure

This tool will ask you a few questions like this:

$ aws configure
AWS Access Key ID [None]: YOUR_KEY_ID_HERE
AWS Secret Access Key [None]: YOUR_SECRET_ACCESS_KEY_HERE
Default region name [None]: us-west-2
Default output format [None]: json

Copy and paste your access key id and secret key into the tool. This will create a couple files in the hidden ~/.aws directory. The AWS CLI will use these configuration files when accessing your account. You can find even more detailed information on the AWS CLI configuration website.

Confirming Your AWS Client Installation

Once you’ve installed and configured the CLI correctly, you should be able to work with your account resources immediately. Give it a try:

aws iam list-users

This should return at least two users from your account, including the recently created deploy user. The output looks like this:

{
    "Users": [
        {
            "UserName": "deploy",
            "PasswordLastUsed": "2016-03-25T04:34:00Z",
            "CreateDate": "2016-03-07T00:51:35Z",
            "UserId": "ABCD1234ABCD1234",
            "Path": "/",
            "Arn": "arn:aws:iam::123412341234:user/deploy"
        },
        {
            "UserName": "jsoconner",
            "PasswordLastUsed": "2015-04-14T04:41:18Z",
            "CreateDate": "2015-04-11T07:21:45Z",
            "UserId": "ABCD1234ABCD1234",
            "Path": "/",
            "Arn": "arn:aws:iam::123412341234:user/jsoconner"
        }
    ]
}

If you got this far and are still reading, you’re ready to do something even better with the AWS CLI. In the next article, we’ll create a complete stack of resources using Cloudformation.

Thanks for reading. If you enjoy this type of content, please provide feedback.

Starting with Amazon Web Services

I recently began working with Amazon Web Services (AWS). AWS truly is an amazing set of services that allow you to create scalable, durable applications in the cloud. AWS provides a powerful set of capabilities to create, deploy, and manage cloud resources with a convenient command line interface (CLI) and a browser console. Over the next few weeks, I will explore some of these AWS capabilities and bring you along for the ride. I’ll tackle a few tasks in quick succession:
1. Create a virtual private cloud (VPC) for a demo app.
2. Create a security group to limit access to compute instances.
3. Spin up an elastic compute cloud (EC2) instance.
4. Deploy a sample application. It should be both fun and educational…mostly for me, but I hope you’ll get something out of it too.

Category: AWS

Headless Raspberry Pi

Although my very first experience with the Raspberry Pi was less than impressive, my subsequent encounter proved fruitful. After following a youtube tutorial, I learned that my initial 8GB micro SD card had NOOBS installed. While it is intended for noobies maybe, I guess my use-case was sufficiently off-track to derail my noob attempt. I don’t have an extra monitor, keyboard and mouse lying around the house, so I needed to interact with a “headless” raspberry pi. I needed to interact with the pi using only my laptop, a network cable, and eventually a wifi adapter. My first step was to burn my own Raspian OS image onto a 64GB micro SD card. A 64GB is not needed. Raspbian doesn’t require anywhere near that much; an 8GB card will do nicely. Being Christmas, I just happened to have access to a new camera’s 64GB card…sorry Robyn (my spouse). I’ll replace it, promise. Pick up the OS image and instructions up from the Raspberry Pi site. Rather than recreate the entire experience here, I’ll point you again to the Youtube video above. It worked as advertised. Use Part 2 of the tutorial to add remote GUI and VNC abilities. Then, check out Part 4 of that tutorial when you’re ready to add a wifi USB adapter to the mix. I’m not certain how long this process took, but it happened while re-watching Star Wars IV (A New Hope). So, there you have it. I’m up and running on my new Raspberry Pi! RaspPI Desktop

First Experience With Raspberry Pi

Today my wife presented me with a Raspberry Pi Cana Kit. Out of the box, I have:

  • the Pi itself
  • a power adapter
  • an HDMI cable
  • a micro-sd card
  • a Wifi adapter

It doesn’t have a keyboard or display of course. I figured the SD card had the OS on it, so I inserted the SD card into the Pi. Then I stuck a network cable into it, connected it to my router, and tried to find it on my network from my laptop. My idea was that I’d connect via ssh. No bueno.

Hmmm…what’s going on? Isn’t this thing booting automatically and connecting to the network?

Swallowing my pride, I picked up the small set of docs. It appears that no OS is running yet. Instead, I think it is running a boot loader called NOOBS, so I can’t SSH into it…yet. I think it’s just sitting there waiting for me to answer a couple questions before it loads an OS.

Sigh…maybe I’ll have to actually connect a monitor and a keyboard directly to it for the first boot and config step? Or maybe I’ll have to to read page 2 of the docs? Heaven forbid.