Hackle's blog
between the abstractions we want and the abstractions we get.

Check out my workshops at NDC { Minnesota }
Nov 15-16 TypeScript Power Users
Nov 17-18 Simple by Design: Declutter Your Architecture, Code and Test

Serialize like javascript - the idea

I have Javascript envy when it comes to consuming JSON, because it needs no serialization / deserialization. This is not the case for most languages, which usually require some sort of serialization, and such operations are usually lossy and may result in unintended effects.

We try to explore a way of serialization / deserialization so this statement can hold: when API / data schema is concerned, if the server can promise to be backward-compatible, then it's possible to build the client side application to be forward-compatible, just like that in Javascript.

(This post is inspired by Rich Hickey's talk Spec-ulation, as unrelated as it may seem)

Let's say we want to build an awesome app called plus1 whose job is to make any user one year older.

plus1 in Javascript

httpClient.get('api/user/123.json')
	.then(user => this.user = user);

// http returns
GET api/user/123.json
{
	name: 'George',
	age: 11
}
	// later in the controller
	this.user.age = this.user.age + 1;
httpClient.put('api/user/123.json', this.user)
	.then(response => celebrateWith10Toasters('success!'));

// HTTP 200 OK

Pretty simple right? However, things are not quite as intuitive in some other languages

the same application in C# as an example

var response = httpClient.Get("api/user/123.json");
var user = Deserialize<User>(response);

// User is defined as
class User
{
	String Name;
	Int Age;
}
user.Age = user.Age + 1;
var payload = Serialize<User>(user);
httpClient.Put("api/user/123.json", payload);
// HTTP 200 OK

Two extra steps for serialization and deserialization, not great, but not the end of the world either.

when it becomes not so great

Assume the schema of user profile changes, an address is now added.

Note this is not a breaking change (which means the server either requires more, or gives less)

GET api/user/123.json
{
 name: "George",
 age: 11,
 address: "thunderbolt street" // <-- new
}

Our plus1 app in javascript will have no problem dealing with this, as the new field will simply flow all the way through to the time we send the modified user back to the API. In fact, nothing needs to be done!

The plus1 in C# though, will now have a problem...

This presents the versioning problem - limited by serialization / deserialization, client side can only deal with the old version / schema, and will result in unintended deletion when dealing with the new version.

One way to work around this, is to demand the server not change the schema of the API response at all. Instead, create a new version of API and serve it separately through a different URI, e.g. api/user/v2/123.json. Client-side applications can switch to the new version whenever it is ready.

This is a sound solution. However, the server side can rightfully argue that such changes are not breaking changes, so there is no need for versioning. Indeed, if we version each of such non-breaking changes, and if the API is a fast evolving (and popular) one, the client-side applications will have a hard time (if ever) catching up with the latest version. And before we know it, we are on v30 and chances are, many (if not all) of these versions need to be maintained!

a more robust solution

Think about it, if the client-side application has been written in Javascript, we would never have such a headache (even though we might have others) - new fields can be freely introduced and they will freely flow through (and back), never to be lost in translation. Not even a single line of code needs changing at the client side to deal with this!

Is the moral, then, that we should all desert our beloved programming language and use Javascript only, so that we can have free-flowing JSON? Tempting... but I for one am not sure about giving up the likes of Idris, Haskell and F#, yet, not even C#.

But how do we make up for the loss in serialization / deserialization? Well, it's simple, we just keep JSON on the side!

// user is defined as
class User
{
	String Name;
	Int Age;
	String OriginalJsonString;
}

(I am going for simplicity here, you don't have to make your implementation this crude. By all means, go for generics or reflection)

string modified = Serialize<User>(modifiedUser);
string merged = MergeJSON(modifiedUser.OriginalJsonString, modified);

// OriginalJsonString {  name: "George", age: 11, address: "thunderbolt street" }
// serialized: { name: "George", age: 12 }
// combined: { name: "George", age: 12, address: "thunderbolt street" }

httpClient.Post("api/user/123.json", merged); // nothing is lost!

That's the idea! Wouldn't you agree that it's really, really simple?

Now you may be wondering how we can implement this MergeJSON method - honestly I am not really worried but as a developer I do enjoy getting my hands dirty, so I will soon get my hands dirty in writing MergeJSON in Idris. (edite: and it's right here MergeJSON in Idris))

In the mean time, you might have better luck finding that it's already there or there for free!