We all know that null is a “billion-dollar mistake”, that it creates a lot of easy ways to make terrible mistakes. But it’s only so bad when it’s not checked by anyone and the compiler doesn’t force you to check it, right? Well, the title might be a spoiler, but let’s find out…

Context / What are you talking about?⌗

In this article, I’m specifically talking about the Kotlin and C# post v8 approach to null. I’m more familiar with Kotlin than C#, so I’ll mostly be talking about it, but their approaches are similar, so that doesn’t really matter.

Basically, you can’t assign null to just any reference type in Kotlin (and C# post v8 with certain settings). For example, this is a compilation error:

// compilation error
val a: String = null;

// ok
var b: String = "not null";

// compilation error (again)
b = null;

And since you can’t assign null to non-nullable types, null can’t screw you! You can’t get a null pointer (reference) exception, etc. Cool!

If you really want to assign null to something, you need to explicitly mark the type of the variable as nullable with T? syntax:

// ok
val a: String? = null;

// also ok
var b: String? = "not null";
b = null;

But when you have a T? (nullable) type, you also need to explicitly check for null. E.g. given a class Mine with a cat field and a variable mine of type Mine?, you can’t access mine.cat, it would be a compilation error:

val mine: Mine? = /* ... */
val your: Mine  = /* ... */

// compilation error
val a: Cat = mine.cat

// elvis operator, if `mine` is null,
// then `your` will be used
val b: Cat = (mine ?: your).cat  

// safe call operator, if `mine` is null, 
// then so be `c`        
val c: Cat? = mine?.cat

// “trust me, I'm an engineer” operator, 
// throws an exception if `mine` is null 
val d: Cat = mine!!.cat

if mine != null {
    // Special Kotlin trick: 
    // in this block `mine` actually has type `Mine`.

    // perfectly fine (we've checked)
    val x: Cat = mine.cat
}

(The same applies for calling a function with an argument of non-nullable type, you need to either check that something is not null or explicitly assume so)

The “special kotlin trick” is called “smart cast”. Basically, if you check that a variable is not null (or has a particular type), then the compiler changes type of the variable in the block, where the check holds. You can read more about it in Kotlin docs: typecasts, null-safety.

And it’s all great, don’t get me wrong! This behaviour is a lot better than the plain old “every reference can be null, just crash if it is and it’ll be fine”. However…

Why it’s still meh⌗

In my opinion, this comes down to 2 things: generality and extensibility.

Generality⌗

In modern languages, we have ways to abstract over types. Usually, the mechanism to do so is called generics. For example, a HashMap doesn’t care what the key and value types are, it only cares that the key is comparable and hashable. Thus you can have HashMap<String, i32>, HashMap<User, Settings>, etc while only writing HashMap once.

That’s all good and all, but what’s the problem? You may ask. Well… Consider this example: the HashMap has a method get which accepts key and returns the value associated with the key if there is one. But what should it return if there is no value associated with the key? It seems reasonable to return null since there is no value. So something like this:

class HashMap<K, V> {
    // If `key` is present in the hash map, 
    // returns the value associated with it. 
    // Otherwise returns `null`.
    fun get(key: K): V? { /* ... */ }
}

But what happens if we use HashMap<_, T?>? Well, then V is T? and so V? is T?? which is the same as T? because there is only one null value.

The crux of the problem is this: you can’t distinguish between “absence of value” (no key in the map) and “presence of absence of value” (value is null). And if you need to distinguish them, then you need to call containsKey or something which is suboptimal.

This is just a simple example, but in general, you can’t use null in generic code because null is too special and unique.

Extensibility⌗

null is not extensible. This mechanism is only useful when you have an optional value: either there is a value, or there isn’t.

It isn’t useful when you need to express errors (null is used for this anyway, but this is problematic), exclusive or (either you have A or B, but not both and not neither), etc.

It may seem like this isn’t a problem — we were talking about expressing optional values from the very beginning. But bear with me, and for now just understand that null is not extensible and there may be other, more extensible, mechanisms.

The saviour?⌗

As I said before, there may be a more general and extensible way to handle optional values. And there is! It has a lot of different names — sum types, enumerations, tagged unions, discriminated unions, etc. They’re all conceptually similar, but I like the “sum types” name the most, so that’s what I’ll call it.

In any case, the concept stays the same — a type that can represent one out of multiple choices. Here are some well-known examples of definitions of sum types:

enum Option<T> { None, Some(T) }
enum Either<L, R> { Left(L), Right(R) }

data Maybe a = Nothing | Just a
data Either a b = Left a | Right b

Option (Maybe) is exactly what we’re looking for! It can be either None (Nothing) or Some(value) (Just value). So either there is nothing or there is some value, exactly the same as with null so far.

But it actually solves all the null problems I’ve mentioned!

Generality⌗

Option<Option<T>> is a meaningful type, unlike T??. In the same example with HashMap::get there isn’t any problems if it returns Option<V>.

impl<K, V> HashMap<K, V> {
    // If `key` is present in the hash map, 
    // returns the value associated with it. 
    // Otherwise returns `None`.
    fn get(&self, key: &K) -> Option<V> { /* ... */ }
}

If the key isn’t present in the HashMap<K, Option<V>>, then None is returned. If the key is present but the associated value is None, Some(None) is returned. Otherwise, if the key is present and the value isn’t None, then Some(Some(value)) is returned.

The use of sum types gives us 3 distinct kinds of values that can be distinguished from one another, instead of an ambiguous null.

Extensibility⌗

Sum types can be used for optional values via Option-like types. But they are not limited to only this. You can define your own sum types. It’s very handy when you need to return errors (See Rust Result for example), define the errors themselves or just in general when you need to hold different kinds (types) of data in one place.

Explicitness⌗

There is one thing you may prefer about null over sum types: T can be coerced to T?. That means that you don’t need to explicitly wrap values in Some (Just).

This may be handy, because you need to type ~6 characters less if you want to pass T to a function that accepts T?. However, I don’t think that wrapping inconvenience outweighs the benefits of sum types.

fun f(x: Int?): Int? = x?.let { it + 1 }
val res = f(1)

fn f(x: Option<i32>) -> Option<i32> { x.map(|it| it + 1) }
let res = f(Some(1));
//          ^^^^^ ^

It would be interesting to see a language with sum types and T to Option<T> coercion though 👀 (see note about Swift down below)

Niche optimization⌗

There are some types that have unused space in their memory representation — a niche. For example, bool is only 1 bit of information, but it uses 1 byte of space, so 7 bits are unused. References (&T, &mut T) and NonNull can never be null, meaning that they have an unused bit pattern — all zeroes. The same goes with NonZero*. Enumerations, just as bool, can have unused bits/bit patterns.

What if we could use this for something actually useful? Well, Rust can.

If a sum type has one of its variants being a type with a niche, then it can use it to represent other variants, instead of using a tag. I.e. Option<&T> isn’t identical to struct { value: union { Some(&T), None }, tag: u8 } (pseudo syntax), but instead it’s just union { Some(&T), None } where the None is encoded as 0 (since reference can never be 0 you can distinguish variants without a tag). Moreover, this exact optimization of Option<&T> is even guaranteed, so it is fully layout compatible with *const T. This allows using it in C-ffi, making None on the Rust side a null on the C side.

This optimization can be observed using mem::size_of function:

// `u8`, `bool` and `Option<bool>` all occupy 1 byte.
// `Option::<bool>::None` is encoded as 2. It's fine, since `bool`
// can only ever by 1 or 0.
size_of::<u8>() = 1
size_of::<bool>() = 1
size_of::<Option<bool>>() = 1
unsafe { transmute::<Option<bool>, u8>(None) } = 2
unsafe { transmute::<Option<bool>, u8>(Some(true)) } = 1
unsafe { transmute::<Option<bool>, u8>(Some(false)) } = 0

// `Option<Option<bool>>` behaves similarly, it just needs 2 patterns.
size_of::<Option<Option<bool>>>() = 1
unsafe { transmute::<Option<Option<bool>>, u8>(None) } = 3
unsafe { transmute::<Option<Option<bool>>, u8>(Some(None)) } = 2

// On 64 bit machines references take 8 bytes.
// `Option::<&_>::None` is encoded as a `0`/`null`.
size_of::<&u8>() = 8
size_of::<Option<&u8>>() = 8
unsafe { transmute::<Option<&u8>, *mut u8>(None) } = 0x0000000000000000

// `u8` has no niche, so `Option` needs a tag (in this case: first byte)
size_of::<Option<u8>>() = 2
unsafe { transmute::<Option<u8>, Pair<u8, MaybeUninit<u8>>>(None) } = (0, MaybeUninit<u8>)
unsafe { transmute::<Option<u8>, [u8; 2]>(Some(0xAA)) } = [1, 170]

// You may have expected 9, instead of 16, 
// but in Rust, size is always a multiple of the alignment.
// References are 8-bytes aligned. The nearest 
// multiple of 8 bigger than 8 is 16.
size_of::<Option<Option<&u8>>>() = 16
// It seems like the first word is tag and the second
// is optional reference.
unsafe { transmute::<Option<Option<&u8>>, Pair<usize, MaybeUninit<usize>>>(None) } = (0, MaybeUninit<u8>)
unsafe { transmute::<Option<Option<&u8>>, [usize; 2]>(Some(None)) } = [1, 0]
unsafe { transmute::<Option<Option<&u8>>, [usize; 2]>(Some(Some(&1))) } = [1, 93965956011124]

Edited output of the test program.

A sad note: Kotlin⌗

Kotlin does support sum types via sealed classes. Here is an example how you could define and use Option (play):

sealed class Option<T> {
    class None<T>() : Option<T>() {}
    class Some<T>(val value: T) : Option<T>() {}
}

fun main() {
    val a: Option<Int> = Option.Some(12);
    val b = when(a) {
        is Option.None -> "None"

        // `a` is smart-casted to `Some`, so `value` is accessible.
        // IMO pattern matching would be better, but that's ok.
        is Option.Some -> a.value.toString()
    }
    println(b)
}

Nevertheless, Kotlin doesn’t use this for optional values! In fact, it doesn’t even have an Option class in the standard library. For me, it seems like a big omission. It seems like making T? equivalent to Option<T> and Option<N> (where N is not Option) be layout compatible with Java’s N (i.e. Java’s null being the same as None<N>) would be sufficient…

A sad note: Java & Scala⌗

Originally I didn’t plan to say anything about Java (or Scala) — after all it is a classic example of why unchecked null is problematic. Everyone (?) already knows that it’s hard to always remember to check for null by hand and that it causes NullPointerExceptions all of the time. So it felt like brining up Java here wouldn’t be valuable.

But! Recently I was reminded that Java has an interesting unsoundness in its type system that is related to null.

In a nutshell Java (and similarly Scala) assumes that if a value of a type exists, then this type is well-formed. Well-formedness means that the type is sensible, that all its type parameters satisfy their bounds. When coupled with other features like wildcards in Java or path-dependant types in Scala, this allows you to create a type that proves a relationship between two types. I.e. that one type is a subtype of another type. You can then create a function that accepts such a proof of type relationship and coerce one type to another.

This is all sensible, because normally to create a value of a type you need to prove that it is well-formed. However there is a flaw in this assumption: null (unsurprisingly, given the theme of this post).

You see, null makes all reference types inhabited, even the ones that are not well-formed. As shown in the Java and Scala’s Type Systems are Unsound paper null allows you to falsify the proof of type relationship, allowing to coerce any two types to each other.

This is another place where implicitness of null caused issues, quoting the paper:

It [implicit-null feature] adds a case that is easy to forget and difficult to keep track of and reason about. Interestingly, here it causes the same problem for the same reasons, but at the type level. The reasoning for wildcards and path-dependent types would be perfectly valid if not for implicit null values

A sad note: C#⌗

C# doesn’t support sum types. To some extent they can be simulated using inheritance from an interface or abstract class, however, such an approach lacks one of the greatest benefits of sum types, namely the exhaustiveness check.

It’s especially sad since F# (which is running on the same VM) supports sum types (F# docs call them Discriminated Unions), but C# doesn’t.

type Option<'a> = None | Some of 'a
type Either<'a, 'b> = Left of 'a | Right of 'b

A sad note: Go⌗

Go doesn’t support sum types. It’s a little sad on its own, but Go also doesn’t have exceptions and all reference types are implicitly nullable. This means that if a function wants to return an error, it needs to return a tuple of success and error values. This not only makes checking which is null (actually nil, but it’s the same thing) pretty annoying, but also leaves the possibility for an invalid state where neither success nor error values are null.

I think it’s inexcusable to have such error-prone design flaws for a language created in 2012.

A happy note: Swift⌗

I’m not very familiar with Swift, however I often hear that it has some interesting/ergonomic/nice features. This time is not an exception — I was told that swift has both sum types and T -> Option<T> coercion.

From what I’ve gathered from here and here in Swift the type is called Optional<Wrapped>. It has two variants (or cases as they are called in Swift) — some(Wrapped) and none. switch can be used to pattern match.

let a: Optional<Int> = Optional.none
let b: Optional<Int> = Optional.some(1)

switch b {
case let Optional.some(x):
//   ^^^ --- let is needed to introduce new variables (`x`).
//           honestly, this is quite a good choice.
//           it allows to use `Optional.some(pat)`
//           to compare the wrapped value to `pat` variable.
    print(x)
case Optional.none:
    print(":(")
}

And then on top of that Swift adds a whole bunch of syntax sugar

1. T? can be used instead of Optional<T>
2. T can be coerced to T? (and coercions are transitive it seems, so T -> T??? is also supported)
3. nil can be used instead of Optional.none (there is a coercion from nil literal to T?)
4. .variant can be used if the type can be inferred (e.g. .none and .some(16))
5. a?.b can be used to do chaining / map access (a.map(|x| x.b) in Rust terms)
6. a ?? b can be used to provide a default
7. if let x = a assigns x to the inner value if a is some (syntax sugar for if case let Optional.some(x) = a {)
8. guard let x = a else { return } can be used to early-return if a value is not some
9. a! can be used to unwrap a variable crashing if it’s none
10. and more (probably (I’m a little tired of writing this list))

Personally, I’d say that’s a bit too much sugar and I’m a little concerned how coercions interact with generic code. But overall I think that’s a pretty nice feature set and I guess it’s nice to use all of this. I wish more languages went in direction “make Optional nicer” instead of “make null” nicer (yes, I’m still looking at you, Kotlin).

Conclusion⌗

There is no nice conclusion. The reality is painful as usual. We are stuck with a 56-year old abstraction that has proven itself to be error-prone for a while now. We are stuck with it, even though a better solution exists for longer than I have.

bye.