Lossy decodable for arrays

August 3, 2019

General Coding, Reference

Comments Off on Lossy decodable for arrays


This article is about exploring a way to decode only the “good” items from arrays.

It’s common for apps to decode arrays of data. For example, you may have a feed of user-generated content or a list of items for sale. To get this data, the app will make a network request to some backend API. Then, that API will most likely send the data back as JSON.

Swift gives us a great way to decode such data. You can simply set your objects to conform to the Decodable protocol. Then use JSONDecoder to build your objects.

Unfortunately, the data isn’t always perfect. If the data in the JSON doesn’t match your model the decoder will throw an error. And if just one field of an object or sub-object isn’t right, the entire list is thrown out.

So, what can we do about it? How can we allow the good items to go through and only reject the bad items without rejecting the entire list?

Starting with good data

Let’s look at an example. We’re going to decode a page of messages.

struct Message: Decodable {
    let sender: String
    let subject: String?
    let body: String
}

struct MessagePage: Decodable {
    let page: Int
    let limit: Int
    let items: [Message]
}

We have a page of message items which each contain a sender, optional subject and body.

Lets look at some good test data:

let goodTestData = """
{
    "page": 1,
    "limit": 10,
    "items": [
        {
            "sender": "Sender One",
            "subject": null,
            "body": "Body one."
        },
        {
            "sender": "Sender Two",
            "body": "Body two."
        },
        {
            "sender": "Sender Three",
            "subject": "Third subject",
            "body": "Body three."
        }
    ]
}
""".data(using: .utf8)!

Since the subject is optional, it can be null, missing, or contain a valid value and the decoder will easily handle each of those cases.

Finally, we can test decoding this data with something like this:

let jsonDecoder: JSONDecoder = {
    let decoder = JSONDecoder()
    decoder.keyDecodingStrategy = .convertFromSnakeCase
    return decoder
}()

do {
    let posts = try jsonDecoder.decode(MessagePage.self, from: testData)
    posts.items.forEach { print($0) }
} catch {
    print(error)
}

So far, there’s no issue. We tested the expected good data and everything is properly decoded. Ship it! Unfortunately, the real world is full of poorly constructed data. Our assumptions may be false.

Dealing with bad data

What if we were working with this data instead:

let testData = """
{
    "page": 1,
    "limit": 10,
    "items": [
        {
            "sender": "Sender One",
            "subject": null,
            "body": "Body one."
        },
        {
            "sender": "Sender Two",
            "body": "Body two."
        },
        {
            "subject": "Third subject",
            "body": "Body three."
        }
    ]
}
""".data(using: .utf8)!

Now we don’t get any items. Instead, we get an error telling us that our third item is missing a value for “sender”.

We could go back to our model and make sender optional. In some cases, it might make sense, but what does it mean if a message has no sender? This data may be required by our UI or even for another API that the app is using. Besides, if we go make all the fields optional we might as well just have used the older JSONSerialization instead.

One option would be to use manually implement the initializer and skip over any bad items by decoding into a dummy object.

Custom Decodable

struct Dummy: Decodable { }

struct MessagePage: Decodable {
    let page: Int
    let limit: Int
    let items: [Message]
    
    enum CodingKeys: CodingKey {
        case page, limit, items
    }
    
    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        page = try container.decode(Int.self, forKey: .page)
        limit = try container.decode(Int.self, forKey: .limit)
        var items = [Message]()
        var itemsContainer = try container.nestedUnkeyedContainer(forKey: .items)
        while !itemsContainer.isAtEnd {
            do {
                let item = try itemsContainer.decode(Message.self)
                items.append(item)
            } catch {
                _ = try? itemsContainer.decode(Dummy.self)
            }
        }
        self.items = items
    }
}

Now we get the good items in the array and allow the bad items to drop off.

We need the dummy object because the index of the decoder doesn’t increment when a decode fails. We can abstract that away with a failable decodable object.

Failable Decodable

struct FailableDecodable<Element: Decodable>: Decodable {
    var element: Element?
    init(from decoder: Decoder) throws {
        let container = try decoder.singleValueContainer()
        element = try? container.decode(Element.self)
    }
}

This allows us to remove the Dummy object and rewrite the initializer as follows:

init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        page = try container.decode(Int.self, forKey: .page)
        limit = try container.decode(Int.self, forKey: .limit)
        var items = [Message]()
        var itemsContainer = try container.nestedUnkeyedContainer(forKey: .items)
        while !itemsContainer.isAtEnd {
            if let item = try itemsContainer.decode(FailableDecodable<Message>.self).element {
                items.append(item)
            }
        }
        self.items = items
    }

This isn’t scalable yet, it’s a lot of boilerplate to write anytime you have an array that could fail. Let’s fix that.

Lossy Decodable Array

struct LossyDecodableArray<Element: Decodable>: Decodable {
    let elements: [Element]

    init(from decoder: Decoder) throws {
        var elements = [Element?]()
        var container = try decoder.unkeyedContainer()
        while !container.isAtEnd {
            let item = try container.decode(FailableDecodable<Element>.self).element
            elements.append(item)
        }
        self.elements = elements.compactMap { $0 }
    }
}

Now we can greatly simplify the MessagePage object.

struct MessagePage: Decodable {
    let page: Int
    let limit: Int
    let items: LossyDecodableArray<Message>
}

Looking pretty good. The thing I don’t like is that you have to access the messages using items.elements. So let’s fix it.

extension LossyDecodableArray: RandomAccessCollection {
    var startIndex: Int { return elements.startIndex }
    var endIndex: Int { return elements.endIndex }
    
    subscript(_ index: Int) -> Element {
        return elements[index]
    }
}

Now we can access the elements as we did originally and everything is working nicely.

Conclusion

I like using Decodable models to represent objects returned by backend APIs. Unfortunately, the data isn’t always perfect, and it can be challenging to find a clean solution using Decodable. However, with some persistence and the right abstractions, we can create scalable solutions. I really like how this case turned out, and I hope you do too.

I’d love to hear your feedback, questions or thoughts; find me on twitter @kenboreham

đŸ”„ Thanks for reading! 👍


Best strategies for configuring multiple environments in Xcode projects

October 21, 2018

General Coding

Comments Off on Best strategies for configuring multiple environments in Xcode projects


What things do you configure based on different environments? You might have views that are only for debugging, or you might want to turn off logging for release builds. You probably have multiple backend environments to configure like dev, QA, UAT, stage, prod, etc. Each of these requires different root URLs, API keys and app secrets. The app may also integrate with social media, crash reporting tool, or other analytics tools and we shouldn’t pollute this data with our testing efforts. We might also want to change the app icon and app name to make it visible which environment an installed app is running.

It’s easy enough to develop simple iOS apps and not worry too much about configurations. When you are just starting, it’s probably ok to do some setup with code, modifying the values as needed. You might even try commenting/uncommenting lines of code to switch between different configurations. Some people use #if DEBUG. Either of these will become problematic, quickly. It’s error-prone and time-consuming. So, what do we do?

Read More »


Add some color to your tvOS buttons

September 30, 2018

Button, Reference, tvOS, UIKit

Comments Off on Add some color to your tvOS buttons


The interface for UIKit is the same between iOS and tvOS, but the behavior is a little different. The interaction on tvOS is distinctly different than iOS because tvOS doesn’t offer direct interaction. Instead of tapping the screen you use a remote. This is an important point when deciding on how to customize interactive elements.

In this article, I’m going to discuss how you can add color to your buttons on tvOS. This is often required for consistency and branding purposes.

There are a couple of options depending on what your requirements are. The problem is that you might have to try a few approaches to get the desired look.

I will quickly go through what I’ve tried, what worked and what didn’t.

Read More »


How to customize UISegmentedControl without losing your mind

March 25, 2018

Playground, UIKit

Comments Off on How to customize UISegmentedControl without losing your mind


The UISegmentedControl provided by UIKit is a ubiquitous control. However, when it comes to customizing this control, it can get pretty tricky. So I’m going to try to explain how this widget can be styled to better match your app by walking through a few different customizations.

The image below shows the different styles we will build.

Read More »


Master map, compactMap, flatMap, reduce and filter by creating your own implementation

February 21, 2018

General Coding, Playground, Reference

Comments Off on Master map, compactMap, flatMap, reduce and filter by creating your own implementation


There are a few higher-order functions for collections that can be rather difficult to fully understand. While there are several good resources out there to learn the basics, it can still be confusing and seem a bit like magic. And when things seem like magic they tend to not get used effectively or maybe not at all.

In this post, I’m going to try explaining map, compactMap, flatMap, reduce and filter by walking through their inputs, outputs and deriving an implementation for each. For simplicity, we will focus on arrays but this can be expanded to most data structures. The goal is only learning the ideas, not to implement the best possible solution.Read More »


Beyond 128-bit integers

December 24, 2017

General Coding, Reference

Comments Off on Beyond 128-bit integers


I just finished a post related to the top range of integer values that are supported by Swift.

https://kenb.us/big-integers

The conclusion was that we can use the Decimal type to hold integer values with up to nearly 128 bits. That’s an impressively large number.

It’s hard to imagine a case where you would require both a larger number and precision at the least significant figures. However, it does make an interesting exercise so let’s give it a shot.

Read More »


How to store large integer values without losing precision

December 23, 2017

General Coding, Reference

Comments Off on How to store large integer values without losing precision


I recently wanted to use some very large numbers in Swift and quickly came across some limitations.

 

Int

My first implementation started out naively with Int (Int64) type. This is, after all, the easiest way to use integers.

The limitations are also easy to understand.
Read More »


3 ways to extract numbers from a string

December 3, 2017

General Coding, Performance, Reference

Comments Off on 3 ways to extract numbers from a string


I’m working on a project which requires me to parse some data that was formatted more for readability than parsing. Similar to NSStringFromCGRect which outputs in the format “{{x,y},{width,height}}”

let rect = CGRect(x: 124, y: 387, width: 74, height: 74)
let text = NSStringFromCGRect(rect)
// "{{124, 387}, {74, 74}}"

This string can easily be read back again using it’s counterpart function NSStringFromRect.

let text = "{{124.123, 387}, {74, 74}}"
let rect = NSRectFromString(text)

Read More »


Quick and dirty spotlight silhouette

November 25, 2017

Lighting, Playground, SpriteKit

Comments Off on Quick and dirty spotlight silhouette


This is a very simple spotlight effect that can be used to show silhouettes. No shaders or blend modes, just a couple sprites. The way this effect works is that you make the foreground and background elements the same color and then simply add an element with contrasting color between them.

Read More »

How to get UITextView to behave like UILabel

November 20, 2017

Reference, Text, UIKit

Comments Off on How to get UITextView to behave like UILabel


UILabel and UITextView both rely on TextKit to render text. So why do they behave differently and how can they work together?

Below is a view with a UILabel (blue) and two UITextViews (red) laid out in IB.

The Label is constrained to the top right corner of the view and uses it’s intrinsic content size to determine the width and height. The size of the label matches closely to the size of the text.

Read More »