Drawing the Line Where Engineering Becomes Over-Engineering

I’m always thrilled to discover a new software principle. when I come across a good one, I experience a eureka. I know I’ll never see the code the same way.

In this article, I’m going to discuss the problem of over-engineering. I think the problem is that we don’t have a concrete definition, of when software engineering becomes an over-engineering. When we do know, it might be too late.

‍

‍

I’m going to explain an awesome principle addressing that problem. It suggested by John Ousterhout, Professor of Computer Science at Stanford University, and the author of the book A Philosophy of Software Design. This guy isn’t another computer-science professor who’s disconnected from the software industry. He lives and breaths code. Folks like him, are exactly what our community needs!

We’re about to start. But first, it’s important we’ll clarify a thing on abstraction.

What is abstraction?

I believe there’s a misperception in the software industry about what abstraction really is: Some believe abstraction is about polymorphism, abstract classes, and interfaces. I myself used to think so.

In fact, abstraction is about a much greater idea: Encapsulating complexity and providing a way to communicate with it simply.

Take for example a laptop. It’s encapsulating an enormous amount of software and hardware layers. This complexity is spared from the user by a keyboard and screen, making communication with it easy.

In OOP languages, we create abstractions by extracting code to classes and functions with meaningful names. We don’t need any abstract classes or interfaces to do that. In other words, abstraction is modeling or, engineering. And over-abstraction is over-engineering.

For the rest of the article, I’m going to refer to abstraction as a Class. But, the same ideas apply to other abstraction techniques, such as components, modules, and micro-services.

As discussed, software abstractions are everywhere. it’s one of our fundamental tools to handle complexity. without it, the codebase will be a big ball of mud, that is impossible to understand.

But abstractions are not free. To add abstraction we need to add classes and functions, which adds some complexity to the codebase. There are many scenarios in which that complexity can do more damage than good. This raises a dilemma: How can we know if the complexity of adding the abstraction is worth the complexity it reduces?

A Java-made over-engineering case-study

One of the over-engineered API’s ever made is Java’s original file’s IO mechanism. Someone at Oracle thought that providing a dedicated API to read and write to files is not generic enough. So they came up with the idea of the generic input/output streams. As result, this is how you use the API:

try {
        FileInputStream file = new FileInputStream("input.txt");
        BufferedInputStream input = new BufferedInputStream(file);

        int i = input .read();
        while (i != -1) {
            System.out.print((char) i);
            i = input.read();
        }
        input.close();
    }

    catch (Exception e) {
        e.getStackTrace();
    }
}

I’m 15 years programming in Java and still confuse between InputStream to OutputStream. And what the hell is BufferedInputStream anyway?
This API is very hard to understand.

What went wrong?

Looking only at Java’s API doesn’t tell the entire story. Java’s Sun JVM is written in C, which is a much lower-level language. You would expect from that to see a much more complex code to do the same in C. The problem is that this is the code you need to write in C to do the same:

FILE *f = fopen("textfile.txt", "rb");
fseek(f, 0, SEEK_END);
long fsize = ftell(f);
fseek(f, 0, SEEK_SET);

char *string = malloc(fsize + 1);
fread(string, 1, fsize, f);
fclose(f);

The code in C is actually simpler and more elegant. This is crazy!

The Deep Abstractions Principle

According to Ousterhout, in order to reduce the overall software’s complexity, a class’s interface should be significantly simpler than its implementation. The only way to achieve that is to leave classes with some “meat”. Otherwise, you won’t gain anything out of them.

‍

As you can see, the class’s internal implementation should have some depth of complexity. How much depth? That’s relative to the complexity of its interface. It means, that API like Java’s streams, that requires it’s client to build five objects, should have a pretty deep implementation to justify that.

Deep class example

class NewsNotificationsPublisherService(
    private val userPreferencesService: UserPreferencesService,
    private val connectivityService: ConnectivityService,
    private val notificationManager: NotificationManager
) {
    fun isEligibleForPublish(): Boolean {
        
        val isConnectedToMobileNetwork = connectivityService.isConnectedToMobileNetwork()
        val isRoaming = connectivityService.isRoaming()
        val isRefreshOverMobileEnabled = userPreferencesService.isRefreshOverMobileEnabled()
        val isRefreshOverRoamingEnabled = userPreferencesService.isRefreshOverRoamingEnabled()

        if (userPreferencesService.isNewsNotificationsEnabled()) {
            return false
        }

        if (isConnectedToMobileNetwork && !isRefreshOverMobileEnabled) {
            return false
        }

        if (isRoaming && !isRefreshOverRoamingEnabled) {
            return false
        }

        if (notificationManager.areNotificationsEnabled()) {
            return false
        }

        return true
    }
}

You can see clearly that the class’s interface isEligibleForPublish() is simple. Actually, it doesn’t get any simpler than a boolean function with no arguments…
The point is, that calling this function will replace a significantly more complex code. So the class is definitely deep. Therefore, this abstraction will likely reduce complexity.

Shallow class example

class UserService(
    private val persistentDataStore: persistentDataStore
) {
    
    fun updateUserName(name: String) {
        persistentDataStore.update(KEY_USER_NAME, name)
    }
}

While there is some information hiding here of the PersistencDataStore, and updateUserName() improves the readability, the function’s implementation is very shallow. The ratio between the interface to the implementation is almost one-to-one. According to the Deep Abstractions principle, there are high chances this class will add more complexity than do any good. In other words: Adding UserService is over-engineering.

Some thoughts

I think that the deep-abstractions principle makes perfect sense. What I love about it is, that it provides a great contrast to the single responsibility principle, which when taken to too literally, encourage tearing the code recursively into smaller parts. So the deep-abstractions principle keeps us from crossing the red line.

Don’t let speculation and fears drive your design decisions

Have you considered wrapping framework API’s just because their interface might change in the future? Have you added an abstraction layer just to make a code testable? And how many times you’ve added abstractions in the name of reusability, without knowing when or how you’ll need it?

In some scenarios, it makes sense to do so. Just keep in mind that adding an abstraction has a price, and it might slow you down.

The deep-abstractions principle will be useful in those cases. You could ask yourself questions like:

• Will this “Unplugger” device do any good?
• What I had to do to unplug a device without having this “Unplugger” button?

Watch the video, and think of it:

Summary

The deep-abstractions principle helps to avoid over-engineering by drawing a line where additional modeling does more damage than good. Finding the right balance of where to apply it is a skill that should be practiced. I believe that keeping it in mind together with other programming principles brings the best outcomes.

Thank you for reading, and hope you find it useful. Feel free to share your thoughts or ask me questions on that matter.

For further reading:

‍

https://www.amazon.com/t/dp/1732102201#ace-9859629705

https://medium.com/@nakabonne/depth-of-module-f62dac3c2fdb