Go Conway

By Avi Deitcher

2015 Aug 24

There is a famous saying, known as Conway's Law, which states that:

organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations

It means that when your organization builds a system, its structure will reflect the organization that created it. If you have 3 teams - database administrators, system administrators and Web developers - then your system architecture will have 3 distinct components: databases, servers and Web UI.

When I have helped companies change their technology design to become much more nimble, reliable and responsive, the most challenging part often is not the technology itself. Rather it is the organizational changes required to support the new design. It is for this very reason that I market my services as operations consulting; technology, as fascinating as it is, is just the tool.

This week, I came across another example of Conway at work: the language Go.

Go is very popular. Besides the full support of its creator, Google (who has plenty of resources to back it), it has a few key features that make it appealing. Among them:

Binary code: unlike interpretive languages like Ruby or JavaScript or Python, and unlike byte-compiled languages like Java and Scala, Go's output (its "deployment artifact") is a native application for the platform. When I deploy a Go application on my Mac, it is a Mac app; on Linux, a Linux app; etc.
Cross compilation: I work on a Mac, but want to build it for Windows or Linux? No problem, just build it right here on my Mac.
No dependencies: Often, the single deployment artifact itself is easy to deploy; it is all of the requirements and pre-requisites - app servers, libraries, etc. - that are difficult to manage. Go compiles it all into a single file that just runs.
Threading: Multi-threading - having multiple parallel streams of execution for your application - is very important to performant applications, yet is quite difficult to do correctly. Go uses "goroutines" and "channels" to make it simple to manage.

The idea of "no dependencies", of course, is a fiction. Everything has dependencies; it just is a question of when and how you resolve them.

In many development environments, you need to resolve them twice: at build time and at deploy time. With Go, you resolve them only once, at build time.

The question, then, is how do you resolve them? How do you make sure that my application A, which depends upon X, Y and Z, can find everything it needs?

Every development environment has its own way of both declaring a dependency and getting it.

Java uses "import" statements to declare dependencies, while those dependencies are distributed as "jar" files that you import manually or use tools like maven
Nodejs uses "require" statements and npm to retrieve the dependencies
Ruby uses the bundler to declare and import dependencies.

Go uses "import" statements, wherein you declare what you depend upon and import it:

import "net/http" - declare a dependency upon and import the official "http" package
import "github.com/someone/something" - declare a dependency upon and import the package at "github.com/someone/something"

How is go different, then? In how it lets you declare dependencies relative to your project. In just about every development environment, you can declare a dependency on one of three types of packages:

Standard: Every language has an standard library that is available with the language.
External: By using the absolute, globally unique identifier of a package, you declare your dependency on that package wherever it is.
Internal: By using a relative path, you declare that your bit of code depends on another bit of code that is an internal part of the structure you are working on.

Each of these serves a purpose. The official packages are a crucial part of the language, basic utilities and services; external packages provide flexibility to use the broader community's or your internal organization partners' contributions.

And "internal" packages allow you to organize your code. As anyone who has built anything larger than a sample "hello, world!" application knows, the ability to support and maintain an application is directly proportional to the organizational structure of the code.

Here is an example from searchjs, a small project with which I was involved:

This structure allows you to distribute not just the final deployment artifact, i.e. the single final file, but also the source code, which is crucial to collaboration.

In Go, however, the directories mean nothing other than "another package." It is impossible (or, at the very least, frowned upon strongly, with indication that it will be removed in the future) to say, "include the package that is in this directory." All you can say is, "include this external absolute package."

Why should it matter? Who cares if the sub-directory needs to be referred to by its relative path "./util" or its absolute path "github.com/someone/something/util"?

It breaks encapsulation!

If someone copies the entire directory tree, the entire "capsule", of your project, they cannot assume it that it works just fine on its own. Every file that imports something else will contain the absolute path.

Here is an example. I have one main file in a package named github.com/me/myproject, that includes a bunch of other packages in my project. If I am well encapsulated, it would look something like this.

import "package1"
import "package2"
import "package3"

But since Go's thrust is absolute only, it looks more like:

import "github.com/me/myproject/package1"
import "github.com/me/myproject/package2
import "github.com/me/myproject/package3

When someone copies the entire project - or uses the single most popular method of doing so, a "github fork", the modifications will not work. The project now depends on a different project.

And if the above imports appear in 5 places in each of 20 different files in 3 different directories in the project? Nightmare.

Go's structure has unit of distribution and encapsulation directly at odds with every other language in common use, and every code distribution and collaboration mechanism.

However, I come not to criticize Go's architects' choices, nor Google itself, but to understand why and how this came to be.

Go's genesis is inside a single monolithic company. There may be multiple people collaborating on a project, but they all work against a single source structure for a single boss, somewhere up the chain.

The idea of multiple completely independent people and organizations sometimes working independently, sometimes collaborating, of their own free will and time, each with their own style, just does not fit within their mindset.

Inside Google, it is safe to presume that every little directory, no matter how small, is an absolute package, because it really is.

Reading the forum discussions about supporting relative packages is very enlightening. The people behind Go simply cannot fathom why anyone would want sub-packages, or to encapsulate all of the directories into a single self-sustained project.

While Go is an excellent language and platform in many ways, and it is being adopted widely, its package management is the subject of a lot of complaints and grumbling.

Ironically, its growth ultimately will constrain its success, as larger scale teams struggle to work with and around the package management system. I suspect someone will fork it fairly soon, if only to get around these constraints.

Summary

Go's design reflects its roots inside Google, while projecting the organization and constraining it outside.

While your technology architectures will reflect your organization, per Conway, your technology architectures also will constrain your ability to organize and the usefulness of your technology, until you and your technology grow together.

Does your technology design enable or restrict your growth? How could you change your technology to enable better processes and ultimately a more nimble and faster-delivering organization? Ask us.