🦄 Unicorns and the Shifting Landscape of Computing
I've noticed an interesting shift in computing over the last few years. It's one of those changes that becomes obvious in hindsight, though while in the moment it was confusing and disorientating. It turns out I was blinded by my own path of coming up in the industry.
I'm never surprised when a bias gets in the way of seeing something. It's a constant of human progression and the fundamental reason we science. We're aware of our limits, and we pursue knowledge from the perspective of disproving ourselves so that we can uncover the truth despite the limits of our ability to observe and think. Science is cool.
That same set of filters are much more difficult to apply to industry trends, as the facts are fluid, and the reasons for those facts are based on a complex set of interactions between people (who themselves are gooey masses of crazy).
The search for sparkles ✨
Finding talent in computing has always been difficult. There is a large gap in how well people perform while building software, where one developer can be between 5 and 10 times more productive than another. Individuals may also perform better or worse depending on how they fit into their team, and how they mesh with the software stack in use. Measuring these differences in interviews is difficult.
I've used a number of tools when interviewing developers and have found what most researchers have found, that standard measurement tools like brain teasers aren't really that useful. I've used written tests, whiteboard tests, brain teasers, and even off-the-wall questions when interviewing people. I found that the standard tools failed to gauge actual job performance, rather it measured the ability for people to perform in interviews.
I have found a few less rigid techniques for measuring potential hires, mostly in gauging trajectory and passion. Is the developer excited about learning? Are they excited about making software? Do they actually do both of those things? Combined with the standard do-they-fit-who-we-are gut reaction, I've been moderately successful in finding some great people.
â›° The landscape is not a constant
Now this is where it gets interesting. I've been interviewing and hiring for almost 20 years. I have accumulated a bunch of questions I ask people. And while the technologies I talk about have changed, I have always expected certain skills and behaviours from specific levels of software developers. In terms of soft skills this has been very successful, but recently I've noticed that the hard skills have dissipated.
This Spring I've been hunting for a few senior developers. I've talked to dozens of incredible people, but have been confused by the lack of experience in system design techniques. Looking back I noticed signs of it years ago, though in recent interviews it's a set of skills that have mostly disappeared.
When my own perspective fails me, I look to other pillars of the local scene. I talked about it with a good friend, asking him if he had noticed the shift in any of their recent hires. He's also seen a lack of system design understanding. We emailed back and forth for a few days and came up with some plausible reasons for the gap.
✨ Sparkles become more rare over time
We agreed that skills like system design are (and should be) more rare now than in the 90s. There are a bunch of reasons for this, but mostly it's the result of faster and larger computing resources, combined with tools that are much more capable. While this seems very obvious in retrospect, it was blindingly difficult to see while interviewing people.
Most software development problems in the 90s required some amount of system design. Resources were usually constrained by some aspect of the problem, and tools like capacity planning and careful plotting around the inherent system limits were absolutely necessary. Junior developers had to learn these design techniques to progress to mid-level developers. Senior programmers taught these skills, and were responsible in the many cases where planning failed to foresee limits that blocked the project.
As the industry matures, the risk of failure due to limited resources decreases. Additionally, the tools have progressed, reducing another common friction from 20 years ago. There are fewer times when system design is required by every team member.
And as hardware, tools, and the industry mature further, the business of software evolves with it. Many businesses now focus on little-a-agile techniques, which are iterative and fast. We call it moving fast, which is no longer the desperation of the Valley startup, but a viable approach to building software. It wasn't possible in previous eras of computing, as the technical risks required moving slowly. As those technical hurdles melt away, early startups are freed to obsess over features and utility instead of spending all of their time on figuring out if the problem is soluble at all.
We of course still do system design, though there are fewer people that need to learn it. Businesses can hold off on system level design until they're scaling, as the risks justify the tradeoff in development speed. It's a crazy and wonderful time to be in computing.
🦄 Unicorns see other unicorns 🦄
Now the question remains: how was this shift in computing knowledge surprising? I came up in the industry at a time when most of the problems required careful systems design. My incredible mentor came up in the proterozoic era of computing, where even the smallest of problems required careful design. I learned to think about software in terms of capacity, data, and flow first, using diagrams to think and communicate more clearly.
Today? Most mid and senior level developers can work through problems with less rigour. There is less need to understand how the engines work, and there are many problems that the current generations of developers will never need to think about. Questions about systems planning, measurement, and design are above the experience of these generations.
Deep knowledge has shifted up over each generation of developers and designers over the last 50 years. Deeper design knowledge is now relegated to unicorns and architects. And that's okay. We'll still use it, and we will continue to teach it as deep understanding always pays off over the long term.