I find that software developers struggle to sketch diagrams of their software. They get lost in the specifics of diagraming techniques, in choosing from the many available tools, or in the futility of drawing diagrams at all. I understand their pain, as there are many standards for diagrams and many (often obtuse) tools for drawing with. It's not very motivating to have a sea of choices, none of which looks particularly appealing.
I think about software design by diagramming and writing. The act itself improves the result. It forces me to decompose and organize the problem, and attempt to explain it back to myself. I have always been able to think about software through this process of sketching, refining, and describing diagrams, even when I didn't know anything about what was standard or what should be done. I started by doing what made sense to me.
Over the years, I found that people's milage varied with my diagrams and documentation. Sometimes a diagram would communicate clearly and other times it would baffle. I took the time to figure what I didn't know, reading and trying pretty much everything I could get my hands on. And oddly enough, many of my earliest diagrams were the most useful, before I started to adopt the more specific technical styles. Why was that?
There are as many diagraming standards as there are development languages. There is a common subset that is much more manageable, but it's still easy to get lost in the choices if you don't have an understanding of what the history and classifications are. It's easy to be fooled too, as even an obtuse way of drawing will start to make sense to you if you practice it often enough.
One of my early mistakes was using diagrams that didn't suite the audience, the problem, the level of detail, or even my way of thinking. A great example was when UML became popular. I drew all sorts of architecture and design diagrams using its parts, and for my own thinking it was fairly productive. But I found that these diagrams communicated poorly, as they captured the wrong level of detail for many types of conversations. They also missed other details that are important in higher level thinking about a system. It wasn't just a problem with UML, I was applying it poorly too.
It turned out I was focusing on the trees before the forest. Much of UML is, for example, great for showing the precise details of things. These detailed schematics don't always show the hierarchical or proportional relationships well, but in terms of the finer aspects of design they are great. But try to use a class diagram to explain how a larger system functions and you'll be losing out on all sorts of important hints and cues.
Now if you're detailing the design of a class library, or the interactions of a protocol, then many of the UML diagram types are great. If you're discussing features and architecture, you'll be missing important parts of your story. You need to understand what you're trying to capture and communicate before picking a style of diagram.
Let's take a simple problem of design: how do chopsticks work? We can imagine them as a mechanical problem, as a software service problem, or as a manufacturing problem.
Consider the humble utensil: chopsticks. Two pieces of bamboo, plastic, or steel. Tapered. Textured. Packaged. Done.
If we were defining how to build chopsticks, we might describe what they were made of; their length, their properties, or their use. If we were coding up a chopstick abstraction for our latest game, we might draw their properties, or relationship to other utensils, or their data storage. If we were selling a web service that provided RESTful chopsticks, we might show their architecture or APIs. If we were Apple, we might describe the detailed manufacturing process that makes them what they are.
Which style of diagram would help you come up with a design for some choice chopsticks? Which would help you build them as a web service? Which would help you explain their purpose to your customers?
We could define chopsticks in terms of their utility:
Of course this is a Venn diagram, a joke (and a bad one at that). Feature diagrams are often nonsensical, as they show features in boxes and circles. They're meaningful in terms of showing features and their relationships, but are certainly not something you could build software from.
Feature diagrams are useful, however, especially to users who want to understand your product. Finding a visual language for the people who use your software is important, as it can simplify your documentation and support. It lays a foundation of icons, flow, and terminology. Users think about what they need from the software and what they create with it. Helping them think in the language of your software can make it easier for them to become productive with it. This is good.
Or, we could define the constituent components of chopsticks:
These are great diagrams for understanding the gravity of features or components. It could just as easily represent the types of languages or services used, or the number of types of data in the system, or the composition of the user base.
Or we could define how chopsticks work mechanically:
I use mechanical style drawings for thinking through states, sequences and componentization. This diagram, of course, is a silly summary of states in Venn form.
Functional diagrams allow you to use space and size to define how components relate. Lines and arrows can show flow, and containment can show composition. These mechanical drawings have standard forms in UML and various architecture standards, so it's worth reading up on what's done elsewhere if working with teams familiar with specific diagramming standards. You're also free to forge your own way with these diagrams, though they will work better if you rely on at least a few familiar conventions.
For sketching, obviously, you can pick and choose what works best for you. This is mostly how I think in design. I love to draw software and interfaces in terms of machines and ad hoc drawings. The freeform style is fast, expressive, and fun. These are important attributes for thinking in.
Now if we were building chopsticks as software, we could define their architecture:
I love architecture diagrams. They represent the polished, high level thinking about a system. They look a lot like things you'd see in marketing materials, high level enough to be approachable, detailed enough to provide insight. You might not be able to build software directly from such diagrams, but they are great for giving people an overview of your software or service. They are especially useful when bringing on new staff at all levels, as well as showing clients, customers, and investors what you do. They provide a vocabulary for everyone to share, with a spacial sense, both visually and in terminology.
I find that architecture diagrams work best when they borrow concepts from lower level diagramming languages, but simplified and more iconic. This aids familiarity, and provides an expressive way to pack information into the limited format. It also is a way to lead people into your detailed designs, anchoring their understanding in the simpler overview.
I believe high level diagrams are the most important development artifact when they accurately represent your software and its ecosystem. They represent your core values in terms of a defined language, and they map your way through design and construction. Your teams and management run blind without this shared understanding.
We could also dive into more detail and define a pair of chopsticks as a web service API:
API diagrams are spacial maps of your web service components and their namespaces. They are a lower-level slice of functionality specific to the web, but similar to component diagrams. These diagrams extend the vocabulary of your software, and decompose it into spheres of influence, and layers of implementation.
I also love API diagrams, as they help me think about what goes where, and about the actions and data. Thinking about decomposition is helpful for finding holes in your understanding and for finding organization before you build out the system (or when adding to it).
There are many standard diagram styles for component diagrams, depending on your audience and specific purpose.
As we continue to describe each part of a system, we find our way to lower level diagrams. These include things like:
We also start to wonder about how these pieces interact, for which we use lower level sequence, state, and transition diagrams. Tables are also useful for states and transitions, and I have found no single approach useful for all types of detailed planning.
When considering interactions we use wireframes, information hierarchies, sites maps, user interface mockups, and so on. These diagrams are just as important, and should also share pieces of your visual design language.
I have applied many types of diagrams to various stages of the design and development process. Some have worked and others have not. Part of my learning had me figuring out what worked for my own exploratory thinking and part of it had to do with finding out how to improve what other people understood of my designs. Some of the challenge was in the type of diagram used, some of it in the level of detail, and some of it in the style applied to the drawings themselves.
If you look at my simplified examples you'll see a few common themes:
I tend to vary the colours and shapes in my design diagrams, though there are common themes. For example, I tend to use icons from standard diagramming languages in higher level diagrams. I also tend to pick fonts and colours that suit the domain or product branding, to bolster familiarity and ownership of such things. I also pack in a bit of humour, where appropriate, to keep people alert and enjoying themselves.
I find that these elements translate nicely to pencil sketches and whiteboard discussions too. If you start to work on a language for your software, your entire team will gain from the tools you have provided for them. They will be able to leverage existing definitions and extend them with their own design details.
Navigating through the universe of diagramming techniques and ways of applying them is in itself a huge undertaking. Add to that the difficulty in recording and producing attractive diagrams and you have a task that seems impossible.
I'll let you in on a secret. It's really very simple. You just draw. Keep drawing, enjoy it, learn your tools, read a bit, and before you know it you'll be thinking visually.
So how do I produce diagrams? Is there a magic tool that will save us all?
I learned dozens of tools before I realized there was no perfect tool. Today I still use a variety of tools to translate what I see in my head to paper (and I, of course, also use paper).
At first learning multiple tools seemed counterintuitive. It felt like a waste of time. I mean, why learn several tools when I could learn one really well? It turned out that learning various tools taught me the fundamentals of drawing in a way that makes it simpler to use any drawing tool. It gives me choice, dexterity, and a richer palette to work from.
The principles of making technical drawings are simple, once you realize there is no magic. You just do it. You work around the limitations. You nudge things by hand. You use what the tool does well and ignore what it doesn't. For me, I would get frustrated with a tool and quit, when I really should have been frustrated at my lack of own lack of persistence. Stopping only guaranteed my failure.
Incorporating diagrams into documentation can also seem tedious. Given time, however, I found that you just do it. As my agility improved, the cost of the tedium went down. I also have a stronger vision of what I want in my head and I use that to brush off the more annoying aspects of the process. Eventually the annoying parts fade away, and you just make stuff.
Design is difficult at first. Too much choice, too much to learn. But, it boils down to a few principles:
In the end, defining a way to talk about your design is more important than what you don't know. Your design sketches will unify how people think and talk about your software and will lay the groundwork for to extend and improve the things you build. The learning will also speed up your own thinking about design and improve the polish of what you build.
As designers, software developers, and business geeks we thrive on a few things. We seek inspiration. We obsess over details. We work long hours. And we need focus.
Computers and software have evolved to the point where nearly anything is possible. It's an amazing threshold, as we are rarely limited by the hardware, OS, and languages. We're mostly limited by our own ability to produce. We are the weakest link in our productivity.
Enter the era of focus applications. We've seen focus tools over the years, but the market has matured and there are dozens of tools available to help our focus.
One such tool is Hocus Focus. It is a simple tool. It minimizes applications that you're not currently using after a fixed timeout. That's it.
It's an interesting way to work. I usually have several windows open at once as work, ideas, and life are interruptive. A tool like Hocus Focus lets the interruptions happen, cleaning up after the chaos.
Hocus Focus may not be an every day tool, so it lets you disable it when you're working on something that requires a bit of mess. It also lets you customize the timeouts in a few ways, in case you have applications that need to be more persistent.
Does it work? I think so. Application disappearing takes a bit of getting used to (a fade might be better), but once you stop noticing applications dropping like flies you're left with a desktop that is more focused.
As a software designer, I love to design software. As software developer, I love building software. When I dream of better things, I funnel my ideas into new projects. When I don't have time for all of my side projects, the projects pile up in a corner and gather dust. And when a side project is an unfinished publishing platform it gets in the way of writing.
I'm intensly interested in writing and publishing on the web. There are just so many possibilities for saying interesting things and making the presentation of those things more useful. I've dedicated a chunk of my consulting time to making beautiful interactive sites for non-profits, and I use a slice of my spare time for maintaining a number of my own sites (and those of friends and family). I believe that what we're seeing on the web today is a shadow of what we will publish in the future, and it excites me.
My love for writing and publishing led me to building my own minimalist tool. Chronicle.md is a Markdown app written in PHP. It works as well as a first version needs to work, but it isn't productive enough yet to invite me to write daily or weekly. It requires twiddling and poking to get posts out, and it doesn't support any of the common publishing APIs. And as it's not at the top of the pile of awe inspiring ideas, it's not going to get those features any time soon.
And when I weigh the importance of building a better tool with my need to write, writing wins.
So I'm back to WordPress. It's a reasonable tool, with a vast, stable API, with hundreds of tools and plugins to make it sing. And as I've built hundreds of custom themes for it, I can spend some of my fun development time making WordPress better.
The tipping point this time around was the simple lack of writing apps that supported my custom tool (hint: this number was 0). As I have been itching to try Desk.pm (a writing tool that supports CMS's like WordPress), and as I have been publishing sporatically, it was time for a change.
Now I can dream of ways to make my writing flow better, instead of building everything at once. This fits the character of side projects better, as I can take bite sized problems, related to my writing needs (or my client's needs), and solve them really well. It's all about focus.
If I do decide to build a CMS in the future, I will make it my main gig, instead of trying to fit it inbetween the other stuff. Letting side projects interfere with your flow is a sure sign that the balance is wrong.
In the interim, when I get the itch to invent a new CMS, I will aim my powers at trinkets, plugins, and themes for WordPress (and other CMS's).
I wrote my first computer program when I was 8. It was a simple adventure game written in BASIC, a mess of GOTO
and PRINT
statements. I saved it to tape a few times, proud enough of my first creation that I didn't want to lose it. And despite what I didn't understand my mind exploded with ideas.
I continued to write games and curios into my early teens. I wrote text adventures and arcade ripoffs. I hacked at biometric data collection using a busted set of paddles and mixed mode graphics. I designed and built a large (but clunky) side scrolling adventure game using only a set of customized fonts. I plagiarized code from magazines and hacked at my machine until the wee hours of the night.
I learned almost nothing about writing production code in my early years. And while discovered that I loved to imagine and build things in code, I barely brushed against the principles that would later be required to make real learning possible. My skills didn't improve much either as my time was largely unfocused.
I was a hobbyist. I discovered the joy and love of software, but entirely missed the nuts and bolts of it. I didn't learn anything about the science, the skills, the mechanics, or the business behind software, which limited how much I could learn and improve. It's a limit that prevents most of us from eclipsing hobbyist, even those who end up making a profession of it. Failing to escape hobbyist damns you to a shallow relationship with understanding.
“You want to know how to paint a perfect painting? It's easy. Make > yourself perfect and then just paint naturally.” > #
But it's more than just knowledge. After you discovery where to start learning the right things, you must then endeavour to follow it with practice. Failing to practice leads to a lack of reinforcement and eventual atrophy.
I shied away from computers as high school progressed. I was bored with my slow progress and was naturally gravitating away from solitary activities. Instead, I spent time with friends and worked in various menial jobs. I enjoyed working too, especially in restaurants where I was making food with my hands (even fast food).
I made my way to college, and wandered through it mostly without direction. I passively avoided computer science, though I found myself programming in various capacities anyways. I wrote software for restaurants, built data analysis code for agriculture and biology courses, and coded simple tools for print operations. I was still sure that I wasn't a programmer, so I continued working in restaurants and studying anything but computer science.
I also considered myself an artist at the time, even though I spent very little time being artistic. I would binge on art from time to time, especially enjoying block printing and photography. I also enjoyed sketching, though I never reached the escape velocity required to do much more than doodle icons and simple caricatures.
It was around this time that I realized that I wasn't really very good at anything I did. I mistakenly suspected that I wasn't trying hard enough, so I doubled down on my studies and changed my major to computer science, as it seemed somewhat inevitable.
“The truth knocks on the door and you say, “Go away, I'm looking for > the truth,” and so it goes away. Puzzling.” > #
Unfortunately, trying harder was a vague proposition, and I found myself bored when my lack of skill eclipsed my need to produce useful software. I did well in my studies and project work, but I didn't really learn how to write software despite writing quite a lot of it.
I developed a schizophrenic relationship with making things though college. My mind would race at the prospect of building something, excited to get started. I would enjoy the tedium of construction and I would revel in finishing. But at some point in every project I would realize how far from great it was.
I tried to ignore the reality: I didn't enjoy thinking about how much I didn't know and how limited my abilities were, even though the thinking about those things would have pointed me toward improvement sooner. I continued along a superficial path of creation for many years. It was a dichotomy that would follow me for some time, the excitement and passion for making things, and the inevidable disappointment of knowing they weren't great.
I continued meandering through college, bored with my classes and part time work. I left my program early for an opportunity to work in the real world, hoping that it would nudge me out my rut. And it did.
When you want to hurry something, that means you no longer care about > it and want to get on to other things. > #
My first few years as a full time professional were liberating. I was working as part of a team where my lack of skill and knowledge were no longer limiting. We were given simple, defined problems to solve, mostly tied to specific customer support requests. The team was knowledgable, at least in terms of how to solve specific problems, so I was able to start and finish projects quickly and with some quality.
Smaller, more defined work was the first step to successful practice. Our time was limited by customer deadlines, which forced us to keep our approaches and designs simple as well. Combine small and simple with compiled languages, and we kept the code clean too. This principle drilled a few things into my head that were important, though I wouldn't fully grasp them until later.
This first real software job was important in another way: it showed me how little I knew. Both through the practice of churning out a release every few weeks and the fellowship of our early staff I came to realize that I was still just a fledgling in the craft. And as luck would have it, the employer embraced continued learning and they graced us with a very ample book budget. I spent every penny I could of that budget, and read every book cover to cover numerous times. I felt like I was finally learning.
Of course I had already learned more than I realized, but by this time I had some experience and perspective. My experience helped me focus the time I spent learning, and it helped me filter the learning materials too. Both focused time and quality inputs are important for learning, as is having enough perspective to see the value in what's being learned.
After my first few years as a professional I became aware of how making things affected me. I learned that when I made something the experience was deeply satisfying. And as my results improved, the experience was better. In short, making things felt good. Getting better at making things felt incredible.
“Logic presumes a separation of subject from object; therefore logic > is not final wisdom.This is Zen. This is my motorcycle maintenance. “
There is a problem, however, with associating satisfaction with making, especially when it's your profession. Sometimes you are forced into unsuccessful projects, either through simple missteps or through external pressures. When we are attached to the our successes, our failures become part of us too, and we suffer emotionally and physically.
Over the years I've had to fight the attachment to the things I make. On one hand there is great satisfaction in the process. On the other hand the failures eat away at who you are. Conflating satisfaction with value is an expensive mistake that I'm only now learning how to avoid.
I am neither my success nor my failure, nor am I the journey. In the end I am who I choose to be, in how I allow the experience affect me, and how I perceive the journey itself. I'm not suggesting rose-coloured glasses, rather that you accept the reality (whatever it be) and choose who you become (or who you don't become), without prejudice. This plays for both success and failure, and for the things in between, as lot of what we do in computing is on the long road between inception and shipping.
I look back on those early successes when I get lost in software development. I also look back on the time when I started to understand learning as a reminder of what I can do when the inputs and circumstances are just right. These experiences help me find my way back to reality, inspiring me, and reminding me what can be done.
Looking back on some of my earliest, successful projects also gives me a sense of perspective, one I could not have found at the time. Those early projects were not as bad as I thought they were. They were a balance of pragmatic, functional, and creative (at least the ones I remember). I often find myself wishing I could find that balance again, a balance that I was forced into by my lack of knowledge and experience, a balance that is now more difficult to find.
Some of our art is in learning how to learn, learning how to practice and hone our craft, and in exercising our experience and knowledge. The craft is also in finding that child-like simplicity, where the balance hinges on the limitations more than the infinite possibilities, and where that youthful passion is clearly seen in the things we make.
Friction works against you. It exceeds your ability to progress. It deflates you, stripping your motivation. It's the hill you see before starting a difficult project. It's the gravity that pulls you away when interest wanes.
I'm starting a small project for a client today. It's a simple set of improvements to something I built for them last year, using a toolkit I wouldn't normally use (but one that fit their uses well enough). I had forgotten a lot about the toolkit when quoting on the work, and by the time I remember it's clear that there are challenges ahead of me.
When it comes time to start the work I find that I'm stuck. Unmotivated. Paralyzed. I'm staring up the barrel of an uphill battle.
To get unstuck I need to start moving. I need to realize that the problems are soluble and mostly in my head. And the best way to get started is to do something, anything at all.
So I work my way around the edges of the harder problems. I simplify development by working on a scratch copy of the project, isolating and reducing the size of each individual piece. I work outside of the warty toolkit and test my work carefully before integrating it. I carefully break my work down into lists of small achievements so that I can make measured progress hourly. Once I get a piece working I test it, and put it up on the shelf. By the time I finish the small bits I'm moving, and I can attack the mountain without distraction and with gusto.
And the mountain? It was a projection of my imagination. I thought the work was going to be difficult. I believed it. I didn't feel like re-learning the old, warty tools. I remembered the flaws more than the good qualities. Or maybe it was one of hundreds of other human flaws manifested as part of my fictitious friction. The key was to realize that friction was just a glitch in my perception, that if I shed my perception I could do it, that once I started the work it would be easy. And it was.
In other words: just fucking do it already. It won't be as hard as you think. And if it is, you'll be done before you realize it.
One of my favourite JavaScript features is its literal object notation.
It allows you to declare data structures in JSON
, which is a very
succinct, C-like syntax. You can use this notation to declare SETs, and
the a in b
syntax to verify a variable is in a set:
/* Declare your set as a JSON constant */
var valid_widgets = {
'widget-viewer': 1,
'special-viewer': 1,
'widget-editor': 1
};
/* Fail if something isn't in the set */
if ( ! (options.widget in valid_widgets) )
return false;
/* Profit! */
As the in
keyword operates on the index of the referred object, we set
the value of the constants to 1
(true). A bit of a hack, but it
results in a very clean way of checking if something is part of a set of
valid values.
Somehow I missed the recent larger EC2 unit sizes:
When I saw this, I immediately thought: these numbers will have K, M, and G suffixes within the next 5 years. Compute units will be sold as 88 mega-ECUs. Imagine 88 giga-ECUs?
The ECU is similar to our old measure of computing: boxen (except more virtual). The idea of giga-boxen boggles, but is inevitable (boxen is so much more inspiring than ECUs), or maybe it should be boxels?
My best guess is that K-boxels will be available within 2 years, M-boxels within 4, and G-boxels within 7. By the time my kids hit university, we'll be working with giga-scale utility computing units.
In the words of the great philosopher Ted Logan, “Whoah!”
Giga-boxels!
While writing a spec earlier today the last few years of progress in server land hit me: uptime is a facade.
In the early days, server resources were expensive and scarce. Uptime was sacred. Long running hardware was celebrated, UNIX tools were born, beer was consumed.
The problem of focusing on the hardware is that it detracts from the time you spend developing software. Remember those late nights configuring RAID setups? What about hunting down faster drives, or terminating SCSI cables?
With utility computing like Amazon Web Services, hardware no longer exists. Slices of computing are sold by the hour, by the byte, and by the feature. Provisioning is nearly instant, and can be controlled by API, from any machine connected to the net.
Machine uptime doesn't matter. It's been replaced by ubiquity.
It ends up that service uptime doesn't matter either. Instead of thinking about keeping continuous instances running, we need to focus on ways to make it easier to replace instances, and mechanisms for pooling service resources in ways that fail over gracefully.
As the cost of swapping servers approaches zero, it becomes the ultimate form of stability and parallelization.
While I was thinking about uptime tonight, an impending software update threatened the honour of my computer. My first reaction was to avoid running the updates: this machine was nearing 3 months between reboots.
So you can't write. You're empty, tired, and unfocused. You've tried exercise, meditation, caffeine, binge sleeping, and a dozen other cures—but nothing comes out. And when you do happen to squeeze out a few paragraphs, you stall, spiralling further into your uninspired daze. You are burnt out.
Writer's block and burnout are the same problem: undue mental exhaustion brought on by overuse, misuse, and other stress. It's important to understand that you cause your own burnout, either directly through your actions, or indirectly through your lack of action. It's your choice.
While it isn't just writing that's affected by burnout, writing is one of the fundamental skills subdued by mental exhaustion. If you can't write, you can't design, create, express, or inspire. For me, writing is the real measure of my creative mojo.
Don't worry about the root cause of your exhaustion. Your job and your life have very little to do with how you react to stress. Your reaction to stress might feed your exhaustion, but learning to dodge them are only a temporary measure. The real problem is that you're stagnant, stuck in a place where you allow these natural forces to erode your being. While the erosion might cause the burnout, allowing it at all is the real problem.
Your goal is to get unstuck. Ignore the bullshit of addressing stress and the remedies that promise to manage it. The stress doesn't matter. Your balance isn't important. You need to get unstuck. You need to strengthen. You need to create.
Don't try to find balance. Balance isn't something you can concentrate on; it's what happens when you become stronger, when you create things, when you polish them, and when you enjoy the result. If you focus on balance, you'll find it impossible to stay balanced at all. Let balance happen.
Don't try to focus. It's not something you can do directly, rather it's what happens when you do. Build something. Learn to perform each step well. Train yourself to understand the history and science behind each concept. Enjoy polishing each piece until you're proud of it. Just do it. The burnout melts away.
Skills are fundamental. Skills are required to do, even if secondary to the doing. Learn, practice, and play regularly in the domains that matter to you. This is a form of doing that precedes and complements the act of creating things.
You need to learn to develop your skills as part of your regular regimen. You should include it in your project work, when things you're building require better understanding. You should also include it in those times where it's not practical (or possible) to create. Between projects, when especially demotivated, and when the opportunity encourages it.
Burnout is the result of becoming stagnant and the natural erosion that results. You can't cure burnout by coping with the erosion, you transcend it when you make the erosion irrelevant. Grow, practice, and do.
Just do it. Do it well. Do it now, and repeatedly, and with passion. Throw yourself at making things and understanding every part, until you breathe their essence. Enjoy the process of practice, in a way that complements your work. Everything else ceases to exist as you grow stronger and do more.