If I Could Create Worlds…

I’m Mark Nadal & I want to tell you stories of an awe-inspiring future.

I am one of the types that has intense day dreams of vibrantly detailed SciFi universes, full of unique never-imagined before creatures, cities, societies, economies, and ecosystems. My day job, though, is instead stuck writing nuanced code. One day I hope it will power utopian civilizations, but in the meanwhile, getting it to reliably scale for tens of millions of users to do their cat video thing is, well, the less glorious “one small step” for humankind.

So here, here I wanted to express some of the endless ideas I have for such a forward facing future and why it matters to me. This is Part 1 that focuses on various unturned stones of human-computer psychology and interactions.

#1: UX

I originally wanted to be an artist/actor, but the tools I kept using were infuriatingly slow to go from the idea/vision I had in my head to output/reality. I kept thinking “why doesn’t it work this way?”, shocked to find nobody had built what seemed to me stunningly dumbingly obvious.

Let’s ignore science-fiction devices for a second, and work with what we’re currently limited to — screens, keyboards, phones, gamepads, VR. These ideally are not the human-machine interfaces we want, but they’re ubiquitous and where we must evolve from if we want accessibility for everyone to be included in this journey.

So to start, when you turn on the screen, it should have a way to both create and discover. The lack of built-in creative tooling on most devices is abysmal. So let’s look at the UX of productivity tools. The singular gist is simple:

Why doesn’t Photoshop have video game controls?

Video games are fundamentally immersive, fun, compelling, and extremely precise yet easy for newcomers to pick up.

Why should we even focus on this as the first point? Because if we have really good digital creation tools, it becomes the easiest and most powerful way to bootstrap all sorts of other physical goods and societal/civilization-level structures. You could 3D print products and communicate economic data across countries. The internet and the interface to it has the highest ratio of from-nothing to world-impact possibilities.

Throwback 10 years ago, my decade younger high school self tried to build this “video game for Photoshop” type tool. This demo doesn’t give it justice, but the results were very cool, including being able to rebuild any website in minutes — including investors & journalists we pitched to as teenagers.

It supported multiplayer experiences across touchscreen mobile  — mind you, this was 2011. The hacks I had to go to were insane back then.

Interaction & Information

So there are two key design principles to understand, Interaction & Information. Here are the basic ways we can interact with a device:

Interaction

1. Stare
2. Scroll
3. Tap
4. Type
5. Talk

Scrolling is what the eyes do, they look around to collect more information, because not all of it can fit in the same space you’re staring at. Tapping includes any range of gestures, but poking something is the most basic place to start. Typing refers to external inputs, like a keyboard or gamepad, which translate different types of “tapping” into more complex interactions. Finally, talking includes recorded input, capturing the world around us and trying to shove it into the digital world.

If you notice, the degrees of dimensionality and precision increases drastically over each item. For the most part, even paralyzed people can stare and scroll. However, recording audio or video or even 3D body motion performances requires not only a whole lot more data to process, but also talent on the part of the human.

This is why some tools can be infuriating, getting the precision of a pixel via a trackpad is not something most people are good at — we’re clumsy. And you shouldn’t have to be rockstar drummer in order to “time” things perfectly.

So this is why tools should always reduce interactions down to a single dimension at a time. That way it is easy to move just the X coordinate of something, without wiggling into the Y space. However, each tool should be “composable”, just like a piano keyboard, where as a user gets more proficient with the tool, they can move both X and Y simultaneously by “tapping” down on 2 controls at once.

Video games are really good at this type of incremental nudging, letting you unlock more “combos” as you learn the basics. I even tried to turn this whole “staring to tapping” concept into a game itself with a full on kid-friendly Space Opera plot to it:

As you see, the game would slowly evolve from a text-based choose-your-own adventure into eventually a full blown 3D FPS MMORPG, where the tooling to create anything from a tweet or blog post to a video game itself was baked into the game and taught as part of the campaign tutorial.

However this journey was unfortunately stopped when I discovered that the even-more-basic and boring tools of infrastructure were broken. Simple things, like storing, syncing, and scaling data, was enormously complex. So I turned from building UX tools into building information tools. From design to databases. I couldn’t trust anything anymore.

Information

Let’s pause and talk about this at a grand scale first. Infrastructure is still a type of UX. How we drive on a freeway, exchange money, or how we file books or photo albums on a shelf, or how we organize our closet, are all information management problems. These protocols are “drive on the left side” (unless you’re in England), “arrange things by color”, and even stuff like “Capitalism or Socialism”.

How we do these things, are all infrastructure. Culturally prescribed UX decisions. Yes, we want roads that let you go anywhere if you happen to have a car; “No”, we want free public transit; Yes, we want drinking water delivered in aqueducts and pipes straight to the homes of everyone for free; “No”, we want to have to sell water in bajillions of little plastic bottles.

So many of these decisions are engrained into us they’re perceived as the nature and fabric of reality itself. You know, like Socialism and Capitalism, despite being only 170 & 250 years-ish old, and invented by just two guys. Let’s not forget the roaring transition from the Mercantilism of the British East India Trading slavery company, to the skyscrapers & automobiles of America in just a span of a few hundred years:

Yes, even how we use and think about money will radically change. Money was a way to scale humanity up past a village. But now that we have the internet, most of what money as a technology was used for is now irrelevant. Instead, we can design a better UX for managing economic information — we can change the definition of wealth from “how much you hoard” to “how many people you’ve helped.” The math for this is actually quite simple.

But getting back to our UX point, the design principle of information is to do the same thing we did with interaction, to reduce everything down to one dimension. So now the goal becomes “how do you organize information to fit on a line?” Such that this will work for everything from audio editing, to website layouts, and the beyond.

The power of words is that it discretizes language into symbols that can be manipulated on a line, endlessly, until the perfect sequence of unique combinations arise. Sheet music tries to do the same thing for songs, but we don’t see very many music editing apps adopting this technique. But I actually think this is an extraordinarily powerful approach. Windows Movie Maker kinda got this right, although could have been made a million times better.

Rather than infinitely scrolling to the right, music and video should wrap like words. This is the same reason we don’t use Egyptian hieroglyphics anymore. Walking down a corridor sucks too much energy compared to scanning your eyes down some papyrus. Sideways scrolling in a tiny timeline viewport is tiring.

Let’s take this to an extreme though, the editor should literally be laid out like this article. Where would you place the video preview then? Why not just as a fullscreen transparent overlay. You should be able to edit, cut, and trim the video just like you can with text. For basic commands, such as navigation and deletion, you’d be able to use standard scrolling and caret selection.

Ideally, a song gets automatically split into “words” based on silence spaces, or if that cannot be determined, in 1 second chunks. These words can be clicked on and scrubbed through for more precision, and if there is multiple tracks or layers, the active line would expand to accommodate the height of the word.

You can even encode website layouts on a line, using the same concepts as a link. A link contains nested data, information, structure — you can always pause a stream and jump into a link, or optionally skip the link and continue on your way. All information, websites, layouts, etc. can be rendered as a series of wrapped and nested lines. The web is the closest to achieving this, but bottlenecked by poor performance and shoddy design structures.

Information Design Theory

The above approach can be applied to a tremendous number of applications, such that it not only unifies our interactions across informational systems, but also comes to consolidate and replace countless isolated and siloed tools. The reason why we have so many “productivity” tools is because they do not speak a common underlying composable language. Can we reduce design, layout, and creativity tools down to a set of lower level primitives?

This is what I call “Information Design Theory” that has 6 rules for making sure data looks good on larger displays or on handhelds. They’re pretty basic:

1. Flow & Wrap. 
2. Max & Min.
3. Focus & Divide.
4. Leak & Hold.
5. Align & Center.
6. Zoom & Detail.

The details that describe a shape’s geometry will often nest these 6 rules in a series of different arrangements or combinations. I have much more detailed thoughts on these subjects, but I’m not going to explore them here — if you’re curious, come join our community and ask me about it there.

Unifying II

What I’d like you to ponder is a few explorations of how we can integrate interactivity & information. First, Bret Victor’s “Inventing on Principle” did an excellent job in capturing programmer’s imagination, laying out the basic foundations that many tools should include by default.

I strive to take this a step further and discuss how we can combine this with Machine Learning tools, natural language programming, and “tactile-spatial” syntaxes that reduce the burden of coding for humans. Here I explore the past, present, and “Future of Programming” while showcasing some cool demos of what could be possible.

Next up is actually mixing the bits into layout, allowing different Interactions with different Information. Here’s a short example of what it could look like:

Finally, we want to take this to the extreme. This should not be just some web page or a network of web apps, it should be the browser itself. Accessibility and security information should always be available wherever you go, and the ability to create and author content should be baked into it, core to the OS. This last touch is to use that to form an overlay network on top of the existing internet, so we get the best of both worlds. The old and the new.

#2 : Appliances

The real app is hardware & appliances. Things that reduce human labor, sweat, pain. These are the real life savers, not the software aids. They free humanity up to do what we’re good at, creating innovation and art.

…but this is a teaser for Part II in this series. It and beyond will address: Appliances, Housing, Transportation, Aerospace, and Molecular Materials. Each subject as a peculiar twist that I haven’t heard others talk about. 

Generally speaking, I prefer building these things rather than writing or talking about them. Partly because when I go to write about them, it takes years to finish due to my obsession with explaining every detail. Naturally, this pairs better just creating it. However, this article and the next ones are my failed attempts at trying to briefly touch on a variety of deeper ideas.

Conclusion

I have an energetic brain with an endless number of excruciatingly detailed worlds that I tried to create but quickly became frustrated with the unintuitive interfaces prevalent everywhere that sap an immense amount of time and energy to express the precision an beauty imagined in my head. At first I became trapped in the hope that others smarter than me where on the cusp of releasing tools with the “obvious” features I assumed, only to despair when those projects disappointed, succeeded but were later shut down, or turned into greedy services.

You have to be the change in the world you want to see.

And so I got building, often failing and disappointing myself and others, but through hard struggle I’m slowly evolving towards a compounding set of tools that accelerate on top of each other. Despite the fact that I’ve had to work backwards all the way down to database infrastructure, and at some point will have to incrementally switch over operating systems and graphics hardware, this process has allowed me to shape technology such that each layer on top of it becomes exponentially more powerful. So while it is a seemingly audacious and ambitious task to tackle a variety of subjects that industries of people have spent their lives dedicated to, this is equally proportional to the awe-inspiring nature of technology itself. It equips, enables, and empowers increasingly smaller teams to achieve magnitudes larger global impact.

These tools should be for the children. Not just the certified adult wage maker. They are toys for expression, to shape forward humanity. Not just the means for the penance of our rent and inherited debts. Free as in people and free as in product. We have to lay that foundation.

So I invite you to make dance and play games, and to not stop, until the world becomes the dreams we crave.

~ @marknadal