Over a two month period I submitted four PRs that added support for the WebP image format to filament’s GLTF loader. As I was not a project owner it demonstrates solid technical judgement combined with good guest manners.

view PRs ↗

After I had delivered the initial versions of the mobile engine, the CEO challenged the leaders of the graphics teams to produce a vision for the future. My response was a presentation titled “Geomagical Galore: Our Data Everywhere” which described a future in which our room and design data became not only standards-based (GLTF), but highly repurposable. In addition to resolving the single data source & consumer debugging deadlock via third party tools, it would open the door to a huge number of benefits. It would enable a clean path to our v2 scanning tech, to creating magazine quality ‘beauty shots’ using off the shelf renderers, to sharing omnibus designed room models with users and partners.

After the presentation there was a lot of general agreement, but also a lot of people deeply occupied in their current work. It was clear I’d need to break the inertia with an initial implementation. When it was ready I did a series of one on one demos with key team members to show the basic shape of it and solicit feedback and additional considerations. A key element of the demo was showing that the formats were A/B switchable via a config flag. No one was going to have to bet their evenings and weekends on sticking the landing. After the leads were aligned I held another company-wide presentation. This not only allowed everyone else to ask questions, it also let them see other leaders answering them. Showing, not telling: leadership was aligned and confident.

The transition was across two formats: rooms (base scenes) and compositions (placement of items in the scenes). What followed was two months of coordinated work across five teams that covered the platform clients, backend infrastructure, asset processing, and room perception. The QA staff were embedded with their respective disciplines. Rolling this change out to our dev environment was a significant exercise in coordination. In preparation for the final all-hands dogfooding I wrote a detailed testing procedure for validating behavior with the new formats enabled and disabled (first, do no harm). Pushing the big green button for rollout to dev was ultimately rewarding, but low stress. The dogfooding went well and we had the ripcord of the A/B switch to fall back on.

After the rollout every tech benefit was realized, from tool availability to v2 scanning development. Even better, ideas came from others as they saw the possibilities. Our head of product suggested adding ‘View in BabylonJS’ buttons for the room and composition database pages, and it was an instant win. It provided fast visualization and scene inspection, so answering the “who’s right and who’s broken” riddle between client and data became a single click. It was an intended benefit, but with better surfacing. ‘Beauty shot’ rendering was prototyped in a week and developed into a feature in a couple months. An earlier attempt at the same task with the old formats had involved multiple contractors over multiple months and ended in capitulation to bad cost/benefit dynamics.

While the data sharing benefits didn’t play out, mostly due to IP concerns, that door is open if the situation changes, and capitalizing will cost next to nothing. The other benefits paid off immediately and continue to do so for the entire Kreativ project.

My initial work at Telltale was porting their engine to the PS4 and getting the back catalog running on it. In addition to the engine work it provided a crash course in what Telltale’s games were all about. Roughly a year in, I took on the tech lead role for the upcoming series, Minecraft: Story Mode. By this time I had contributed engine support for Tales From the Borderlands and had seen the production process from prototype & pre-production to production ramp-up to episode golden master for an entire season.

On previous titles the process had been to onboard audio, visual FX, QA, and build engineering teams at the very end of pre-production, after prototyping was complete. This introduced a lag as everyone got their asset revision control system configured, pulled the new project and got the editing tools squared away for the new effort. It was not uncommon for episode work on different titles to overlap, so switching the Telltale Tool’s focus was a critical capability, but getting the initial run of a new project took some finagling. The prototyping period was relatively slack for an engine programmer, even one tasked with lead work because it takes time for the creative team to determine what adjustments or capabilities they’ll need. Once the prototyping crew had the basic project far enough along to launch a placeholder scene I used that low-load time to visit the audio, visual FX, QA, and build engineering teams to get their RCS and tooling set up for the new project. When the ramp-up phase began they’d be able to jump in hot.

Another issue I had noted on previous titles was that the dev build debug tooling, toggled by special screen taps or key combos, was not available until a scene was fully loaded. This introduced significant gaps where build version info and debug tooling were not accessible. It provided bugs places to hide and made identifying bad/wrong builds slow by requiring team members to fully load a level before they could check. This was another fine use for the prototype-phase slack time and I dug into the painful work that had been the primary reason it hadn’t been dealt with before. In all fairness to predecessors, when you’re grinding through crunch this is exactly the sort of thing that ends up at the bottom of the stack. Once the debug tooling coverage encompassed the entire execution life cycle I did another round with the discipline teams. I had them update to the latest version of the prototype, including the debug tooling improvements and had the lead of each team verify that it worked as expected. “It works on my machine” is the cardinal sin of coding for others.

The producer was a star and very good at her job. The writing, sound, VFX, and scripting teams were all outstanding, and the animatics team transfused their blood into the project. While I was on the technical team and had a ringside seat to what the creative teams did, their work was wholly its own magic, and I was as big a fan of their work as any of our players. I can’t take credit for how smoothly the project landed or how well it was received, but I can say that the technical foundation stayed out of their way and delivered their work cleanly and on time.

When I arrived at Leap Motion my primary work was contributing to the Leap SDK and the example code library. I built their CI/CD system as well and was generally taking on the “whatever needs getting done” job that goes with startup life. The hand tracking sensor device driver and SDK were updated in tandem each release. The release process involved each engineer taking a turn at the release management job. As our product was pre-v1.0, releases were intermittently scheduled, depending on recent development progress. Once designated they were intended to go out after a one week fix and test period. This was a startup with a young engineering team. Everyone was 3 sigma smart, but several were on their first jobs out of school.

As we approached the first release date of my tenure one of the other engineers had the baton and managed the process. There was a grave concern for getting every last hoped-for thing in and a habit of not just adding late changes, but big ones. The fear was having to wait for the next release to get ‘my important feature’ out. This resulted in a release process that ran a week or more over schedule and still managed to ship rough edges. Catastrophic bugs were avoided, but it was a hitchy process because every delay to fix something was treated as an opportunity to get one more thing in. This focus on getting every feature into the current release was compounded by the delays. “It takes so long to get a release out I really don’t want to wait for the next one.”

Managing releases was a job no one wanted because it cut into coding time, hence the merry-go-round. My turn with the baton came around, and when I ran the process I insisted on no new features after the release branch was cut. I also required bug fixes to be smaller in scope as the release period progressed. Fixes that were too large were deferred if the bug was minor. If it wasn’t, the pending feature didn’t make the cut. There was some grumbling, but the release went out cleanly and on time. The QA team, which mostly consisted of game industry veterans, scattered rose petals before my feet.

Afterward, the engineering team asked me to manage the next one, since this one had gone so well. But they said they wanted to get more velocity out of the next one, so I should run it the way they did. There are moments when calm silence outperforms obvious incredulity and I judged this to be one of those times. They were serious. I counter-argued that the reason it had gone smoothly was the discipline I had enforced, but they were not convinced. Realizing that an empirical approach was the only path out of this, I offered a deal. I’d manage the next release and do it their way. Then I’d do the one after that my way again, and we’d have our data. They agreed and I managed the next release their way. Unsurprisingly, it was late and rough. I observed the difference to them, but was not pointed about it. Then we did the third release and we had another clean, timely delivery. When I explained the principle behind the approach they were prepared to hear it.

When the bus runs on time you can always catch the next one.

I took on the role of production manager, handling all releases and coding as much as I could make time for. It wasn’t that I wanted that job, it was that it was clearly where I could make the biggest contribution. I managed five major releases, each with point releases. The major releases included our v1.0, which shipped on time, without major issues, and to much internal celebration. The acid test for the bus schedule came when we needed to ship a sequence of three critical point releases, backed up against the holiday break. We nailed each one and ate our turkey in peace.

Panscopic was a startup that built a business analytics and intelligence product that had a back end server component built on Apache and a front end component in the form of a plugin for Macromedia DreamWeaver (before Adobe acquisition). I was familiarizing myself with the code base and getting my first edit/compile/test cycles going and asked how we made internal test builds and release builds. I was informed that everyone built the parts they were responsible for and then pushed them to a network shared folder. Internal and release packages were created by zipping up the folder contents and copying them where they needed to go for distribution.

I indicated a bit of confusion at this process and they assured me that was how they did it and it was fine. A brief conversation attempting to give a little pushback made it clear that management’s opinion was that working on an automated build system would take more engineering time than they were willing to spend. This was something that would definitely change after the first major consequence occurred, but I had a strong preference to not wait for that sequence of events. During the process of learning the particulars of developing for and building the various pieces I encapsulated the knowledge as a series of scripts that automated the whole process end to end. I put those scripts on the existing designated ‘build’ machine and added a cron job that checked for changes in CVS to run them (Hudson/Jenkins did not come into existence until years later).

When I presented the system I explained that I had just encoded my onboarding study of the build procedure into scripts as I went along and that it only added an extra ten percent to the process. That wasn’t strictly true, but if you don’t count the after-hours time I spent, it probably came out to less than that. They used that automation for internal and production builds for the remainder of my tenure and beyond.

The use of plain HTML+CSS with minimal JavaScript is a deliberate choice. This site doesn’t need whizzbang, view counters, or a feedback forum. Simplicity serves its focused purpose. The most important engineering decisions frequently revolve around knowing what to leave out, which requires understanding where the knee of the complexity/utility curve lies.