Resilient TechEd is a startup founded at the beginning of the pandemic to provide STEM education to children and students through the best-selling Minecraft sandbox game. Its founder had successfully sold a previous vocational education startup and took a prototype developed by an intern around digital logic circuitry and turned it into a product sold as a subscription service, first targeting parents through social media presences and collaborations with YouTubers, later selling to schools and universities. ReLoAd is a Minecraft server plugin that allows to build and simulate digital logic and can serve as an engine for interactive games that teach Python programming or binary numbers; students are provided with a tutorial, quests, and minigames to experiment and verify their understanding. CodeQuest is a Minecraft-based game that teaches programming through a story-driven adventure centered around robots in a planetary power station wreaked by a natural disaster.
Lead Software Developer
May 2022 - Aug 2024
Driven the development and software engineering of a Minecraft-based Edutainment prototype through beta release and productization.
 |
Like many software developers, I was initially driven into this profession by a desire to program my own games. After 25 years at an international tech company with 300,000+ employees world-wide, this was an exciting new challenge in marked contrast to my previous job. As a father of a 14-year-old daughter not interested in technical stuff at all, I was very much aware of the shortcomings of Germany's outdated education system, and eager to improve this situation. I started as one of the first 10 employees, within the first round of hiring permanent developers, shaping and establishing the development process, culture, and raw product. Employees worked remotely all across the country, but interestingly the company's headquarters were located in my home town, a stone's throw from where I live.
The existing prototype had been developed by a genius in his late teens; making up for his lack of experience in software engineering with raw talent and late-night coding. Unfortunately, he had to decide whether to devote his time to studies or to the startup, and (prudently) chose the former, so the code was quickly handed over to an external developer at the end of 2021, and hiring of permanent developers started soon thereafter. I joined in May 2022 along with 2 younger developers. Together, we expanded the existing CI/CD pipeline, established coding guidelines, evaluated and then refactored the code base, fixing bugs and extending the product along the way.
 |
The first six months were a phase where the team had to find its identity, evaluate the existing prototype, and take first steps towards sustainable product development. The very diverse backgrounds and characters reminded me of my very first team, with the potential to become a formidable force if we managed to develop shared goals and respect each other. That was a lot of work — meetings were anarchic and rife with I know best fights over direction (and often took much longer than scheduled). The external developer had a lot of experience (and was even older than me), and had strong but often quite peculiar opinions on robustness and programming style. With a head start of just a few months, he had the benefit of having directly worked with the original developer and created a few essential supporting components, but now quickly was scrutinized by the new developers.
 |
And there was a lot to question. There was not a single automated test — I knew from the interviews that they were looking for someone with a testing background, but I would not have imagined that this meant starting from scratch! While the original hacker had often reinvented the wheel in clever if unusual ways, the external developer wrote idiosyncratic code that often was overly complex and buggy. In a hurry to deliver, shortcuts were taken (like duplicating code into three separate modules instead of extracting a shared library). Refactoring was difficult because everything was interwoven in a few god classes with long methods for which it was impossible to write tests for. Luckily, I spent a lot of time with one of the very first technical employees, giving me first-hand insight and stories about the product's initiation.
Fortunately, I immediately had good rapport with one fellow developer (who also had a background of working in a corporate environment and on open source projects), and we quickly set up code style and verification rules, building on top of the initial clean up and de-duplication that I had done when I had taken my first steps with the code. Several bugs in the logic engine were solved by completely rewriting (now with proper unit test support, for which I had laid the groundwork). This was time-consuming and intense, but we were rewarded with much higher quality. Several "features" had only been implemented to alleviate poor performance caused by the complex and inelegant design; with the new stable and tested components and the insights into the correct design we had gained, we were able to simply delete or reimplement much of the legacy code. So what initially looked like a daunting task of refactoring a huge mess of legacy code took less than a year, and didn't have to be done to completion, as the focus shifted as well (from greatly extending the logic engine with customer-visible features to using it internally for our game control).
 |
I still get cold shivers thinking of the huge amount of code written in vain (and by a contractor costing more than all staffed developers together); that kind of misinvestment could have easily bankrupted us; it certainly was a painful learning experience (but that's what a startup is for: learning quickly and pivoting). The contractor had written a lot of code in a short amount of time; some of the code became obsolete by our rewrites, some we fortunately didn't have to touch much, but for some parts we first had to do big and complex bugfixes (all without automated tests)! I still vividly remember logic engine scoping bugs (it did not detect a lot of scopes) and missing functionality (scopes completely embedded inside another scope were forgotten to handle) that took a long time to sort out — later we completely dropped scoping, as performance improvements made them unnecessary. The whole feature was ill-defined from the start — many of the design problems actually pointed to poor requirements definitions (and the contractor just did what he was told); more on that later.
 |
I succeeded in bringing the team together, creating a culture where everyone is highly motivated to contribute their particular talents while respecting the others' opinions. I knew that this is the only way to make such talented (and often highly technical, bordering on autistic) people work together, having experienced that at the beginning of my career, and having seen how quickly a team can dissolve in conflict through the bad habits that were apparent when I joined. It took a lot of restraint and patience to give everyone (and that means confident and highly skilled youngsters with hardly any working experience) enough room when decisions have to be made quickly, but I had seen how demotivating it can be when the external developer loftily declared that his way is best, and everyone else has no clue.
 |
Most other employees had been hired due to specific knowledge: about IT administration, Minecraft subject-matter expertise, Java programming. In contrast, my role was more of general technical stewardship: Understanding the overall architecture, various technologies used, and (crucially in such a small single team) getting the hands dirty and contributing to the product's evolution wherever it's necessary. In the first few months, I've also done usability testing of our website's onboarding and user management process, working with the offshore team that maintains that website, prepared for and attended a trade fair as an exhibitor, set up gaming PCs for that, troubleshooted and fixed the wireless networking in the office building, spent time with the founder to soak up his vision and background.
Despite these very diverse and interesting tasks, my main focus had always been on the technical side. Like in past jobs, I invested heavily in my own tooling, automating and scripting development and infrastructure tasks. This forced me to focus on each topic and learn what's behind it (the dynamic environment makes it very tempting to just do the bare minimum and move on). That deep knowledge was the currency I could spend on gaining respect within the team for my initiatives (like getting them to adopt test-driven development). I would calmly resolve a problem that previously had them flailing, and then explain the root cause and encourage them to invest in the right things to prevent such problems from happening again in the future.
 |
For example, the build setup was held together by Git submodules; I wrote scripts to manage those, built a checklist of steps on how to do reintegration properly (the external developer had worked with long-lived branches; the history often was a complete mess — we agreed on short-lived feature branches for the future). The checklist became the official guidelines in our Wiki. I then automated the (repetitive) reintegration steps, saving me some tedious work and avoiding errors (so it wasn't that nerve-wracking when I got interrupted during one). Based on that, I developed another checklist on how to do quasi-transactional reintegrations — with only a handful of contributors, concurrency rarely is an issue, but this enabled me to easily roll out metadata changes across all our repos (where the chance of collision is highest). With that I became the lead maintainer of the metadata, ensuring consistency. Still seeing potential for automation, I extended my scripts to offer transactional reintegration (and support (partial) integration in GitHub (which most of my colleagues did, creating unnecessary pull requests, but that's one of the things I let slide)).
 |
Next stops on my journey through build and configuration management then were the Makefile (agreeing on Bash scripting for all tooling; developers were free to use Windows, Linux, or Mac), Docker setup, GitHub actions (splitting a dozen custom actions off our monolithic workflow).
The first developers had checked in a couple of Batch files and shell scripts to start up a local server for testing; I reverse-engineered the use cases, cleaned up and consolidated the scripts, so they could keep up with the rapid changes that the product underwent. This greatly helped with developer efficiency. Ever on the lookout for further improvements, I maintained my own wrappers with experimental extensions and convenience features (contributing them back to the team when it made sense).
The team was led by a people-/project-manager / product owner in personal union; a tough job with such a diverse team in a fast-changing environment. Brimming with energy and excitement, work and also the tracking of such was chaotic. The company didn't use an off-the-shelf project management system, but rather used a low-code platform to build an all-encompassing custom in-house tool that combines project management, bug tracking, test management, CRM, and anything else we'd need. While easy to use and customize, considerable (ongoing) effort went into adapting that system; we've spent days discussing ticket prioritization and flow. The founder also sold customization and consulting services as a side business, so ours was a strategic in-house use as well. It still felt weird that we spent so much time on what essentially was a second product, while focus is so important for a startup, but the project manager also was fully sold on that tool, and liked to fiddle with it. I appreciated the adaptability of a home-grown tool (initially everyone even had full customization rights!), but saw this as a huge distraction.
 |
The team loosely practiced Scrum when I joined, but the two-week iterations didn't fit very well; development was mostly doing refactoring for months on end, and the rest of the company was occupied with various, often quite spontaneous tasks. Tasks weren't estimated or tracked, even changed in scope on the fly, and as such often dragged on for several iterations. I suggested that we switch to Kanban instead, as that was much closer to what we were practicing, anyway. Despite my lobbying, we unfortunately didn't introduce work-in-progress limits from the start, but developers did a trial doing task estimations (which naturally led me to adapt my processes, add a small tool, and write a few scripts to automatically derive the tracking information from my calendar; I was the only one who kept tracking tasks after that experiment was over).
The team had initially used Sprint Goals not for a single sprint, but rather as long-term goals (chaotic naming was a permanent source of confusion for me, coming from a much more strict corporate background). When we moved to Kanban this was (slowly) replaced by the ability for tasks to be contained in other tasks (i.e. the Aggregate Design Pattern); later this deep hierarchy was flattened again into effectively goals (named Arbeitspakete) containing tasks. Paired with the tool's big lack in searching (basically just full-text search; you couldn't even exclude done tasks so any results were spammed by outdated stuff), you basically had to know about relevant tickets (the aficionados each had their own custom reports, but nobody set out to generalize them for broad use), or spend a lot of time finding them (so of course we had a huge amount of duplicate and outdated tickets). Fortunately, for development, we were mostly working on our own, and after the refactoring there were few actual bugs in the code, so we could ignore the flawed system for the most part. But I knew that once the training wheels came off and we had to rely on the system, it would be a huge drag. But before that happened, a lot more changes came to be…
 |
Despite the relative mess in the backlog, I'm pleasantly surprised by how many long-lived feature requests I was able to implement over time. Often, something that started as a small convenience feature or design improvement later paved the way for a groundbreaking enhancement. I managed to refactor a home-grown I18N framework to support dynamic lookups, replaced a binary persistence mechanism with JSON while keeping backwards compatibility, modularized the server startup scripts, and generalized logic I/O components to take arbitrary identifiers when enabled via feature flag. When a customer asked for better scoping support (the original ill-defined buggy feature had been removed), I was able to combine the latter two and whip up a solution while we were still discussing the issue.
The initial offering consisted of a subscription to a single server where you could invite up to five friends to tinker with digital electronics, supported by self-study material and a Discord channel, for €10 / mo. Short of going viral world-wide, it was hard to imagine how this could become profitable on its own — the numbers simply don't add up. This was seen more as a testing ground, with the positive feedback from the small but hard-core group of young, highly gifted users driving more content that could then be sold to schools and universities for much larger sums. But even there, digital electronics was just a tiny part of education (and we just convinced one prof to sign up), so we looked into other technical areas, and developed content for serial and parallel combinations of resistors, debuting a full small game with a surrounding story (a confused professor asks the player to reconnect a generator to the grid, and the janitor has to teach the player about the need for resistors to make this happen). This showed that we could do more than just digital electronics, and allowed us to offer (well, we brazenly did this from the beginning) companies or institutions our services to build any kind of game-based content (e.g. for appealing to young applicants). Frankly, this was very much a put up anything and see what sticks approach; we still lacked a sales strategy and product direction.
 |
When in 2023 programming courses became mandatory for all secondary schools in Germany, we saw an opening with a Minecraft-based introduction to programming sold to schools. Deciding on Python allowed us to leverage web-based Jupyter as the development environment. We developers resuscitated an open source API and Minecraft plugin that bridges Python into Minecraft, and put together a first working prototype within a few weeks. By pure chance I saw a YouTube ad from a competitor that used the exact same setup for private one-on-one tutoring (and that German startup shared so many striking similarities with us it could have been our twin (apart from the fact that we were much more focused on Minecraft)).
Still lacking a driving force behind the product (sales was occupied with cold-calling schools, and learning about the differences in each of the 16 federal states of Germany), that prototype underhandedly morphed first into a demo shown on trade fairs and to schools, and (after overwhelmingly positive feedback) into a full-fledged complete game, to be ready in summer '24. I had already invested in basic security (e.g. sandboxing the Python process so that students could only kill their own development environment, but not crash the entire server); now development had to keep step with rapidly progressing game development; quickly reacting to feature requests at short notice, and spending the rest of the time trying to invest in robustness without knowing where this would all lead us to. Game features also required a lot of Minecraft functionality for which we leveraged various plugins, each of which were in different states of maturity and support, which further complicated the challenge of schools having to install different kinds of client software, depending on whether they were using tablets or PCs. Behind the official world of Minecraft (which was bought by Microsoft, and has since fragmented into the original Java edition, a Bedrock edition for consoles and mobile devices, and an Education Edition bastard derived from the latter), there's a vast ecosystem of enthusiast-driven servers and plugins, with vastly varying levels of commitment, quality, and legality. For some of those plugins, we had very close in-house experience; other plugins weren't maintained and we had to fork them and make them our own. Quite a bit of our refactoring effort had the goal of making those adoptions as easy for us as possible.
 |
The because'a anything that the 1000 the that because is €123a about the the is the that. Is 1000, is should is anything the vestibulum the that about the is That (the that'a should the about is the the, is suspendisse a should is the anything, is that that the that is because vestibulum because).
 |
The Python programming demo was very well received at a trade fair in February, generating lots of leads, earning the team an emotional well done from the founder. On the next trade fair in June (which I had now attended the third time in a row), we received an award and presented both school and business ventures at a big booth, but unfortunately there was much less interest (the year prior we were at a bad location, and it was also disappointing). (However, I used the opportunity for ad-doc usability testing with interested pupils and grown-ups.) Converting those leads into demos and contracts still proved difficult. On the technical side, we frantically tried to make a web-based client work for us (despite it being a one-man fork currently under heavy development and with dubious legality), assuming that the technical difficulties in having to install a Minecraft client were preventing us from success.
 |
We had tried to stall, keep ourselves afloat, but everyone knew that the final reckoning was due. I received my redundancy dismissal at the end of June, took my remaining vacation in August, and left the company at the end of that month.
 |
Ingo Karkat; last update 09-Oct-2024