That is accurate. It is also incomplete.

So far, there are two states for NavEngine; the hosted version - whose database I am responsible for whether it fails at three in the afternoon or three in the morning and the one a customer installs.

For this read, I share the story of what I built to hold it all together and why it ended up being more than a place to put files.

The Decision I Didn't Know I Was Making

I had a lot of questions at the start of the project.

• How would the customers download the product?

• Where would it be stored?

• How do companies like Canonical handle distribution?

• What does distribution actually imply?

I had set out to solve a storage problem. I needed a place that was reliable, versioned, and reachable from anywhere I needed to reach it from. S3 was the obvious answer. Most things are obvious in retrospect. I did not think much about the decision at the time.

What I did not anticipate was that a storage decision is also an architecture decision, and an architecture decision at the distribution layer has a way of spreading.

Job One: The Product Suite

NavEngine started as an ISO. Before a customer would run anything on their infrastructure, they needed to have an existing artifact . They needed somewhere they could fetch it from, with integrity guarantees, across whatever network conditions exist on their end.

Indulge This raised a different constraint: multiple products would need the same guarantees.

I needed a place to store iterations as I built towards UAT - User Acceptance Testing. The registry handled images. The bucket handled everything else.

The decision to version the artifacts explicitly - not overwrite, not float - came from the same reasoning behind the floating tag in the update pipeline: I need to know exactly what was shipped.

Versioning, but without guarantees...

If v4.5.0 ever needed to be reproduced or rolled back to, it should already exist. Not rebuilt. Not reconstructed. Present.

Indulge The hardest part of distributing software you do not host is preserving the integrity of what you shipped. Tags drift. References move. The artifact you think you gave a customer can diverge from what they actually received. Versioned, immutable storage is the only reliable paper trail when something breaks on a system you cannot access.

Job Two: The Backup

We initially served backups of NavEngine by running weekly and monthly VM snapshots. It was tedious, human-dependent and error-prone.

Example: If an engineer needed to manually update a database record, they would have to manually create a backup for that before that individual operation. This backup would then live on the same machine.

This meant that while the backup was present, if anything were to hit the fan, it was game over.

I have been here. The only grace I had was that it was on the staging environment. The story of how I got back is for another day.

The issue wasn’t that we lacked backups. It was that restoring them required the same precision that introduced the risk.

A backup that lives in a different system than your product artifacts is a backup that gets forgotten about during an incident, when you are moving fast and your attention is already split. Having everything in one place means one set of access credentials, one retention policy to reason about, one place to look when something is wrong.

It was anything but elegant. It was reliable.

Indulge Backup strategies fail at the moment of failure, not at the moment of setup. The question to ask is not "do I have backups" but "can I restore from them at two in the morning, under pressure, with the right people asking questions?"

If the answer is uncertain, the backup strategy is incomplete regardless of how often the job runs.

Job Three: The Agnostic Product Registry

This is the job I had not planned for.

What if we had other products to ship that had the same constraints?

Beyond hosting the versioned images of NavEngine, the storage option needed to be agnostic enough to store other images while retaining the ability to version those products too.

Somewhere in the process of building the first two jobs, I noticed that the structure I was laying down did not have to be NavEngine-specific. The bucket had opinions - a versioning scheme, an artifact layout, a naming convention - but those opinions were not tied to NavEngine. They were tied to the idea of a distributable product. Any product that needed the same things NavEngine needs - versioned artifacts, clean separation between releases, a retrievable history - could sit in the same bucket structure without the structure caring what the product was.

I did not set out to build a product registry. I set out to store some files. The registry emerged from the constraints.

As the team builds BusinessAI, the need to account for portability became obvious. Different pricing. Different licensing. Same distribution problem.

This matters because distribution surfaces compound. When the second or third product needed the same infrastructure, the cost should be near-zero. The structure would already exist, the access policies already reasoned about and the retention logic existing.

I wanted to reuse the shape I had already committed to without building new infrastructure for a new product.

I am building around invariants, not specifics. This isn’t storage for NavEngine. It’s storage for products that require versioning, integrity and retrievability from unknown environments.

The Bucket That Became Infrastructure

At some point the bucket stopped being a storage decision and became infrastructure.

Storage holds data. Infrastructure carries consequences.

For this use, I went for Garage - a self-hostable and distributed S3 store. Migrations, such as our port from v2 to v3 no longer required moving 60GB+ of assets. The problem reduced to managing references - ASCII text pointing to stable objects.

Indulge: In the case of migration, moving from v2 to v3 meant that while the database rows were ported, pointing the assets for each organisation including white-labelled data was a manual process. It relied on knowing the file system storage path and identifying what asset belonged to what organisation.

This meant moving large files over a network from server A to server B - a process I would not like to do again.

Three jobs, one system - not because it was the cleanest architecture, but because the jobs turned out to share more than I initially gave them credit for -versioning, integrity, global access, and durability beyond any single system.

What I built was, inadvertently, the distribution layer for the company's technical output.

It holds what we ship, what we back up and what we might ship next.

That is more than I planned for when I sat down to store some files.

I later ported the same S3 infrastructure to my Homelabbing series, extending it with POSIX-compliant storage via versity.

This means that even as I develop software for my own use case, I can safely extend anything pre-existing to use the volumes I already have. That's it. No config. No workaround. And most importantly, no downtime.

What This Changes

As I dive deeper into product agnostic infrastructure and look back at what this quarter has meant, my thinking about distribution has shifted.

Uptime isn’t just about keeping systems running. It’s about designing for the moment you lose control of them - backups, extensibility, storage.

The CI/CD piece ended with designing for absence - for the reality that once the software leaves your hands, you have no control over what happens to it. The S3 structure is the complementary question: what must never leave your control?

Artifacts. Backups. Release history - the source of truth.

What the customer runs is a copy. If the copy breaks, you come back to the source. If the source is solid, the copy is recoverable.

That is the job the bucket is actually doing.

Not storage. Infrastructure.

I say "shipping" loosely. There is no deployment script I run, no SSH session I open and no Kubernetes rollout I watch. The software lives on infrastructure I have never seen, behind firewalls I cannot reach, on machines whose specs I do not know. Shipping, in this context, means pushing an image to a registry and trusting that a process running inside a container on a customer's server will eventually notice and do something about it.

That gap - between what I push and what the customer runs - is what this piece is about.

The Assumption Collapse

Every CI/CD tool I have ever used was built on a premise so foundational that nobody thought to state it: you control the deployment target. You own the server. You have the keys. Deployment, in the conventional sense, is just automation wrapped around access you already have.

NavEngine is quite the opposite. It exists as a custom image - qcow2 - shipped onto the customer's infrastructure. The customer owns the machine. I do not have SSH access unless I go through DWService and even then, that is a support channel, not a deployment one. Yet somehow, we need to continuously deliver updates to machines we cannot reach, across connections we cannot guarantee, without breaking software that is actively in use.

So the question became: if you cannot push, how do you deliver?

Indulge The moment you decide to ship software you don't host, you have made a decision with consequences that will follow you for the life of the product. Not just operationally. Architecturally. Every assumption your codebase makes about the environment it runs in now belongs to someone else's infrastructure. That is not a deployment problem. It is a design problem that shows up at deployment time.

The Answer is Pull

Watchtower. It runs as a container alongside the rest of the stack, polls the Docker registry on a configured interval, and when it detects a new image digest on the tag it is watching, it pulls and restarts the relevant containers. No webhook, no push, no SSH. The installation phones home for updates and takes what it finds.

The key design decision here was the floating tag. Every customer's Butane config ships with core:stable. Not core:v3.0.39. Not a digest pin. stable. When Watchtower polls the registry and sees that stable now resolves to a different digest than what is currently running, it pulls. What "stable" points to is entirely under my control, from the registry side, without touching anything on the customer's machine.

This sounds obvious once you say it. It took longer than I would like to admit to get there.

Two Registries, Two floating tags, One gate

Here is the full pipeline as it actually runs.

Every push to main triggers a build on the dev registry. The image gets tagged with a version identifier and a floating tag - staging. A staging environment - running the same Compose stack, same Butane configuration, same structure as a customer installation - pulls from staging. This is where the image lives until I am satisfied it works.

When staging looks good, I create a release. The production registry builds from that release, tags the image with the version (v3.0.40) and overwrites the floating stable tag. Customer installations, on their next Watchtower poll interval, see a new digest on stable and update.

The critical detail: stable is never overwritten by a push to main. Only by a release. The staging gate is the only thing standing between a broken image and a customer's running installation. There is no automated rollout percentage, no canary fleet, no gradual traffic shift. The gate is a human decision, made after watching staging run and deciding it is ready.

For a solo-operated product at this stage, that is the right call. Complexity in release infrastructure that you do not need is just surface area for things to go wrong.

Indulge A staging environment that does not accurately reflect production is a very expensive placebo. It gives you confidence without giving you information. The hardest thing about shipping self-hosted software is that your staging environment runs on infrastructure you understand, with data you created, on a network you control. Your customer's environment is none of those things. No pipeline fully closes that gap. The best you can do is know exactly where your confidence ends.

What Happens When stable is broken

Backups(monkeyuser.com).

It will happen. An image that passes staging will break in a customer environment for a reason that staging did not surface - a schema migration that assumed a clean database, a dependency that behaves differently on older hardware, a configuration value that was present in staging and absent in the field.

The recovery flow is: fix on main, watch it pass staging, cut a new release. v3.0.41 overwrites stable. Watchtower picks it up on the next poll interval. The customer, who may or may not have noticed anything, is now running the fixed image.

The window between the broken image landing and the fix arriving is real. Depending on how fast the hotfix moves through staging and how long the Watchtower poll interval is, a customer could be running broken software for anywhere from minutes to hours. There is no remote kill switch. There is no way to reach in and restart a service. There is DWService if the situation is bad enough to warrant it, but that is a support escalation, not a deployment tool.

This is the honest cost of not controlling the deployment target. You accept a recovery latency that you cannot compress below a certain floor. The mitigation is not a cleverer pipeline. It is investing deeply in staging fidelity and in making sure the image fails loudly rather than silently - health checks that surface problems immediately, startup validation that refuses to run on bad configuration rather than running badly.

A system that fails loudly is a system that can be fixed. A system that degrades quietly is a system that erodes trust before anyone knows there is a problem.

Enterprise and Standard Licenses: different cadences, same pattern

How do we manage customers that diverge from the main product line with an enterprise license?

NavEngine has two license tiers. Enterprise customers are on a separate release cadence from standard customers. The mechanism is straightforward: separate floating tags on the production registry. core-enterprise:stable and core-standard:stable. The Butane config shipped to each customer points at the appropriate tag. Enterprise releases can go out on a different schedule, carry different feature sets, and move more cautiously than standard releases.

What prevents a standard customer from pointing Watchtower at the enterprise tag? Mostly friction. The Compose file is baked into the Butane config at provisioning time. There is no SSH access to change it. A customer would need console access and the motivation to go looking - unlikely for most, impossible to rule out for all.

The proper answer is registry-level access control: pull tokens scoped to the tags each customer is entitled to, issued at license activation and revoked at expiry. This means the registry enforces entitlement, not just the application. An expired license means an expired pull token means no updates, enforced at the point of delivery rather than after the fact.

This is on the roadmap. For v4, the answer is friction and trust.

Indulge License enforcement in self-hosted software is a negotiation between what you can technically control and what you have to trust. You cannot fully control what runs on a machine you do not own. At some point, a sufficiently motivated customer can circumvent almost any enforcement mechanism you build. The goal is not to make circumvention impossible. It is to make compliance easier than circumvention, and to make the value proposition strong enough that the question rarely comes up.

The license server is not in the update path

One decision I am glad we made early: the licensing server and the Docker registry are separate infrastructure. They do not share a failure domain.

Watchtower polls the registry. The license server is called from within one of the running containers as part of normal application operation. If the registry is unreachable, the software keeps running. If the license server is unreachable, the backend falls back to its last known state - persisted to disk, not held in memory, so it survives a container restart. The check runs periodically. The grace window is generous enough that a license server outage does not immediately affect customers, but not so generous that expired licenses can run indefinitely.

This matters because the failure modes compound. A product update that requires a license check to proceed has just made the license server a dependency of your deployment pipeline. Any outage that hits your license infrastructure also hits your ability to ship updates to paying customers. Keeping these paths separate means they fail independently, and independent failures are recoverable in ways that cascading ones are not.

What the pipeline actually looks like

The immutable version tags are not just for auditing. They are the rollback reference. If v3.0.40 breaks everything, v3.0.39 still exists in the registry. I can retag it as stable manually and customers will roll back on the next poll. This has not been needed yet. It is there for the day it is.

Indulge Most CI/CD writing treats deployment as the end of the story. Ship it, watch the metrics, move on. Self-hosted software inverts this. Deployment is the beginning of a period during which software you cannot reach is running in an environment you cannot see, on behalf of a customer whose experience you will only hear about if something goes wrong. The pipeline is not a delivery mechanism. It is a trust mechanism. Every decision in it is a decision about how much you trust the image before it leaves your hands.

Three Weeks Out

NavEngine v4 is three weeks away. The pipeline is running. Staging has held. The tags are in place.

None of that answers the only question that matters: what happens when the software leaves you?

It will run on machines you have never touched, against data you have never seen, in environments that do not care about your assumptions. By the time it fails, if it fails, it will already be someone else’s problem - and still entirely yours. The customer notices before you do. That is the thing. I am shit scared.

I suppose that this is the essence of these notes - to document real systems in real time. This is the inversion self-hosted systems force on you: treating deployment not as the end of control but as the beginning of its absence.

So you design for that absence.

You design for recovery over prevention.

For visibility over certainty and trust over control.

Everything else is just what it takes to make that possible.

Software is the means by which businesses achieve mass distribution. Beyond technical acumen and efficiency, software is the bus through which businesses achieve scale.

Over the quarter, I have been working on software distribution. Specifically: licensing. Think of it in terms of introducing a new avenue of revenue. We dare ask: Do the businesses that value this solution want control over where their data lives? Are they willing to pay a premium over this control?

The answer, in both cases, was yes and that opened a door I had not thought to knock on before.

I had to stop thinking about features and lean towards markets and distribution surfaces. Distribution, in this sense, is not about deployments and pipelines. It is about understanding that different buyers have fundamentally different relationships with trust, compliance, and infrastructure ownership. Your architecture either accommodates those differences or silently forecloses them.

Being technical helped me deploy features product needed. Being business-minded led me to more questions. What would it take to ship this as a product someone else operates? What are we assuming about shared infrastructure that a regulated industry, a government agency, or a security-conscious enterprise would never accept?

The answers to those questions are not found in a ticket or a sprint. They surface when you sit long enough with the business model to let it ask questions back at you.

Licensing as a pricing model

For Navengine, the questions came back as licensing. Once you accept that a buyer might want to run your software in their own environment - air-gapped, on-premise, inside a VPS they control - you need a mechanism that lets the software know it is legitimate without phoning home in ways that violate their security posture. You need entitlements, activation flows, machine-bound licenses and expiry.

It turns out the problem I was solving was not purely technical.

Somewhere in the middle of designing the activation flow, I realised the license schema was not a technical artifact. It was a pricing model. The fields I was choosing - seats, features, expiry dates, activation limits - were decisions about how we intended to capture value, just expressed as database columns. It was literally us deciding how we wanted to make money, written in schema form. And no product manager or founder had handed me a neat spec sheet. I was the one typing those business decisions into code.

That kind of realisation doesn’t land with a dramatic “aha!” moment and applause. It creeps in quietly. You’re just fiddling with fields one afternoon… and then, almost without noticing, the way you look at your own codebase has already changed. You start reading the same migrations and models like they’re business strategy documents instead of “just” backend plumbing.

The second-order effects arrive

When you decide to support self-hosted deployment, you are not making a DevOps decision. You are ratifying a set of architectural choices that were either made deliberately or inherited by accident. Configuration that was hardcoded because "we only have one environment" now needs to be externalized. Database assumptions that worked fine in a shared-infrastructure model now need isolation boundaries. Upgrade paths that were previously your problem - a migration you ran on a schedule you controlled - become your customer's problem, which means they become your support problem, which means they were always your design problem.

The business-minded engineer understands this before the contract is signed. They look at a feature and ask not just "how do we build this" but "how does this behave when someone else is operating it?" That is a different question and it produces different code. It produces code that fails loudly with actionable errors instead of silently with corrupted state. It produces configuration schema that are documented because they have to be, not because someone found time. It produces container images that carry their dependencies rather than assuming a shared environment that may not exist on the other side of a firewall.

Containerizing Navengine using Flatcar images was not primarily a reliability choice - it was a distribution choice. The image had to be self-contained because I could not know what the environment on the other side looked like. Keygen had to be reachable at activation but not required at runtime, because an installation behind a firewall cannot tolerate a dependency on an external service it may never reach. None of that came from an architecture review. It came from thinking hard about the buyer and working backwards from their constraints.

That backward motion - from buyer to architecture - is the discipline that business-minded engineers practice. It is less intuitive than building forward from requirements, and it is rarely taught, because most engineering education treats the business as a context for the technical problem rather than as a force that shapes the technical solution. But the shape of a system tells you exactly what assumptions its builders made about how it would be used, and by whom, and under what conditions.

If Navengine could only really be operated, understood, or debugged by the people who built it… it would cease to be a product. It would be a service that’s been dressed up in product clothing.

The distinction matters because services and products scale differently. A service scales with people - support engineers, customer success managers, implementation specialists. A product scales with copies. When you ship a licensed, self-hosted deployment, you are betting that the copy can stand on its own. That bet is won or lost in the architecture, long before the first installation.

None of this is free.

I will not pretend this is clean. Supporting two distribution surfaces - a managed SaaS and a self-hosted licensed product - means every feature has to answer for itself in both worlds. Every abstraction either holds across deployment models or eventually creates a seam you will debug at the worst possible time. The surface area grows and so does the discipline required to keep it coherent.

This is why most teams never do it. Not because it is technically beyond them, but because it requires a kind of thinking that does not come naturally when you are moving fast and the backlog is full. It requires you to slow down and ask what you are actually building - not in terms of features, but in terms of the thing a customer would take ownership of, operate independently, and stake their own reputation on. That is a different standard than passing your test suite. It is a harder one.

But it is also where engineering becomes interesting in a way that pure technical depth rarely matches. When you understand the business well enough to make architectural decisions that preserve future revenue options, you are no longer just implementing someone else's vision. You are shaping the product's possible futures. That is leverage of a different kind - not the leverage of writing faster code or designing more elegant abstractions, but the leverage of making choices today that compound into distribution tomorrow.

John Carmack once said the most important thing an engineer can do is build full product skills. I used to read that as advice about career positioning. Working through this, I think it means something more specific: learn to see the economic system your technical system lives in. Features are temporary. Architectural commitments are long. A business model encoded in a data structure outlasts the engineer who designed it.

The business-minded engineer does not stop caring about the craft. If anything, they care more - because they understand what the craft is in service of. Every clean abstraction is a future operator's sanity. Every well-chosen failure mode is a support ticket that never gets filed. Every deployment boundary that was thought through carefully is a market that remains reachable.

Software is the means by which businesses achieve mass distribution. You already knew that. The question is whether the software you are writing is ready to be distributed - not just deployed, not just maintained, but handed to someone else, in a corner of the internet you will never see, running a business you will never fully understand, solving a problem you helped make solvable.

Becoming a business-minded engineer doesn't mean every engineer should be writing pricing decks. It means understanding that every architectural choice is also a business choice - and that some of the most impactful technical work you'll do is quietly expanding the distribution surface of the thing you're building.

That is the bar. Build to it.

“And I took the road less travelled. And that made all the difference.”

Time management is something I find myself working through every once in a while whether it involves writing down weekly items to check off and crossing 2 of the twenty, crossing a 3/4 percentage or throwing it all together.

My read on effective engineering dubbed, The Effective Engineer by Edmond Lau, has been as inspirational as it has been eye-opening. From this, a few thought processes came up that would be worth noting, specifically regarding impact, sustainability and effectiveness.

To quote:

I worked the long hours because I wanted to make a meaningful impact, but I couldn’t help but wonder: Was putting in 70-80 hour weeks really the most effective way of ensuring our startup’s success? Our intentions were sound, but could we have worked smarter? - Edmond Lau

With this piece, and in a builder’s pursuit of mastery, I will interweave a few ideas, thoughts and research points to challenge convention and support the contrarian or vice versa. It is a leap out of the day-to-day routine of the builder and into the vastness unknown.

The Always Be Shipping mantra

The most captivating moment from the book, at least for me, was the mention and talk of experimentation. The goal is building, not with having a specific audience or market in mind but for the sake of it. I call it flow.

I can’t tell you why you need a home computer right now. I mean, people ask me, “Why should I buy a computer in my home?” And I say, “Well, to learn about it, to run some fun simulations. If you’ve got some kids, they should probably know about it in terms of literacy. They can probably get some good educational software, especially if they’re younger. “You can hook up to the source and, you know, do whatever you’re going to do. Meet women, I don’t know. But other than that, there’s no good reason to buy one for your house right now. But there will be. There will be.” - Steve Jobs (Speech at the International Design Conference in Aspen, June 1983)

In the early 2000s, Google pioneered what was called “The 20% time” - a chip of time within which engineers in its domain would be free to leverage Google’s technology to build their own products unhindered. Suffice it to say, this led to the development of products such as Google News, Google Maps and Gmail.

Consequently, this fostered a culture of innovation and creativity which contributed to its relish within the engineering community as the ‘go-to’ company to work with. LinkedIn, Atlassian and Adobe followed suit with hopes of catching up.

I don't know what the future will bring, but if you stop investing in basic research today, you won't have a future. - Neil deGrasse Tyson

From my experience, it is always the builders that do more than is within their scope that end up being the great men and women in their domain; the innovators of the future. To mention a few of these; ProductHunt, Slack, Twitter, Pinterest (hello Elle), Shopify, AppSumo and GitHub. For the founders, it was the little something on the side that made all the difference.

Remarkably, by the time the aforementioned companies were either acquired, went public or reached a certain user-base marking, their ratio of engineer to user was in the scale of one to thousands or even millions. In other words, they had built successful enterprises not with an abundance of human resources but with a vision for a product that addressed their immediate need.

The reason we[Woz and I] built a computer was that we wanted one, and we couldn’t afford to buy one… We were just two teenagers. We started trying to build them and scrounging parts around Silicon Valley where we could…All our friends wanted them too. It was taking up all of our spare time because our friends were not that skilled at building them, so Woz and I were building them for them. - Steve Jobs, 1996, Apple 20th anniversary.

Thought: If someone does not think you are insane for how relentlessly you work to achieve your dreams, you might want to re-evaluate your choices. Being called crazy and being rejected is not such a bad thing.

Conclusion

In the end, you might never know when or where your project might be of use to the rest of the populous. The least you can do, however, is to take a chunk of time within your day to build, innovate and imagine the possibilities.

Reference notes

For some notes on how some of the products above came to be, I leave some starter resources here.

• How GitHub Democratized Coding, Built a $2 Billion Business, and Found a New Home at Microsoft

• Steve Jobs Speech at the International Design Conference 1983

• The origin of ProductHunt

• Make Something Wonderful - Steve Jobs

An opinion piece that caught my attention was one by Adam Molai entitled "Africa lacks the right kind of entrepreneurs," along with the paper "Jobs, Economic Growth and Capacity Development for Youth in Africa," both of which form the basis of my thoughts today.

Indulge

With the world at 8 billion persons as of 15th November 2022, Africa remains one of the poorest continents on the globe, despite the fact that it will account for 2 out of 5 working-age individuals by the end of the 21st century. The peculiar part of this statement is the quality and purposes with which we build our ventures within the African economic zones.

The crux? Was this business built out of a pure innovative spirit - to create what once was not - or was it a means to an end?

Returning to the population question, with a 245.0% increase in population within the working age group, we expect a rise in the number of graduates and professionals alike. All this while, the market is not primed to absorb this number of individuals. If anything, the tech lay-offs that have happened over the last couple of months have been a testament to what happens when projects fail and companies are bloated.

Was this business built out of a pure innovative spirit - to create what once was not - or was it a means to an end?

This raises the question, what does this mean for education?

In a culture where entrepreneurship is often viewed as a 'side-hustle' for those unable to secure traditional jobs, we risk neglecting the creation of strong foundations for the future.

The next Bill Gates will not build an operating system. The next Larry Page or Sergey Brin won’t make a search engine. And the next Mark Zuckerberg won’t create a social network. If you are copying these guys, you aren’t learning from them. - Peter Thiel (Zero To One)

It is high time we create sustainable solutions that navigate from bringing in the everyday meal to those that impart beyond sustenance. To leave you something to nibble and ponder, here is an article from TechCabal about tech talent navigating its way back to Africa.

Thought:

Perhaps the danger of Artificial Intelligence is not in it taking away employment ( unless you champion for bullshit jobs) but that with the increased use of the same technology, AI relearns from itself and other old content rather than what is made a-new. The limit remains; Artificial Intelligence will only learn from what already exists.

I have a confession to make; I am a huge fan of sci-fi.

Whether it is another episode of The Minority Report, Upload or Travelers, I am all for it. Perhaps the underlying cause is my fascination with how these societies use technology in their everyday lives. Moreso, perhaps it is in the chance to witness something that is on the brink of discovery in our modern-day information era.

One of the common theme that appears across these films is the ability of machines to predict what we want.

An observer need only take one look at how, say the show, The Minority Report, uses its predictive algorithm Hawk-Eye, to predict the occurrence of crime and compare it to Essex University’s KeyCrime, to see a closeness between fiction and reality.

What if we could predict what would happen before it does? What if a machine could tell what you would buy before you actually get to it?

What is a Recommendation System?

A recommendation system is an information filtering technique that predicts what a user might prefer based on their historical interactions or preferences.

As an example, and coming back from our high on KeyCrime, if a user frequently watches romantic comedies on a streaming service, a recommendation system may suggest similar movies or TV shows. These systems are based on machine learning algorithms that analyze user behaviour to generate other details that they might be interested in.

They have become commonplace in industries such as e-commerce, social media platforms, security and healthcare. Let’s talk about this and the different aspects that come to play around recommendation engines.

Types of Recommendation Systems

For brevity, there are different types of recommendation systems, including content-based, collaborative-based, hybrid, and demographic-based recommendations.

Content-based Recommendation

Say, for instance, you are called Ian, and you have a thing for apple products. You log into Instagram and double-tap on two or three images around apple products or advertisements. What happens in the background?

Content-based recommendation systems make recommendations based on the similarity of items and are attached to the particular user. the more interaction you have with a system, the more accurate the data gets. In essence, a content-based recommendation engine needs a base item(s) around which a user’s actions(reviews, likes etc…) can be pegged.

“We see you liked an image that has wheat. Here are other wheat-based products you might relish.”

Collaborative-based Recommendation

What if we have a whole community of foodies? Could they ping each other based on what they enjoyed preparing?

Collaborative-based recommendation systems generate recommendations based on the behaviour of similar users. If two users have similar preferences, the system will recommend items that one user has liked to the other.

The Netflix Prize was one such event where builders across the board were invited to a collaborative filtering algorithm challenge to improve upon its own algorithm, Cinematch. A grand prize of $1000 000 for any team that came through with the solution.

Broadly speaking, collaborative-based recommendation systems are grouped into memory and model-based. For purposes of this guide, we shall focus on memory-based collaborative filtering within which we can further break into Item and user-based collaborative filtering.

User-based collaborative filtering

At its core, user-based collaborative filtering is based on similar consumption patterns.

Example: Let’s say Rita finally set up her restaurant. We pay a visit, and you give it a 5-star rating. Now, your friend, Peter, does the same. Already, we have two users with a similar rating on the same location. It is very likely that because Rita and Peter have a 5-star rating on a specific item, they will have other items in common. Thus, we look for people with a rating of between 5 and 3.5 for the restaurant and suggest other restaurants they like to you.

This ‘envelope’ we just created in our example is called a threshold. So, a user with a rating of 1 on the said restaurant will not be a source of suggestions for our next restaurant hopping experience.

Item-based collaborative filtering

Example: Most users that buy bread also buy butter. therefore, these items must be similar. Or if user X likes films with ‘Iron man’ then they will like those with ‘Spiderman’. You get the gist.

Introduced by Amazon in 1998, item-based filtering bases recommendations on similarities between items. This decision will affect our ‘pre-sales’ option during checkout. The suggestion is to get this customer to buy one more item, that is, spend more. This is the reason why everything in retail stores is where it is. Nothing by chance, nothing random.

Are there challenges to collaborative filtering? Well, certainly. Depending on which of the aforementioned you use you might encounter the following:

1. The early rater problem

So, you successfully have a user log in to your platform with 20 000 books. They rate 1 / 20 000. We literally have nothing to recommend.

1. Sparsity

You have a number of users all of whom have rated a significant count of products. However, these items are too far spread apart. Say, for instance, back to books, only two have rated the same books. All the others have rated items with no duplication.

1. A gray sheep

You know yourself. You like to remain unpredictable, you open browsers in incognito all the time, like random things every time and leave the platforms you visit. Just like Elle(laughs like Mojo Jojo).

1. The shilling attack

This happens when a single user or group of people create multiple accounts on the same platform and rate , say a product in a certain way in order to sway other users’ interest to the items they are rating highly. It can also happen to sway users away from a product by giving it bad ratings.

Hybrid Recommendation

Certainly, we must have come up with a workaround content and collaborative-based recommendation engines. In comes Hybrid recommendations.

Hybrid engines combine both content-based and collaborative-based approaches to generate recommendations.

Demographic-based Recommendation

Do you remember the piece, Up and Away with Scalability - specifically on caching and content delivery servers?

Demographic-based recommendation systems make recommendations based on demographic information such as age, gender, or location.

For one, we could have a location that is known for a type of music. Thus, instead of having our data centre closest to them caching everything and all other genres, why not cache what people actually listen to?

Pro tip: Bongo, a music genre, is popular in the coastal regions of Kenya and Tanzania. My music service would therefore hold a cache for this genre on servers closest to these regions as opposed to , say Alaska.

Demographic-based recommendation systems are often used for marketing purposes and can be particularly useful for identifying and targeting specific groups of users.

Applications of Recommendation Systems

A favourite example I like to reference when it comes to recommendation engines is one highlighted in the book, The Power of Habit, by Charles Duhigg, of the American retail store, Target.

Target’s data analytics team got so good at building these systems, it got into the public limelight when it suggested diapers in one of its marketing campaigns to a father who was sure his daughter was not. It recommended products users might want to purchase based on their patterns and on the data of other people they collected using their gift cards.

Other examples you might find interesting include how, for instance, LinkedIn uses the “You may also know” or “You may also like” for companies or profiles to follow.

Thought: Did you know , Netflix has gone through phases in its Personalization using recommendation systems?

What does this mean for the products in the build process?

For a preview, you can take a sneak peek into marvinkweyu/marastore - an e-commerce platform for travellers and backpackers. It includes within it, an item-based collaborative filtering engine. We dare answer;

Does user A need product Y given user B bought X and Y?

Back to you, the reader: How would you rank the aforementioned results?

Another example of a similar implementation is the library project, "Urbanlibrary" - the Afrocentric literature bookshelf. Within it, I implemented a mix-and-match if you may of recommendation algorithms. Is it based on history? Is it based on rating? You’ll never know unless you find out.

Conclusion

Recommendation systems have evolved to suit different needs and algorithms have been built on top of each other to meet this demand. You can combine, break away from, build your own or adopt an existing solution. Either way, it offers a path to understanding your customers and business. Take advantage. Personalize this experience.

Indulge: Perhaps all you need to be a mind reader is your browser history.

I needed a central place for all this. I needed a localized address where I could get all my code and the requirements to, well, ‘just work’.

Scenario:

I deploy projects A and B to the same server. We might have two domain names, or each might use a different subdomain. To be sure, or if I wanted to change them, I would have to modify the nginx file(s) on the server.

Would it not be easier to look at these locally such that I deployed with confidence each time?

Come Ye Docker

Consequently, I packaged my projects into their respective images and deployed them. All the while, I was keen on performance and memory consumption.

Questions:

How long does it take to build?

How big are the respective images?

What does CPU usage look like?

Are they truly that much safer than bare metal?

In this journey of discovery, I built upon the following recommended practices of containerization - modifying each to suit what it needed to accomplish.

Minimizing the number of layers

At its core, docker sits on the premise of layers. Case in point; creating a software engineering project will require an operating system, the core components that will let your programming library/language work, the programming language itself and the files that you write.

Similarly, when you go the docker way, you create a system where you are in control of these layers. Do you need library X or would Y work better? What about cron tasks? Does my system require cron or curl to be installed somewhere within?

Across the commands that we have in docker files, the following modify the image size:

COPY

RUN

ADD

What is common amongst the above is that they involve the movement of files either from the internet (RUN or ADD) or within your directory(COPY).

Example:

Take the following docker file.

# many layers
FROM python:3

RUN python -m pip install --upgrade pip
RUN python -m pip install --upgrade setuptools
RUN pip install -r requirements.txt

To optimize this, we can merge multiple RUN commands and or use ADD instead

Image 2:

FROM python:3
# notice the merge of RUN commands TO update the system and install any packages needed
RUN python -m pip install --upgrade pip \\
&& python -m pip install --upgrade setuptools \\
&& pip install -r requirements.txt

You notice that every time we run either of the commands highlighted above, the image size changes. The same does not apply to other docker file commands (CMD, ENTRYPOINT)as they are limited to creating intermediate layers (0 bytes).

Choosing the right image

When it comes to choosing a base image for your containers, it is advised to use an image that matches your environment as closely as possible.

Example:

If deploying a java-based application, you would use the JDK image upfront rather than using an ubuntu-based image and installing the JDK on top of it. In this case, your Dockerfile would look as below:

FROM ubuntu:latest vs FROM openjdk:latest

This way, you free the image from having to install some apt dependency W all because it was needed by some background task that is recommended.

Using the exact image tag rather than choosing the latest tag for the image base

Correct. The previous Docker image file is wildly flawed.

Example:

Consider for a moment, that we are running a Django 1.17 project. However, our Dockerfile insists that it wants to build based on the latest version of python out there, so we add the following image.

FROM python:latest

What are the project dependencies that were deprecated? What module was moved or merged? What package had to change because the libraries of the latest python version no longer behave the same?

Instead, you could run, your product with a specified version:

FROM rustc:1.64.0

(The above snippet assumes you are a rustacean, but the same principle applies to any code base you might have).

Using a minimal-sized base image

Getting back to layers, an image can be built based on another. To this end, your stack of choice, say python3 might come as is, that is python 3 or be based on the Debian or alpine images. These image versions can appear as below:

python:3.9.16

python:3.9.16-slim

python:3.9.16-alpine

python: 3.9.6-slim-bullseye

To identify the base of your images, you observe the tags that follow. In the above case, images tagged with the keywords alpine are based on alpine-linux while those with the tags Jessie, stretch, buster and bullseye rely on Debian 8, 9, 10 and 11 respectively. The tag, slim*, is the trimmed version of the default image, that is, python:3.9.16 and may be included in either tag, Debian or just python default tags.

Note: The names jessie, stretch, buster and bullseye are not random but rather the actual names of the Debian Linux versions released at that time. Thus, if we had a new Debian linux version named, strange-happenings, we would consequently get a python image tagged: python3.9.16-strange-happenings.

Of the mentioned, the safest bet is on using the default image. In this case, python:3.9.16. We use alpine when we want to have the bare minimum setup and are willing to install only the required packages manually.

Remember, however, that alpine-based images will have challenges in terms of libc dependencies ( musl-based vs not glibc). For reference, these are common C libraries that you might find in data science packages like pandas, scipy and so forth.

Let’s take a look at how these images might compare by pulling and doing an image size comparison.

From your terminal, pull the images, replacing the tags where needed.

docker pull python:3.9.16
docker pull python:3.9.16-slim
docker pull python:3.9.16-alpine
# add --quiet flag to keep the logs out of your screen or at a minimal
docker pull --quiet python: 3.9.6-slim-bullseye

To compare their sizes:

docker images

A great thread that you might be interested in is this: Super small images based on alpine linux. To cap a comment that stood out for me:

As another example, a co-worker recently was working with some (out-of-tree) gstreamer plugins, and the most convenient way to do so was with a docker image in which all the major gstreamer dependencies, the latest version of gstreamer, and the out-of-tree plugins were built from source. The offered image was over 10GB and 30 layers, took quite a while to download, and a surprising number of seconds to run. With just a few tweaks it was reduced to 1.1GB and a handful of layers which runs in less than a second. It was just a total lack of care for efficiency that made it 10x less efficient in every way, enough to actually reduce developer productivity … ploxiln on Dec 23, 2015

Remember; choose the right image based on your needs.

Use multi-stage docker files

The concept of multi-stage docker files entails breaking your project build process into more than one. In this case, if building a project with say golang, the build files and the dependencies needed for them would be on one image while the compiled file would be on another, ready to be run.

Likewise, if building a python project, the installation of the dependencies would be on one image while the actual production-ready application would be on another. The result would look something like this.

You can read more about this in the piece: Multi-stage docker files vs builder patterns

Thought Digest: Did you know that when a Docker image is first built or used, Docker retains components that haven't changed, so it doesn't have to rebuild everything from scratch again? That’s called caching.

Only install the required dependencies

Alluding to installing the image that closely resembles the stack you are using, having only the required dependencies is key. For instance, using an alpine-based image and installing git, node js, and java jdk to a program that needs none of them is baggage brought forward.

Have a .dockerignore file

Your gitignore file knows to keep node_modules out of version control. Your docker image, however, does not. We keep any items that the image might add during installation later on or items that are not needed for the docker image to function.

Running your docker container commands as a non-root user

Every time we create a Dockerfile and build its image, the base-image file places you in the place of the root user. That is, you know what you are doing. This leaves you vulnerable to attackers(users/processes) that can gain access to your host system. They can, consequently access the project files, copy or add more scripts and so forth.

In the same way, you are not expected to use a VPS as root nor do I expect to see the almighty # on your terminal(an indication that you are navigating as root).

To avert this, you create a non-root user in your docker image by adding a USER directive to your Dockerfile.

For example, to create a non-root user called appuser101:

FROM python:3.9.16
# Create user
RUN useradd -m appuser101
# Set user as default
USER appuser101

To run commands in the container as this new user, you can use the docker exec command as below:

docker exec -u appuser101 -it <container_name> <command>

Where we stand

So far, I have significantly reduced the sizes of my images and their build times as well as improved their maintainability. My billing has also changed. I get the room to have one or more projects for the price of one.

Is there more? I think so. I am pushing this to see how it goes but I will come back with more along this trail.

A word to the wise:

Caching is your friend. Caching is here to stay.

Consider, for one moment, that you were building a project that required shipping to a different platform from the one we were using. Better yet, we want it to act exactly the same way as it does locally.

Sidenote: I have had my fair share of deployments during which some library was perfectly fine in development only to discover ubuntu 22.0 or some other version did not support it anymore.

For this illustration, we shall take the program, marvinkweyu/minigrep, a command-line tool that lets you search via text files; a mimic of the popular UNIX grep command-line utility.

How would you containerize this?

What method would be the most efficient given the resources?

Let us understand this.

(Clone the minigrep command-line program to your local file system if you have not done so already)

A single dockerfile

Create a dockerfile at the root of the project directory that looks like below:

FROM rust:latest
WORKDIR /myawesomeapp
# copy the source file onto the /app directory
COPY . .
# compile into executable
RUN cargo build
# run the project
ENTRYPOINT ["./target/debug/minigrep", "to" , "poem.txt"]

Above, we tell minigrep to look for characters that match the consecutive to.

Finally, build your image as the first of its kind by running this in the same root directory:

docker image build -t minigrep:v1 .

What is the possible size of this build?

docker images minigrep

REPOSITORY                                TAG                   IMAGE ID       CREATED              SIZE
minigrep                                  v1                    c4c56d10c4b2   About a minute ago   1.41GB

You can play around with a container of the image to see what minigrep finds:

docker container run minigrep:v1

Observe: minigrep:v1

Using the builder pattern

The v2 to reproducible code

Earlier, we build our image using a single dockerfile. Along with the executable, it had non-essential build files. That is, files it did not need at run-time to actually work.

We could change this by creating two images instead. the first would contain all the necessary files we needed before the build and act as the image build. The second would contain only what was needed for the package to run.

The builder image would look similar to the normal build:

FROM rust:latest
WORKDIR /myawesomeapp
# copy the source file onto the /app directory
COPY . .
# compile into executable
RUN cargo build
# run the project
ENTRYPOINT ["./target/debug/minigrep", "to" , "poem.txt"]

In comes the runtime image:

FROM rust:latest
WORKDIR /myawesomeapp
# copy the source file onto the /app directory
COPY . .
# compile into executable
RUN cargo build
# run the project
ENTRYPOINT ["./target/debug/minigrep", "to" , "poem.txt"]

We minimize our image even further by using an alpine image, 5MB in size, rather than the 800MB default rust image.

Remember, we have already compiled our code and all that it requires.

To manage assets from one image to the other, we need to copy the compiled code onto the runtime image. A transfer of files between two images if you may.

To achieve this, we would navigate into a container of the build image, copy the compiled code onto the current working directory and let the runtime image dockerfile copy the required file(s) into itself as it creates the final image.

The command would look something like this:

# notice the path of the main built app. It is the same as where we copied our file in our working directory
docker container cp minigrepv:/myawesomeapp/main .

Once this is done, we would then delete the initial build image as it is no longer required, giving us that one single docker image for spawning containers.

To shorten this and the cycle of commands you would have to run, create a develop.sh file with the following contents: (This assumes you have bash on your system)

# !/bin/bash
docker image build -t minigrep-build-image -f Dockerfile.build .
# Create container from the build Docker image
docker container create --name minigrepv2-build-container minigrep-build-image

# Copy build items from build container to the local filesystem 
docker container cp minigrepv2-build-container:/myawesomeapp/target/debug/minigrep .
docker container cp minigrepv2-build-container:/myawesomeapp/poem.txt .
# Build the runtime Docker image 
docker image build -t minigrep-runtime-image . 
# Remove the build Docker container
docker container rm -f minigrepv2-build-container 

rm minigrep

Our directory would look as below:

src
target
.gitignore
poem.txt
Cargo.lock
Cargo.toml
README.md
Dockerfile.build
Dockerfile
develop.sh

Run the develop.sh file to build the images:

./develop.sh

This will build the two images, copy the compiled code from one container to the other and delete the build container.

Let’s see how our images differ: docker images ls

REPOSITORY       TAG                  IMAGE ID       CREATED              SIZE
minigrep-runtime-image                    latest                0ef6f1b87748   14 seconds ago       11.8MB
minigrep-build-image                      latest                66d4a2a395c3   16 seconds ago       1.41GB

Walah! A whopping difference of over 1000MB!

In our use of the builder pattern, we have managed to save up to 1GB of file storage in our final image size. The trade-off was that we had to create two docker files, build the first, copy the final file to the local directory, and only the n move this t the final image. Perhaps there is a shorter way.

To observe: minigrep: v2

Multi-stage dockerfiles

Much like the builder pattern, a multi-stage dockerfile will create intermediate images and have our final image of equivalent size. The pro to this methodology, however, is the use of a single Dockerfile. Behold, say goodbye to the bash script and the need to have files copied to the local filesystem before transferring to the final image.

# give the initial image a name
FROM rust:latest as build-image
# Set the working directory 
WORKDIR /myawesomeapp
# Copy source file from current directory to container 
COPY . .
# Build the application
RUN cargo build
#* build the final image 
# give this a name
FROM alpine:latest as runtime-image
# Set the working directory 
WORKDIR /myawesomeapp
# copy the compiled code from the initial image onto the runtime image
COPY --from=build-image /myawesomeapp/target/debug/minigrep .
COPY --from=build-image /myawesomeapp/poem.txt .
# Run the application 
ENTRYPOINT ["./minigrep", "to" , "poem.txt"]

Go ahead and inspect your image.

REPOSITORY                                TAG                   IMAGE ID       CREATED              SIZE
minigrep                                  v3                    2c4927420843   19 seconds ago       11.8MB

What a difference.

What does this look like in other projects? Can we build dependencies separately from the final image in interpreted languages?

Hint: TambuaShamba

Conclusion

We could create a simple dockerfile for smaller products and applications, or go ahead and use multi-stage docker files. The choice remains to the engineer that #builds. Along the way, what you intend to #create , your capacity, your development experience and what you are willing to take differently all sum up to your ideal. Iterate.

Introduction

The Covid-19 epidemic revealed the large gaps in our social systems, infrastructure, and outlook on life. We lost jobs, we lost loved ones, and we had to prioritize basic needs over impulse purchases; understanding that that extra broccoli is not so bad after all.

Soil organic carbon is the measure of carbon stored in soil organic matter. It plays an essential role in determining the quality of soil and the productivity of a farm. Poor soil organic content can lead to soil erosion, reduced crop yields, and decreased water infiltration. Therefore, it is important for farmers to track the soil organic content of their farms and identify areas for improvement.

What if there was a way to map this data out? What if you could get a glimpse of which farms were performing better based on their soil organic carbon content?

Tambua Shamba

I developed Tambua Shamba, an analytical dashboard view of farms based on their soil organic content. This dashboard will enable users to view the best and worst farms based on their content measures, as well as to identify trends in soil measurements over time.

The Build

I had to decide which tools to use to build this project. I could have used Phoenix LiveView, an elixir framework whose concurrency would have been beneficial if the project had multiple users at once. On the other hand, I could have gone with a REST API approach and opened the product up to different clients around the globe. I chose the former.

To maintain this project, I needed to answer a few questions:

1. How would farms be added?

2. Who would this product speak to?

3. Would it be as easy for a farmer to understand as it would be for a soil analyst?

4. How would anyone across the project trace their steps?

5. How would the data be rendered?

From a system design perspective, I wanted to record the actual files of the sources of data. I also needed a mechanism to track which farms belonged to which instance of a data update, as well as which farms existed in that specific update. Additionally, I needed to know who did the update and when it was done. All this would eventually need to be rendered on a map, plotted as a multi-polygon.

It’s never what it seems — The Challenges

The main challenge I encountered was ensuring data integrity. We could not have random file formats uploaded. For example, a video file recording should not be treated as a source of truth for a quantifiable entity. Additionally, if there are specific identifiers the farms require, then all farms must meet this spec.

Moreover, since data can be collected in multiple formats, we need to account for when a scientist uses polygons to demarcate farms on one occasion and a multipolygon on the next.

To render this information, we need to consider what happens when we have two farms to record data on. Would we get the same result if we queried for the best and worst farms? What would happen if we had two files with the exact same farms? What if we forgot to add a field on one of them?

Hitting the nail on the wall — Navigating product edge-cases

To address these challenges, I overwrote the ModelViewSet create method to read and check the file before saving. I let the API accept CSV files as a data source as they are much easier to work with when inputting data into a database table. To ensure I did not get a duplicate list of best and worst farms, I filtered through the already returned list of best farms; Warning: By no means is this scalable.

I also color-coded the farms to show what kind of farms we had and added popups to give more detail about specific farms. Conventionally, the project is split into modules and lazy loaded to reduce the bundle size and increase its speed.

The Horizon

For the next release, I plan to add features such as asynchronous file management and a standard user management system to the dashboard. Additionally, I will look into providing more user-friendly features such as searching for farms by name and pinpointing their location on the map and updating information on farms.

This should let users have a comprehensive understanding of the soil organic content in their farms, as well as allow them to compare their farms to others in the same region.

Indulge: Can we include a data analytics engine that will enable users to track trends in their soil measurements over time?

For reference, the project can be found on marvinkweyu/tambua-shamba

Conclusion

As I continue to build Tambua Shamba and explore the relationship between soil and food security, I invite you to join in the conversation. #build

Originally published on marvinkweyu/projects/tambua-shamba

In the packaging of one of my projects, I came across a scenario that caught me off guard: versioning. Specifically, my challenge came with the choice of what versioning system I would use or how I would combine multiple versioning techniques if the need arose.

Working with multiple teams and in different demographics, packaging has always been different. For one, we have the SemVer evangelists, the CalVer antagonists(e.g Ubuntu packaging), the build version engineers and my just discovered git commit hashing brooders. Lo! How The choices go on.

I want to share with you the two that have piqued my interest so far and what options I use to curate experiences for the web.

A Case for Semantic Versioning

Semantic Versioning or simply SemVer, is a system of packaging software in which major, minor and bug fixes or enhancements are noted.

MAJOR**.MINOR.**FIXES

Examples:

Django 4.1.5

ColorDetect v1.6.0

fancy-git v7.1.9

Here, we would bump each version according to what purpose it serves. With this in mind, one of the major advantages that comes with SemVer is human readability. Engineers and users alike can clearly tell what came after the other. User A, will see that your product, ‘Legendary Lemons’ has a version, v3.0.0 while they have v1.2.0. Subsequently, they can tell that what is on your site is of a higher value if we were counting from 1 to 10. The same goes for your team members, albeit with a little more technicality.

Git commit Hashing strikes back

When it comes to using git commit hashing for software versioning, Fred Simon’s 7 Deadly sins of Versioning comes to mind.

Scenario

What if I needed to consider a component that had no impact on either functionality, performance or quality? Would I release v1.4.5 due to a spelling mistake in the README? Clearly, this is a developer-centred update in which a config file on the main branch needs to be updated for the project to run on the local environment. Should this be transferred to the user? What is the extent of items that should be? Ahh... the conundrum. Release management can be messy.

However, the challenge comes with interpretation. Surely 5517829933722 should mean something more than an array of digits. Is it higher than the others? Is it lower than the package I have?

Release management can be messy.

The pill

I packaged The Urbanlibrary as a docker container. We have dev-centered versioning and what is made known to the public. A mix and match if you may. Taking what works, modifying what I needed and sharing the much-needed value.

Example:

Consider the Windows operating system packaging. We have Windows 11, Windows 10 and so forth. In between these, we constantly have updates for the software. Patch upon patch each modifying the previous build.

I might as well merge these two schemas. Ultimately, what matters is that:

1. The product has clear documentation

2. I can see what happens across multiple builds

3. Automation sits at the center of deployment and changelogs happen on the fly as much as they can (I am not going to do an SSH every single time ).

Conclusion

What I have come to understand is that just because something is well-known, does not always mean it is the right way to do it. There is a plethora of choices on the table and I welcome thoughts and conversations on the same. I am keen to discover what your team or you personally use or would use to version your project. Reach out, touch base and let’s #build.

PS: For some anecdote, I stumbled on Julie’s video on how she chose to version her projects in general.

Image credits:

People illustrations by Storyset
Monkeyuser

It’s been a week since we had our first hackathon here at Value Chain Factory. A week since we had a buzz of activity with young engineers teeming with ideas on how to create the next solution that would address Africa’s most daunting challenges.

Over the weekend, my team had the pleasure of hosting one of the most exciting hackathons we have had this quarter. Our aim was to call upon innovators in the technical space to share and build upon their ideas on agriculture, health and transport; matters that are paramount in a mineral and nutrient-rich continent.

Of the possible startups brought forth, I thought I would mention a few notable ones. That is, that in conjunction with my team and the participants, would provide more value and be impactful in their own right.

Lifeline medicare

To assist emergency responders, team #lifelinemedicare created a solution that stores key details of their patients. Succinctly, the determining whether a particular treatment would be viable for an incapacitated patient.

SmartBins

With an ever-growing population, currently, at 8 billion, our primary approach to waste disposal and management has been to bury it. Practically equivalent to burying our heads in the sand and letting the tides of time do its thing. To address this, the smartbins startup would provide waste management services. Taking up this role from collection, recycling and selling of compost manure back to the local farmer.

The Farmer-market bridge

To complete the cycle, the market bridge team would open up the marketplace to expose farmers to their immediate sellers. In a way, making the farmer aware of their produce’s movement through to the consumer.

What stood out was how these builders were willing to take a leap in validating their enterprises. An emerging trend amongst this lot of African engineers, was their embrace of new technology. From the use of NFC tags in medicine, the leverage of blockchain technology for transparent transactions to the use of geo-sensors in garbage collection. They took a step beyond ideation to provide the next possible solution.

Remember:

More important than starting a particular startup, is getting to meet a number of potential cofounders. From there, instead of working on what you intend to make and then finding the audience, work in the reverse by thinking of the public. Ideate.

You should only start a startup if you feel compelled by a certain problem and you think starting a startup is the only way to solve it. The passion should come first, and the startup comes second. - Sam Altman

Thus, to you the reader: If you knew success was a certainty, what would you do?

PS: Yes. My team came second. Because winning is what we do. Hi Wachira.

During this time, I got to interact with the best in their fields; lawyers, psychologists, mechanical engineers , security analysts, fellow startup founders and so forth. Each of us was brought together to spark sustainable solutions that would impact our communities in one way or another.

Truth be told, it was a nerve-wracking 54-hour brain-storming session.

Suffice it to say, from this, we brought to life a number of products:

1. ABCs of mental Health (A place to get mental to the corporate workforce)
2. Knock knock(Accessible emergency services to those with hearing disabilities.)
3. Iko Network (A solution that provides easy registration to visitors within a premises)
4. Vamva (A #fintech solution solving the barrier-to-growth problems faced by ride-hailing drivers and other mobility )
5. Caes International (An organisation that gives the local boda-boda rider rechargeable batteries.)
6. Instruct Kenya - An easy-to-access legal advisory platform
7. ShopOkoa - money management services to university students

One point that resonated with me, however, was the fact that technology should not be the #1 go-to to solving the that Africa faces.

Technology should not be the #1 go-to to solving the challenges that Africa faces.

Rather than shoving technology down the throat of our future product users, our question should bring us back to one single question:

How will this product affect the local citizen(mama mboga)?

Along with the network built and the products created, I got #first-hand information as to the success points and pitfalls of pushing a successful product through the African market. How do you identify the different types of users, and tell whether that specific cluster needs what you have to offer?

Do the people want this? Talk with the people. Who are your users? How are they different? What are their needs?

The best feedback you're going to get on your product is in the 3 seconds after you tell them the price.

Ultimately, the baseline sits with where you intend to take your product and what solution it intends to tackle. Removing the technical jargon and complexity, we ask:

'Is this the most suitable way to solve this problem?'

Originally published at: https://www.marvinkweyu.net/indulge/the_place_of_technology_in_africa

This guide acts as a follow-up to the talk Binary vs Interpolation Search.

Time complexity and search algorithms. A walk through their definition, purpose, use cases and constraints. A search algorithm is a technique used to locate an item in a certain data structure.

Introduction

A search algorithm is a technique used to locate an item in a certain data structure.

Before we begin, you should have an understanding of the below:

• Arrays
• Iteration and recursion

Let's give an overview of time complexity and what it entails.

Time Complexity

A way to show how the runtime of a function increases.

Broadly speaking, the time complexity of a program is grouped into three:

• Linear time
• Constant time
• Quadratic time

Linear Time O(n)

The processing time of a program will increase as the size of the input increases.

Example We can create a program that processes a list of items stored in an array. A case example can be a program to process locally stored files.

Constant Time O(1)

Here, no matter what input we parse to our program, we get the results in the same time frame. Hence, a process taking 10_000_000 items returns in the same time as the one taking in 7 items.

Quadratic time O(n2)

Where n2 is n squared. As the name suggests, a program will return results in a time period that resembles a quadratic curve. This time is common with multi-dimensional arrays/lists of lists, since:

• We loop over the list(say A) to get individual lists
• We loop over a specific list(say B) in this list to get the item and perform an operation on it.

For reference, the equation:

a + bx + c

Example: A program to calculate the product of integers in an array.

int array_products(int[] a){
    int product = 0; // O(1)
    for(int number: a){
       product = product * number; // O(1)
   }
return product; // O(1)
}

We can get an overview of how long a function will take, based on how large of an array we have in this particular function/ method.

In our case, since we can get the time complexity of this function as below:

O(1) + n * O(1) + O(1)

Since we are adding a value to the summation x number of times, we have O(1) happening for each of those occurrences. Thus n O(1)*.

Since a constant + another constant is still a constant.

O(1) + n * O(1)

Get the fastest-growing term from the equation, i.e:

n * O(1)

Then remove the coefficient to get O(n) (linear time i.e, the function takes longer to complete the larger the array)

Binary Search

Given a sorted array, binary search locates an item in question, using the 'divide and conquor'

Time complexity: O(log n)

Binary Search pseudocode

1. Get the first and last item of the sorted array.
2. Add the first and last item indexes and divide by 2
3. Get the element in the middle from the step above i.e middle
4. If the value at the middle index is equal to the searched item, return the index
5. If the value at the index is greater than the searched item, set the middle item to be the new high and discard the right
6. If the value at the index is less than searched item, make the item the new low and discard the left side
7. Repeat the process until the index is obtained
8. Return -1 if the item is absent

Visual representation

Implementation of binary search

int search( int[] a, target){
    int arrayLength = a.length;
    int left = 0;
    int right = a.length;

    while(left <= right){ 
       int middle = (left + right) / 2;

       if(a[middle] == target){ 
          return middle;
        }

       if(target < a[middle]){
             right = middle - 1;
        }else{
         left = middle + 1;
    }

  return -1;
}

Applications of Binary Search

1. Database indexing

Within your database of choice, a binary tree will be used where it will precompute the middle point on each step, until reaching the single item. More details of database indexing can be found here

1. 3D games and applications.

Binary space partitioning occurs as space is divided into a tree structure and a binary search is used to retrieve which subdivisions to display according to a 3D position and camera.

1. Git debugging with git bisect

2. Autocomplete search

Consider a user typing in a URL on a browser. Given a history of searches, instead of going through each one and comparing, a binary search is used to perform a faster lookup across the history.

1. Searching for prefixes

Instead of an array of integers, you can have an array of strings where you intend to search for an item that starts with a certain string or character sequence.

1. Going through a dictionary to find a word

2. Git cherry picking

A clever example of where this is used is by Sarv Shakti, who used it to identify what commit broke the application

Example: Consider a scenario where you have a list of maps storing student records in your classroom. Given a name, or phone number, which algorithm would you use to search through this? What if the records got to 10_000 or more? How would this grow?

Interpolation Search

This is an improvement over binary search. Gets an element from where it is more likely to exist.

Time complexity: O(log log N)

Constraints of interpolation search

1. The array should be sorted
2. The array should be uniformly distributed.

Note: Uniform distribution in an array implies that the difference between subsequent elements should be relatively equal. In this case, an array [10, 30, 300000, 23_000_000] would fail to meet this condition.

Example: Searching for the word 'zelda' in a dictionary of words from letters 'a' to 'z'.

Instead of starting from the middle item in the array, we go closer to the end of the array.

Time complexity and its relation to interpolation search

A walk through the equations

y = m*x + c

arr[index] = m*index + c

where: index: the position of the element in the array arr[index]: the value at this index in the array

arr[low] = m*low + c —– (1)

arr[high] = m*high + c —– (2)

Subtract equation (1) from (2), and we get

arr[high] – arr[low] = m * (high – low)

m = (arr[high] – arr[low]) / (high – low) —– (3)

Say we are searching for an element, K in this array.

arr[index] = m*index + c

Then replacing target in the above equation, we get

target = m*index + c —– (4)

target – arr[low] = m * (index – low)

index – low = (target – arr[low]) / m

index = low + (target – arr[low]) / m

index = low + (target - arr[low])*(high - low) / (arr[high] - arr[low])

Example

Array to search through:

[32, 35, 37, 39, 42, 44, 46, 48]

We will search for target = 42 in the above array.

The size of the array is 8.

low = 0,

high = 7,

arr[low] = 32,

arr[high] = 48

index = 0 + (42 - 32) * (7 - 0) / (48 - 22) = 2.6923 ~ 2

arr[2] = 37 < 42

Remember to floor the value of the index

Interpolation search pseudocode

1. Calculate the Index using the interpolation index formula.
2. If the target is smaller than the item at that index, search in the lower sub-list. Calculate the index for the left sub-array. Let low = index - 1
3. If the target is greater than the item at that index, search in higher sub-list. Calculate the position of the right sub-array by assigning low = index + 1.
4. If the target is a match, return the index of the item from the array and exit.
5. If it is not a match, probe position.
6. Repeat until the match or array is eventually 0

Implementation of Interpolation search

int search(int[] array, int target) {
    int n = array.length;
    int low = 0;
    int high = n - 1;

    while (low <= high) {

        int index = low + ((target - array[low]) * (high - low)) / (array[high] - array[low]);

        if (array[index] < target) {
            low = index + 1;
        } else if (array[index] > target) {
            high = index - 1;
        } else {

            return index;
        }
    }

    return -1;
}

Applications of interpolation search

Interpolation search can be used in the same areas where binary search is used. The only caveat is that this data must be uniformly distributed. A failure to meet this condition will give you a time complexity of O(n).

Benchmark

Comparatively, given uniformly distributed data, this difference might occur as below:

Resources and questions

These are locations to practice your data structures along with links to various challenges that may hold the above mentioned concepts or pivot them accordingly.

• Leetcode
• Exercism
• Hackerrank
• Algoexpert

• The two sum problem
• How many numbers are smaller than the current number
• Middle of a linked list
• Container with most water
• OpenSource collection - TheAlgorithmCore

Other types of search algorithms

• Linear search / sequential search
• Breadth-first search
• Depth-first search
• Jump search
• Exponential search
• Fibonacci Search

Conclusion

There are more search algorithms out there. This is by no means a comprehensive list, and by no means are you expected to know all of them. A single algorithm can be used as the basis for other searches. With that said, there is no silver bullet. You choose what works based on the constraints at hand.

References:

Mohammed, A. S., Amrahov, Ş. E., & Çelebi, F. V. (2021). Interpolated binary search: An efficient hybrid search algorithm on ordered datasets. Engineering Science and Technology, an International Journal, 24(5), 1072–1079. https://doi.org/10.1016/j.jestch.2021.02.009

Manolopoulos, Y., Theodoridis, Y., & Tsotras, V. (2012). Advanced Database Indexing. In Google Books. Springer Science & Business Media. https://books.google.com/books?hl=en&lr=&id=pD3tBwAAQBAJ&oi=fnd&pg=PP13&dq=+database+indexing+using+binary+search+&ots=HecPStLWUK&sig=9A5wLiIHxg1ygaiZ-UxiSob7-Qs

Originally published at https://www.marvinkweyu.net/talks/binary_vs_interpolation_search

I realized I had to completely relearn these systems because of the cloud.
~Satya Nadella

There are a number of considerations that you as a developer, architect or engineering team have to make as your user base grows. Going from products with 10 to say 30 or a million and so forth has its considerations. This, of course, depends on the user base we are talking about.

Remember astute reader, a customer can be an organisation with a plethora of other members inside it, or just as well be a single user called Jack.

Explore with me

Imagine thousands, half a million and so forth people performing a search query against a server. If those queries land on your end, would it break? Would it be able to hold?

We should be able to handle more data, more concurrent connections and higher interaction rates.

A note on interaction rate:

How often a user changes the interface on a blog application vs a game.

Before we begin, it is important to note that setting up our product for scalability should be as easy and cheap to grow 10x as it should at 0.2x as is common with startups whose needs change rapidly over a short period of time.

Foremost, there are two types of scaling we need to observe.

a) Vertical scalability

b) Horizontal scalability

Both of the aforementioned have their pros and cons - each with a use case which would be deemed fit. Let's narrow this down.

Example :

For solo and relatively small ventures, you might opt for a VPS(virtual private server) which is essentially a physical machine holding multiple 'servers', yours included, each with its own OS and so forth. Once at a certain cap, you might see the need to increase resources and hence, the incremental hunt; more RAM, more storage, more CPU and so forth. almost everything gets more.

The example above, is one implementing vertical scaling, where, we upgrade the hardware and or the network throughput. In short, you do not need to modify your application architecture.

Vertical Scalability

Broken to bits, we resolve to a number of techniques.

• Adding more drives in RAID arrays and thus distributing our reads and writes during database interaction.
• Switching to solid-state drives from hard drives
• Increasing the RAM
• Adding more cores to the server to reduce the context switching.
• Upgrading network adapters, especially for systems reliant on media file processing.

There are a number of drawbacks, however. For instance, the change to a solid-state drive might not show a significant difference if working with MySQL which uses its own data structures for sequential disk operations or Cassandra. At a certain point, lock contention is bound to catch up and here, more CPU does little to any good. Vertical scaling also comes with a steep financial cost as you add more and more given the cash input doubles or triples as a unit of resource is added.

Example:

A RAM chip will double in cost as it grows in size. That is, x GB is $z , 2x GB is $2z and so forth. A reality check hits us once we get to a stage, say 128GB where the rules are not quite the same.

Horizontal Scalability

The alternative and preferred method - horizontal scaling. While it may come at a steep cost initially, it does get significantly cheaper as your enterprise grows to reach more users. Let's see how.

Caching

Use of content delivery networks (CDN)

To manage our assets, we save the hassle of having to perform the read and write operations on our own servers. Ever wondered why we have CDNs for most packages(Bootstrap, Material etc)?

Example:

We have our servers for, say UrbanDesk in South Africa. For access, an audacious Sergei (hint - thy knowest thyself) an enthusiastic early adopter from Germany, will hit the CDN(which are just data centres on their own). The CDN, acknowledging it lacks this data - images in our case, will hit our server for the resources it needs (CSS, images, javascript) and serve it to him while storing the same data and resources for further calls to the site. In this way, the next time, the call does not have to go a whole continent away.

Services like Cloudflare, Cloudinary or GoogleCloud CDN should ring a bell.

We have our static assets sorted out, but what about our data?

Edge Cache Servers

I've expounded on the data server to include functional partitioning, yet , another enhancement to our scaling needs. The concept behind it is that our project should be decomposed into individual actions those components need to perform - to group components of our application based on what they do.

For instance, we have sections that address this page you are currently reading (thegreencodes.com), then others that speak to different types of users, say, those interested in merch (merch.thegreencodes.com), or the admin (admin.thegreencodes.com). Each of these sections has been compartmentalized.

We might also move the database to its own server and update our configs to point to it, hence to each its own, that is, the server the application runs and the server the database runs are different but still communicating.

You will notice that I also added the GeoDNS. It works very similarly to an ordinary DNS, the only difference being that it is ' geo-tagged '.

The DNS

A Domain Name Server(DNS) is a way to identify your server's IP address on the web. In this way, if your server has an IP of say 127.0.0.1, it is displayed as example.com, to whoever else is looking at it.

Back to GeoDNS

Essentially, when you attempt to connect to thegreencodes.com, the server IP will be resolved to the data centre closest to you.

Elaborately, once your product has a certain cap of users , serving all across the globe, having a single server in a single location might degrade the experience these very users get.

Example

Google has multiple servers. While you might connect to its root domain from Australia and person B from Germany, these might not be served from the same server. The GeoDNS works to resolve the IP addresses of these servers/ data centers based on the location of the end user.

You will access the same data, only without knowledge of what IP the domain you are visiting is currently resolving to.

What makes horizontal scaling stand out is the basis on which it stands. At its core, horizontal scaling lets you deploy and maintain your project on several medium or small servers. A single server does not necessarily have to be extensive with vast amounts of resources - we split the load between different servers.

Of note, is that we need to consider whether our project is stateful or stateless. What this means is to identify whether the split needs data and whether that data needs to be synchronised.

Managing data in stateful horizontally scaled products

Behold, the CAP theorem and the ability to identify whether we need more reading or more writing of our database.

If we indeed have data that needs management, we might opt for three techniques.

a) Sharding by data b) Sharding by hash c) Data replication

On the one hand, we have sharding by data. In a case example similar to the one shown, we can have a database that holds data from those in Europe and another that holds data from those in America.

On the other hand, we can opt to use a consistent technique of hashing specific keys on our model and use those hashes to store data in different databases. For instance, hashing the profile_id and stating that data for users with the hash algorithm ABC will sit on this server and not the other.

The above are optimizations meant to prevent hitting the same database for all requests.

While not in-depth, I do intend to come back to this in future. So keep an eye up for this.

More than the type of scaling, we have to consider a number of techniques and practices to maintain.

As these platforms grow, the following have to be considered.

Testing

There is more to testing than unit tests. Larger systems eventually comprise of larger subsystems. How these systems work together (integration testing), whether the application is behaving according to the specification(functional testing), whether it can handle different loads (performance testing) and so forth.

Logging

The more the components, the more integration is needed and the more the system is prone to fail. The more servers we have, the higher the chances. We hope for the best, but prepare for the worst. Your team needs to know when a module went down, whether the data maintained its integrity, which modules are still up and so on.

Choose to go for the Crash-only approach; let the system always be ready for a crash and whenever it reboots, it should be able to continue without human interaction. An example of such a system is Netflix's Chaos Monkey which runs to kill random components of the system within working hours to test its reliability.

Conclusion

The practices above are intertwined. Functional partitioning would lead to the decomposition of the project, decomposition would lead to a 'maybe' use of microservices, this would bring a restructure of your team to smaller domain-driven teams, a change in how data is cached and managed and so forth. Build upon this. Go agile and iterate. This path is not linear, nor should it be treated as such.

Treat these principles as your northern star, but feel free to morph them based on the value they create for your business.

Understand

In the creation of sustainable software applications, we place the business logic at the centre. The inverse, you'll find, will get you great code, rather than solving the need to be addressed.

By placing technology first, we may get a great rails application, but it may not be a great pharmaceutical application ~ Robert C. Martin

Onto Architecture - the stuff we wish we'd gotten right at the start of our project.

Next, we have to get through fetching, displaying this information and manipulating it.

Let's use beloved axios to grab and interact with our API.

npm install --save axios

For our application, we'll use an already created database - TheGreenCodes data source. In the same manner of thinking as before, we create tests for the non-existent feature; listing to-do items.

We shall display uncategorised items on one page, all the incomplete items on another, and then follow and display those that have actually been completed on the last page.

As we are starting off, we ensure, just like before, that since we have nothing in our database, yet, we display an appropriate message. Among others, here are the questions we need to ask:

• Does it show an appropriate message if no item exists in the database yet?
• Does our application fetch to-do items if any?
• Does the app show any error message in case it cannot get to the server?
• Can I even create a task?

Bear with me on this one. In the spirit of TDD, we are going to write our application based on our tests. So from here on out, no UI until we have something complete. That is, after all, the whole point of testing; pushing code with the confidence it is not going to break.

Modify the test suit to watch for changes in our test files. That way, we do not have to run npm run test: unit after every modification.

Load your package.json and add the below:

...
"test:unit": "vue-cli-service test:unit",
"test:unit:watch": "vue-cli-service test:unit --watchAll", // line to add
...

From here, we open our console and run:

npm run test:unit:watch

Leave it open for now. You can always open it to see any changes that may occur. Let's fulfil our queries as we go.

We want our home page to display the 'empty database' message for an empty database. Let's test that.

To start, we do some modifications to our test suite, making it easier to work with as it grows.

import { BootstrapVue, BootstrapVueIcons } from 'bootstrap-vue'
import Vue from 'vue'


describe('Home.vue', () => {
// before any test case in this test suite, import and use bootstrap UI components 
  beforeEach(() => {
    Vue.use(BootstrapVue)
    Vue.use(BootstrapVueIcons)
  })
  it('Displays welcome message', () => {
    const wrapper = shallowMount(Home)
    const welcomeMessage = 'Welcome to TheGreenCodes awesome to-do list'
    expect(wrapper.text()).toMatch(welcomeMessage)
  })
)}

Along with the other previous imports, we import bootstrap UI components.

Remember:

Our test case is assuming a black box. We are testing this specific component. Nothing outside. So it does not know what we have. Without the use of the UI import, the test would surely work but will show a warning as to what b-container or b-card means. It doesn't know.

Let's test for a message in case there are no to-do items. We expect this to fail at the first instance as we have not implemented any feature to display to-do items.

Append the below code snippet to your suite:

  it('Displays a message when there are no todo items', () => {
    const wrapper = shallowMount(Home, {
      data () {
        return {
          todoItems: null
        }
      }
    })

    const titleFound = wrapper.find('h4')
    const emptyTodoListMessage = 'You have no existing todo!'
    expect(titleFound.text()).toMatch(emptyTodoListMessage)
  })

Reasoning

We mount the Home component and set a data property to hold the data from the API. We also grab the h4 tag, an element we expect to hold the message when there are no items on the to-do list.

We should also check that this 'You have no existing todo!' message does not display if there are indeed items on our to-do list.

  it('Does not display message when there are todo items', () => {
    const wrapper = shallowMount(Home, {
      data () {
        return {
          todoItems: [{ id: '1', title: 'new item', content: 'make awesome content' }]
        }
      }
    })
    expect(wrapper.find('h4').exists()).toBe(false)
  })

Run the tests, in case you have not done so yet, and take a look at the spectacular failure.

PS to self:

Stop logging 'madness' on Sentry servers.

Let's fix this.

<template>
  <div class="home">
    <h2>Welcome to TheGreenCodes awesome to-do list</h2>
    <div class="container">
      <b-card class="list-container">
        <h4 v-if="!todoItems">You have no existing todo!</h4>

      </b-card>
    </div>
  </div>
</template>

<script>
import axios from 'axios'
export default {
  name: 'Home',
  data () {
    return {
      todoItems: null
    }
  }
}
</script>

<style scoped>
.list-container{
    max-width: 170%;
}
</style>

As you may have noticed, we have redundant code. We create a new wrapper each time we have a test. We can have this centralized. Modify your test to look as below. We explain, as always what each means.

For brevity, we ignore other components of the test file.

...

describe("Home", () => {
  const build = () => {
    const wrapper = shallowMount(Home, {});
    return { wrapper };
  };
  beforeEach(() => {
    Vue.use(BootstrapVue);
    Vue.use(BootstrapVueIcons);

  });

...

At the very top of the description statement of the test, we shallowMount the Home component. That said, further tests can be used as below:

 it("Displays application title", () => {
    const { wrapper } = build();

    const welcomeMessage = "Welcome to TheGreenCodes awesome to-do list";
    expect(wrapper.text()).toMatch(welcomeMessage);
  });

  it("Displays a message when there are no todo items", () => {
    const { wrapper } = build({});

    wrapper.setData({
      todoItems: []
    });

    const titleFound = wrapper.find("h4");
    const emptyTodoListMessage = "You have no existing todo!";
    expect(titleFound.text()).toMatch(emptyTodoListMessage);
  });

With this refactor, we declare at the very start that it is the Home component in testing mode. We make the code cleaner and reduce having to remember that every time since we know that only the component's properties would be changing every time.

To set these individual properties, we can call the methods: setData, setMethods, setProps and so forth directly.

Because our component has no children, we navigated to shallowMount. However, if it did and we wanted to show whatever these children have in them as well, we could as well call mount and import it from the same path we got shallowMount.

What about the listing of all items?

// add imports
...
import axios from "axios";
import flushPromises from "flush-promises";


// more tests go here
...

it("Gets all todo items regardless of status", async () => {
    const { wrapper } = build();

    const expectedItems = {
      // ToDo: our items contain articles to read
      data: [
        {
          id: "1",
          title: "TheGreenCodes: Unit testing in Vue",
          content: "A guide to better predictable code.",
          complete: true
        },
        {
          id: "2",
          title: "TheGreenCodes: Tests must fail",
          content: "Building blocks to test driven development",
          complete: false
        },
        {
          id: "3",
          title: "TheGreenCodes: Tests must also pass",
          content: "This share",
          complete: false
        }
      ]
    };
    // on axios call, use expectedItems object
    jest.spyOn(axios, "get").mockResolvedValue(expectedItems);

    await wrapper.vm.getToDoItems();
    // make sure API request promises are complete before looking for articles to read
    await flushPromises();

    expect(axios.get).toHaveBeenCalledTimes(1);
    expect(axios.get).toHaveBeenCalledWith("todos/");

    expect(wrapper.vm.todoItems).toEqual(expectedItems.data);

    // Finally, we make sure we've rendered the content from the API.
    const todos = wrapper.findAll('[data-test="todo"]');
    expect(todos).toHaveLength(3);


    // we have articles now
    expect(wrapper.html().includes("You have no existing todo!")).toBe(false);
    expect(wrapper.html().includes("TheGreenCodes: Unit testing in Vue")).toBe(
      true
    );
  expect(
      wrapper.html().includes("Building blocks to test driven development")
    ).toBe(true);
  });

Through this case, we build an object we expect to get as a response. It returns a list of the series you are reading right now.

We say, ' Whenever you call on axios with a get request, return data that looks similar to this - '

Since our component calls the getToDoItems method, we can call it directly by using await wrapper.vm.getToDoItems(). This is almost similar to how we edited or set our data before.

Because the call is asynchronous, i.e, a request is made to the server and data sent back, we have to ensure our data replicates a wait time as well. For this, we call to flushPromises after adding it to the list of imports.

We are also ascertain that the get request is made only once and to the specific URL we need it to.

As axios will return the information wrapped in an inner data object, we want our data to get out of that to be compared directly to an array we intend to store. Thus the line:

expect(wrapper.vm.todoItems).toEqual(expectedItems.data);

For the last part, this should come out clearly.

If you have seen the light at the end, you might have seen the errors all snarly and already changed your Home component. For reference:

<template>
  <div class="home">
    <h2>Welcome to TheGreenCodes awesome to-do list</h2>
    <div class="container">
      <b-card class="list-container">
        <h4 v-if="todoItems.length < 1" class="empty-list">
          You have no existing todo!
        </h4>
        <div v-else>
          <b-list-group>
            <b-list-group-item
              v-for="(item, index) in todoItems"
              :key="index"
              data-test="todo"
            >
              <h5 class="d-flex justify-start">
                {{ item.title }}
              </h5>

              <article class="d-flex justify-start">
                {{ item.content }}
              </article>
            </b-list-group-item>
          </b-list-group>
        </div>
      </b-card>
    </div>
  </div>
</template>

<script>
import axios from "axios";
export default {
  name: "Home",
  data() {
    return {
      todoItems: [],
      errorFound: null,
    };
  },
  created() {
    this.getToDoItems();
  },
  methods: {
    getToDoItems() {
      axios
        .get("todos/")
        .then((res) => {
          this.todoItems = res.data;
        })
        .catch((error) => {
          this.errorFound = error;
        });
    },
  },
};
</script>

<style scoped>
.list-container {
  max-width: 170%;
}
</style>

This change should make our current tests pass.

Zoom in on this line:

...
<b-list-group-item
              v-for="(item, index) in todoItems"
              :key="index"
              data-test="todo" 
            >
...

To be specific data-test="todo".

This is a property added to the list of items for the articles yet to be read. This helps us call:

 const todos = wrapper.findAll('[data-test="todo"]');

You can equate this to a CSS class that is relevant to writing unit tests. We all the items in one swoop and check their length.

We can go on with the tests and features but this is the baseline;

• Write your failing tests
• Make the tests pass
• Refactor the code.

Working iteratively through this , enables not just error-free code, but also prevents us from writing extra redundant code.

Over the lifetime of the repository,awesome-to-do, we shall iterate through this process and come up with a complete application. I invite both you and your counterpart in crime, to join in and build better software together.

Alas! Let the OpenSource contributions begin.

Regards,

Marvin K.

PS: Picture creds: monkeyuser.com

This here goes out to all those who have reached out or are yet to, with regards to articles shared, both new and old.

To you, the reader that you are and builder that keeps at it.

I took a moment, a beat, to reflect on why I got started. The reason as to why the wheels turned to this particular alley, consequently bringing with it an array of an audience.

This here outlines an engineer's path to a targetted mastery. A nudge towards paths, journeys and challenges he encounters on his quest; a story by all means.

For the frequents, 'welcome back ' and to the visitors 'we are glad you made it'.

Over the last three years or so, I have journeyed with engineers across the board towards writing pages, useful to both us and those around us. It has been a tremendous journey, down to its hours of reading and research to beckon what I thought best for the audience at the time. Three years in, and here we are.

Series that have sparked conversation or passionate debates and those once-in-a-while articles that open up doors to collaboration and open source contribution. Hurrah!

A heads up to all those unexpecting eyes and ears, and in the words of those before me:

I refuse to be nothing but a full-time blogger.

As I take the time to pass this along, I open up the box of what is to come through this next phase; the builder's era.

With this, comes a voyage through which I shall not only share the nuggets of my findings but build around those same principles and guidelines.

For the start of this, however, I share with you Project Urbanlibrary to lead you into the world of content aggregation.

Along the way, I share routines and practices that have helped and still help to keep me sane in both mind and body. Keep healthy at all times and make time for your wellness, lest you are forced to make time for your illness - balanced, as all things should be.

Note to a future self and anyone else reading this:

Good work isn't created in a vacuum and the experience of art is a two-way street; incomplete without feedback. - Kleon Austin

I keep it short. I keep it sweet. Just like Thanksgiving.

So who are we? We are TheGreenCodes and at its heart, is a drive to build and talk about what we have built and continue to build. I urge both you and the friend you share this with, to join us and discover a place that could be.

Regards,

Marvin Kweyu

Broken down, a monolithic system is one that is deployed as a single unit of interconnected components.

Let's give an example.

We are creating a hotel management system for our local favourite spot in Django. Our application will hold, amongst other things, account management , rooms, rating and so forth. Break it into as many modular applications as you might come up with.

Essentially, a monolithic system is one in which multiple smaller applications live within the same server, share a file system and same database. Sound familiar? To easily identify if a project is monolithic, ask, can you scale an individual modular part of the application (memory or CPU) while leaving other apps unattended? If not, then this is a monolithic application.

Projects built like this are known to be simple in terms of development, easy to scale up to a certain cap, easy to deploy and be fast during the development phase. Fast, because all its components are together hence easy to collect and it would be written majorly in one language hence no need to learn another language/framework.

Have in mind , for instance, a java application using a jar file that contains all the apps logic for deployment.

A common architectural pattern , one we aim to discuss , that follows this , is the layered architectural pattern.

The Layered Architectural Pattern (n-tier architecture)

Governed by the Single responsibility Principle - a class or module has one and only one function - the layered approach leverages the separation of concerns. Broadly speaking, it splits software products into the presentation layer, business logic layer , persistence layer and the database layer.

These are modular sections of the application.

Each section/layer outlined has a specific function without needing to know what the other layer entails.

The presentation layer, for instance, deals with the display of information to the user. The business layer gets data from the persistence layer and processes it against application logic (say merge with another table or filter against a third party ) without the need to know anything about the user interface. Hence, you might as well change the presentation without changing the logic of what your project aims to do or change from an SQLite database to MySQL without your business logic requiring to know.

On a number of smaller applications, the business and persistent layer may be merged into one business layer. As observed, we get to have 3 layers in one case, four in another and so forth, hence the name n-tier, where n is the number of layers.

Requests flow from one layer to the next, say business logic to the persistence layer. This is a classic example of a closed layered system. That is, one in which, a lower-level layer cannot be accessed directly.

Example:

Not letting the user access the database without walking through the business logic. What is handled in this 'logic' you might ask?

For one, authentication and authorization. No one should just access our local hotel database. at the same time, an ordinary user should not be able to perform managerial actions within our platform. That is left to the manager and only the manager.

In its basic form, a layered architectural system works in a closed circle.

We may, however, need to have access to lower-level layers without the intermediaries - an open layered system.

Example:

Given the premise of the n-tier architecture, we have an extra layer, the services layer . Our project is now a 5-tier system. Despite having this layer, we might not use it as often. Hence, it does not serve our purpose for every request to go through it and have no change or mutation. What this layer would be adding to this specific request, is time; and time is of the essence. We have it, yes, but only use it when needed.

We bypass all that, and keep this layer open. This allows our request to access the level lower to it without having to go through it.

Note:

Within your project, aim to have at most one open layer. Having more will create interdependencies between these layers, and you lose the whole point of having the layered system in the first place.

Another variation of the n-tier architecture is the use of cached layers. Here, requests may be cached in between layers so that concurrent requests do not have to go all the way down. Plainly put, not every request made must hit the database (save the user some time).

Let's take our app idea from the top

A guest wants to make a reservation. They log onto our page and see that we have , on top of rooms, fun activities and games for our guests (the presentation layer). They go ahead to make a reservation. Our screen gets their request and sends it to our module within the business logic layer. Here, we want all available rooms and as a bonus, the activities available within the time frame of booking (notice the filtering of time, price and so forth). Now, call the room and activities data access object (dao) within the persistence layer, our ORM. These will, in turn, execute the queries necessary from the room and activity database table. Push this data back to the user and let them know they can have sushi on the house!

A little perspective

So, we've given a primer on monolith architectures using the n-tier approach. We have seen a case application that uses the monolith system and used it to get our new client a room. Our system works and our local hotel pays in kind. What happens when they get new outlets? More activities and more traffic? Will it scale?

Monoliths are good, and monoliths are great. How you use them depends on your needs. Use a monolith when you want simplicity in development, deployment and scaling , work in a relatively small team or are creating a prototype. That works 100%.

A monolith will, however, bite you as the system gets larger due to a number of reasons.

Framework /Language lock

The whole codebase is in one framework that might not be suited for future feature additions. For example: we wrote our application in one language that is good for speed but we now need a section of it to use image processing libraries. How suited is it for image processing?

Inefficient resource allocation

If one endpoint needs more time either due to IO operations or processing, other functions may be locked.

Scaling becomes difficult

1000 or so requests to the search functionality have been made. In monolithic architecture, a new instance of the running application might be created during load balancing to handle multiple accesses. Here, we have a whole new instance of the same application created, when all we needed was the search to grow .

Slowed development

Take it that we now have a chain of hotels across the town. Heck, across the country. We have added feature 1 and gotten to feature 300. We now want to change a variable name within an application in our project. We have not touched anywhere else but that section. The whole codebase, including the untouched , has to be compiled once more. We are still developing, remember that! That's 10 minutes for build and compilation that could have been better served elsewhere not to mention an IDE load time that puts us all to sleep.

Conclusion

So far, we have a birds-eye view of what we have been creating. A capture of what monoliths involve and what they do not - layer upon layer. Great start. Is there more? Stick around a while longer.