🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #130 - Sort Visualiser

This challenge is to build your own sorting algorithm visualiser, a tool that animates how common sorting algorithms operate on an array of values.

Sorting is one of the first topics every programmer meets, and it’s a topic many of us never properly see. We learn that quick sort is faster than bubble sort, that merge sort divides and conquers, that heap sort uses a clever tree structure, but the names blur together until you watch them work.

A visualiser turns each algorithm into a little dance: bars or numbers shuffling, settling, and finally clicking into order. Once you’ve watched insertion sort sweep across a nearly-sorted array, or seen quick sort partition a chaotic mess into neatly halved buckets, the differences between the algorithms become intuitive in a way no Big-O table can match. Building the visualiser yourself is even better, you’ll gain a deep, hands-on understanding of eight classic algorithms and pick up some practical skills around UI rendering, animation timing, and clean abstractions along the way.

Win 💰💰💰 - Join The Ready, Spec, Ship Hackathon I’m Running!

I’m pleased to be hosting the Ready, Spec, Ship Hackathon alongside Angie Jones, and; and sponsored by Kiro. You can enter solo or as a team of up to three people, and you’re welcome to submit more than one project.

There’s a $9,600 prize pool to be won, plus 2000 FREE Kiro credits for every verified entry. Submissions close on 23 August, with the winners announced on 6 September.

Find out more and take part:https://codingagents.fyi/hackathon/kiro/

It’s a great chance to checkout the Kiro agentic coding environment!

OK, on with the coding challenge!

The Challenge - Building Your Own Sorting Visualiser

In this challenge you’re going to build a tool that animates a sorting algorithm at work on an array of values. By the time you’re done, your tool will support eight classic algorithms: bubble, insertion, selection, merge, quick, heap, shell, and radix. And let the user pick the algorithm, the sample size, the initial order of the data, the display mode, and the animation speed.

Something like this:

You can pick the platform that suits you best. A terminal/CLI implementation (using something like ncurses, blessed, or a similar library) is a great fit if you want to stay close to the keyboard. A browser/web implementation gives you the most flexibility for visuals and is easy to share. A native desktop GUI (using something like Qt, Tauri, GTK, or your language’s standard toolkit) is somewhere in between. Pick the one that interests you every step in this challenge applies to all three.

A small note before we start: you’ll be tempted to just run the algorithm and update the screen at each iteration. That works at first, but quickly tangles your sorting logic with your rendering logic. We’ll structure things so that algorithms produce a stream of events (compared these two indices, wrote this value to that index) and the renderer consumes them. That separation is what’ll make adding new algorithms and new display modes painless later on.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

Choose your target platform, terminal/CLI, browser/web, or native desktop GUI, and the programming language to go with it. I’d encourage you to pick a stack you’re comfortable with for both UI rendering and handling user input, because you’ll be doing plenty of both.

Have a think about how you’ll handle the animation loop. Most platforms have a natural answer, a render loop tied to the frame rate in a browser or a GUI toolkit, a timed redraw in a terminal. Whichever route you take, you’ll want a way to do work, redraw, and pause for a configurable interval between steps.

Step 1

In this step your goal is to render an array of values on screen and watch a single sorting algorithm run end-to-end.

Start by generating an array of, say, 30 random integers. Render each value as a vertical bar whose height is proportional to its value, with the bars laid out left-to-right across the screen. Don’t worry about colours, controls, or configuration yet, get the rendering working first.

Now implement bubble sort and animate it. After every comparison and every swap, redraw the array and pause for a short delay (something like 50–100 ms is a good starting point). When the algorithm finishes, the bars should be in ascending order from left to right.

This is your end-to-end skeleton. Everything else in the challenge will hang off this foundation.

Testing: Run your tool and watch the array sort itself. The bars should start in random order and finish in ascending order. Try a few different sample sizes (10, 30, 50) and confirm the layout still looks right. If your bars overflow the screen or look squashed, adjust your scaling so the tallest bar uses most of the available height.

Step 2

In this step your goal is to highlight what the algorithm is doing right now and show some live statistics.

Up to now your bars have all been the same colour. That makes it hard to see what the algorithm is up to at any given moment. Pick three colours: a resting/idle colour for bars that aren’t being touched, a comparison colour for the two indices the algorithm is currently comparing, and a swap/write colour for indices that are currently being swapped or written. Update your bubble sort so that the right indices are highlighted at the right moment.

Next, add a small statistics overlay somewhere on the screen showing the number of comparisons, the number of swaps/writes, the number of array accesses, and the elapsed wall-clock time. Update these as the algorithm runs so the user can watch the counters tick up.

Finally, when the algorithm finishes, validate that the array is actually sorted. If it isn’t, surface a clear error, this’ll save you a lot of pain later when you’re implementing trickier algorithms and a subtle bug slips in.

Testing: Run your tool and confirm the highlighted bars match what bubble sort is doing, the two compared bars should light up in the comparison colour, and when a swap happens you should see the swap colour. Watch the stats counters and sanity check them: for a small array, count a few comparisons by eye and confirm the counter agrees. At the end of the run, deliberately break the algorithm (e.g. comment out the swap) and confirm your validation catches it.

Step 3

In this step your goal is to give the user playback controls and a configurable speed.

Right now the visualisation runs straight through from start to finish with no way to pause or replay it. Add controls that let the user start a run, pause it mid-way, resume from where it paused, restart it from the beginning, and step forward by a single operation at a time. The exact bindings are your call, keys in a terminal, buttons in a GUI, both in a web app.

Add a speed control too. Provide at minimum slow, medium, and fast presets, plus a way to set a custom value. Speed can be expressed as a frames-per-second rate or as a delay-per-step, pick whichever feels natural for your platform.

Make sure your controls remain responsive at all speeds. At the fastest preset, it’s tempting to do many algorithm steps per redraw to keep up, but the user still needs to be able to hit pause and have it actually pause. Test that.

Testing: Start a run, pause it half-way through, and confirm the bars freeze in place with the comparison/swap highlights still visible. Resume and confirm it picks up from where it left off. Hit restart and confirm the array goes back to its original state and the stats reset to zero. Use the step control to advance one operation at a time and watch the comparison/swap highlights move with each press. Try the fast preset and confirm pause still responds promptly.

Step 4

In this step your goal is to refactor bubble sort behind a clean algorithm interface, then add two more algorithms.

So far your sorting code and your rendering code are probably mixed together. We’re going to separate them. Define an interface for a “sortable run”, something that, given an array, produces a stream of events describing what the algorithm is doing. The two essential events are compared these two indices and wrote this value to that index. A generator (or coroutine, or async iterator, depending on your language) is a natural fit: each yield is one event, and the renderer pulls events out one at a time.

Refactor your bubble sort to fit this interface. Your renderer should now consume events from the algorithm rather than calling sort logic directly. The stats counters become trivial to maintain, every comparison event increments the comparison counter, every write event increments the write counter, and so on.

With the interface in place, add insertion sort and selection sort. Each one should be its own implementation of the interface, with no changes needed to your renderer. Add an algorithm selector to your UI so the user can pick which one to run.

Testing: Run each of the three algorithms on the same input and compare. Bubble sort should swap adjacent elements; insertion sort should sweep one element at a time into its correct place in the sorted prefix; selection sort should make far fewer swaps but lots of comparisons. The stats counters should reflect the differences, selection sort, for example, will show roughly the same number of comparisons as bubble sort but far fewer writes.

Step 5

In this step your goal is to add merge sort, quick sort, and heap sort.

Each algorithm should slot into the same algorithm interface you defined in Step 4. If the interface is right, your renderer shouldn’t need any changes, every algorithm is just a different sequence of comparison and write events.

A small word on merge sort: because it uses an auxiliary array, you’ll need to decide how to represent writes back into the main array. The simplest approach is to emit a write event for each element placed back into the main array during the merge. The user will see the array reform piece by piece as the merges complete, which is exactly the behaviour you want.

Testing: Run each of the three algorithms and watch them. Merge sort should produce the classic “halves coming back together” pattern. Quick sort should show pivots being placed and partitions taking shape. Heap sort should show a chaotic phase (heap construction) followed by an orderly phase (extracting the max from the back of the array). Confirm the correctness check from Step 2 passes for all three on a variety of input sizes.

Step 6

In this step your goal is to add shell sort and radix sort.

Radix sort is the odd one out. It’s not comparison-based, it sorts by repeatedly distributing values into buckets based on individual digits. You’ll need to decide how to express what radix sort is doing through your event stream. One approach is to emit write events as values move from the array into the buckets and back. Another is to extend your event vocabulary with something like a bucketed this value event. Either is a reasonable design choice, the goal is for the viewer to be able to see the algorithm working.

Testing: Run shell sort and watch how the array goes from chaotic to “fairly ordered” to fully sorted as the gap shrinks. Run radix sort on a fixed-width integer array and confirm it sorts correctly. Compare radix sort’s stats counters to the comparison sorts, radix sort should have zero comparisons (or very few, depending on how you count) and a number of writes proportional to the number of digits times the array size.

Step 7

In this step your goal is to add the numbers display mode behind a pluggable rendering interface.

Up to now everything has been bars. The numbers display mode renders each value as a small labelled tile or box showing the numeric value, laid out in a row. When two indices are being compared or swapped, the corresponding tiles should be visually lifted or separated from the main row so the active operation is unambiguous, with the comparison/swap colours from Step 2 applied.

That should look something like this:

Just as you separated algorithms from rendering in Step 4, now separate the bars renderer from a generic rendering interface. Each renderer should accept the same array and event stream and decide how to draw it. Adding the numbers renderer should not require changes to any of the algorithms.

Add a display mode selector to your UI so the user can switch between bars and numbers.

Testing: Run each of your eight algorithms in numbers mode and confirm the lift behaviour matches what you expect, when bubble sort compares two adjacent elements, those two tiles should lift; when it swaps them, both should be highlighted in the swap colour. Switch between bars and numbers mid-run if you can; the algorithm should keep running and just the rendering should change.

Step 8

In this step your goal is to add sample size and sample order configuration, polish the user experience, and finish the documentation.

Add controls that let the user select the sample size (the number of elements in the array, with sensible minimum and maximum bounds for your platform) and the initial sample order. Support at least three orders: random, reversed, and already sorted. When the user restarts a run, regenerate a fresh sample of the configured size and order, except for already sorted, which is deterministic by definition. This means consecutive runs of the same algorithm exercise different inputs.

Show the current configuration on screen at all times: the chosen algorithm name, the sample size, the sample order, and the display mode. The user should never have to wonder what they’re looking at.

When a sort completes, present a clear “sorted” state. A nice touch is a final sweep through the array, a brief animation where each element is briefly highlighted in turn, left to right, to confirm visually that everything is in order. Freeze the final stats so the user can read them.

Finally, write a short README documenting how to run your tool, how to choose between platforms (if you’ve implemented more than one), and, importantly, how a future contributor would add a new algorithm or a new display mode. If your interfaces from Steps 4 and 7 are clean, this should be easy to write. If it’s hard to write, that’s a signal that the interfaces could be better.

Testing: Try every combination of configuration: each algorithm, each sample size from your minimum to your maximum, each sample order, each display mode. Confirm the on-screen configuration always matches what you’ve selected. Confirm that two consecutive runs of bubble sort with random order produce different starting arrays, but two consecutive runs with already sorted order produce the same one. Watch the completion sweep at the end of a run and confirm the stats freeze on the final values. Hand your README to a friend and ask them to add a hypothetical “cocktail sort” algorithm; if they can do it without reading your renderer code, you’ve nailed the abstraction.

Going Further

Here are some ideas to take your sorting visualiser further:

• Add more algorithms: cocktail sort, comb sort, gnome sort, Tim sort, intro sort, or even bogo sort for fun.

• Add a head-to-head mode that runs two algorithms side-by-side on the same input array, so the user can race them.

• Add an export-to-video or export-to-GIF option so users can share their favourite runs.

• Build an algorithm-comparison view that runs every algorithm across a range of sample sizes and plots comparisons or writes against input size, giving a visual confirmation of each algorithm’s Big-O behaviour.

• Visualise additional algorithm-specific metadata: recursion depth for merge and quick sort, the heap structure for heap sort, or the gap sequence for shell sort.

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

I’m pleased to be hosting the Ready, Spec, Ship Hackathon alongside Angie Jones, , and sponsored by Kiro. You can enter solo or as a team of up to three people, and you’re welcome to submit more than one project.

There’s a $9,600 prize pool, plus 2000 FREE Kiro credits for every verified entry. Submissions close on 23 August, with the winners announced on 6 September.

Find out more and take part:
https://codingagents.fyi/hackathon/kiro/

It’s a great chance to checkout the Kiro agentic coding environment!

I hope to see you taking part.

Please let me know if you have any questions.

Regards, John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #129 - Coding Challenges Coach

This challenge is to build your own AI coaching agent that helps developers learn by working through the Coding Challenges projects.

We’ve all wished for a patient mentor who sits with us while we learn: someone who helps us pick the right thing to work on, talks through our approach, lets us run our code, and remembers where we got to last time. In this challenge you’ll build exactly that, an AI coaching agent built with Google’s Agent Development Kit (ADK) and the Gemini Enterprise Agent Platform.

You’ll start with a small prototype running on your own machine, then progressively harden it into a production-ready service that runs on Google Cloud, tracks its own costs, watches its own behaviour, and passes a security audit. This coding challenge is all about the process of taking a demo through to something you’d be happy to run for real.

This project is kindly sponsored by Google Cloud.

The Challenge - Building Your Own Coding Challenges Coach

You’re going to build a coaching agent that can hold a conversation with a developer, help them choose a Coding Challenge that suits their interests and skill level, reads the challenge specifications, searches the web for extra context, runs and tests the developer’s code in a safe sandbox, and remembers the whole journey so they can stop and come back later. Then you’ll take that agent and turn it into a proper cloud service: deployed, identified, secured, costed, and observed.

This is a challenge that’s aimed at engineers who are comfortable writing code and now want to learn what it takes to build and operate an AI agent responsibly. Going from a local prototype through to a hardened, audited production service. The core skills here, working with an agent framework, wiring up tools, managing state, deploying to the cloud, and thinking about security and cost, transfer to any agent platform. We’ll use the Google stack throughout because it gives us a complete, coherent path from prototype to production, all driven from local CLI tools.

The interesting and tricky parts are in the second half. Getting an agent to chat is easy. Giving it tools it can safely run, making it remember the right things without repeating itself, deploying it without a single static credential in sight, keeping its costs under control, and proving to an auditor that it’s secure, that’s where the real learning is.

Each step below covers one whole area of the agent, so treat a step as a mini-project of its own rather than a quick hour of work. Get each area solid before you move on, because the later steps build directly on the earlier ones.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

This challenge uses Google’s Agent Development Kit, you can find installation instructions here: ADK Installation. You’ll also need a Google Cloud account and the gcloud CLI, because the later steps deploy to and lean on Google Cloud. Installation instructions for the gcloud CLI are here.

Everything you do will run from your local command line, so get comfortable with the adk CLI commands (adk run, adk web, and adk deploy).

It’s also worth spending some time reading through the ADK documentation so you understand the core building blocks: agents, tools, and the runtime. Have a look at how the built-in Dev UI works, as that’s where you’ll do most of your local testing.

Finally, gather a handful of Coding Challenge specification files to coach against. Grab two or three challenge descriptions and save them locally as markdown, JSON, or YAML files in a folder your agent will be able to read. These are your test data for the rest of the challenge.

Testing: Confirm your environment is ready by creating an empty starter agent and launching the Dev UI:

adk web

You should be able to open the Dev UI in your browser and see your agent listed, ready to chat, even if it doesn’t do anything useful yet:

Step 1

In this step your goal is to build a coaching agent that runs entirely on your machine and helps a developer choose a Coding Challenge through conversation.

Using the ADK, create an agent that holds a natural-language conversation, asking about the developer’s interests, what they want to learn, and their experience level, then recommending a challenge that suits them. Give the agent a clear coaching persona through its ADK configuration, including a system prompt that makes it behave like a mentor: encouraging, Socratic, and focused on guiding the developer to the answer rather than handing it over. All of your development and testing at this stage should happen locally through the ADK Dev UI. You’ll find some Google ADK examples to build on here.

Check out the previous coding challenge, building an AI-powered chatbot for some guidance on creating this agent and setting the persona via the system prompt.

Testing: Run your agent with adk web and open the Dev UI. Tell the agent you’re a beginner interested in networking and confirm it asks sensible follow-up questions and recommends a suitable challenge. Try again as an experienced developer wanting to learn about databases and confirm it recommends something more advanced. Then ask it to “just give me the full solution” and confirm it declines politely and offers hints or questions instead of the answer.

Your ADK session should look something like this:

And your agent should refuse to provide the full solution:

Step 2

In this step your goal is to give your agent tools so it can ground its coaching in real information rather than guessing.

Implement a custom tool that reads Coding Challenge specification files (markdown, JSON, or YAML) from the local filesystem and returns the content relevant to the developer’s question.

Implement a second tool that performs a web search using an external search API of your choice, so the agent can find extra context, documentation, or alternative approaches when the developer is stuck. Bind both tools to the agent so it can chain them together within a single turn, for example reading the relevant part of a challenge spec and then searching the web for a related concept before it replies.

When building agents there’s a fine line between providing too few tools, reducing the agent’s flexibility, and too many tools, resulting in the agent picking the wrong tool. If several tools are likely to be used in the same order repeatedly, see if you can build a workflow or a composite tool, this saves both time and tokens.

For guidance, refer to the ADK docs on building a multi-tool agent.

Testing: In the Dev UI, ask a question about one of your saved challenge specs, for example “what does step 3 of this challenge ask me to do?”, and confirm the agent reads the file and answers from its real content. Ask something not in your specs, for example “what’s a good library for parsing this format?”, and confirm it runs a web search. Finally ask a question that needs both, for example “I’m on step 2, what background reading would help?”, and confirm you can see both tool calls in the trace, the spec read first and the search second, in a single turn.

Step 3

In this step your goal is to give your agent a safe place to run the developer’s code, in more than one language.

ADK handles code execution through the BaseCodeExecutor interface, and ships a GkeCodeExecutor that runs each snippet in a gVisor-sandboxed, ephemeral GKE workload. Every run should get a fresh, isolated container from a custom image with Python, Node, and Go toolchains installed. The tool should write the developer’s code into that sandbox, run it, and return the standard output, standard error, and exit code. Support at least Python, JavaScript, and Go, with the agent choosing the language from the challenge context or the developer’s stated preference.

Testing: In the Dev UI, ask the agent to run a small snippet that prints something and exits cleanly, and confirm you get the output and an exit code of 0. Give it code that raises an error and confirm you see the error on standard error and a non-zero exit code. Run a “hello world” style snippet in each of Python, JavaScript, and Go and confirm each runs in the right language. Tell the agent you’re on a Go challenge, give it a Go snippet without naming the language, and confirm it picks Go.

Step 4

In this step your goal is to let your agent run tests and coach the developer through the results, while keeping execution under control.

Allow the agent to run a test suite against the user’s code, report which tests fail, and coach the developer through debugging, helping them understand and fix the problem themselves rather than rewriting the code for them. Add safety limits to every execution: a timeout, defaulting to 30 seconds, and a maximum output size, defaulting to 64 KB, both configurable, so a runaway process or a flood of output can’t disrupt the session. You’ll have to provide a way for the user’s code to reach the agent.

Testing: Give the agent code with a known bug and a test that catches it. Confirm it runs the tests, reports the failure clearly, and responds with guiding questions rather than the corrected code, even when you ask it to “just fix it”. Then give it code with an infinite loop and confirm execution is cut off at the timeout with a clear message. Give it code that prints a huge amount of output and confirm the returned output is capped. Change the limits in your configuration and confirm the new values take effect.

Step 5

In this step your goal is to let a developer stop mid-challenge and pick up later, without the agent losing the thread or repeating itself.

Persist the conversation state using Session service and also enable the Memory Bank (VertexAiMemoryBankService) so a session can be paused and resumed. The Session service will capture which challenge the developer is working on, the code they’ve written, the coaching advice they’ve already been given, and which concepts have been covered. The Memory Bank will consolidate related memories across sessions so that user preferences are stored and as the notes accumulate the agent merges overlapping ones and avoids giving the developer the same advice twice.

Testing: Have a session where you pick a challenge, write some code, and receive a couple of pieces of advice, then end it. Start a fresh session and confirm the agent remembers which challenge you’re on and refers back to your earlier code and advice rather than starting over. Across two or three sessions, steer it towards giving the same category of advice and confirm that by the later sessions it recognises it has already covered this and builds on it instead of repeating it.

Step 6

In this step your goal is to take your local agent and run it as a proper cloud service, still driven from your local repository, now easily running on the cloud.

Use adk deploy to deploy the agent from your local ADK project to the Google Cloud Agent Runtime, keeping your local repository as the single source of truth. Configure the deployment to use Gemini models served from Agent Runtime rather than a local or third-party API key. Expose the deployed agent through a public HTTPS endpoint, and set up continuous deployment so that pushing changes from your local repository automatically updates the running agent.

Testing: Deploy with adk deploy and confirm it succeeds. Call the agent’s public HTTPS endpoint from your machine, for example with curl, and confirm you get a valid response over HTTPS using the Gemini models served from Agent Runtime, with no local or third-party model API key in the configuration. Make a small change locally, push it, and confirm the deployed agent updates without you running a manual deploy.

Step 7

In this step your goal is to give your deployed agent a real identity, so you can both restrict what it reaches and attribute everything it does, with no static credentials anywhere.

To do this, look into deploying to Agent Runtime with Agent Identity enabled.

Testing: Confirm the deployed agent can still read specs, run searches, and execute code using its SPIFFE identity (A SPIFFE identity is a secure, cryptographically verifiable identifier assigned to specific software services (workloads) rather than machines - you can learn more on the SPIFFE website), and that there are no static API keys or long-lived credentials anywhere in the deployment.

Enable Data Access audit logs on the specs storage, run a full session, then filter Cloud Audit Logs by the agent’s principal and confirm the spec read, the search, and the sandbox execution all appear under the same SPIFFE ID.

Deploy a second agent and confirm its actions carry a different SPIFFE ID, so you can separate the two agents’ activity from the logs alone.

Capture a valid token and try to use it from a different workload without the agent’s certificate, and confirm it is rejected while the agent’s own requests still succeed. Through the agent’s identity, try to reach a Google Cloud resource it shouldn’t need and confirm access is denied, and that the denial is recorded against the agent’s identity in the audit log.

Step 8

In this step your goal is to prove your hardening actually holds up, by passing an audit.

Build and run a mock third-party security audit against your deployment: a script that only ever reads, the same posture a real external auditor without console access would take. Check for static credentials confirming zero user-managed keys. Check for over-permissive IAM roles, flagging anything broader than the least-privilege set you defined in Step 7. Check for unencrypted state storage. Check for exposed debug endpoints by probing your deployed HTTPS endpoint for known debug paths, such as the ADK Dev UI routes, /docs, and /openapi.json, and confirming each returns 404 or 401 rather than 200. Your agent should pass all four checks.

Testing: Run your audit and confirm it reports a clean pass on all four checks. Then deliberately introduce a problem, for example add a static credential or open up an IAM role, and confirm the audit catches it and fails. Fix it and confirm the audit passes again.

Step 9

In this step your goal is to make your agent pick the right model for the job and keep its spending under control.

Configure the agent to select models from Model Garden based on task complexity, using a lightweight model for chat and challenge selection and a more capable model for code review and debugging. Because an ADK LlmAgent has a fixed model, structure this as a multi-agent setup: a lightweight coordinator agent that handles the conversation and delegates to a more capable sub-agent (via ADK sub-agents and auto-transfer, or by wrapping it as an AgentTool) whenever the developer needs code review or debugging. This makes the model boundary explicit and keeps each model’s usage cleanly attributable in the trace.

Track token usage per request and per session, and report the cost both in the Dev UI during development and in your production logs. Add a configurable per-session cost cap that gracefully ends a session when it’s reached, telling the developer clearly, rather than quietly running up charges.

Testing: Have a simple challenge-selection conversation and confirm the lightweight model is used, then ask for a detailed code review and confirm the more capable model is used, checking the trace to see which model handled each request. Confirm you can see per-request and per-session cost in the Dev UI and in your production logs. Set a low cost cap, run a session until it’s reached, and confirm the session ends gracefully with a clear message and makes no further billable requests. Raise the cap and confirm the session can continue.

Step 10

In this step your goal is to make your agent’s behaviour visible and to keep its recommendations within policy.

Instrument your agent with OpenTelemetry and export its traces to Cloud Trace, so that each agent interaction becomes a single trace whose child spans capture every model call and tool call, each span carrying the user query, the tool used, the model that served the request, and the token counts.

Finally, implement a governance control that restricts which challenges the agent can recommend based on a configurable policy, for example blocking challenges tagged as “enterprise” or “paid”, and record each policy block as a span event so enforcement is auditable in the trace.

Testing: Run some interactions against your deployed agent, including at least one that triggers an error, then open Cloud Trace and confirm each interaction shows up as a single trace with child spans for its model and tool calls, that the model-call spans carry token-count attributes, and that the errored interaction has a span marked with an error status.

Configure the policy to block a particular challenge or tag, steer the conversation towards it, and confirm the agent won’t recommend it and offers an allowed alternative instead, then confirm the block is recorded as a span event on the trace. Then allow it and confirm the agent will recommend it.

Going Further

Once you’ve built the core coaching agent and hardened it for production, here are some ideas to take it further, ordered roughly from easiest to most ambitious:

• Add a richer set of challenge specs and let the coach recommend a learning path across several challenges rather than one at a time.

• Add automatic grading that assesses how close a developer’s solution is to passing all the challenge’s testing steps.

• Support voice or video coaching modes as well as text.

• Add multi-user support and team coaching sessions, with per-user memory and shared team progress.

• Let the agent use an alternative sandbox provider and compare its behaviour and cost against the GKE-based executor.

• Build a proper front end beyond the Dev UI, tailored to the coaching experience.

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #128 - Mock Interview Coach

This challenge is to build your own mock interview coach, a tool that lets you practise behavioural interviews face-to-face speaking to a photorealistic AI interviewer.

Behavioural interviews are the part of the hiring process that engineers practise least. Reading sample STAR answers is nothing like composing one out loud, under time pressure, while someone watches you. In this challenge you’ll build a coach that puts a live, talking interviewer on your screen using Anam’s real-time persona platform and their latest cara-4 model. It will ask you behavioural questions, probe your weak answers with follow-ups, and then score your performance and tell you how to improve. Along the way you’ll work with real-time avatar streaming, persona and prompt design, session-based authentication, and post-session transcript analysis.

This project is kindly sponsored by Anam.

The Challenge - Building Building Your Own Mock Interview Coach

You’re going to build a web application with two halves. The backend, in any language you like, will manage interview setup, issue Anam session tokens, and analyse transcripts to produce feedback reports. The frontend is a thin web page that streams the interviewer’s face and voice into a video element and lets you talk to them over your microphone.

The interesting parts are the persona design (writing a system prompt that makes an AI interview you rather than help you), the security model (keeping your API key server-side while the browser streams in real time), and the feedback loop (turning a raw conversation transcript into a structured STAR assessment).

This is an intermediate to advanced challenge. The backend can be written in any language that can serve HTTP and call a REST API, so pick whatever you’re most comfortable with. The frontend needs only a small amount of JavaScript, so don’t be put off if you’re not a frontend developer.

What you’ll build will look something like this:

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

First, sign up for a free Anam account. The free tier gives you API access, 30 minutes of conversation per month, and a 3 minute limit per conversation. That’s enough to complete this challenge, but it does mean each test session is short, so plan to test with brief exchanges rather than full-length interviews.

From your Anam account, create an API key and make a note of it. We’ll be keeping it server-side, so think about how you’ll provide it to your backend, an environment variable named ANAM_API_KEY is the convention we’ll use in this challenge.

Next, pick your backend language and an HTTP server library you’re comfortable with. Then spend a little time with the Anam documentation, particularly the JavaScript SDK quickstart, to understand the moving parts: personas, session tokens, and the streaming client.

Finally, make sure you have a browser with a working microphone and camera permissions you can grant to localhost.

Step 1

In this step your goal is to design your interviewer persona.

A persona on the Anam platform combines a face, a voice, an LLM, and a system prompt. The system prompt is where your coaching product lives, it’s what turns a general-purpose talking avatar into a behavioural interviewer.

Define a persona configuration (name, avatar, voice, LLM, and system prompt) for an interviewer that:

• Asks one behavioural question at a time and waits for the answer.

• Probes with a follow-up when an answer lacks specifics, for example asking “what was the outcome?” when the candidate doesn’t say.

• Never suggests answers or completes the candidate’s stories for them.

• Stays in role, politely steering the conversation back to the interview if the candidate drifts off topic.

Getting this behaviour reliable is a prompt engineering exercise, expect to iterate. If you want some great ideas for what to include in the system prompt, check out my interview with Austen McDonald about mastering behavioural interviews. Also check out Austen’s newsletter: Mastering Behavioral Interviews. He also has a guide to doing mock interviews.

Testing: Use the Anam Lab to create the persona and talk to it. Try three conversations: give a vague answer (”we fixed it and it was fine”) and check you get a probing follow-up; ask the interviewer “what would you say in my situation?” and check it declines to answer for you; try changing the subject to the weather and check it brings you back to the interview. The lab is a great no-code option for refining the avatar before you build the full system.

Here’s what it looks like:

Use the prompt tab to enter your prompt, the avatar tab to upload / take a photo of you, the voice tab to clone your voice if you want to. Otherwise pick suitable ones from the existing options.

Step 2

In this step your goal is to build the session token service, the backend half of the security model.

Anam’s streaming sessions are authorised with short-lived session tokens, created by calling POST <https://api.anam.ai/v1/auth/session-token> with your API key and a persona configuration. The API key must never reach the browser, so your backend will act as the exchange point.

Build a backend with an endpoint that, when called by your frontend, requests a session token from the Anam API using the persona configuration from Step 1, and returns that token to the caller. Read the API key from the ANAM_API_KEY environment variable.

Your endpoint should also apply a time limit to each interview session. Make the limit configurable, and default it to fit inside the free tier’s 3 minute conversation limit.

Testing: Start your backend and request a token from the command line:

curl -X POST <http://localhost:8080/api/session-token>

You should get back a JSON response containing a session token. Check your frontend code and browser network tab later in the challenge to confirm the Anam API key itself never appears anywhere client-side. Try setting the time limit configuration to a different value and confirm the session created with the token respects it.

Step 3

In this step your goal is to get your interviewer on screen, streaming in real time.

Build a minimal web page, served by your backend, that:

• Fetches a session token from the endpoint you built in Step 2.

• Creates a streaming client using Anam’s @anam-ai/js-sdk package and attaches it to a video element with streamToVideoElement.

• Has controls to start the interview and to end the session at any time.

Once connected, the candidate speaks their answers over their microphone and the interviewer responds with voice and video. Have the interviewer open the session by greeting the candidate and explaining the format: how many questions to expect and how the interview will run. That behaviour belongs in your persona’s system prompt or greeting configuration, not in your JavaScript.

Testing: Open the page in your browser, click start, and grant microphone access. You should see your interviewer appear in the video element, greet you, explain the format, and ask the first question within a few seconds. Answer out loud and confirm the interviewer hears you and responds. Click your end control mid-conversation and confirm the stream stops cleanly.

Step 4

In this step your goal is to make each interview a real, configurable interview rather than a single hard-coded conversation.

First, build a question bank of at least 20 behavioural questions, each tagged with a competency such as leadership, conflict, failure, teamwork, or delivery. Store it however you like, this is data your backend owns.

Then add an interview setup screen where, before the session starts, the candidate chooses:

• The target role they’re practising for.

• The seniority level.

• Which competencies they want to be assessed on.

Your backend should generate each interview from those choices, drawing suitable questions from the bank, so that repeated sessions produce different interviews rather than the same script every time.

Testing: Set up an interview targeting a senior role with the “conflict” and “failure” competencies. Confirm the questions asked relate to those competencies. Run a second interview with the same setup and confirm you get different questions from the bank. Change the competencies and confirm the questions change to match.

Step 5

In this step your goal is to capture and store the interview transcript.

The feedback report in the next step needs the full conversation, both the interviewer’s questions and the candidate’s answers. Anam gives you two routes to it: the SDK emits message history events while the session runs, and there is a session transcript API you can call after the session ends. Pick whichever suits your design.

Once you have the transcript, persist each completed interview with its date, the setup choices from Step 4, and the full transcript, so past sessions can be reviewed later.

Testing: Complete a short interview, answering at least two questions. Then inspect your stored data and confirm it contains the date, the role, seniority and competencies you chose, and the complete conversation with both the interviewer’s and your own words. Complete a second interview and confirm both sessions are stored separately.

Step 6

In this step your goal is to turn the transcript into a feedback report, this is where your tool becomes a coach.

The standard framework for behavioural answers is the STAR method: Situation, Task, Action, Result. Strong answers cover all four; weak answers typically describe a situation and then trail off without actions or results.

Analyse each answer in the stored transcript and produce a written feedback report that includes:

• Which STAR elements were present and which were missing from each answer.

• A score from 1 to 5 for each answer.

• The candidate’s overall strengths.

• One specific improvement suggestion per question.

You’ll want an LLM to do the analysis, any provider works, this part is independent of the live Anam session.

Finally, track scores across sessions so the candidate can see whether they’re improving over time.

Testing: Run an interview and deliberately give one incomplete answer, describe a situation but never say what the result was. The report should flag the missing Result for that answer and score it lower. Give one full STAR answer in the same interview and confirm it scores higher. Complete a few sessions and confirm you can see your scores across them.

Step 7

In this step your goal is to add interviewer styles and give the candidate control over the flow of the interview.

Different interviewers put different pressure on candidates, and practising with only one style leaves you unprepared for the others. Offer at least two selectable interviewer personas with distinct voices and personalities, for example a supportive coach who encourages you along, and a formal bar-raiser who is polite but gives you nothing back.

Then add flow control: the candidate should be able to skip a question they don’t want to answer, and end the interview early. When they do, the feedback report should cover only the questions actually answered, with no scores or feedback invented for questions that were skipped.

Testing: Run an interview with each persona and confirm the voice and manner are noticeably different while the interview structure stays the same. In another session, skip the second question and end the interview after the third. Check the report covers exactly the questions you answered, with the skipped question absent.

Going Further

Once you have the core coach working, here are some ideas to take it further:

• Add a per-answer timer so candidates can practise keeping answers to two or three minutes.

• Generate a printable or PDF version of the feedback report to share with a mentor.

• Add technical and system design interview modes with their own question banks and scoring rubrics.

• Bring your own LLM to the live conversation, so the same brain that scores the answers also drives the interviewer’s follow-up questions.

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #127 - Tmux

This challenge is to build your own version of tmux, the terminal multiplexer.

If you spend any time working in a terminal, sooner or later you’ll meet tmux. It lets you run multiple terminal sessions inside a single window, split that window into panes, and, best of all, detach from a session and come back to it later with everything still running. It’s the tool that saves your work when your SSH connection drops. Under the hood it brings together some fascinating systems programming: pseudo-terminals, client-server architecture over Unix domain sockets, raw terminal input handling, and screen rendering.

Building a simplified version yourself is one of the best ways I know to really understand how a terminal works.

The Challenge - Building Your Own Tmux

In this challenge you’re going to build a simplified terminal multiplexer. Your tool will let you create named sessions that keep running in the background, detach from them and reattach later, open multiple windows within a session, and split windows into panes, each running its own shell.

This is an advanced challenge. The individual steps are manageable, but you’ll be working with pseudo-terminals, raw terminal modes, Unix domain sockets, and screen rendering, so it helps to be comfortable with systems programming.

The concepts are the same in any language, so pick one you know well that gives you good access to operating system APIs. This challenge is best tackled on Linux, macOS, or another Unix-like platform, since that is where pseudo-terminals and Unix domain sockets live.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

Choose your target platform and programming language. You’ll need access to pseudo-terminal (PTY) APIs, Unix domain sockets, and the ability to put the terminal into raw mode, so check your chosen language has libraries for these before you commit.

If you haven’t used tmux before, install it and spend some time with it. Create a session, detach with Ctrl+b d, reattach, split some panes, and switch between windows. Getting a feel for the real thing will make every step of this challenge clearer. The tmux man page and the tmux getting started guide are both excellent references.

It’s also worth reading up on how pseudo-terminals work (man pty on most systems) and how terminal escape sequences control the screen. These two topics are the foundation of everything you’ll build here.

Step 1

In this step your goal is to allocate a pseudo-terminal, spawn a shell inside it, and connect it to your own terminal.

This is the core building block of a terminal multiplexer. Your program should create a PTY, launch the user’s shell as a child process attached to it, put your own terminal into raw mode, and then shuttle data in both directions: keystrokes from your terminal go to the shell, and the shell’s output is displayed on your screen. When the shell exits, your program should restore the terminal to its original state and exit cleanly.

Testing: Run your program:

cctmux

You should see a shell prompt. Try running some commands, including full-screen programs:

ls -la
vim
top

Everything should behave exactly as it would in a normal terminal. Type exit and verify your program shuts down and your terminal is restored to a usable state.

Step 2

In this step your goal is to split your program into a server and a client that communicate over a Unix domain socket.

Real tmux is built as a client-server system. The server owns the sessions and the shells running inside them; the client is a thin process that connects to the server, sends it your keystrokes, and displays whatever the server tells it to. This separation is what makes detaching possible later: the shells belong to the server, so they survive when a client goes away.

When your client starts, it should check whether a server is already running. If not, it should start one in the background. The two should communicate over a Unix domain socket, and you’ll need a simple protocol for passing keyboard input in one direction and screen output in the other.

Testing: Run your client:

cctmux

You should get a working shell, just like in Step 1. In another terminal, verify the socket exists (for example with ls -l on wherever you chose to put it, such as /tmp/cctmux-*) and that the server appears in ps as a separate process from your client.

Step 3

In this step your goal is to support named sessions.

A session is a collection of shells managed by the server under a single name. Your tool should support creating a new named session:

cctmux new-session -s mysession

The server should be able to hold several sessions at once, each with its own shell. If a user tries to create a session with a name that already exists, they should get a clear error message.

Testing: Create a session and verify you get a shell:

cctmux new-session -s work

Run a command like echo hello to confirm the shell works. Then try creating a duplicate:

cctmux new-session -s work

You should see an error telling you the session already exists.

Step 4

In this step your goal is to implement the prefix key and detaching from a session.

Tmux is driven by a prefix key, Ctrl+b by default. When the user presses the prefix, the next keystroke is a command for the multiplexer instead of input for the shell. Your first command is d for detach: the client should disconnect and exit, restoring the terminal, while the session and its shell keep running on the server.

Make the prefix key configurable so users can choose something other than Ctrl+b, for example via a command line flag or a simple configuration file. Any key that isn’t a recognised command after the prefix should be handled gracefully, and pressing the prefix twice should send the prefix key itself through to the shell.

Testing: Create a session and start something long-running in it:

cctmux new-session -s work

Inside the session run:

sleep 300 &
echo "still here"

Press Ctrl+b then d. You should be back at your normal terminal prompt. Verify with ps that the shell and the sleep process are still running. Then configure a different prefix key, start a new client, and verify detaching works with the new prefix and that Ctrl+b now passes through to the shell.

Step 5

In this step your goal is to support reattaching to a running session.

Detaching is only half the magic. Your tool should let the user reconnect to a session that’s running in the background:

cctmux attach-session -t mysession

On attaching, the user should see the session’s current screen content, including output that was produced while they were detached, and be able to carry on working as if they never left. Attaching to a session that doesn’t exist should produce a clear error.

Testing: Create a session, run echo before detach, then detach with Ctrl+b d. Reattach:

cctmux attach-session -t work

You should see the screen as you left it, including before detach. Detach again, then try attaching to a session that doesn’t exist:

cctmux attach-session -t nosuchsession

You should get an error message, not a hang or a crash.

Step 6

In this step your goal is to support listing sessions, killing sessions, and cleaning up the server.

Your tool should report the sessions the server is currently managing:

cctmux list-sessions

The output should show each session’s name and some useful detail, such as how many windows it has and whether a client is attached, similar to the real tmux output.

It should also support killing a session:

cctmux kill-session -t mysession

Killing a session should terminate the shells running inside it. When the last session is closed, whether by kill-session or by all of its shells exiting, the server should shut itself down and remove its socket, leaving nothing behind.

Testing: Create two sessions, work and play, detaching from each. Run:

cctmux list-sessions

You should see both sessions listed. Kill one:

cctmux kill-session -t play
cctmux list-sessions

Only work should remain. Kill work too, then verify with ps that the server process has exited and that the socket file has been removed.

Step 7

In this step your goal is to add a status bar.

Tmux reserves the bottom line of the terminal for a status bar. Yours should show the session name on the left, the list of windows in the middle (just one window for now), and the current time on the right. The status bar should update as things change, including ticking over as the time changes, and the shell should now render in the remaining lines above it without ever overwriting the bar.

Testing: Attach to a session. You should see the status bar on the bottom line showing the session name and the time. Run a command that produces lots of output, such as:

seq 1 100

The output should scroll in the area above the status bar and the bar should stay intact. Watch for a minute and confirm the clock updates.

Step 8

In this step your goal is to support multiple windows within a session.

A window is like a tab: each window has its own shell, but only one window is visible at a time. Pressing the prefix key followed by c should create a new window with its own shell and switch to it. The status bar should list all the windows with their numbers, starting from 0, and indicate which one is currently active.

Testing: Attach to a session and run echo window zero. Press Ctrl+b c. You should get a fresh shell and the status bar should now show two windows, with the second one marked as active. Run echo window one in the new window. Create a third window with Ctrl+b c and confirm the status bar shows three windows.

Step 9

In this step your goal is to support switching between windows.

With multiple windows created, the user needs to move between them:

• The prefix key followed by a digit (0 to 9) should switch directly to that window number.

• The prefix key followed by n should switch to the next window, wrapping around from the last back to the first.

• The prefix key followed by p should switch to the previous window, wrapping the other way.

When switching to a window, its screen content should be restored exactly as it was, and the status bar should update to show the newly active window.

Testing: Using the session from Step 8, press Ctrl+b 0. You should see the first window with window zero still on screen. Press Ctrl+b 1 and verify you see window one. Press Ctrl+b n twice and Ctrl+b p twice, checking the status bar each time to confirm you cycle through the windows in order and wrap around at each end. Try Ctrl+b 9 and confirm switching to a window that doesn’t exist is handled gracefully.

Step 10

In this step your goal is to split a window into two panes, one above the other, with the prefix key followed by ".

This is where your multiplexer really starts to earn its name. Pressing Ctrl+b " should split the current pane horizontally, creating a new pane below it with its own PTY and shell. Both panes run at the same time: output arriving in either pane should render in that pane’s region of the screen, whichever pane is active.

Draw a border between the panes using box-drawing characters (such as ─ and │), and make sure each shell knows its actual pane size so that programs running inside it wrap and draw correctly. Keyboard input should go to the active pane.

Testing: Attach to a session and press Ctrl+b ". The screen should split into a top and bottom pane with a horizontal border between them. In the active pane run:

seq 1 5

The output should appear only in that pane. Start a long-running command in one pane, for example top, and confirm it keeps updating its own region while you type in the other pane after switching (for now you can test switching in the next step, or verify the inactive pane keeps rendering). Run ls and confirm the columns fit the pane width, not the full terminal width.

Step 11

In this step your goal is to split a window into two panes side by side with the prefix key followed by %.

Pressing Ctrl+b % should split the current pane vertically, creating a new pane to the right with its own shell. Vertical borders should be drawn between side-by-side panes, and splits should compose: the user should be able to mix " and % splits to build up a grid of panes, each rendering independently in its own region.

Testing: In a fresh window press Ctrl+b %. You should see two panes side by side with a vertical border. Run seq 1 5 and confirm the output stays within the pane. Now press Ctrl+b " and confirm the current pane splits into top and bottom, giving you three panes. Run a different command in each and verify all three render correctly in their own regions.

Step 12

In this step your goal is to support navigating between panes with the prefix key followed by the arrow keys.

Pressing Ctrl+b followed by an arrow key should move the active pane in that direction: up, down, left, or right. Your tool should give a visual indication of which pane is active, for example by highlighting its border. Keyboard input should always go to the active pane.

Testing: Build a layout with at least three panes. Use Ctrl+b followed by the arrow keys to move between them, confirming the active pane indicator follows you. In each pane type a different command, such as echo top left, and verify the input lands in the pane you expect. Try navigating in a direction where there is no pane and confirm nothing breaks.

Step 13

In this step your goal is to support resizing panes with the prefix key followed by Ctrl and the arrow keys.

Pressing the prefix followed by Ctrl+Up, Ctrl+Down, Ctrl+Left, or Ctrl+Right should move the border of the active pane in that direction, making it bigger or smaller. The neighbouring pane should shrink or grow to match, borders should be redrawn, and the shells in the affected panes should be told their new size so that running programs adapt.

Be warned if you’re working on a Mac Ctrl+Arrow gets captured by the OS, you’ll want to add support for the Alt key just like tmux itself does.

Testing: Split a window with Ctrl+b ". Press Ctrl+b Ctrl+Up a few times and watch the border move up, making the bottom pane taller. Run top in one pane, resize it, and confirm top redraws itself to fit the new size. Repeat with a side-by-side split using Ctrl+Left and Ctrl+Right. Confirm you can’t resize a pane away to nothing.

Step 14

In this step your goal is to handle the terminal itself being resized.

When the user resizes their terminal emulator window, your client should notice, tell the server, and the server should redistribute the space: recalculating the pane layout, informing every shell of its new size, redrawing all borders, and keeping the status bar on the bottom line.

Testing: Attach to a session with several panes and resize your terminal window by dragging its corner. The layout should adapt smoothly: panes share the new space, borders are redrawn correctly, and the status bar stays on the bottom row. Run top in a pane while resizing and confirm it adapts. Make the terminal very small and then large again, and confirm nothing is corrupted.

Step 15

In this step your goal is to add a command prompt.

Pressing the prefix key followed by : should open a prompt on the status bar line where the user can type a command, edit it, and press Enter to run it, or Escape to cancel. Support the commands your multiplexer already understands, for example new-window, split-window, list-sessions, and kill-session. Unknown commands should show an error message on the status line rather than doing anything surprising.

Testing: Press Ctrl+b : and the status bar should turn into a prompt. Type new-window and press Enter; a new window should be created just as if you had pressed Ctrl+b c. Open the prompt again, type nonsense, and confirm you get an error message on the status line. Finally, press Ctrl+b : then Escape and confirm the prompt closes without running anything.

Step 16

In this step your goal is to add copy mode, so users can scroll back through a pane’s output history.

Each pane should keep a scrollback buffer of output that has moved off the top of the screen. Pressing the prefix key followed by [ should enter copy mode for the active pane: the pane freezes, and the user can scroll back through the history using the arrow keys and page up/page down. Give a visual indication that copy mode is active, such as a position indicator showing where the user is in the history. Pressing q should leave copy mode and return the pane to live output.

Testing: In a pane, generate more output than fits on the screen:

seq 1 500

Press Ctrl+b [ and scroll up. You should be able to scroll back and find line 1. Scroll down again, then press q and confirm the pane returns to the live view and typing works normally. Verify that output arriving in another pane doesn’t disturb your position while you’re scrolling.

Going Further

Once you have the core multiplexer working, here are some ideas to take it further:

• Add text selection and copying in copy mode, with a paste buffer and a paste key binding (prefix followed by ]).

• Support renaming sessions and windows, and show the custom names in the status bar.

• Add a session chooser (like tmux’s prefix followed by s) for switching sessions interactively.

• Support a configuration file that sets key bindings, colours, and status bar content, like .tmux.conf.

• Add zooming a pane to full screen and back with prefix followed by z.

• Implement mouse support: clicking to select a pane, dragging borders to resize, and scrolling with the mouse wheel.

• Add synchronised panes, where input is sent to every pane in the window at once.

• Support layout presets such as even-horizontal, even-vertical, and tiled.

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #126 - Background Removal Tool

This challenge is to build your own image background removal tool that takes a photograph of a person and removes the background, keeping the person and making everything else transparent.

You have probably seen this feature everywhere by now. You give it a photo of yourself, and a moment later you have a clean cut-out that you can drop onto a new background, turn into a profile picture, or use in a design. It feels like magic, but underneath it is a satisfying mix of image decoding, machine learning, and careful pixel work. Getting the body right is the easy part. Handling the fine wisps of hair less so!

In this challenge you are going to build a tool that does all of this on the user’s own device, with no hosted service doing the heavy lifting for you. By the end you will have a tool that can process a single image or a whole directory tree of them.

The Challenge - Building a Background Removal Tool

You are going to build a tool that accepts a photograph of a person, works out which pixels belong to the person and which belong to the background, and writes out a new image where the person is preserved and everything else is fully transparent.

The interesting parts of this challenge are the details. You have to decode common image formats, automatically find the person without any help from the user, produce edges that are smooth enough to look natural against a new background, and do all of the clever work on the user’s own device rather than shipping the image off to someone else’s server/API. Once the single-image case works, you will scale it up to handle whole directories at once while behaving gracefully when one image causes trouble.

This is an intermediate to advanced challenge. The core pipeline is the same whatever you build it with, so pick whatever you are most comfortable with for image processing and, if you go down the machine learning route, running a model without calling out to a hosted service.

Step Zero

In this introductory step you are going to set your environment up ready to begin developing and testing your solution.

Choose your target platform and programming language. I would encourage you to pick a language that has good libraries for decoding and encoding images, because you do not want to be writing a JPG decoder by hand this week. You will also want to think ahead to the person detection step. Many languages have bindings for running pre-trained segmentation models, so it is worth checking what is available before you commit.

It is entirely up to you how the user interacts with your tool. A command-line tool, a native desktop application, and a web app are all perfectly good choices, so pick whichever suits you and the way you like to work.

One constraint to keep in mind from the very start is that the background removal itself must not be handed off to a hosted third-party API. The important thing is that no one else’s server does the work of finding the person and removing the background. Where that computation runs is up to you, as long as it is under your control: running a model you have downloaded on the user’s machine is fine, and so is compiling it to WebAssembly and running it entirely in the browser so the image never leaves the user’s device. What is not in the spirit of things is uploading the image to a hosted background removal service and getting the answer back. Part of the fun here is understanding how the magic actually works, so you want to be the one doing it.

If you go down the machine learning route, be aware that a good segmentation model can be a sizeable download, often in the region of a couple of hundred megabytes. If you are on a slow or metered connection it is worth pulling the model down early so it is ready when you reach the detection step. This is worth bearing in mind for a browser-based build too, where that download becomes something the user’s browser has to fetch.

Gather a small set of test photographs to work with. You will want a few pictures of people, ideally with a mix of easy cases (a person against a plain wall) and harder ones (someone with loose or curly hair, or a busy background). Keep them somewhere handy, because you will run your tool against them again and again.

Step 1

In this step your goal is to load an image and write it back out as a PNG with the same dimensions and an alpha channel.

Before we go any further, it is worth being clear about what the alpha channel is, because it is central to the whole challenge. A colour image is made up of pixels, and each pixel usually has three values: how much red, green, and blue it contains. Each of those values normally runs from 0 to 255.

The alpha channel adds a fourth value to each pixel, on that same 0 to 255 scale, that records how opaque or transparent the pixel is. An alpha of 255 means the pixel is fully opaque and you see its colour completely. An alpha of 0 means it is fully transparent and you see straight through it to whatever is behind, as if that pixel were not there at all. Values in between give partial transparency, which is exactly what you will need later for soft edges.

So when this step asks you to make the whole image fully opaque, you are giving every pixel an alpha of 255. JPG images do not have an alpha channel, which is why they can never be transparent, but PNG images do. That is the reason the output of this challenge is always a PNG: it is the transparency, stored in the alpha channel, that lets the background disappear while the person stays.

Your tool should accept a single input image in either JPG or PNG format. Once it has decoded the image, it should write out a PNG that has an alpha channel and exactly the same pixel dimensions as the input. For now, do not try to remove anything. Just keep every pixel as it is and make the whole image fully opaque, giving every pixel the maximum alpha of 255. This gets your decode and encode pipeline working end to end before you add any of the clever detection logic.

Getting this foundation right matters. The rest of the challenge builds on top of a reliable “read an image, produce a same-sized PNG with alpha” pipeline, so it is worth making sure this is solid.

Testing: Run your tool against a JPG and a PNG from your test set. Open the output files and confirm they look identical to the inputs. To check the dimensions and the alpha channel without having to eyeball them, reach for a command-line tool like exiftool:

exiftool -ImageWidth -ImageHeight -ColorType output.png

You should see the width and height matching your input exactly, and a colour type of RGB with Alpha, which confirms the alpha channel is present. If you prefer ImageMagick, magick identify -format '%wx%h %[channels]\n' output.png does the same job, printing something like 1200x800 srgba, where the a on the end tells you there is an alpha channel.

Step 2

In this step your goal is to automatically detect the person in the image and produce a mask.

This is the heart of the challenge. Your tool needs to work out, for every pixel, whether it belongs to the person or to the background, and it must do this automatically. The user should not have to click, draw, or mark anything to point out where the person is. Given a photo, your tool figures it out on its own.

The output of this step is a mask, which is simply a second image the same size as the input where each pixel says how much that location belongs to the person. A common approach is to run a pre-trained person segmentation model yourself and use its output as your mask. Remember the no-hosted-service rule from Step Zero applies here.

You do not have to invent the detection method from scratch, and there is a lot of prior art to lean on. Here are some starting points worth a look, and it is up to you which route you take:

• U²-Net is a salient object detection network that powers a lot of open-source background removers. See the paper “U^2-Net: Going Deeper with Nested U-Structure for Salient Object Detection” (arxiv.org/abs/2005.09007) and the code at github.com/xuebinqin/U-2-Net. The popular rembg library (github.com/danielgatis/rembg) is built on it and makes a good reference point for what “good enough” looks like.

• DeepLabv3 and DeepLabv3+ are general semantic segmentation models that include a “person” class and ship ready to use in libraries such as torchvision. The paper is “Rethinking Atrous Convolution for Semantic Image Segmentation” (arxiv.org/abs/1706.05587).

• MediaPipe Image Segmenter and Selfie Segmentation from Google (ai.google.dev/edge/mediapipe/solutions/vision/image_segmenter) are designed to run efficiently on-device, including in the browser through WebAssembly, which makes them a natural fit if you are building a web app.

• Segment Anything (SAM) from Meta (segment-anything.com, paper at arxiv.org/abs/2304.02643) is a powerful, more recent option that can produce very clean masks.

• MODNet (arxiv.org/abs/2011.11961, github.com/ZHKKKe/MODNet) and BiRefNet (github.com/ZhengPeng7/BiRefNet) focus on high-quality matting and fine edges, which will be especially handy when you reach the hair in Step 4.

Whichever you pick, you will typically load and run it with a runtime such as ONNX Runtime, PyTorch, or TensorFlow, all of which can run on the user’s own device and so keep you within the no-hosted-service rule.

If you would rather build the detector yourself from scratch instead of reaching for a trained model, that is a brilliant way to learn, and the trick is to start with the easy case and work up. A good first option is to lean on the plain background in your easy test photo: treat the pixels around the border of the image as background, then grow outwards to every connected pixel of a similar colour, and whatever is left in the middle is your person. That is really just two classic ideas stitched together, colour keying and a flood fill, and it will give you a usable mask on a simple photo with no machine learning at all. From there, a natural step up is GrabCut, a well-known graph-cut foreground extraction algorithm (the paper is “GrabCut: Interactive Foreground Extraction using Iterated Graph Cuts”, dl.acm.org/doi/10.1145/1015706.1015720, and there is a clear walkthrough in the OpenCV docs at docs.opencv.org/4.x/d8/d83/tutorial_py_grabcut.html). GrabCut normally wants a rough box around the subject, which you can supply automatically by assuming the person is roughly central or by running a lightweight face detector such as an OpenCV Haar cascade to locate them. Other classic building blocks worth knowing are Otsu’s thresholding (en.wikipedia.org/wiki/Otsu%27s_method) and the watershed algorithm. A from-scratch approach will not match a modern model on messy backgrounds or fine hair, but it will teach you a great deal about why those models exist.

To make this easy to inspect, have your tool save the mask out as its own greyscale image while you are developing. That way you can look at it directly and see how well the detection is working before you use it for anything.

Testing: Run your detection against your test photos and save the masks. Open them and check that the person shows up as the bright, filled-in region and the background shows up as dark. Try your easy photo first (a person against a plain wall) and then a harder one. The mask does not have to be perfect yet, but the overall shape of the person should be clearly recognisable.

Step 3

In this step your goal is to use the mask to make the background transparent.

Now that you have a mask, combine it with the original image. Every pixel that the mask says belongs to the person should keep its original colour value untouched. Every pixel that belongs to the background should be made fully transparent by setting its alpha to zero. The result is a PNG, still the same dimensions as the input, where the person floats on a transparent background.

Testing: Run the full pipeline on your test photos and open the output PNGs. The quickest visual check is to composite each cut-out onto a bright, solid colour that contrasts with the original photo, for example a strong red or green. With ImageMagick you can build that composite from the command line:

magick output.png -background magenta -flatten check.png

Open check.png and the person should appear against magenta with none of the original background showing through, and their colours should match the original photo.

You can also automate a few checks with ImageMagick so you do not have to rely on your eyes at all:

# There should now be some transparent pixels, so this prints False
magick identify -format '%[opaque]\n' output.png

# A corner pixel is almost always background, so its alpha should be 0
magick output.png -format '%[pixel:p{0,0}]\n' info:

# The average alpha across the image should sit between 0 and 1,
# roughly the fraction of the picture the person fills
magick output.png -alpha extract -format '%[fx:mean]\n' info:

The first command should print False, confirming the image is no longer fully opaque. The corner pixel should come back fully transparent, something like srgba(0,0,0,0). The mean alpha should be greater than 0 and less than 1, which tells you some of the picture has been kept and some made transparent, rather than the whole thing being wiped out or left untouched.

Step 4

In this step your goal is to produce smooth edges around the person, including the tricky areas like hair.

If you composited your Step 3 output onto a new background, you probably noticed a hard, jagged edge, and hair almost certainly looked like it had been cut out with blunt scissors. Real photographs do not have a crisp line where a person ends and the background begins. Strands of hair are semi-transparent, and edges blend gradually. To make a cut-out that composites cleanly onto a new background, the alpha channel around the person needs to transition smoothly rather than jumping straight from fully opaque to fully transparent.

Refine the edges of your cut-out so that boundary pixels take on partial transparency. Pay special attention to hair, which is the classic hard case and the thing that separates a convincing cut-out from an obvious one.

Testing: Composite your refined output onto a new background again, ideally one that is quite different in colour from the original. The edges should look soft and natural, and wisps of hair should blend into the new background rather than showing a harsh outline or a fringe of the old background colour. Compare a before and after against your Step 3 result on a photo of someone with loose hair. The difference should be obvious. It is also worth peeking at the alpha channel on its own, for example with magick output.png -alpha extract edge-check.png, where you should see a soft grey halo around the person rather than a hard black-and-white boundary.

Step 5

In this step your goal is to give the user control over the output filename. We’re half way so this is an easy one to give you a quick rest.

Your tool should let the user specify where the output should be written. When they do not specify a name, your tool should choose a sensible default derived from the input filename. Crucially, the default must never overwrite the input file. Someone processing photo.jpg should get their cut-out saved somewhere else, not have their original silently replaced.

Testing: Run your tool on an input without specifying an output name and confirm a sensibly named output file appears and that the original input is untouched. Then run it again specifying an explicit output name and confirm the file lands exactly where you asked. For good measure, try an input that is already a PNG and confirm the default output name does not collide with it.

Step 6

In this step your goal is to handle invalid input files cleanly.

Not every file with a .jpg or .png extension is actually a valid image, and users will inevitably point your tool at something that is not an image at all. When your tool is given a file that is not a valid JPG or PNG, it should report a clear error message explaining what went wrong and fail with a non-zero exit status (or the equivalent error signal for whatever interface you built). It should not crash with an unhelpful stack trace, and it should certainly not write out a broken or empty output file.

One thing that catches people out here is truncated JPEGs. Many mainstream JPEG decoders are deliberately tolerant and will happily decode a truncated file into a partial image without raising any error at all, so a test you expect to fail passes instead. If you hit this, you may need to validate the file yourself rather than relying on the decoder, for example by checking that a JPEG ends with its end-of-image marker. Truncated PNGs, by contrast, tend to fail cleanly almost everywhere, so they are a gentler test case.

Testing: Start with the easy case. Create a text file, rename it to have a .jpg extension, and run your tool against it. You should see a clear error message and, if your tool is a command-line one, a non-zero exit status.

Next, make a truncated JPEG by keeping only the first slice of a valid one, which chops off the end of the image data and the end-of-image marker:

head -c 2000 valid.jpg > truncated.jpg

Do the same for a PNG:

head -c 500 valid.png > truncated.png

Run your tool against truncated.jpg and truncated.png. Both should be rejected with a clear error and a non-zero exit status, and no output file should be produced in any of these cases. The truncated JPEG is the one to watch, since a tolerant decoder may quietly accept it. If that happens, this is exactly the case the note above is warning you about, and it is a sign you need to validate the file yourself.

Step 7

In this step your goal is to process an entire directory of images.

Instead of just a single image, your tool should also accept a directory as input. When given a directory, it should find and process every JPG and PNG image within it, and it should do so recursively, descending into subdirectories to find images nested further down. Each image gets the same background removal treatment you built in the earlier steps.

Testing: Create a directory with a few images in it, plus a subdirectory containing a couple more. Point your tool at the top-level directory and confirm that every image, including the ones in the subdirectory, gets processed. Drop a non-image file into the directory as well and confirm your tool sensibly ignores it rather than trying to process it.

Step 8

In this step your goal is to mirror the input directory structure in the output.

When processing a directory, the output needs to be organised, not dumped into a single flat folder where files from different subdirectories collide. Your tool should mirror the input directory structure under the output directory. In other words, each output image should be written to the same relative path as its source image. An image at people/holiday/beach.jpg in the input should produce a output file at the matching relative location under the output directory.

Testing: Run your tool against the nested directory from Step 7. Inspect the output directory and confirm its structure mirrors the input. The output image for the image in the subdirectory should sit inside a matching subdirectory in the output folder, at the same relative path as its source. There is a subtle collision to watch for as well: because every output is a PNG, an input folder containing both portrait.jpg and portrait.png would map both output images to the same output name and one would clobber the other. Put a matching pair like this in a folder, run your tool, and decide how you want to handle it so that no image is silently lost.

Step 9

In this step your goal is to keep going when one image fails.

When you are batch processing a whole directory, it is frustrating if the entire run stops dead because one image out of a hundred was corrupt. Your tool should continue processing the remaining images when one of them fails, rather than aborting the whole run. Then, once it has finished working through everything, it should report the failures at the end so the user knows exactly which images did not get processed and why. Since a single bad file fails with a non-zero exit status (as you built in Step 6), a batch run that had any failures should also finish with a non-zero exit status once it has processed everything, so that scripts calling your tool can tell the run was not completely clean.

Testing: Build a directory containing several valid images plus one invalid file (reuse the corrupt file from Step 6). Run your tool over the directory and confirm that all of the valid images are processed and written out correctly. At the end of the run, you should see a clear summary listing the file that failed, and, for a command-line tool, a non-zero exit status. Confirm that the one bad file did not prevent any of the good ones from being processed.

Going Further

Once you have the core tool working, here are some ideas to take it further:

• Add a preview or “trimap” mode that shows the mask and the refined edges side by side so you can debug tricky images.

• Let the user replace the transparent background with a solid colour or another image, turning your tool into a full background replacement tool.

• Support additional input formats such as WebP or HEIC.

• Add a progress indicator when batch processing large directories, and parallelise the work across multiple CPU cores or a GPU.

• Try more than one segmentation model and let the user choose, or compare their results to see which handles hair and fine detail best.

• Handle photos with more than one person in them, and give the user a way to keep only a specific person.

• Benchmark how long each image takes and see where the time goes, then look for ways to speed up the slowest stage.

• If you built a native tool, try a browser-based version compiled to WebAssembly (or the other way round) so people can use it without installing anything.

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Thank You For All The Feedback!

Before we kick of this week’s coding challenge, I’d like to thank everyone for all the feedback. I got a lot and am still working my way through it. I really appreciate everyone taking the time to get let me know your thoughts.

Thank you! 🙏😍

Coding Challenge #125 - Online Diff Viewer

This challenge is to build your own online diff viewer that compares two text inputs and displays the differences in a clear, visual format.

If you’ve ever used GitHub, GitLab, or any code review tool, you’ve seen a diff viewer. It’s the side-by-side or unified view that shows you exactly what changed between two versions of a file - green for additions, red for deletions. But how do these tools actually work? In this challenge you’ll find out by building one yourself. You’ll implement a diff algorithm, render the results in multiple visual formats, and add all the quality-of-life features that make a diff viewer truly useful.

The Challenge - Building Your Own Online Diff Viewer

You’re going to build an online diff viewer that runs entirely in the browser. Users will paste, upload, or drag-and-drop two pieces of text, and your tool will show them exactly what changed. Over several steps you’ll add side-by-side and unified views, syntax highlighting, navigation tools, export options, and accessibility features.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

This is a web application, so you’ll need HTML, CSS, and JavaScript (or a framework you’re comfortable with). You’ll also need to understand how a diff algorithm works.

Take some time to study the Myers diff algorithm. It’s the algorithm used by Git and many other tools. Your diff viewer will need to take two sequences of lines, compute the shortest edit script that transforms one into the other, and produce a set of additions, deletions, and unchanged lines.

Step 1

In this step your goal is to compute and display a basic line-level diff between two text inputs.

Create a simple interface with two text areas for input and a diff button. When the user clicks the button, your tool should compute the diff between the two inputs and display the result. Colour-code additions in green, deletions in red, and leave unchanged lines uncoloured or in a neutral colour.

At this stage a simple vertical, unified-style output is fine - we’ll add proper views in the next steps.

Testing: Paste the following into your two inputs:

Left (original):

function greet(name) {
  console.log("Hello, " + name);
}

function farewell(name) {
  console.log("Goodbye, " + name);
}

Right (modified):

function greet(name) {
  console.log("Hello, " + name + "!");
  console.log("Have a great day!");
}

function farewell(name) {
  console.log("Goodbye, " + name + "!");
}

You should see the added lines in green and any lines that exist only in the original in red. Unchanged lines like function greet(name) { should appear without colour.

Step 2

In this step your goal is to add a side-by-side diff view with properly aligned rows.

Side-by-side views show both inputs next to each other, with matching lines on the same row and gaps where lines were added or removed. This is the most intuitive way to view diffs for many people.

Your side-by-side view should:

• Show the original on the left and the modified version on the right.

• Align unchanged lines on the same row.

• Leave blank gaps on one side when lines are added or removed.

• Show line numbers for both sides.

Testing: Use the same test data from Step 1. In the side-by-side view, the two console.log lines in the modified version should appear opposite blank space on the left, and the original console.log("Goodbye, " + name) line should appear opposite blank space on the right.

Step 3

In this step your goal is to implement a unified diff view and let users toggle between the two views.

The unified diff view is the format used by Git and many command-line tools. It shows a single continuous output where additions are prefixed with +, deletions with -, and context lines with a space. Hunks are separated by @@ headers showing the line ranges.

Your tool should allow the user to switch between side-by-side and unified views with a simple toggle button or tab.

Testing: Switch between views with the same test data. In unified view, verify that added lines start with +, deleted lines with -, and context lines with a space. The hunk header format should show something like @@ -2,5 +2,6 @@.

Step 4

In this step your goal is to display diff summary statistics.

Users want to quickly understand the scope of changes. Your viewer should show a summary panel that displays:

• Lines added

• Lines removed

• Lines unchanged

• Total number of changes

Update this summary whenever the diff is recomputed.

Testing: With the test data from Step 1, verify your summary shows 2 lines added, 1 line removed, and the correct number of unchanged lines. Change the inputs and confirm the statistics update accordingly.

Step 5

In this step your goal is to add syntax highlighting for at least five common programming languages.

Diff output is much easier to read when the code is syntax-highlighted. Your tool should highlight at least JavaScript, Python, HTML, CSS, and JSON.

The syntax highlighting should apply to both the original and modified sides. It should also work with the colour coding from the diff - so a highlighted Python keyword that appears in an added line would show with both the syntax colour and a green background.

Your viewer should attempt to auto-detect the language based on file extension or content, and also allow the user to manually override the language selection.

Testing: Create test files in each supported language and verify the highlighting works correctly. Try mixing file types (JavaScript on the left, Python on the right) and verify the language override lets you pick the correct one. Test auto-detection by uploading a .py file and confirming Python highlighting is applied.

Step 6

In this step your goal is to support multiple input methods for loading text into the diff viewer.

Typing or pasting into text areas works, but users will also want to:

• Upload files from their computer using a file picker.

• Drag and drop files directly onto the viewer.

• Fetch content from a URL.

When a file is loaded, display basic metadata such as the filename, file size, detected language, and line count.

Testing: Create two text files on your computer and upload them using the file picker. Verify the content appears correctly in both sides and the metadata is displayed. Test drag and drop by dragging a file from your desktop onto the viewer. Test URL fetch by providing the URL of a publicly accessible text file.

Step 7

In this step your goal is to add diff precision modes: smart, word, and character-level highlighting.

So far your diff works at the line level. But within a changed line, only part of the line might have changed. Precision modes let users see exactly which words or characters changed:

• Smart mode: Your best guess at what changed - typically word-level changes.

• Word mode: Highlight changed words within a line.

• Character mode: Highlight individual changed characters within a line.

Testing: Use a line like changing Hello World to Hello Universe. In word mode, only World should be highlighted on the left and Universe on the right. In character mode, individual characters like W vs U should be highlighted. Verify smart mode makes a sensible choice.

Step 8

In this step your goal is to add diff control options that let users customise what counts as a change.

Sometimes you want to focus on meaningful changes and ignore formatting differences. Add toggleable options for:

• Ignoring whitespace changes (e.g., extra spaces that don’t affect the code).

• Ignoring case differences (e.g., HELLO vs hello).

• Ignoring blank lines (empty lines added or removed).

• Ignoring indentation-only changes (tabs vs spaces, changes in indent level).

• Ignoring line ending differences (CRLF vs LF).

These options should be available as checkboxes or a settings panel and take effect immediately when changed.

Testing: Create two inputs that differ only in whitespace. With “ignore whitespace” enabled, the diff should show no changes. Disable it and the changes should reappear. Repeat for each option.

Step 9

In this step your goal is to improve large diff readability by letting users hide unchanged sections.

When there are hundreds of lines of unchanged code between changes, scrolling through all of them is tedious. Add the ability to collapse unchanged sections, showing a clickable indicator like “... 47 unchanged lines ...”. Users should be able to click the indicator to expand the section.

Testing: Create two files with at least 50 unchanged lines between two change blocks. Verify the middle section is collapsed by default or can be collapsed, and that clicking it expands to show the full content.

Step 10

In this step your goal is to add a line wrap toggle and synchronised scrolling.

When lines are long, users need to be able to toggle line wrapping on and off. With wrapping off, a horizontal scrollbar lets them view long lines without breaking formatting.

Synchronised scrolling keeps both sides of the side-by-side view scrolling together. When the user scrolls one pane, the other follows. Users should also be able to disable this if they prefer independent scrolling.

Testing: Create a file with very long lines (200+ characters) and verify toggling line wrap works correctly. In side-by-side view, scroll one pane and verify the other scrolls in sync. Disable sync scrolling and verify the panes scroll independently.

Step 11

In this step your goal is to add comprehensive navigation features.

Navigating a large diff efficiently is critical. Add the following:

• Next change and previous change buttons that jump between diff hunks.

• First change and last change buttons that jump to the start and end of the diff.

• A change navigator panel that lists all change blocks, showing the line range and a preview. Clicking a block in the panel should jump to that location.

• Gutter markers (coloured indicators in the margin) for additions, deletions, and modifications.

• Keyboard shortcuts for all navigation actions (e.g., J for next, K for previous).

Testing: Create a diff with at least 10 change blocks spread across the document. Verify each navigation control works correctly. Test all keyboard shortcuts and confirm the change navigator panel updates to highlight the current change as you navigate.

Step 12

In this step your goal is to handle large inputs efficiently.

Diffs of 10,000+ lines can freeze the browser if not handled carefully. To keep the UI responsive:

• Run the diff computation in a Web Worker so the main thread stays responsive.

• Use virtualised rendering - only render the lines currently visible in the viewport instead of all 10,000 at once.

• Show a progress indicator while the diff is being computed.

Testing: Generate two large text files (10,000+ lines) with differences scattered throughout. Verify the diff computation doesn’t freeze the browser - you should be able to click buttons and scroll while it’s processing. Verify scrolling through the output is smooth.

Step 13

In this step your goal is to add export and copy options.

Users need to get the diff output out of your tool and into other tools or documents. Add the ability to:

• Copy the full diff output to the clipboard.

• Copy individual change blocks.

• Export the diff as a unified patch file, HTML, Markdown, and JSON.

• Generate Git-compatible patch output that can be applied with git apply.

Testing: Create a diff and test each export format. For unified patch, verify you can save it to a file and apply it with git apply. For HTML and Markdown, verify the output renders correctly with colour coding preserved. For JSON, verify the structure includes all change information.

Step 14

In this step your goal is to make your diff viewer a Progressive Web App (PWA) that works offline.

Turn your diff viewer into a PWA with a service worker so users can use it without an internet connection. Additionally:

• Persist the user’s recent sessions using localStorage or IndexedDB.

• Show a history of previous diffs the user can revisit.

• Let users save named diff sessions they can come back to later.

Testing: Load your diff viewer, then disconnect from the internet. Verify the page still loads and the diff viewer works. Run some diffs, close the browser, reopen it offline, and verify your history is still there.

Step 15

In this step your goal is to ensure your diff viewer is fully accessible and works well on all devices.

Your viewer should be usable by everyone, regardless of ability or device. This means:

• All functionality should be keyboard-accessible.

• Proper ARIA labels and roles should be used throughout.

• Screen readers should announce change summaries when the diff is computed.

• Line numbers and file headers should remain visible (stick to the top) when scrolling.

• The layout should be responsive - usable on desktop, tablet, and mobile.

• Clear empty states and error messages should be shown when something goes wrong.

Testing: Navigate the entire viewer using only the keyboard. Run a screen reader and verify it announces the diff results. Resize the browser to mobile width and verify everything remains usable. Test with an empty input to verify the empty state message is shown.

Step 16

In this step your goal is to add theme support with light, dark, and high-contrast modes.

Your diff viewer should respect the user’s system theme preference by default. Add a theme selector that lets users choose between:

• Light mode

• Dark mode

• High-contrast mode (for users with visual impairments)

The theme should apply to all UI elements including the diff output, syntax highlighting colours, and control panels.

Testing: Change your system theme and verify the viewer matches. Manually switch themes and verify all colours update correctly. Test high-contrast mode with a screen reader or accessibility testing tool.

Going Further

Once you’ve built the core diff viewer, consider adding these advanced features:

• AI-powered diff explanation: Use an LLM to explain what a change does in plain English.

• Code-review mode: Let users add comments and annotations to specific lines or change blocks.

• Semantic diffs: For JSON, compare the structure rather than the text. The same for HTML - compare the DOM tree.

• Syntax validation: Highlight JSON or HTML that is syntactically invalid in the input.

• Three-way merge view: Show a base version and two modified versions, useful for resolving merge conflicts.

• Image diff: Support uploading two images and showing a visual diff (pixel-by-pixel or structural comparison).

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

I’d love your help with something.

I’ve noticed that fewer people are opening and reading Coding Challenges each week. That tells me one of two things is probably true:

Either life is busy and inboxes are noisy, which I completely understand.

Or the newsletter isn’t giving you enough value right now.

I don’t want to keep sending you things that aren’t useful. I’d much rather shape this into something you genuinely look forward to opening.

So I have one question:

What would make Coding Challenges worth reading every week?

You can reply with a number if that’s easiest:

1. More coding challenges and practical projects

2. More AI, LLMs, and generative AI content

3. More software engineering, architecture, testing, and system design

Or reply with your own answer.

Some things I’d be happy to write about include:

• Learning to code

• Learning AI

• Building AI products

• LLMs and generative AI

• Deploying AI systems.

• System design and architecture

• Testing and maintenance

• Engineering excellence

• Programming languages

• Databases

• Cloud platforms

• AI tooling

• Developer productivity

I know that’s a broad list, so don’t feel limited by it. The main thing I want to know is what would actually help you.

If you’re happy to share publicly, please leave a comment. It would help build a better conversation around where Coding Challenges should go next.

If you’d rather keep it private, just hit reply and email me directly.

And if this newsletter isn’t for you anymore, no hard feelings at all. Please unsubscribe. I’d rather not keep bothering people who don’t want these emails.

Thanks for reading, and thanks in advance if you reply.

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #124 - Du

This challenge is to build your own version of du, the classic POSIX disk usage estimation utility.

du is one of those utilities you reach for when your disk is filling up and you need to know where all the space has gone. It walks a directory tree, adds up how much disk space each directory consumes, and prints a summary. It handles symbolic links, hard links, glob exclusions, human-readable units, depth limits, and more. Building your own version is a fantastic exercise in filesystem traversal, careful bookkeeping of inodes, and writing a tool that plays nicely with the rest of the Unix ecosystem.

The Challenge - Building Du

In this challenge you’re going to build your own version of du, a disk usage estimation tool. Your tool will recursively walk directory trees, calculate disk usage for files and directories, and display the results in a variety of formats. It will be compatible with the POSIX du utility, which means you’ll be able to test your work directly against the system du.

The interesting parts of du are the details: counting disk blocks rather than byte sizes, avoiding double-counting hard-linked files by tracking inodes, following or skipping symbolic links, and presenting the data in ways that are actually useful when you’re trying to figure out why your disk is full.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

Choose your target platform and programming language. I’d encourage you to pick a language that gives you good access to filesystem metadata, things like file sizes, inode numbers, and modification times. Most general-purpose languages have libraries for this, so you should be well covered.

Before you start coding, spend some time with the system du. Run it on a few directories, try the various flags, and get a feel for the output format. Have a read through the POSIX du specification and the man page on your own machine (man du).

Create a small test directory structure to use throughout the challenge:

mkdir -p testdir/subdir1/deep
mkdir -p testdir/subdir2
echo "hello world" > testdir/file1.txt
echo "this is a slightly longer file with more content in it" > testdir/subdir1/file2.txt
echo "deep file" > testdir/subdir1/deep/file3.txt
echo "another file" > testdir/subdir2/file4.txt

We’ll use testdir as our playground throughout the challenge.

Step 1

In this step your goal is to recursively calculate and display the disk usage of directories.

Your tool should walk a directory tree starting from a given path, sum up the disk usage of all files it finds, and print the total for each directory. The output should be in 512-byte blocks (the POSIX standard default), with the size followed by the directory path. Directories should be printed in traversal order, deepest entries first, with parent directories printed after their children.

When no path is specified, your tool should default to the current directory.

A 512-byte block is the traditional Unix unit for disk usage. On most systems you can calculate this by taking the file size in bytes, dividing by 512, and rounding up. A zero-byte file still occupies one block on most filesystems, but for this step you can treat empty files as zero blocks to keep things simple.

Testing: Run your tool against the test directory:

ccdu testdir

You should see output similar to:

8	testdir/subdir1/deep
16	testdir/subdir1
8	testdir/subdir2
32	testdir

The exact numbers will vary depending on your filesystem and file contents. The key thing is that each directory shows a total, deeper directories appear before their parents, and the final line shows the total for the root of the tree.

Step 2

In this step your goal is to add the -h, -k, and -m flags for different size units.

By default your tool outputs in 512-byte blocks. The -h flag should switch to human-readable format, showing sizes with suffixes like K, M, G, and T (so 1.5M for 1.5 megabytes, for example). The -k flag should show sizes in kilobytes (1024-byte units), and the -m flag should show sizes in megabytes.

Have a play with the system du -h to see how it formats sizes, it rounds to a sensible number of decimal places and picks the largest unit that keeps the number readable.

Testing:

ccdu -h testdir
ccdu -k testdir
ccdu -m testdir

The -h output should show sizes like 4.0K or 1.2M. The -k output should show whole numbers in kilobytes. The -m output should show sizes in megabytes.

Step 3

In this step your goal is to add the -s (summary) flag.

When -s is passed, your tool should display only a total for each argument path, skipping the per-directory breakdown entirely. This is the flag people reach for when they just want to know how big a directory is without seeing the whole tree.

Testing:

ccdu -s testdir

This should produce a single line with the total size of testdir. Try it with multiple paths:

ccdu -s testdir testdir/subdir1

This should produce one summary line per path.

Step 4

In this step your goal is to add the -c flag for a grand total.

When -c is passed alongside multiple path arguments, your tool should print a grand total line at the end of the output, labelled total. This is useful when you want to know the combined size of several directories at once.

Testing:

ccdu -s -c testdir testdir/subdir1 testdir/subdir2

You should see a summary line for each path, followed by a total line that is the sum of all three.

Step 5

In this step your goal is to add the -a flag to show individual file sizes.

By default du only shows sizes for directories. With -a, it should also print a line for every file it encounters, showing that file’s disk usage alongside its path. Files appear in traversal order, interleaved with the directory totals.

Testing:

ccdu -a testdir

You should see lines for each file (file1.txt, file2.txt, etc.) as well as for each directory.

Step 6

In this step your goal is to add the -d / --max-depth flag to limit output depth.

The -d flag takes a number and restricts output to entries at or below that depth relative to the starting path. -d 0 is equivalent to -s (summary only). -d 1 shows the root directory and its immediate children, but nothing deeper. This is incredibly useful when you have a deep tree and only care about the top-level breakdown.

Testing:

ccdu -d 0 testdir
ccdu -d 1 testdir

The first should show only the total for testdir. The second should show testdir plus subdir1 and subdir2, but not subdir1/deep or any files.

Step 7

In this step your goal is to add the --exclude flag to skip files matching a glob pattern.

The --exclude flag takes a glob pattern (like *.log or tmp*) and any file or directory whose name matches the pattern should be skipped entirely, it should not be counted in any totals and should not appear in the output. Multiple --exclude flags can be specified.

Testing:

ccdu -a exclude "*.txt" testdir

This should produce no file lines (since all our test files are .txt) and all directory totals should be zero or minimal.

Step 8

In this step your goal is to add the -L flag to follow symbolic links.

By default du does not follow symbolic links, it reports the size of the link itself, not the target. With -L, symbolic links should be followed and the size of the target file or directory should be counted instead.

Create a symlink to test with:

ln -s testdir/subdir1 testdir/link-to-subdir1

Testing: Without -L, the symlink should contribute very little to the total. With -L, the contents of subdir1 should be counted again through the link path.

ccdu -s testdir
ccdu -L -s testdir

The second total should be larger because it includes the contents reachable through the symlink.

Step 9

In this step your goal is to avoid double-counting hard-linked files.

When two or more hard links point to the same inode, du should only count that file’s disk usage once. This means you need to track which inodes you have already counted and skip any file whose inode you have seen before.

Create a hard link to test with:

ln testdir/file1.txt testdir/subdir2/hardlink-to-file1.txt

Testing:

ccdu -a testdir

The total for testdir should not include file1.txt twice. The hard link should not appear in the output (even with -a) because its inode was already counted for the first link encountered, and du skips all subsequent links to the same inode entirely.

Step 10

In this step your goal is to add the -t / --threshold flag.

The -t flag takes a size value (with optional unit suffix like K, M, G) and filters the output so that only entries at or above that size are shown. This is useful for finding the big consumers of disk space in a large tree without wading through hundreds of tiny directories.

Testing:

ccdu -t 1K testdir
ccdu -t 10K -h testdir

The first should show only directories using at least 1 kilobyte. The second should show only directories using at least 10 kilobytes.

Going Further

Here are some ideas to take your du implementation further:

• Add the -inodes flag to count the number of inodes (files and directories) rather than disk usage

• Add the x / -one-file-system flag to skip directories that are on a different filesystem or mount point

• Add the -dereference-args flag to follow symlinks only for the command-line arguments, not for symlinks found during traversal

• Support the -block-size=SIZE flag to let the user specify an arbitrary block size

• Add coloured output when writing to a terminal, highlighting the largest directories

• Support JSON or CSV output format for piping into other tools or visualisation

• Add a -sort flag to order output by size, name, or time

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #123 - Database Driven LLM Wiki

This challenge is to build your own personal knowledge base tool - a system where an LLM agent reads your curated sources, extracts the key information, and builds you a living, interlinked wiki that grows smarter with everything you add.

Most people’s experience with LLMs and documents is RAG: upload files, ask questions, get answers stitched together from retrieved chunks. It works, but the LLM rediscovers knowledge from scratch every time. Ask a subtle question that spans five documents, and the system has to find and piece together fragments it’s seen before. Nothing accumulates.

LLM Wiki takes a different approach. Instead of just retrieving from raw documents at query time, an LLM agent incrementally builds and maintains a persistent wiki - a structured collection of markdown files that sits between you and your sources. When you add a new source, the agent reads it, extracts key information, and integrates it into the existing wiki: updating entity pages, revising topic summaries, noting where new data contradicts old claims, strengthening or challenging the evolving synthesis. The knowledge is compiled once and then kept current, not re-derived on every query.

You’re in charge of sourcing, exploration, and asking good questions. The LLM does all the grunt work: summarising, cross-referencing, filing, and the bookkeeping that makes a knowledge base actually useful over time.

This challenge is inspired by Andrej Karpathy’s LLM Wiki concept - a pattern for building personal knowledge bases using LLMs. Karpathy describes the idea in the abstract; our job is to build a working implementation.

Under the hood, the system stores vector embeddings and full-text indexes for every wiki page in Oracle AI Database. When you ask a question, it finds relevant pages using hybrid search, and an LLM synthesises an answer with citations. The agentic behaviour - ingestion workflows, multi-step querying, lint passes - is orchestrated using LangGraph’s state machine model, while LangChain handles the LLM integration. It’s a practical introduction to agent-based knowledge management, vector search, full-text search, and building tools that genuinely compound in value over time.

The Challenge - Building LLM Wiki

You’re going to build a personal knowledge base that an LLM agent writes and maintains for you. It starts by ingesting source documents into a wiki of markdown files, then lets you query it through a web interface or CLI chat. Step by step you’ll add wiki scaffolding, source ingestion, index and log management, vector storage, semantic retrieval, project management, linting, and a user interface. By the end, you’ll have a tool that genuinely helps you build and navigate a growing body of knowledge.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

You’ll need to make a few decisions and get some infrastructure running:

1. Set up your vector and full-text database. You’ll need Oracle Database 26ai running in a local Docker container. Pull the container-registry.oracle.com/database/free:latest image, start the container, and set a password for the admin account. You can find full setup instructions in the Oracle Database Free Get Started guide. Once the container is running, connect using a SQL client and verify you can create a table. Store all credentials in an environment file, not hardcoded anywhere.

docker pull container-registry.oracle.com/database/free:latest
docker run -d -p 1521:1521 -e ORACLE_PWD=<your-password> container-registry.oracle.com/database/free:latest

1. Choose your embedding model. You need a text embedding model that captures semantic meaning. Nomic’s nomic-embed-text is open source (Apache 2.0) and runs locally on CPU. It produces 768-dimensional vectors. Install it via Hugging Face: pip install sentence-transformers and load it as nomic-ai/nomic-embed-text-v2-moe. Any general-purpose embedding model with reasonable semantic quality will work - the key requirement is that it can capture the meaning of wiki pages well enough to find relevant ones for a given query.

2. Set up your LLM provider. You’ll need a language model for writing wiki pages, summarising sources, and answering questions. Any provider with a chat API will work - Anthropic, OpenAI, Google, Mistral, or a local model via Ollama. The model needs to be capable enough to write coherent markdown and extract structured information from source documents.

3. Set up LangChain and LangGraph. You’ll use LangChain for LLM integration and LangGraph for orchestrating the agent’s workflows. Install both: pip install langchain langgraph. LangChain handles the plumbing of prompting and response parsing. LangGraph handles the agentic flows - the multi-step processes of ingesting a source (read, extract, write summaries, update pages, update index, log), answering queries (search, read pages, synthesise), and running lint passes.

4. Pick a topic for your first wiki. Choose a domain you’re genuinely curious about and gather 50-100 source documents - articles, papers, blog posts - about it. Save them as markdown or plain text files. This will be your test corpus. Pick something where the articles naturally reference the same entities and concepts, so there are connections for your wiki to surface. A technical topic works well (e.g. database internals, a specific ML technique, a programming language), but any domain with depth will do - history, cooking, fitness, whatever interests you.

Testing: Verify your Oracle Database container is running and you can connect to it. Load your embedding model and generate a test embedding to confirm it returns a vector of 768 dimensions. Make a test call to your LLM API to confirm it returns a valid response. Verify your environment file is being read correctly and no credentials are in your source code.

Step 1

In this step your goal is to build the wiki scaffolding - the system that creates and manages the directory of markdown files that will become your knowledge base.

The wiki lives on disk as a directory of markdown files. Before you can ingest sources or answer questions, you need the infrastructure to create pages, write content to them, and link them together. Think of this as the file system layer of your knowledge base.

Start by defining what a new wiki looks like on disk. When you create a wiki, the system should scaffold a directory structure with subdirectories for different page types: summaries for source summaries, entities for pages about people/companies/concepts, topics for overview pages, and a raw directory for the original source files. Alongside the directories, create a schema file (call it SCHEMA.md) that defines the conventions for this wiki - what the directories are for, what naming conventions to use, what frontmatter fields pages should have, and how cross-references should be formatted.

Pages should carry YAML frontmatter at the top with metadata: title, type (entity, concept, summary, overview), date created, date updated, tags, and a list of sources the page draws from. The frontmatter makes pages queryable later and lets tools like Obsidian’s Dataview plugin generate dynamic views.

Cross-references between pages should use standard markdown links or wikilinks ([[Page Name]]). Which style you use is up to you, but the schema file should record the convention so the agent can be consistent. When one page references another, the agent should be able to follow that link, read the target page, and update both sides of the relationship.

Build a simple CLI that lets you create a new wiki, list existing wikis, and inspect a wiki’s structure - how many pages it has, what directories exist, and what the schema says. This CLI is just for development and testing; you’ll replace it with a proper interface later.

Testing:

• Create a wiki. Verify the directory structure and SCHEMA.md file were created on disk.

• Manually add a few test pages with frontmatter and cross-references. Verify the frontmatter parses correctly and links resolve to the expected paths.

• Create a second wiki with a different schema configuration (different frontmatter fields, different link style). Verify both wikis coexist and respect their own conventions.

• List your wikis and verify both appear.

• Delete a wiki directory manually and verify the listing correctly reflects its absence.

Step 2

In this step your goal is to build the source ingestion pipeline - the agent workflow that reads a source document and integrates its knowledge into the wiki.

This is the heart of the system. When you drop a new source into the raw directory and tell the agent to ingest it, a multi-step workflow begins. The agent reads the source, extracts key information, discusses the takeaways with you (in interactive mode), and then updates the wiki across multiple pages.

Model the ingestion workflow as a graph in LangGraph. Each node handles one concern: read the source, extract entities and concepts, identify claims and key information, write a summary page in the summaries directory, update or create entity pages for each entity found, update or create concept pages for each concept, revise topic overview pages, flag contradictions with existing content, update the index, and append an entry to the log. A single source might touch 10-15 wiki pages.

The agent should be able to detect contradictions. When a new source makes a claim that conflicts with something already in the wiki, the agent should note the discrepancy on the relevant page rather than silently overwriting or ignoring it. The user should be able to see where sources disagree and make their own judgement.

The agent should also identify gaps - entities or concepts referenced in the source that don’t yet have pages - and create stub pages or flag them for later attention.

Think carefully about how you prompt the LLM for each of these tasks. The quality of the wiki depends entirely on the quality of the extraction and synthesis. You’ll likely need different prompts for different page types: a summary page prompt, an entity page prompt, a concept page prompt, and so on. The schema file you built in Step 1 should guide these prompts.

The original source file should go into the raw directory and never be modified. The agent reads from it but never writes to it. This is your source of truth.

Testing:

• Ingest a single source document (a short article, 500-1000 words) into a test wiki. Verify the agent creates a summary page that captures the key points without hallucinating facts not in the source.

• Verify the agent creates or updates entity pages for the key people, companies, or concepts mentioned in the source.

• Verify the agent creates or updates topic overview pages that connect this source to existing knowledge (if the wiki already has content).

• Ingest a second source on the same topic that contradicts something in the first source. Verify the agent flags the contradiction on the relevant page.

• Check the wiki directory after ingestion. It should contain new or updated files in the summaries, entities, and topics directories. The raw directory should contain the original source unchanged.

• Ingest a source that mentions an entity not yet in the wiki. Verify the agent creates a stub page or flags the gap.

Step 3

In this step your goal is to build the index and log - two special files that help the agent (and you) navigate the wiki as it grows.

The index (index.md) is content-oriented. It’s a catalogue of every page in the wiki, organised by category: entities, concepts, sources, overviews. Each entry includes a link to the page, a one-line summary, and optionally metadata like creation date and the number of sources that feed into it. When the agent needs to answer a query, it reads the index first to find candidate pages, then drills into the most relevant ones. This approach works well at moderate scale (hundreds of pages) and avoids the need for embedding-based RAG infrastructure at the browsing level.

The log (log.md) is chronological. It’s an append-only record of everything that happened: ingests, queries, lint passes, schema changes. Each entry starts with a consistent prefix format: ## [YYYY-MM-DD] type | Description. This makes the log parseable with simple command-line tools - grep "^## \\\\[" log.md | tail -5 gives you the last five entries.

The key design decision is that the agent owns both files. Every ingestion should update the index with new pages and revised summaries. Every operation should append to the log. The agent should read the index at the start of every query to know what’s available. The agent should read the log at the start of every session to know what’s been done recently.

Build the index and log maintenance into your LangGraph workflows from Step 2. After the agent finishes writing wiki pages for an ingestion, it should update the index and append to the log as the final nodes in the graph. If an ingestion fails partway through, the log should record the failure.

Testing:

• Ingest a source and verify the index is updated with entries for the new summary page, entity pages, and concept pages. Each entry should have a link and a one-line description.

• Ingest a second source and verify the index reflects both sources, with shared entity pages showing updated descriptions.

• Check the log after several ingestions. Verify each entry has the correct format (## [YYYY-MM-DD] ingest | Title) and appears in chronological order.

• Run grep "^## \\\\[" log.md and verify you get a clean chronological listing.

• Manually delete a wiki page. Ingest a new source and verify the index accounts for the missing page (removes the stale entry rather than leaving dead links).

Step 4

In this step your goal is to add semantic search over your wiki pages using vector embeddings and full-text search stored in Oracle AI Database.

So far the agent navigates the wiki by reading the index and following links. That works at moderate scale, but as your wiki grows to hundreds of pages, you’ll want semantic search - finding pages by meaning, not just by browsing the catalogue.

Take every wiki page (excluding the index and log themselves, and excluding raw source files), generate a vector embedding for it using your embedding model, and store the embedding alongside the page’s path, title, type, tags, and a snippet or summary in Oracle Database. The metadata fields should all be stored and indexed so you can filter by type (”only entity pages”) or by tag.

Create a vector index on the embedding column for fast cosine similarity search. Also create an Oracle Text full-text index on the page content (or at minimum on the title and summary fields). Vector search finds semantically related pages even when the words don’t match. Full-text search catches exact names, technical terms, and phrases that vector search might rank lower. Together they give you robust retrieval.

Think about when embeddings should be generated. Every time the agent creates or updates a page during ingestion, the new or revised page needs to be re-embedded and stored. Pages that weren’t touched by an ingestion should keep their existing embeddings. You’ll need to track which pages changed so you only re-embed those.

Also think about what you embed. You could embed the full page text, but long pages might dilute the semantic signal. You could embed a summary or the first N paragraphs. You could embed both the title and the body separately and combine scores. Experiment and find what works for your test corpus - different approaches suit different types of content.

Testing:

• Run the full pipeline against your test wiki: parse all pages, generate embeddings, store in Oracle Database. Verify that the number of stored embeddings matches the number of wiki pages.

• Query Oracle Database directly to inspect a few stored entries. Verify each contains the embedding vector, page path, title, type, and tags.

• Verify that both the vector index and the full-text index have been created.

• Update a page (by ingesting a new source that modifies an existing entity page). Verify only the changed page is re-embedded; unchanged pages keep their existing embeddings.

• Search for a page by a concept it discusses (not by its exact title). Verify vector search returns it even though the words don’t match.

• Search for an exact technical term or person’s name. Verify full-text search catches it with high confidence.

Step 5

In this step your goal is to build the query system - the agent workflow that answers your questions by searching the wiki and synthesising a response.

Now that you have a searchable wiki, you need the agent to put it to use. When you ask a question, the agent should follow a multi-step process: read the index to identify candidate pages, search for semantically relevant pages using the hybrid search you built in Step 4, read the most relevant pages in full, and synthesise an answer with citations to specific pages and sections.

Model the query workflow as a LangGraph graph. The nodes might include: read index, hybrid search, read candidate pages, and synthesise answer. If the agent finds gaps - the question touches on something the wiki doesn’t cover well - it should say so honestly rather than speculating.

The system should support different answer formats depending on the question. A comparison between two concepts might be best as a table. A timeline of events might be best as a chronological list. A straightforward explanation might be best as prose. Give the LLM the flexibility to choose the format, and provide guidance in your prompts.

An important capability: the system should offer to file good answers back into the wiki as new pages. When you ask a question that generates a useful analysis, comparison, or synthesis, the agent should ask if you want to save it. If you do, it writes the answer as a new wiki page, adds it to the index, logs it, and embeds it. This way your explorations compound in the knowledge base just like ingested sources do.

Support follow-up questions within a session. If you ask “tell me about X” and then “what about Y?”, the agent should understand from context that Y relates to the broader topic X is part of. LangGraph’s state carries conversation context forward between queries.

Testing:

• Ask a question about something well-covered in your wiki. The answer should be accurate, cite specific pages, and not hallucinate facts not in the wiki.

• Ask the same question with different phrasing. Verify you get a similar answer - semantic search should match by meaning, not exact wording.

• Ask a question that spans two or more wiki pages. Verify the agent reads multiple pages and synthesises an answer that connects them.

• Ask a follow-up question without re-stating the topic. Verify the agent maintains context from the previous exchange.

• Ask a question about something not covered in your wiki. Verify the agent honestly reports the gap rather than making things up.

• Ask the agent to save an answer as a wiki page. Verify the page is created on disk, added to the index, logged, and embedded in Oracle Database.

• Ask a comparison question (”what’s the difference between X and Y?”). Verify the answer uses an appropriate format (table, side-by-side, etc.).

Step 6

In this step your goal is to add wiki project management so you can maintain separate knowledge bases for different topics.

A single wiki is useful, but you’ll likely want separate knowledge bases for different areas of your life - one for your research topic, one for book notes, one for health and fitness, one for career learning. Each should be isolated, with its own set of pages, its own embeddings, and its own agent memory.

Add support for named wiki projects. Store project metadata - name, creation date, last ingestion timestamp, page count, source count - in Oracle AI Database. Each project’s embeddings and metadata should be isolated so that searches against one wiki never return pages from another.

The user should be able to create a new wiki project, list existing projects, and select which project to work with. When the user starts the system, they should be able to specify a project name and immediately pick up where they left off.

All wiki data should persist between runs. The markdown files live on disk in their project directories. The embeddings and metadata live in Oracle AI Database. When the user comes back and selects a project, everything should be exactly as they left it - the same pages, the same index, the same log, the same search capability.

Testing:

• Create two wikis on different topics, each with a small set of source documents. Ingest sources into both.

• Query one wiki and verify results come only from that wiki, not the other.

• List your wikis and verify both appear with correct names and metadata (page count, last ingestion time).

• Query the project metadata directly in Oracle Database and verify it matches what the system reports.

• Stop and restart your system. Verify all wiki data is intact and queryable for both projects.

• Add a new source to an existing wiki and re-run ingestion. Verify only new or changed pages are re-embedded; unchanged pages preserve their existing embeddings.

Step 7

In this step your goal is to build a linting system that health-checks your wiki and helps it stay consistent as it grows.

As your wiki accumulates pages and sources, inconsistencies creep in. A page makes a claim that a newer source contradicts, but the older page was never updated. A concept is discussed across five pages but never got its own dedicated page. A page references another that was renamed or deleted. Links only go one way. Gaps appear where you have half the story. Humans abandon wikis because this maintenance burden grows faster than the value. The agent can handle it.

Build a lint operation as a LangGraph workflow. The agent should walk the wiki systematically and check for: contradictions between pages (two pages make incompatible claims), stale claims (a page asserts something that a newer source has revised or disproven), orphan pages (pages with no inbound links from other wiki pages), missing pages (entities or concepts referenced but lacking their own page), broken cross-references (links that point to non-existent pages), and data gaps (areas where the wiki is thin and could benefit from additional sources).

The agent should report its findings as a prioritised list: critical issues first (contradictions, stale claims), then warnings (orphans, missing pages), then suggestions (gaps, possible new sources to look for). Each issue should include the specific pages involved and a suggested action.

The lint pass should also suggest new questions to investigate and new sources to look for - what’s missing from the wiki that would fill important gaps? This turns the lint operation from a bug-finding exercise into a research planning tool.

Make the lint operation interactive by default. The agent presents its findings, and you accept, reject, or modify each suggestion before any changes are made. The agent should not modify pages without confirmation unless you explicitly run in auto-fix mode.

Testing:

• Create a deliberate contradiction: write two entity pages that make incompatible claims about the same thing. Run lint and verify the agent detects and reports the contradiction, citing both pages.

• Create an orphan page: a page with no other pages linking to it. Run lint and verify it’s flagged.

• Reference a page that doesn’t exist (a broken wikilink). Run lint and verify the broken link is reported.

• Have a concept discussed across multiple pages but without its own dedicated page. Run lint and verify the agent suggests creating one.

• Accept a lint suggestion and verify the agent applies the fix correctly.

• Reject a lint suggestion and verify no changes are made.

Step 8

In this step your goal is to build a user interface for your LLM Wiki, providing a chat-based interface for querying and managing your knowledge base.

Your wiki agent currently works through a development CLI. Now give it a proper interface. You have a choice: build a web interface (similar to the Code Sherpa challenge), a CLI chat interface, or both. The core requirements are the same regardless.

The interface should provide a chat panel for querying the wiki. The user types questions in natural language, and the agent responds with synthesised answers that cite specific wiki pages. Responses should render markdown so that tables, lists, and formatted text display clearly. Citations should be clickable links that open the referenced page.

The interface should support the full query workflow you built in Step 5: semantic search, multi-page synthesis, follow-up questions with context, different answer formats, and saving answers as new wiki pages.

Provide a way to browse the wiki structure: a page tree or list showing categories (entities, concepts, summaries, overviews) and the pages within each. This helps the user understand the shape of their knowledge base at a glance.

When the agent is processing a query, show a loading state so the user knows something is happening. LangGraph workflows can take several seconds as the agent reads the index, searches for pages, reads them, and synthesises an answer.

Build a separate CLI tool for source ingestion that doesn’t require launching the full interface. A user should be able to run something like llm-wiki ingest my-research ./new-article.md and have the agent process the source in the background, updating the wiki, index, log, and embeddings. This makes adding sources a quick, low-friction operation.

Testing:

• Launch the interface and verify you can access it.

• Select a wiki project and ask a question. Verify the response appears with markdown rendering and citations to wiki pages.

• Click a citation link. Verify it opens the referenced page.

• Ask a follow-up question and verify the system maintains context from the previous exchange.

• Browse the wiki structure through the interface. Verify it accurately reflects the pages on disk.

• Ask the agent to save an answer as a wiki page. Verify the page appears in the wiki structure immediately.

• Use the CLI ingestion tool to add a new source. Verify the wiki updates without needing to launch the full interface.

• Submit a query and verify a loading indicator appears while the agent processes it.

Going Further

You’ve built a working personal knowledge base with an LLM agent that ingests sources, writes wiki pages, answers questions, and keeps everything consistent. Here are some ways to take it further:

• Cloud database support: Add an option to connect to a cloud-hosted Oracle AI Database instance instead of the local Docker container. Read the connection string from configuration.

• Batch ingestion: Add a batch mode that ingests multiple sources at once with less supervision. The agent processes each source, generates updates, and presents a summary of changes for your review rather than discussing each source individually.

• Marp slide generation: Add the ability to generate slide decks (using Marp format) from wiki content. This turns a collection of pages on a topic into a presentation with a single request.

• Obsidian integration: Build tighter integration with Obsidian. Watch the wiki directory for changes made through Obsidian and update embeddings automatically. Use Obsidian’s Dataview plugin with the frontmatter your agent already writes.

• Multi-format sources: Extend source ingestion to handle PDFs, web URLs (with scraping), and audio transcripts. The more formats you support, the more knowledge you can capture.

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #122 - AI-Powered Contract Review Agent

This challenge is to build your own AI-powered contract review agent using Trigger.dev - an application that takes a PDF contract, breaks it into clauses, analyses each one for risk in parallel using LLMs, pauses for human review, and streams back a final summary. Whilst the example of contract review, the workflow is applicable to many other domains.

This challenge was created in collaboration with Trigger.dev, whose platform provides durable background tasks with no timeouts, built-in retries, concurrency controls, human-in-the-loop pause points, real-time streaming, and full observability - all in TypeScript.

Contract review is one of the most time-consuming parts of legal work. Lawyers spend hours poring over dense documents looking for risky clauses, ambiguous language, and missing terms. An AI agent that can do the first pass, flag issues, and then incorporate human feedback before producing a final report would save enormous amounts of time.

Building this from scratch means you’d need to solve several hard infrastructure problems: job queuing with retries, parallel execution with concurrency control, durable pause-and-resume for human review, real-time streaming to the frontend, and execution tracing. That’s exactly what Trigger.dev handles for you. You define your workflow as a set of tasks - functions that can run for as long as needed - and Trigger.dev takes care of the rest. Your focus stays on the application logic: extracting clauses, analysing risk, and generating summaries.

By the end you’ll have a deep understanding of durable workflow orchestration and how a platform like Trigger.dev makes building AI-powered applications dramatically simpler.

The Challenge - Building Your Own AI-Powered Contract Review Agent

You’re going to build a web application that lets users upload PDF contracts and then kicks off a series of durable Trigger.dev tasks. Those tasks will extract text, split it into clauses, analyse each clause in parallel with automatic retries, pause for human approval, and stream a final summary back to the frontend in real time. The system will handle contracts with 50+ clauses reliably, support multiple LLM providers, and give you complete visibility into every step through Trigger.dev’s built-in dashboard.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

You’ll need to make a few decisions:

1. Set up your Trigger.dev project. Trigger.dev is a TypeScript-first platform, so you’ll be working in that ecosystem. Scaffold a new project using npx create-trigger@latest and follow the quickstart to get a task running. Choose a framework for your web app - Next.js is a natural fit since it pairs well with Trigger.dev, but you can also use Express, Remix, or any Node.js framework. Get your Trigger.dev API key from the dashboard and configure your environment.

2. Choose your LLM providers. Sign up for API keys. You’ll build an abstraction layer so you can swap providers without changing the rest of your code.

3. Choose your database. You’ll need to persist users, contracts, clause analyses, reviewer decisions, and final summaries. PostgreSQL works well and integrates naturally with Prisma. Pick what you’re most comfortable with.

4. Understand the Trigger.dev project structure. Your tasks live in the trigger/ folder. Each file defines one or more tasks using the task() function. These are functions that can run indefinitely with no timeouts, automatic retries on failure, and built-in logging. You trigger tasks from your web app, and you can chain tasks together using triggerAndWait() or batchTriggerAndWait(). Take a few minutes to read through the tasks documentation and get comfortable with the concepts.

Testing: Run the example task that create-trigger generates (usually called hello-world). Trigger it from your web app and verify it appears in the Trigger.dev dashboard. Make a simple API call to each of your chosen LLM providers with a basic prompt and verify you get a coherent response. Set up your database, create a test table, and verify you can read and write data. Once all three are working independently, you’re ready to start building.

Step 1

In this step your goal is to build user authentication and a lightweight homepage.

Create a sign-up and login system using email and password. You’ll need a user model in your database, registration and login forms, and session management. Keep the auth simple - you don’t need OAuth or social login, just email/password with hashed passwords.

Build a lightweight homepage that explains what the product does. It doesn’t need to be a full marketing site - just a clear explanation that this is an AI-powered contract review tool, what it does, and a call-to-action to sign up or log in.

Testing:

• Visit the homepage and verify it renders correctly with the product explanation.

• Register a new account with an email and password. Verify you’re redirected and logged in.

• Log out and log back in with the same credentials.

• Try registering with an email that already exists. You should get an appropriate error.

• Try logging in with the wrong password. You should get an appropriate error, not a crash.

Step 2

In this step your goal is to build the PDF upload and text extraction pipeline.

Create a web UI where logged-in users can drag and drop or select a PDF contract file for upload. Once uploaded, store the file and extract the raw text from the PDF. Use a PDF parsing library - pdf-parse for simpler PDFs or pdf.js for more complex documents.

This is a great place to introduce your first custom task. Create a processContractUpload task in your trigger/ folder. When a user uploads a PDF, your web app should store the file, create a contract record in your database, and then trigger the task with the contract ID. The task should:

• Extract raw text from the PDF

• Store the extracted text back to the database, linked to the contract

• Update the contract status

You don’t need to worry about timeouts - Trigger.dev tasks can run as long as needed, which is important for large PDFs. You also don’t need to worry about what happens if the server restarts mid-processing. The task will resume where it left off.

Testing:

• Upload a multi-page PDF document. Verify the text is extracted and stored in the database.

• Upload a PDF with unusual formatting (headers, footers, columns). Check how well your extraction library handles it. Some garbled text is expected.

• Try uploading a non-PDF file. Your application should reject it with a clear error message.

• Try uploading without being logged in. The application should redirect to the login page.

• Open the Trigger.dev dashboard. You should see your processContractUpload task run with a status, duration, and any logs you emitted.

Step 3

In this step your goal is to split the extracted text into individual clauses using an LLM.

Raw extracted PDF text is rarely clean. You’ll have page numbers, headers, footers, and sometimes text in the wrong order. Your job now is to take that extracted text and use an LLM to identify and split it into individual, well-formed clauses.

Extend your processContractUpload task or create a new child task that takes the extracted text, sends it to an LLM with a prompt instructing the LLM to return the text split into clauses, and stores the results. Each clause should be a distinct logical unit - a paragraph, a condition, a definition, a warranty.

Store the identified clauses in your database, linked to the contract. Each clause should have a reference number (1, 2, 3...) so you can refer to it later.

Testing:

• Upload a contract and verify it gets split into multiple clauses. For a typical multi-page contract, you should get at least 10-15 clauses.

• Inspect the clauses in your database. Each one should be a coherent, self-contained piece of text, not a fragment mid-sentence.

• Upload a very short document (a single paragraph). It should still work, returning one clause.

• Check the Trigger.dev dashboard and verify the clause-splitting step appears in the run timeline.

Step 4

In this step your goal is to analyse each clause in parallel using LLMs to flag risk levels and ambiguous language.

This is the core of the application - and where Trigger.dev’s features really shine. For each clause, you need to send it to an LLM for analysis. The analysis should identify:

• Risk level - high, medium, or low

• Risk explanation - a short explanation of why the clause is risky

• Ambiguous language - any vague terms like “reasonable efforts”, “as soon as practical”, “material adverse change” that could be interpreted differently

• Recommendations - suggested changes to reduce risk or clarify ambiguity

Create an analyseClause task that takes a clause ID and text, calls your LLM, and stores the analysis result in the database. Configure it with retry settings so transient LLM API failures are handled automatically - Trigger.dev will retry with exponential backoff by default.

Now for the parallelism. From your parent task, use analyseClause.batchTriggerAndWait() to trigger all clause analyses in a single batch call. Trigger.dev will execute them in parallel (up to your environment’s concurrency limit), collect all the results, and return them to your parent task. A 50-clause contract is no problem - you get fan-out parallelism without writing any queue infrastructure.

Set a concurrencyLimit on the analyseClause task’s queue if you need to respect LLM API rate limits. For example, if your OpenAI tier allows 10 concurrent requests, set queue: { concurrencyLimit: 10 }.

Testing:

• Upload a contract and verify all clauses are analysed, each with a risk level, explanation, ambiguity flags, and recommendations.

• Check the Trigger.dev dashboard run view. You should see the parent task with all the child analyseClause runs, their individual statuses, durations, and any retries.

• Temporarily use a rate-limited API key and verify that failed analyses are automatically retried and eventually succeed. Watch the retries in the dashboard.

• Upload a 50+ clause contract. Verify it completes reliably. All 50+ analyses should be in the database.

• Inspect a few analyses. A clause that says “The Provider shall not be liable for any damages” should be flagged as high risk. A clause that mentions “reasonable efforts” should be flagged for ambiguous language.

Step 5

In this step your goal is to aggregate the clause analyses into a structured review report and pause for human review.

Once all clause analyses are complete, your parent task should aggregate them into a structured review report. The report should show each clause number, the clause text, the risk level, the analysis explanation, any ambiguous language found, and recommendations. Group clauses by risk level (high first) so the reviewer can tackle the most important issues first.

Now for the human-in-the-loop part - this is where Trigger.dev’s waitpoint system comes in. After aggregating the results, use wait.createToken() to create a pause point. Store the token ID alongside the contract in your database so your review dashboard can reference it later. Then call wait.forToken() - your task will suspend at this point. Trigger.dev checkpoints the task state and releases compute resources. You’re not paying for idle time, and there’s no timeout.

Send an email notification to the user with a link to the review dashboard. Trigger.dev has hooks for this - you can use the onSuccess hook of the analysis task, or send the email before the waitpoint. Use a transactional email service like Resend, SendGrid, or Mailgun.

Testing:

• Complete an analysis on a contract. Verify the aggregated report is stored in the database, ordered by risk level.

• Verify the email notification is sent containing the correct summary statistics and a working link to the dashboard.

• Check the Trigger.dev dashboard. The run should show as WAITING - it’s suspended at the waitpoint, waiting for the review token to be completed.

• Verify that the task does not proceed until someone completes the token.

Step 6

In this step your goal is to build the review dashboard where a reviewer can approve, reject, or annotate each flagged clause.

Build a web dashboard that displays the aggregated review report. For each clause, the reviewer should be able to:

• Approve the clause as is (no changes needed)

• Reject the clause (it needs revision)

• Annotate it with a free-text note explaining their reasoning or providing instructions

The dashboard should show the original clause text alongside the AI’s analysis so the reviewer has full context to make a decision. Make it easy to navigate between clauses and see at a glance which ones have been reviewed and which still need attention.

When the reviewer is done and submits their review, your application should save all decisions and annotations to the database, then complete the waitpoint token to resume the suspended task. Use wait.completeToken() from your web app backend (or send a POST to the token’s URL from the frontend using the public access token). The task will resume exactly where it left off, with all the reviewer’s decisions available from the token’s output.

You can also use Trigger.dev’s Realtime hooks in your dashboard. useRealtimeRun() lets you subscribe to run status changes without polling - so your dashboard can show the live status of the contract review workflow.

Testing:

• Navigate to the review dashboard for a contract. Verify all clauses are displayed with their AI analysis.

• Approve a few clauses, reject a few, and add annotations to some. Verify the decisions are saved to the database.

• Submit the review. Verify the token is completed and the Trigger.dev task resumes.

• Watch the Trigger.dev dashboard during review submission. The run should transition from WAITING to running again.

• Before submitting, check that you can see visually which clauses have been reviewed and which haven’t.

Step 7

In this step your goal is to make the system LLM provider-agnostic with a configurable abstraction layer.

Your clause analysis and summary generation tasks currently call one or two specific LLM providers. Build an abstraction layer so you can swap providers without changing your task code.

Define a common interface for LLM interactions: a function that takes a prompt (or messages), configuration (temperature, max tokens, etc.), and returns a standardised response with the generated text and metadata (tokens used, finish reason, etc.).

Configuration should be externalised. Provider selection, model choice, and API keys should come from environment variables. Your Trigger.dev tasks should interact with the abstraction layer, not with any specific provider’s SDK directly.

Testing:

• Run a full contract analysis using one LLM provider. Verify it works end to end.

• Switch the configuration to use another provider instead. Rerun the same contract. The analysis should complete with comparable results.

• Swap providers without changing a single line of task code (only environment variables).

• Verify your abstraction layer captures provider-agnostic metadata regardless of which provider is underneath.

Step 8

In this step your goal is to generate the final summary report using an LLM, incorporating the clause analyses and reviewer feedback.

Now that the human review is complete, the waitpoint token has been completed, and your task has resumed, it’s time to generate the final summary.

Create a generateSummary task that loads all clause analyses, reviewer decisions, and annotations from the database, sends everything to an LLM, and asks it to synthesise a final report. The final summary should include:

• Executive summary - a high-level overview of the contract’s risk profile

• Key findings - the most important issues identified, incorporating reviewer feedback

• Risk breakdown - a summary of risk levels across the contract

• Clause-by-clause detail - for each clause, the original risk assessment and the reviewer’s decision, combined into a final recommendation

This should be a well-written, professional document that could be shared with a client or colleague. Your parent task should trigger the summary generation using triggerAndWait() so it gets the result back.

Testing:

• Generate a final summary for a contract that has been fully reviewed. Verify it includes all sections and incorporates both the AI analysis and the reviewer feedback.

• Check that clauses the reviewer approved show as “accepted” in the final report, while rejected clauses include the reviewer’s annotations and reasoning.

• Verify the summary is stored in the database and linked to the contract.

• Read the summary from start to finish. It should read as a coherent, professional document, not a jumble of disconnected analyses.

Step 9

In this step your goal is to stream the final summary to the frontend in real time as it is being generated.

Final summaries can be long, and waiting for a complete document before showing anything is a poor user experience. Trigger.dev has first-class support for streaming data from tasks to your frontend. Use Realtime streams to pipe LLM tokens directly to the browser as they’re generated.

Define a stream using streams.define() - give it a clear ID like "summary-output" and a type for the stream chunks. In your generateSummary task, configure your LLM call to stream tokens, and pipe the stream to your defined stream using .pipe().

In your frontend, use the useRealtimeStream() React hook to subscribe to the stream. As tokens arrive, your component renders them incrementally. No polling, no WebSocket management, no SSE wiring - the hook handles the connection automatically.

The user should also be able to receive the final summary via email as an alternative. Once the stream is complete, send the full summary as an email.

Testing:

• Generate a final summary and watch the frontend. Tokens should appear incrementally, not all at once at the end.

• Verify that the stream works across page refreshes - existing chunks should be replayed.

• Check that the streaming handles slow generation gracefully. Partial content should render without freezing.

• Verify the email delivery option works. Trigger an email with the completed summary and check your inbox.

• Generate summaries from different providers. Streaming should work regardless of which provider is configured.

Step 10

In this step your goal is to explore Trigger.dev’s built-in observability - run tracing, logging, and monitoring.

You’ve already been using Trigger.dev’s dashboard throughout this challenge to see your tasks run. Now let’s make the most of its observability features. Unlike building your own tracing system from scratch, Trigger.dev gives you this out of the box.

Add tags to your tasks and runs so you can filter them in the dashboard. For example, tag runs with the contract ID, the user ID, the LLM provider used, and the workflow stage.

Use runs.metadata to attach structured data to your runs that updates as the workflow progresses. For example, set metadata for the number of clauses found, the count of high/medium/low risk clauses, the review status, and any error counts. This metadata appears in the dashboard and is available via the SDK.

Use Trigger.dev’s built-in logger throughout your tasks. It automatically captures log entries with timestamps and attaches them to the run - no log aggregation infrastructure needed.

Finally, explore the dashboard’s run view. You can see the full timeline of your contract review workflow: when each analyseClause child task started and completed, which clauses triggered which retries, how long each LLM call took, and any errors that occurred. The batch trigger view shows all parallel clause analyses at a glance, with individual run statuses and durations.

Testing:

• Run a complete contract review. Open the Trigger.dev dashboard and find your run.

• Verify you can see every step in the timeline: PDF extraction, clause splitting, batch clause analysis (with all child runs), waitpoint pause, waitpoint completion, and summary generation.

• Click into individual analyseClause runs. You should see logs, duration, and whether any retries occurred.

• Apply filters in the dashboard using your tags. Filter by status (failed runs only), by user, or by date range.

• Add metadata to your runs and verify it appears in the dashboard.

Going Further

Want to take this further? Here are some ideas:

• Use wait.for() to schedule follow-ups. Trigger a task that waits 7 days, then sends a reminder to review a contract that hasn’t been actioned.

• Use input streams for cancellation. Add a cancel button to the frontend that uses Trigger.dev’s input streams to abort a running summary generation mid-stream.

• Add support for more file formats. DOCX is even more common than PDF for contracts. Add support for Word documents and other formats.

• Add role-based access control. Different users might need different permissions - uploaders, reviewers, and administrators.

• Add comparison mode. Upload two versions of the same contract and have the LLM identify what changed and whether the changes alter the risk profile.

• Add custom risk categories. Let users define their own risk categories and rules, then use Trigger.dev’s wait.forToken() to collect approval for each category.

• Add a clause library. Build a library of standard, low-risk clause templates that the LLM can suggest as replacements for high-risk clauses.

• Use concurrency keys for multi-tenancy. Leverage Trigger.dev’s concurrencyKey to give each organisation or user their own isolated queue.

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #121 - Dd

This challenge is to build your own version of dd, the low-level data copying and conversion utility found on every Unix-like system.

dd has been around since the early days of Unix - it first appeared in Version 5 Unix in the mid-1970s. The name is a nod to IBM’s JCL (Job Control Language) DD statement, which was used to describe data sets on mainframes. Unlike most Unix tools, dd uses a key=value syntax for its arguments rather than the usual flags, another inheritance from its mainframe roots.

At its heart, dd reads data in fixed-size blocks, optionally transforms it, and writes it out. That simplicity makes it surprisingly powerful: it’s used to copy disk images, create files of a specific size, benchmark storage throughput, convert character encodings, and recover data from failing drives. If you’ve ever created a bootable USB stick with dd if=image.iso of=/dev/sdb, you’ve used it.

If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️, with the bonus that you get 20% off any of my courses.

3. Buy one of my self-paced courses that walk you through a Coding Challenge.

4. Join one of my live courses where I personally teach you Go by building five of the coding challenges or systems software development by building a Redis clone.

The Challenge - Building Dd

In this challenge you’re going to build your own version of dd. You’ll start with the core block-copy loop and progressively add the operands and conversion options that make dd so versatile.

Step Zero

In this introductory step your goal is to set your environment up ready to begin developing and testing your solution.

Choose your target platform and programming language. dd is a low-level tool that benefits from a language with good support for binary I/O and byte-level manipulation all work well.

Before you start coding, spend a few minutes playing with the system dd to get a feel for how it behaves:

# Copy a file
dd if=/etc/hosts of=/tmp/hosts-copy

# Copy from stdin to stdout
echo "Hello, dd!" | dd

# Create a 1 MB file of zeroes
dd if=/dev/zero of=/tmp/zeros bs=1M count=1

Notice the summary that dd prints to stderr when it finishes - something like:

2+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000123 s, 8.3 MB/s

That 2+0 notation means “2 full records and 0 partial records”. You’ll be implementing that too.

Also note that dd uses key=value operands rather than the usual -flag style. if=, of=, bs=, and so on are all positional arguments, not flags.

Step 1

In this step your goal is to implement the core block-copy loop.

Your ccdd should read data from stdin and write it to stdout in fixed-size blocks, defaulting to 512 bytes. It should support the if=FILE operand to read from a file instead of stdin, and the of=FILE operand to write to a file instead of stdout.

When it finishes, it should print a summary to stderr in the same format as the real dd:

<n>+<m> records in
<n>+<m> records out
<bytes> bytes copied, <time> s, <rate> MB/s

Where n is the number of full blocks read and m is the number of partial blocks (blocks where fewer bytes were available than the block size).

Testing: Copy a file and verify the output is identical:

ccdd if=/etc/hosts of=/tmp/hosts-copy
diff /etc/hosts /tmp/hosts-copy

The diff should produce no output. Check the summary printed to stderr matches what the real dd reports. Also test reading from stdin and writing to stdout:

echo "Hello, dd!" | ccdd | cat

Step 2

In this step your goal is to add block size control with the bs=, ibs=, and obs= operands, along with size suffixes.

The bs=BYTES operand sets both the input and output block size simultaneously. The ibs=BYTES and obs=BYTES operands set them independently - useful when you want to read in small chunks but write in large ones, or vice versa.

You should support the following size suffixes on any byte count:

Suffix Multiplier c 1 w 2 b 512 k or K 1024 M 1,048,576 G 1,073,741,824

So bs=4k means a 4096-byte block size, and bs=2M means 2,097,152 bytes.

Testing: Verify that different block sizes produce the same output:

ccdd if=/etc/hosts of=/tmp/out1 bs=1
ccdd if=/etc/hosts of=/tmp/out2 bs=512
ccdd if=/etc/hosts of=/tmp/out3 bs=4k
diff /tmp/out1 /tmp/out2
diff /tmp/out1 /tmp/out3

All three should be identical. Check the records in/out summary changes appropriately - a 1-byte block size will show many more records than a 4k block size for the same file.

Step 3

In this step your goal is to add the count=N, skip=N, and seek=N operands.

count=N limits the copy to N input blocks. skip=N skips N input blocks before starting to copy (seeking forward in the input). seek=N skips N output blocks before starting to write (seeking forward in the output, leaving the beginning of the output file untouched).

These three operands are what make dd useful for working with disk images and binary file formats where you need to operate on a specific region of a file.

Testing: Extract the middle portion of a file:

# Create a test file with known content
printf 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' > /tmp/test.bin  # 64 bytes of A
printf 'BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB' >> /tmp/test.bin # 64 bytes of B
printf 'CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC' >> /tmp/test.bin # 64 bytes of C

# Extract just the B section (skip 1 block of 64 bytes, copy 1 block)
ccdd if=/tmp/test.bin of=/tmp/out.bin bs=64 skip=1 count=1
xxd /tmp/out.bin

The output should be 64 bytes of B. Test seek=N by writing into the middle of an existing file:

dd if=/dev/zero of=/tmp/sparse.bin bs=64 count=3
ccdd if=/tmp/test.bin of=/tmp/sparse.bin bs=64 skip=1 count=1 seek=1
xxd /tmp/sparse.bin

The middle 64 bytes should now be B, with zeroes before and after.

Step 4

In this step your goal is to implement the conv= operand with the text conversion options: ucase, lcase, and swab.

conv=ucase converts all lowercase ASCII letters to uppercase as the data passes through. conv=lcase does the reverse. conv=swab swaps adjacent bytes - byte 0 with byte 1, byte 2 with byte 3, and so on. If an odd number of bytes is read, the last byte is held over and swapped with the first byte of the next block.

Multiple conversions can be combined with commas: conv=ucase,swab.

Testing:

echo "Hello, World!" | ccdd conv=ucase

Should output HELLO, WORLD!. Test lcase:

echo "Hello, World!" | ccdd conv=lcase

Should output hello, world!. Test swab with a known byte sequence:

printf '\\x01\\x02\\x03\\x04' | ccdd conv=swab | xxd

Should show 02 01 04 03 - each pair of bytes swapped.

Step 5

In this step your goal is to implement the remaining conv= options: notrunc, noerror, sync, and sparse.

By default, dd truncates the output file before writing. conv=notrunc disables this, leaving any existing content beyond what dd writes intact. This is essential when patching a specific region of a binary file.

conv=noerror tells dd to continue after a read error rather than stopping. It’s used when recovering data from a failing drive - you’d rather get most of the data than none of it.

conv=sync pads each input block with null bytes (\\x00) to the full input block size when a short read occurs. Combined with noerror, this is the standard recipe for imaging a failing drive: dd if=/dev/sda of=image.img conv=noerror,sync.

conv=sparse is an optimisation: instead of writing blocks that are entirely null bytes, dd seeks past them in the output file. The filesystem records these as “holes”, creating a sparse file that takes up less actual disk space than its apparent size.

Testing: Test notrunc by writing a short string into the middle of a longer file:

echo "Hello, World!" > /tmp/original.txt
echo "Hi" | ccdd of=/tmp/original.txt conv=notrunc
cat /tmp/original.txt

The output should start with Hi but retain the rest of the original content (, World! and the newline). Without notrunc, the file would be truncated to just Hi\\n.

Test sparse by creating a file with large null regions and checking its actual disk usage:

ccdd if=/dev/zero of=/tmp/sparse.img bs=1M count=100 conv=sparse
ls -lh /tmp/sparse.img    # apparent size: 100 MB
du -sh /tmp/sparse.img    # actual disk usage: near 0

Step 6

In this step your goal is to implement the status= operand and SIGUSR1 signal handling.

status=none suppresses all output, including the final summary. status=noxfer suppresses the transfer statistics (bytes, time, rate) but still prints the records in/out counts. status=progress prints periodic transfer statistics to stderr while the copy is running, so you can see progress on long operations.

You should also handle the SIGUSR1 signal: when your process receives it, print the current transfer statistics to stderr without interrupting the copy. This is how you check on a long-running dd without stopping it.

Testing: Verify status=none produces no output at all:

echo "test" | ccdd status=none 2>/tmp/stderr.txt
cat /tmp/stderr.txt  # should be empty

Test status=progress with a slow copy:

ccdd if=/dev/zero of=/tmp/progress-test bs=1M count=500 status=progress

You should see the statistics updating as the copy runs. Test SIGUSR1:

ccdd if=/dev/zero of=/tmp/signal-test bs=1M count=1000 &
PID=$!
sleep 1
kill -USR1 $PID
wait $PID

Sending SIGUSR1 should print the current statistics without stopping the copy.

Step 7

In this step your goal is to add iflag= and oflag= for I/O flags.

iflag=direct opens the input file with O_DIRECT (or the platform equivalent), bypassing the OS page cache. This is useful for benchmarking raw storage throughput without cache effects. oflag=direct does the same for the output.

oflag=dsync opens the output with O_DSYNC, which causes each write to block until the data is physically written to storage. This is slower but guarantees durability.

iflag=fullblock changes how dd handles short reads on the input. Normally, a short read (fewer bytes than the block size) counts as a partial record. With fullblock, dd keeps reading until it has accumulated a full block or reaches end-of-file. This is important when reading from pipes or network sockets, where a single read() call may return less than the requested amount even when more data is coming.

Multiple flags can be combined with commas: iflag=direct,fullblock.

Testing: Test iflag=fullblock with a pipe that delivers data in small chunks:

# Without fullblock, each small write becomes a partial record
yes | head -c 4096 | ccdd bs=512 > /dev/null

# With fullblock, partial reads are accumulated into full blocks
yes | head -c 4096 | ccdd bs=512 iflag=fullblock > /dev/null

Compare the records in/out summary - fullblock should show fewer partial records. For oflag=dsync, copy a file and verify it completes successfully (the main observable effect is that it’s slower, as each write is synchronised to disk).

Going Further

• Add count_bytes=N and skip_bytes=N (GNU extensions) to operate in bytes rather than blocks - useful when the data doesn’t divide evenly into your block size

• Implement conv=ascii and conv=ebcdic to convert between ASCII and EBCDIC character encodings, the original purpose of dd‘s conversion mode

• Add iflag=count_bytes so that count=N counts bytes rather than blocks

• Build a progress bar using ANSI escape codes for status=progress, showing a visual indicator alongside the transfer rate

• Benchmark your implementation against the system dd on a large file and see how close you can get - try different block sizes and see how throughput changes

• Use your ccdd to create and restore a disk image of a USB drive (carefully!) and verify the image is byte-for-byte identical using md5sum

P.S. If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️ twice a month, with the bonus that you also get 20% off any of my courses.

3. Buy one of my courses that walk you through a Coding Challenge.

4. Subscribe to the Coding Challenges YouTube channel!

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #120 - md5sum

This challenge is to build your own version of md5sum, the command-line utility that computes and verifies MD5 message digests.

MD5 (Message-Digest Algorithm 5) was designed by Ronald Rivest in 1991 and published as RFC 1321. For decades it was the go-to hash function for verifying file integrity - you’d download a file, run md5sum on it, and compare the output to the hash published alongside it to make sure nothing got corrupted or tampered with.

These days MD5 is considered cryptographically broken (collisions can be generated in seconds on a modern laptop), but it’s still widely used for checksums, cache keys, and non-security purposes. More importantly for Coding Challenges, MD5 is a simple hash function to implement from scratch. Building it yourself is a wonderful way to demystify how hash functions work.

If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️, with the bonus that you get 20% off any of my courses.

3. Buy one of my self-paced courses that walk you through a Coding Challenge.

4. Join one of my live courses where I personally teach you Go by building five of the coding challenges or systems software development by building a Redis clone.

The Challenge - Building Your Own MD5sum

In this challenge you’re going to build your own version of md5sum. There are two tracks through this challenge. Pick the one that suits you, or do both.

Track 1 (Steps 1 - 3) gets you to a fully working md5sum clone using your language’s standard library or a third-party library for the hash computation. You’ll focus on the command-line interface, file handling, and the check mode.

Track 2 (Steps 4 - 6) takes you deeper. You’ll implement the MD5 algorithm itself from scratch by following RFC 1321, replacing the library you used in Track 1 with your own code. This is where you’ll learn how hash functions actually work - message padding, Merkle-Damgård construction, and the compression function that sits at the heart of MD5 (and SHA-1, and SHA-2).

Both tracks produce the same tool. The only difference is whether the hashing happens inside a library or inside your own code.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

Choose your target platform and programming language. Any language will work for Track 1. For Track 2, pick a language that gives you easy access to 32-bit unsigned integer arithmetic and bitwise operations.

Before you start coding, have a play with the system md5sum so you get a feel for how it behaves:

echo -n "" | md5sum
echo -n "Hello, World!" | md5sum

The output format is the 32-character hex digest, two spaces, and the filename (or - for stdin). Note the -n flag on echo, without it, echo appends a newline, which changes the hash. This is a common gotcha when testing.

If you’re planning to do Track 2, open RFC 1321. It’s short, readable, and includes reference C code and a full test suite in the appendix. You won’t need it until Step 4, but it’s worth a skim now.

Step 1

In this step your goal is to hash the contents of a file and print the result in the standard md5sum output format.

Your tool should accept one or more filenames as command-line arguments, compute the MD5 hash of each file’s contents, and print one line per file in the format:

<32-char hex digest>  <filename>

Note the two spaces between the digest and the filename, this is the md5sum convention for text mode. You can use your language’s built-in MD5 library or any third-party package for the hash computation. The focus in this step is on reading files, formatting the output correctly, and handling errors (for example, printing a message to stderr and continuing if a file doesn’t exist).

Testing: Create a test file and compare your output to the system md5sum:

echo -n "Coding Challenges" > test.txt
md5sum test.txt
ccmd5 test.txt

Both should produce the same hash. Test with multiple files:

echo -n "File one" > a.txt
echo -n "File two" > b.txt
ccmd5 a.txt b.txt
md5sum a.txt b.txt

The output should match line for line. Also test what happens when you pass a file that doesn’t exist - your tool should print an error message for that file and still process the remaining files.

Step 2

In this step your goal is to support reading from standard input and to handle binary files correctly.

When no filenames are given, your tool should read from stdin, compute the hash, and print the result with - as the filename. When - is given explicitly as a filename argument, it should also read from stdin.

Your tool should also support the -b flag for binary mode. In binary mode the output uses * (space-asterisk) between the digest and the filename instead of two spaces. On modern systems the hash is computed the same way regardless of mode, the flag only affects the output format and is a holdover from systems where text and binary file reads differed. But md5sum supports it, so yours should too.

Testing:

echo -n "Hello" | ccmd5
echo -n "Hello" | md5sum

Both should output the same hash followed by -. Test binary mode:

ccmd5 -b test.txt
md5sum -b test.txt

The output should show * before the filename instead of two spaces. Test with a binary file too -- an image, a compiled executable, or /bin/ls -- and verify your hash matches the system md5sum.

Step 3

In this step your goal is to implement check mode with the -c flag.

When called with -c, your tool should read a file containing previously generated checksums (one per line, in the same format your tool produces) and verify each one. For each line, it should read the named file, compute its hash, and compare it to the stored hash. It should print the filename followed by OK or FAILED for each entry.

At the end, if any checksums failed, your tool should print a summary line to stderr saying how many didn’t match, and exit with a non-zero status code. If all checksums match, it should exit with status 0.

Implement the --quiet flag, which suppresses the OK lines and only shows failures. And implement the --status flag, which suppresses all output and only sets the exit code -- useful in scripts.

Testing: Generate a checksum file, then verify it:

ccmd5 a.txt b.txt > checksums.md5
ccmd5 -c checksums.md5

You should see:

a.txt: OK
b.txt: OK

Now tamper with one of the files and re-check:

echo -n "Changed" > a.txt
ccmd5 -c checksums.md5

You should see:

a.txt: FAILED
b.txt: OK
ccmd5: WARNING: 1 computed checksum did NOT match

Test --quiet (only the FAILED line should appear) and --status (no output, but echo $? should show a non-zero exit code).

If you’re happy with a working md5sum clone and aren’t interested in implementing the hash algorithm itself, skip ahead to Going Further. Otherwise, read on.

Step 4

In this step your goal is to implement MD5 message padding and preprocessing, the first stage of the algorithm.

From here on you’re replacing the library hash with your own implementation. By the end of Step 6, your tool should produce identical output using code you wrote yourself.

MD5 operates on the input message in 512-bit (64-byte) blocks. Before processing, the message must be padded so its length is a multiple of 512 bits. The padding works like this:

1. Append a single 1 bit to the message (in practice, append the byte 0x80).

2. Append zero bytes until the message length is 56 bytes short of a multiple of 64 (i.e., length mod 64 equals 56).

3. Append the original message length in bits as a 64-bit little-endian integer.

This padding scheme means the final block always has room for the length field, and the 0x80 byte ensures the padding is unambiguous -- you can always tell where the original message ended.

Implement the padding, then split the padded message into 512-bit blocks. For now, just verify your padding is correct by checking it against the test vectors in RFC 1321 (Appendix A.5). You should also initialise the four 32-bit state variables (A, B, C, D) to the values specified in the RFC:

A = 0x67452301
B = 0xefcdab89
C = 0x98badcfe
D = 0x10325476

These are the starting values of the hash state, sometimes called the initialisation vector. They’re arbitrary constants chosen by Rivest.

Testing: The empty string "" has a length of 0 bits. After padding, you should have exactly one 64-byte block: 0x80, followed by 55 zero bytes, followed by the 64-bit length (0) in little-endian. Print your padded block as hex and verify it looks right.

The string "a" has a length of 8 bits. After padding: 0x61 0x80, 53 zero bytes, then 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00.

Step 5

In this step your goal is to implement the MD5 compression function, the core of the algorithm.

The compression function processes one 512-bit block at a time and updates the four state variables. For each block:

1. Break the 64-byte block into sixteen 32-bit words (little-endian).

2. Initialise working variables a, b, c, d to the current state A, B, C, D.

3. Run 64 rounds, divided into four groups of 16. Each round applies a different auxiliary function to three of the four working variables, adds in one of the message words and a round constant, then rotates the result.

The four auxiliary functions are:

• F (rounds 0-15): F(B, C, D) = (B AND C) OR (NOT B AND D) - a bitwise conditional: “if B then C else D”

• G (rounds 16-31): G(B, C, D) = (B AND D) OR (C AND NOT D) - same idea, different arrangement

• H (rounds 32-47): H(B, C, D) = B XOR C XOR D - a parity function

• I (rounds 48-63): I(B, C, D) = C XOR (B OR NOT D) - another nonlinear mixing function

Each round computes:

a = b + left_rotate((a + func(b,c,d) + message_word + round_constant), shift_amount)

Then the variables are rotated: the old d becomes the new c, the old c becomes the new b, and so on. The shift amounts and round constants are specified in the RFC -- there are 64 of each, and they’re fixed values you can hard-code as a table.

After all 64 rounds, add the working variables back into the state: A += a, B += b, C += c, D += d. This addition step is what makes the construction iterative, each block’s output becomes the next block’s input.

The final hash is the state variables A, B, C, D concatenated in little-endian byte order to produce the 128-bit (16-byte) digest, which is then printed as 32 hex characters.

Testing: RFC 1321 provides test vectors in Appendix A.5. Your implementation should produce these exact hashes:

MD5("") = d41d8cd98f00b204e9800998ecf8427e
MD5("a") = 0cc175b9c0f1b6a831c399e269772661
MD5("abc") = 900150983cd24fb0d6963f7d28e17f72
MD5("message digest") = f96b697d7cb7938d525a2f31aaf161d0
MD5("abcdefghijklmnopqrstuvwxyz") = c3fcd3d76192e4007dfb496cca67e13b

Work through these one at a time. If a hash doesn’t match, check your byte ordering, little-endian is the most common source of bugs. Once all five test vectors pass, swap your library hash for your own implementation and verify that your ccmd5 tool still produces the same output as the system md5sum for every file you test.

Step 6

In this step your goal is to add support for SHA-256, so your tool can operate as both md5sum and sha256sum.

SHA-256 follows the same Merkle-Damgård structure as MD5, pad the message, split into blocks, process each block through a compression function - but with different parameters. The blocks are still 512 bits, but the state is eight 32-bit words instead of four, the compression function runs 64 rounds with different operations (Ch, Maj, and two sigma functions instead of F/G/H/I), and the output is 256 bits instead of 128. The message length in the padding is big-endian rather than little-endian.

Add a --algorithm flag (or -a) that accepts md5 or sha256, defaulting to md5. When SHA-256 is selected, the output format should match the system sha256sum, same layout, just a longer digest.

If you’ve structured your code well, the padding, block processing loop, and I/O code should be shared between both algorithms, with only the compression function and initialisation differing. This is a good test of how cleanly you’ve separated concerns.

Testing: Verify against the system sha256sum and well-known test vectors:

echo -n "" | ccmd5 -a sha256
echo -n "" | sha256sum

Both should produce e3b0c44298fc1c149afbf4c898fbf90a... - (the SHA-256 of the empty string).

echo -n "Hello, World!" | ccmd5 -a sha256
echo -n "Hello, World!" | sha256sum

Test check mode with SHA-256 too -- generate a checksum file with -a sha256 and verify it with -c.

Going Further

Here are some ideas to take your implementation further:

• Add support for SHA-1, SHA-384, and SHA-512 to build a complete family of hash tools

• Add the -tag output format (BSD-style: MD5 (filename) = digest) which is what macOS md5 uses by default

• Implement HMAC-MD5 and HMAC-SHA256 using your hash functions - HMAC is how hash functions are used for message authentication in protocols like TLS

• Benchmark your implementation against the system md5sum on a large file and see how close you can get - then try optimising with SIMD intrinsics or by processing multiple blocks in parallel

• Read about the MD5 collision attacks (Wang et al., 2004) and try to understand how they exploit weaknesses in the auxiliary functions - it’s a fascinating bit of cryptographic history

• Implement SHA-3 (Keccak), which uses a completely different construction (a sponge function) rather than Merkle-Damgård - comparing the two designs is a great way to understand why the cryptographic community moved on from MD5’s family of designs

P.S. If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️ twice a month, with the bonus that you also get 20% off any of my courses.

3. Buy one of my courses that walk you through a Coding Challenge.

4. Subscribe to the Coding Challenges YouTube channel!

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #119 - AI Pong Player

This challenge is to build your own reinforcement learning agent that learns to play Atari Pong directly from the pixels on the screen.

Pong is one of the oldest video games ever made, and it has a special place in the history of artificial intelligence. In 2013, DeepMind used Pong (and a handful of other Atari games) to show that a single algorithm could learn to play games at a human level, just by watching the screen and being told the score. That work kicked off the modern era of deep reinforcement learning. Pong is the friendliest of the Atari games to start with, the rules are simple, the screen is mostly empty, and the agent only needs to choose between moving the paddle up or down. That makes it the perfect first project for going from “I’ve read about reinforcement learning” to “I’ve actually trained an agent from raw pixels and watched it learn to win.” Building this project will introduce you to ideas you’ll come across again and again throughout your career: turning observations into features, sampling from a stochastic policy, computing returns, reducing variance, and the policy gradient itself.

If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️, with the bonus that you get 20% off any of my courses.

3. Buy one of my self-paced courses that walk you through a Coding Challenge.

4. Join one of my live courses where I personally teach you Go by building five of the coding challenges or systems software development by building a Redis clone.

The Challenge - Building Your Own AI Pong Player

In this challenge you’re going to build a policy gradient agent that learns to play Pong from raw pixels using the REINFORCE algorithm. Your agent will start out playing randomly, lose 21-0 over and over, and then, if you’ve wired everything up correctly, gradually start scoring points, then winning rallies, and eventually beating the built-in opponent more often than it loses.

This challenge is a good fit if you’ve written some Python before, are comfortable with NumPy, and have at least a passing acquaintance with neural networks. You don’t need to be an reinforcement learning expert. REINFORCE is one of the simplest deep reinforcement learning algorithms there is, and the version we’ll build here is famously the one Andrej Karpathy described in his “Pong from Pixels“ blog post. A small policy network, no value function, no replay buffer, no target network. Just a policy, some episodes, and a gradient.

A word of warning before you start: training from pixels is slow. Even on a sensible setup, you should expect a few hours of CPU training before the agent really starts to win, and you may want to leave it running overnight. That’s part of the experience, watching the score curve crawl upwards over many thousands of episodes is genuinely exciting once you’ve built the thing yourself.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

Python is the natural choice for this challenge because the reinforcement learning ecosystem lives there, but the ideas transfer cleanly to any language with a deep learning framework.

You’ll need three things installed: Gymnasium (the maintained successor to OpenAI Gym), the Atari environments via ALE-py, and a deep learning framework, PyTorch, TensorFlow, or JAX are all fine, pick whichever you’d like to practise with. You’ll also want NumPy, Matplotlib, and probably opencv-python or Pillow for image work. Have a quick read of the Gymnasium docs and the Atari environment list so you know what’s available.

Before you write any code, spend a few minutes playing Pong yourself if you’ve never seen it. Notice that the only thing that matters is your paddle’s vertical position, the ball’s position, and the ball’s direction of travel. Your agent will need to work this out from the screen, with no idea what any of those concepts mean.

Step 1

In this step your goal is to get a Pong environment running and have a “random agent” play a full game so you can see the data flowing.

Create the ALE/Pong-v5 environment from Gymnasium and run a single episode where, at every step, you pick an action uniformly at random and step the environment with it. For each step, print or log the reward. You should see mostly zeros, with the occasional -1 (the built-in opponent has scored against you) and very rarely a +1 (you got lucky). The episode should end after twenty-one points have been scored on one side.

Have a look at the action space (env.action_space) and the observation space (env.observation_space). The action space has six entries, but for Pong you really only ever need two of them: the action that moves the paddle up and the action that moves it down. Constraining your agent’s choices to just those two actions makes learning much faster, because there are fewer wrong things it can do. Pick the two action indices you’ll use throughout the rest of the challenge and write them down somewhere obvious in your code.

The observation is a 210 x 160 x 3 RGB image - the raw screen. Have a look at one with Matplotlib so you know what your agent is seeing. There’s a lot of pixels there that have nothing to do with playing Pong: the score at the top, the borders down the sides, the colours. We’ll fix all of that in the next step.

Testing: Run your random agent for one episode and confirm that:

• The episode terminates of its own accord (you don’t have to cap the step count)

• The total reward is somewhere between roughly 21 and 15 (random play loses badly)

• The observation shape is (210, 160, 3) with uint8 values

Step 2

In this step your goal is to turn the raw 210 x 160 x 3 screen into a much smaller representation that contains just the information your agent needs.

There are four things to do here, and they should all happen inside a single function that takes a raw frame and returns the preprocessed observation:

1. Crop away the score area at the top of the screen and the borders on each side, leaving just the playing area.

2. Convert the result to greyscale - colour adds nothing useful in Pong.

3. Resize down to 80 x 80 pixels. The image was already mostly empty space; at this resolution you can still clearly see the paddles and the ball.

4. Flatten the 80 x 80 grid into a single 1D vector of length 6400. This is the input format your policy network will expect.

A static frame doesn’t tell your agent anything about which way the ball is moving, and direction is the most important thing in Pong. The classic trick - and the one used in the original Karpathy write-up - is to feed in the difference between the current preprocessed frame and the previous one. Pixels that didn’t change become zero, and pixels that did change show up as positive or negative values. The ball appears as a little bright streak pointing the way it’s travelling. Add this difference computation on top of your preprocessing function.

Testing: Save a few raw frames and their preprocessed versions to disk and look at them with an image viewer. The preprocessed frame should clearly show the two paddles and the ball as bright pixels on a dark background, with nothing else. Display a frame difference - it should be almost entirely black except for the ball and the moving paddle.

A good sanity check: the output of your preprocessing function should be a 1D NumPy array of length 6400 (or whatever shape you’ve chosen) with float32 values, not raw pixel bytes.

Step 3

In this step your goal is to build the neural network that maps a preprocessed observation to a probability distribution over actions, and use it to pick actions.

The policy network Karparthy describes is a tiny network - a single hidden layer with about 200 ReLU units, then an output layer that produces one number per action. Pass that output through a softmax (or a sigmoid if you’ve reduced things to a single output for “probability of moving up”) and you have a probability distribution. To pick an action, sample from that distribution rather than taking the most likely one.

Wire up an “act” function that takes a preprocessed frame, runs it through the network, and returns a sampled action plus whatever extra information you’ll need later for training (typically the log-probability of the action that was taken, or the network output itself).

Once that’s working, run another full episode - this time with your untrained network choosing the actions instead of random.choice. The agent will still lose badly (its weights are random), but the score should be in roughly the same ballpark as the random agent from Step 1. If you see something dramatically different, something is wrong with your preprocessing or your sampling.

Testing: Run a single episode with the untrained policy. The total reward should be in the same -21 to -15 range as the random agent. The action distribution - if you log it - should be close to 50/50 at the start of training. Print the shape of the network output and the sampled action index for the first few steps to make sure everything lines up.

Step 4

In this step your goal is to collect a complete episode of experience and turn the rewards into the returns that will drive learning.

For each step in an episode, store three things: the observation that was fed in, the action that was taken (or its log-probability), and the reward that came back from the environment. At the end of the episode you’ll have three lists, all the same length.

Now compute the discounted return for each step. The return at step t is the sum of all the rewards from step t onwards, with rewards further in the future weighted by a discount factor gamma (use 0.99). You should compute this as a single backwards pass over the reward list - much faster and cleaner than the obvious double loop. There’s one Pong-specific subtlety: every time someone scores a point, the rally ends and a new one begins inside the same episode. You probably want to reset the running sum when a non-zero reward appears, so credit for a point only flows back to the actions in that rally rather than all the way to the start of the game. This makes a big difference to learning speed.

Once you have the per-step returns, normalise them across the whole episode by subtracting the mean and dividing by the standard deviation. Normalised returns put roughly half the actions on the “this was better than average” side and half on the “this was worse” side, which gives the policy gradient a much more stable signal.

Testing: Run an episode, compute the returns, and have a look:

• The length of your returns array matches the number of steps in the episode.

• After normalisation, the mean should be close to zero and the standard deviation close to one.

• For an action that was followed by a +1 reward soon after, the return should be positive; for one followed by a 1, it should be negative.

A nice sanity print is to show, for the last twenty steps of an episode, the reward at that step and the discounted return - you’ll see the return building up smoothly and then jumping when a point is scored.

Step 5

In this step your goal is to actually update the policy in the direction that makes good actions more likely and bad actions less likely. This is the heart of the whole challenge.

The REINFORCE update is delightfully simple. For each step in your collected rollout, compute the loss as -log(probability of the action taken) * normalised return for that step, then sum (or average) across all the steps. Run that through your framework’s autograd, take a gradient step with an optimiser (Adam or RMSProp at a learning rate around 1e-3 to 1e-4 works well), and that’s it. Actions that led to better-than-average returns get pushed up; actions that led to worse-than-average returns get pushed down. You’re doing gradient ascent on expected return, even though you’re calling loss.backward().

A single episode’s worth of gradient is very noisy. Batch up the gradients over multiple episodes - ten is a sensible starting point - before you actually call the optimiser. You can either accumulate gradients across episodes or concatenate the per-step data and do one bigger update; both work.

Now wrap the whole thing in a training loop that runs for thousands of episodes, prints a running average of the score after each one, and just leaves it going. Be patient. For the first few hundred episodes the score will hover around -21 -- the policy is still essentially random and learning very slowly. After that, you should see the running average start to creep upwards. By the time you’ve trained for several thousand episodes (this can be many hours of wall-clock time on CPU), the running average should cross zero, meaning your agent is winning more rallies than it loses.

Testing: This is the step where things either work or they very visibly don’t. A few things to check as training progresses:

• The running average reward should be trending upwards over time, not just bouncing around

• After ~500 episodes, the agent should reliably score some points (running average above 21)

• After a few thousand episodes, the running average should be approaching zero or going positive

• If the loss explodes or the score gets stuck at 21 forever, the most common culprits are: forgetting to reset the discounted return between rallies, an unnormalised return signal, the wrong sign on the loss, or feeding raw frames instead of frame differences

If you’d like a stronger signal that things are alive, log the average length of an episode (in steps). Random play produces short episodes; an agent that’s learning to actually rally produces longer ones, well before the score itself starts to go up.

Step 6

In this step your goal is to make your training run something you can show off, not just a console of numbers scrolling by.

There are four things to add:

1. Save the model weights -- both periodically (every N episodes) and whenever a new best running average reward is achieved. You don’t want to leave a long training run going only to lose the weights.

2. An evaluation mode that loads a saved set of weights, plays a fixed number of episodes with the policy fixed (no learning, ideally with greedy action selection rather than sampling), and reports the average score. This is what you’d use to honestly compare two different training runs.

3. Video recording of the agent playing. Gymnasium has a RecordVideo wrapper that writes MP4s. Record a video of an early-training agent (it’ll be hilariously bad), a mid-training agent (starting to get the idea), and a late-training agent (winning, hopefully). Stitching these together is the single most satisfying artefact of the whole project.

4. A training reward plot -- a simple Matplotlib chart showing the per-episode reward and a rolling average over the run. The shape of this curve, going from a flat line at 21 through random play and up into positive territory, is the picture of an agent learning.

Testing: Once you have all four bits in place:

• Kill your training script and restart it from a saved checkpoint. The running average should pick up roughly where it left off, not crash back to 21.

• Run evaluation on a checkpoint with sampling vs. greedy action selection; greedy should be at least as good as sampled.

• Open one of your recorded videos and watch the agent play. It is unbelievably satisfying to see the paddle that you trained track the ball and put it past the opponent.

Going Further

Here are some ideas to take your Pong agent further:

• Add a baseline to reduce variance. REINFORCE has notoriously noisy gradients. Subtract a baseline - the simplest one is the running average reward, the next-simplest is a learned value function - from the returns before you scale the policy gradient. This is the first step from REINFORCE towards Actor-Critic.

• Replace the MLP with a small CNN. Convolutional layers are a much more natural fit for image input than a flattened MLP. You’ll lose the trick of feeding the frame difference and instead stack the last few frames as channels. Compare training time and final score against the MLP version.

• Try a different algorithm. Once you have all the scaffolding - environment, preprocessing, training loop, logging - you can swap the algorithm out without rewriting the rest. Implement A2C, PPO, or DQN against the same Pong setup and see how they compare on sample efficiency and final score.

• Run multiple environments in parallel. A single CPU core stepping through one game at a time is the bottleneck on most training runs. Use Gymnasium’s AsyncVectorEnv to step several Pong games at once and gather rollouts much faster.

• Train an opponent. The built-in Pong AI is fixed and not very good. Once your agent beats it consistently, you’ve topped out the score. A natural next step is self-play: have two copies of your agent play each other and improve together.

P.S. If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️ twice a month, with the bonus that you also get 20% off any of my courses.

3. Buy one of my courses that walk you through a Coding Challenge.

4. Subscribe to the Coding Challenges YouTube channel!

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #118 - Comm

This challenge is to build your own version of comm, the classic Unix utility that compares two sorted files line by line.

comm is one of those small tools in the Unix toolbox that solves a single problem really well. Given two sorted files, it tells you three things at once: which lines are unique to the first file, which lines are unique to the second file, and which lines appear in both. It does this in a single streaming pass, taking advantage of the fact that the inputs are already sorted, so it never has to load whole files into memory and never has to do an O(n²) comparison.

You’ll find comm used to diff lists of users, find files that exist in one directory tree but not another, work out the intersection of two datasets, and as a building block in countless shell pipelines. Building your own version is a lovely exercise in stream processing, careful state management, and the merge step that sits at the heart of merge sort, ideas you’ll reach for again and again throughout your career.

If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️, with the bonus that you get 20% off any of my courses.

3. Buy one of my self-paced courses that walk you through a Coding Challenge.

4. Join one of my live courses where I personally teach you Go by building five of the coding challenges or systems software development by building a Redis clone.

The Challenge - Building Comm

In this challenge you’re going to build your own version of comm, a streaming, sorted file comparison tool. Your tool will read two sorted files, compare them line by line, and print three columns of output: lines unique to the first file, lines unique to the second file, and lines common to both. It will be compatible with the standard POSIX comm utility, which means you’ll be able to test your work directly against the system comm and use it as a drop-in replacement in shell pipelines.

The clever bit about comm is that it doesn’t sort the files for you, it relies on the fact that they are already sorted to do its work in a single pass with constant memory. Two read pointers, one per file, and a small handful of comparison rules are all you need.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

Choose your target platform and programming language. I’d encourage you to pick a language you’re comfortable with for reading files line by line and parsing command-line arguments. Pretty much any general-purpose language is a good fit for this challenge. The focus is on the algorithm, not on the language.

Before you start coding, have a read through the POSIX comm specification and the man page on your own machine (man comm). Spend some time playing with the system comm so you get a feel for how it behaves, especially around the column indentation and the suppression flags.

Create a couple of small sorted test files to use throughout the challenge:

printf "apple\nbanana\ncherry\ndate\nelderberry\n" > file1.txt
printf "banana\ncherry\nfig\ngrape\n" > file2.txt

file1.txt and file2.txt are both sorted. The two files share banana and cherry, while apple, date, and elderberry are unique to the first and fig and grape are unique to the second. We’ll use these throughout the challenge.

A quick note on locale. Both sort and the system comm only do byte-wise comparison when the locale is C or POSIX. With a default locale (especially on macOS), ordering becomes locale-aware -- Apple can sort between apple and banana rather than before them, which will confuse your test files and any comparisons you run against the system comm. Either set LC_ALL=C in your shell while working on this challenge, or prefix the relevant commands with it (LC_ALL=C sort ..., LC_ALL=C comm ...).

Step 1

In this step your goal is to read two sorted files and produce the three-column output that is the heart of comm.

Your tool should accept exactly two filenames as command-line arguments, open both files, and walk through them in sorted order using a single pass. For each line you should decide whether it belongs to column 1 (lines only in the first file), column 2 (lines only in the second file), or column 3 (lines in both). Column 1 has no leading tab, column 2 has one leading tab, and column 3 has two leading tabs. Lines should be compared byte by byte, the same way LC_ALL=C sort orders them.

You’ll also need to handle files of different lengths -- when one file runs out, the remaining lines from the other file should be emitted in the appropriate column. Empty files are a useful edge case to think about: if one file is empty, every line of the other file goes into its own column. Don’t worry about any flags or options yet; just get the basic three-column comparison working.

Testing: Run your tool against the test files and check your output:

ccomm file1.txt file2.txt
apple
		banana
		cherry
date
elderberry
	fig
	grape

Try it with files of different lengths, with one empty file, and with two completely disjoint files. Lines unique to the first file should appear with no leading tab, lines unique to the second file with one leading tab, and shared lines with two leading tabs.

Step 2

In this step your goal is to support the single-column suppression flags -1, -2, and -3.

comm lets you hide any of the three columns from the output. -1 hides lines unique to the first file, -2 hides lines unique to the second file, and -3 hides lines common to both. When a column is suppressed, the indentation for the remaining columns shifts left so the leading tabs go away. For example, with -1 the second column should no longer have its leading tab, because there is no first column for it to sit next to.

Have a play with the system comm to see exactly how it lays things out. This is one of those details that is easier to copy than to describe.

Testing:

ccomm -1 file1.txt file2.txt
	banana
	cherry
fig
grape

ccomm -2 file1.txt file2.txt
apple
	banana
	cherry
date
elderberry

ccomm -3 file1.txt file2.txt
apple
date
elderberry
	fig
	grape

Step 3

In this step your goal is to support combining the suppression flags.

The flags from Step 2 can be combined to suppress more than one column at a time. -12 shows only the common lines, -23 shows only lines unique to the first file, -13 shows only lines unique to the second, and -123 produces no output at all. The combinations can be supplied as a single argument (-12) or as separate arguments (-1 -2); both should behave identically.

If you wrote Step 2 in a flexible way -- tracking which columns are active rather than special-casing each flag -- this step should be a very small change.

Testing:

ccomm -12 file1.txt file2.txt
banana
cherry

ccomm -23 file1.txt file2.txt
apple
date
elderberry

ccomm -13 file1.txt file2.txt
fig
grape

ccomm -123 file1.txt file2.txt

ccomm -1 -2 file1.txt file2.txt
banana
cherry

The fifth invocation should produce the same output as -12.

Step 4

In this step your goal is to support reading from standard input.

When - is given as one of the two filename arguments, your tool should read that input from stdin instead of opening a file. This is what lets comm slot into shell pipelines. Either of the two arguments can be -, but only one of them at a time -- if both are - your tool should report an error.

Testing:

printf "apple\nbanana\ncherry\n" | ccomm - file2.txt
apple
		banana
		cherry
	fig
	grape

cat file1.txt | ccomm file2.txt -
	apple
		banana
		cherry
	date
	elderberry
fig
grape

The first command pipes a sorted list into your tool as the first input. The second pipes file1.txt in as the second input. Each should produce the same three-column comparison as the equivalent file-based invocation.

Step 5

In this step your goal is to support case-insensitive comparison with the -i flag.

When -i is set, lines that differ only in case (for example Apple and apple) should be treated as equal. The original case of the lines should still be preserved in the output -- only the comparison itself is case-insensitive.

This is the only step where the comparison rule changes, so it’s worth thinking about how you’ve structured your comparison code. If you’ve kept the comparison behind a single function, this should be a small change.

Testing: Build a couple of files that differ only in case:

printf "Apple\nBanana\nCherry\n" > upper.txt
printf "apple\nbanana\ndate\n" > lower.txt

Without -i, every line is unique to one file or the other (because Apple and apple compare differently):

ccomm upper.txt lower.txt
Apple
Banana
Cherry
	apple
	banana
	date

With -i, the matching pairs should appear in column 3:

ccomm -i upper.txt lower.txt
		Apple
		Banana
Cherry
	date

The third invocation combines -i with column suppression:

ccomm -i -1 upper.txt lower.txt
	Apple
	Banana
date

Going Further

Here are some ideas to take your comm implementation further:

• Add the GNU -check-order and -nocheck-order flags so your tool can warn when its inputs aren’t really sorted

• Add the GNU -output-delimiter=STRING flag for choosing a custom string between columns instead of tabs

• Add the GNU -total flag to print a summary line with the count for each column at the end of the output

• Add the GNU z / -zero-terminated flag so records are separated by NUL bytes instead of newlines, which is useful for working with filenames containing newlines or spaces

• Add a -header flag that prints column headings before the output to make it easier to read interactively

• Support comparing more than two files at once (you’ll need to think about what the output even looks like for three or more inputs)

• Add a flag to compare on a specific field rather than the whole line, the way join and sort -k do

• Support files compressed with gzip or zstd transparently, so your tool can read .gz files without an explicit zcat

• Build a streaming library version of your tool so other programs in your chosen language can use the comparison logic without shelling out

• Benchmark your implementation against the system comm on very large files (think hundreds of millions of lines) and see if you can match or beat it

• Try running your comm as part of a real pipeline in your own work -- finding files in one directory tree but not another, or comparing two snapshots of a database export, are good real-world tests

P.S. If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️ twice a month, with the bonus that you also get 20% off any of my courses.

3. Buy one of my courses that walk you through a Coding Challenge.

4. Subscribe to the Coding Challenges YouTube channel!

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #117 - AI Powered Support Bot

This challenge is to build your own AI-powered customer support bot - and then discover, the hard way, why production AI applications need more than just an API key and a system prompt.

This challenge was created in collaboration with Orq.ai, whose Router provides a single API across 400+ models from 20+ providers - with built-in fallbacks, cost routing, and observability. Free to start, no markup on token costs.

Every developer has used a support bot. Most have opinions about them. In this challenge you’ll build one for a fictional version of Coding Challenges, giving it context about the available projects - Build Your Own Redis, Docker, Git and the rest - so it can answer questions about which challenge to tackle, what skills you’ll learn, how to get started, and general troubleshooting.

It starts simple. But step by step you’ll layer on the production concerns that real AI applications face: resilience when a provider goes down, observability so you know what’s happening, and cost routing so you’re not burning money on simple questions. By the time you’ve built all of that yourself, you’ll have a deep appreciation for what an AI gateway does - and you’ll see just how much code disappears when you use one.

If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️, with the bonus that you get 20% off any of my courses.

3. Buy one of my self-paced courses that walk you through a Coding Challenge.

4. Join one of my live courses where I personally teach you Go by building five of the coding challenges or systems software development by building a Redis clone.

The Challenge - Building Your Own AI Powered Support Bot

You’re going to build an AI customer support bot that answers questions about Coding Challenges projects. Along the way you’ll experience the real production pain points of working with LLMs - provider lock-in, reliability, observability, and cost - and then see what happens when you replace your hand-rolled infrastructure with a single gateway endpoint.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

You’ll need to make a few decisions:

1. Choose your programming language. Pick something you’re comfortable building with. You’ll be making HTTP requests, handling JSON, and building a simple interactive loop. Python, TypeScript, Go, Java, and Rust all work well.

2. Choose your first LLM provider. You need an API key from one of the major providers - OpenAI, Anthropic, Google (Gemini), or Mistral. You’ll be using their chat completions API. You can build using the REST API, an AI framework or install their SDK for your chosen language.

3. Grab the context data from here.

Testing: Make a simple API call to your chosen provider with a basic prompt like “Hello, who are you?” and verify you get a coherent response back. If that works, you’re ready to move on.

Step 1

In this step your goal is to build a working support bot using a single LLM provider.

Build an interactive command-line application that takes user questions and responds using your chosen LLM. The bot should have a system prompt that includes the Coding Challenges context you downloaded in Step Zero, instructing it to act as a helpful customer support agent that answers questions based on that context.

Your bot should maintain a conversation history so follow-up questions work naturally. If a user asks “Which challenge should I start with?” and then follows up with “What will I learn from that one?”, the bot should understand what “that one” refers to.

Keep it simple. One provider, one model, one API key hardcoded (or read from an environment variable). No fallbacks, no logging, no clever routing. Just a bot that works.

Testing:

• Ask the bot a factual question about Coding Challenges based on the context data, like “What projects are available?” or “What are the frontend focused projects?” The answer should be accurate and drawn from your context.

• Ask a follow-up question that references the previous answer. The bot should maintain context across the conversation.

• Ask something the context doesn’t cover, like “Can you help me with my tax return?” The bot should politely explain that it can only help with Coding Challenges questions.

• Ask for a personalised recommendation, like “I know Python and want to learn about networking - which challenge should I do?” The bot should give a thoughtful answer based on the available projects.

Step 2

In this step your goal is to add resilience by introducing a fallback to a second LLM provider.

Imagine your primary provider goes down. Right now your bot is completely broken. To simulate this, temporarily use an invalid API key for your primary provider so every request fails.

Now fix it. Sign up for a second LLM provider and integrate their SDK alongside the first. When a request to the primary provider fails, your bot should automatically retry with the fallback provider. The user should get an answer either way.

This sounds straightforward, but pay attention to the friction. You now have two SDKs with different interfaces, two API keys to manage, two different authentication mechanisms, and subtly different request and response shapes. Your code needs to handle the differences, normalise the responses, and manage the error handling for both.

Once your fallback is working, remove the invalid API key and restore normal operation. Keep the fallback logic in place - you’ll want it for reliability.

Testing:

• With both providers working, verify the bot responds normally using the primary provider.

• Invalidate your primary provider’s API key and send a question. The bot should seamlessly fall back to the second provider and still give a good answer. The user shouldn’t see an error - just a response.

• Invalidate both API keys and verify the bot handles the failure gracefully with a clear error message rather than crashing.

• Restore the primary key and verify the bot goes back to using the primary provider.

• Take note of how much code you’ve added to handle two providers. Count the lines if you like. You’ll want that number later.

Step 3

In this step your goal is to add observability by tracking token usage, latency, and cost for every request.

In production, you need to know what’s happening. How many tokens are you using? How much is each request costing? How long are responses taking? Without this information you’re flying blind.

Build a logging layer that captures the following for every LLM request:

• Request ID - a unique identifier for each request, so you can correlate a user-visible response with its log entry.

• Timestamp of the request.

• Provider and model used (especially important now you have two providers).

• Token counts - prompt tokens and completion tokens. Most providers return these in the API response.

• Latency - how long the request took from start to finish.

• Estimated cost - calculate this from the token counts and the provider’s published pricing.

• Whether it was a primary or fallback request.

• Error details - when a request fails, capture the error type, status code, and message. This is what tells you why the primary failed and the fallback kicked in.

• Finish reason - was it stop, length, content-filter, or a tool call? This matters when you’re investigating quality issues.

Store these logs however you like - a local file, an in-memory list, a database. Add a way to view a summary: total requests, total tokens, total cost, average latency, and the breakdown between primary and fallback usage.

You might also consider capturing the request and response payloads (watch out for PII), a conversation or session ID so you can trace a whole support session, retry counts, and time-to-first-token once you’re streaming. These aren’t required, but they’re the kinds of things a production observability stack typically includes.

This is the kind of infrastructure that every production AI application needs, and building it yourself gives you an appreciation for how much work it is to get right. You need to handle it consistently across both providers despite their different response formats.

Testing:

• Send several questions to the bot and then check your logs. Every request should have a complete record with all the fields listed above.

• Force a fallback by invalidating the primary key, send a question, then check the log. The entry should show the fallback provider was used.

• View your summary statistics. They should accurately reflect the requests you’ve made - total cost, average latency, token counts.

• Verify that token counts and cost calculations are consistent with what the providers report in their dashboards.

Step 4

In this step your goal is to add cost-aware routing so that simple questions go to cheaper models and complex questions go to more capable (and more expensive) ones.

Not all questions are equal. “What’s the pricing?” is a simple lookup that any small model can handle. “I’m a backend developer who knows Python but wants to learn systems programming - which challenges should I do and in what order?” needs genuine reasoning ability.

Build a routing layer that classifies incoming questions and directs them to the appropriate model. You’ll need at least two tiers:

• Simple queries - FAQs, greetings, straightforward factual lookups. Route these to a cheaper, faster model.

• Complex queries - personalised recommendations, multi-step reasoning, comparisons across projects, questions that require synthesising information. Route these to a more capable model.

How you classify queries is up to you. You could use keyword matching, a separate lightweight LLM call to classify the question, message length heuristics, or some combination. The point is to reduce cost without noticeably reducing quality.

A note on what’s actually changing: the system prompt and context data stay the same for every request - the cheap model still needs to see the Coding Challenges context to answer “What’s the pricing?” The saving comes from running fewer parameters per token, not from sending less context. Don’t be tempted to trim the context for simple queries; that quickly leads to wrong answers.

This is where your codebase starts to feel the weight. You now have multiple providers, fallback logic, per-request logging across all of them, and routing logic that needs to work with all your models. Take a moment to look at your code. Count the lines dedicated to infrastructure versus the lines dedicated to the actual support bot logic.

Testing:

• Ask a simple question like “What’s the pricing?” and check your logs. It should be routed to the cheaper model.

• Ask a complex question like “I want to learn distributed systems but I’ve never done any systems programming - what’s the best learning path through your challenges?” and check your logs. It should be routed to the more capable model.

• Compare the costs in your logs between simple and complex queries. The simple queries should be noticeably cheaper.

• Verify the quality of responses. Simple questions routed to the cheaper model should still be answered well. If the quality is poor, adjust your classification logic.

• Send a mix of ten questions - some simple, some complex - and review the routing decisions in your logs. Most should be classified correctly.

Step 5

In this step your goal is to replace all of the infrastructure you built in Steps 2 through 4 with the Orq.ai Router.

Sign up for a free Orq.ai account and get your API key. The Router provides an OpenAI-compatible endpoint, which means you can point any OpenAI SDK at it by changing the base URL and API key. That’s it.

Replace your multi-provider setup, your fallback logic, your logging infrastructure, and your routing layer with a single API call to the Router endpoint. The Router handles:

• Fallbacks and retries - if a provider fails, the Router automatically retries with another. You only pay for successful completions.

• Observability - per-request traces with token counts, latency, cost, and error logs are available in the Orq.ai dashboard. No custom logging code needed.

• Cost routing - the Auto Router directs each prompt to the most cost-effective model that meets quality requirements, with a typical 50% cost reduction while retaining 98% quality.

• Access to 400+ models from 20+ providers through a single API key and a single interface.

Now look at your code. The fallback handling from Step 2, the logging layer from Step 3, and the routing logic from Step 4 can all be removed. Your bot should be back to something close to the simplicity of Step 1, but with all the production capabilities you spent three steps building by hand.

Testing:

• Send the same mix of questions you used throughout the challenge. The responses should be at least as good as before.

• Check the Orq.ai dashboard for your request logs. You should see token counts, costs, latency, and which model was used for each request - all without any custom logging code.

• Simulate a provider failure (the Router handles this transparently). Send requests and verify they succeed even when individual providers have issues.

• Compare the total lines of code in your solution now versus at the end of Step 4. The infrastructure code should have largely disappeared.

• Compare the cost of your requests through the Router versus your manual routing from Step 4. The Router’s Auto Router should achieve similar or better cost optimisation.

Going Further

You’ve built a support bot and experienced the full arc from simple prototype to production-ready AI application. Here are some ways to push further:

• Add a web interface: Build a simple chat UI in front of your bot instead of using the command line. The backend stays the same - just add an HTTP API layer.

• Streaming responses: If you haven’t already, add streaming so responses appear token by token. This makes a big difference to the user experience, especially for longer answers.

• Conversation summarisation: For long support sessions, summarise older messages to keep the context window manageable while preserving important information.

• Multi-language support: Add the ability to detect the user’s language and respond in kind. This is a good test of your routing logic - you might want a different model for different languages.

• Evaluation harness: Build a test suite of questions with expected answers and measure your bot’s accuracy, response time, and cost across different configurations. Orq.ai can help you with this.

• RAG integration: Instead of putting all the context in the system prompt, build a retrieval pipeline that fetches relevant documentation chunks based on the user’s question. This scales much better as your knowledge base grows. Orq.ai can help you with this too.

P.S. If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️ twice a month, with the bonus that you also get 20% off any of my courses.

3. Buy one of my courses that walk you through a Coding Challenge.

4. Subscribe to the Coding Challenges YouTube channel!

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #116 - Awk

This challenge is to build your own version of awk, the classic text processing language.

Awk was created in 1977 by Alfred Aho, Peter Weinberger, and Brian Kernighan (the name comes from their initials). It’s a small but remarkably powerful language designed for processing structured text data.

Awk reads input line by line, splits each line into fields, and applies pattern-action rules to produce output. It sits in a sweet spot between sed (which is great for simple substitutions) and a full programming language like Perl or Python. Despite being nearly 50 years old, awk remains one of the most useful tools in a developer’s toolkit. You’ll find it in shell scripts, data pipelines, and one-liners across every Unix system in the world. Building your own awk will teach you about lexing, parsing, interpreters, and the design of small domain-specific languages.

If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️, with the bonus that you get 20% off any of my courses.

3. Buy one of my self-paced courses that walk you through a Coding Challenge.

4. Join one of my live courses where I personally teach you Go by building five of the coding challenges or systems software development by building a Redis clone.

The Challenge - Building Your Own Awk

In this challenge you’re going to build your own version of the awk text processing tool. Your tool will read input line by line, split each line into fields, match lines against patterns, and execute actions -- producing output that is compatible with the standard POSIX awk utility.

Awk programs are built from pattern-action rules that look like this: condition { action }. For each line of input, awk checks every rule. If the condition matches, the action is executed. It’s a simple model that turns out to be surprisingly expressive.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

Choose your target platform and programming language. I’d encourage you to pick a language you’re comfortable with for building interpreters. You’ll be writing a lexer, a parser, and a tree-walking interpreter, so a language with good string handling and data structures will make your life easier.

Before you start coding, have a read through the POSIX awk specification to get a feel for the language. Don’t worry about understanding every detail -- we’ll work through the features step by step. It’s also worth playing with the system awk on your machine to get a sense of how it behaves.

Create a test file to use throughout the challenge. Here’s one you can use:

echo "John 25 London
Jane 30 New York
Bob 22 Paris
Alice 35 Tokyo
Charlie 28 Berlin" > test.txt

Step 1

In this step your goal is to support the basic print action with field splitting.

Your tool should read input from a file (or stdin if no file is given), split each line into fields on whitespace, and support the print statement. The special variable $0 refers to the whole line, $1 to the first field, $2 to the second, and so on.

At this point, you only need to handle a bare action block with no pattern -- meaning the action runs for every line of input. Focus on getting the core loop right: read a line, split it into fields, execute the action, move to the next line.

Testing: Run these commands and compare against the system awk:

ccawk '{ print }' test.txt
ccawk '{ print $0 }' test.txt
ccawk '{ print $1 }' test.txt
ccawk '{ print $1, $3 }' test.txt
echo -e "hello\\nworld" | ccawk '{ print $0 }'

The first two should print every line. The third should print just the first name from each line. The fourth should print the name and city, separated by a space (the default output field separator). The fifth should read from standard input and print each line. Your output should match awk exactly.

Step 2

In this step your goal is to support the -F flag for custom field separators and the built-in variables NR, NF, and FS.

The -F flag lets the user specify a custom field separator. For example, -F: splits on colons, which is useful for parsing files like /etc/passwd.

Implement the built-in variables NR (the current record number, starting at 1), NF (the number of fields in the current record), and FS (the field separator).

Testing: Create a CSV-like test file and test with custom separators:

echo "john:25:london
jane:30:new york
bob:22:paris" > test2.txt

ccawk -F: '{ print $1 }' test2.txt
ccawk '{ print NR, $1 }' test.txt
ccawk '{ print NF }' test.txt

The first command should print just the names from the colon-separated file. The second should print line numbers alongside names. The third should print the number of fields on each line. Compare all output against the system awk.

Step 3

In this step your goal is to support patterns, comparison operators, and regular expression matching.

Awk’s power comes from its pattern-action model. A pattern can be a comparison expression (like $2 > 25), a regular expression (like /London/), or the special patterns BEGIN and END. If a line matches the pattern, the action is executed. If there’s no action, the default is { print }.

Implement comparison operators (==, !=, <, >, <=, >=), regular expression matching with /regex/ patterns and the ~ and !~ operators, logical operators (&&, ||, !), and the BEGIN and END special patterns. BEGIN runs before any input is read, and END runs after all input has been processed. With BEGIN available, you should also support setting FS within the program (e.g. BEGIN { FS = ":" }) as an alternative to the -F flag from Step 2.

Your program should support multiple pattern-action rules. Awk checks every rule for every line, so a single line can trigger multiple actions.

Testing:

ccawk '$2 > 25 { print $1 }' test.txt
ccawk '/London/ { print $1 }' test.txt
ccawk '$1 ~ /^[AJ]/ { print }' test.txt
ccawk 'BEGIN { print "Name Age" } { print $1, $2 } END { print "Done" }' test.txt
ccawk 'BEGIN { FS = ":" } { print $1 }' test2.txt
ccawk '$2 > 25 && $2 < 35 { print $1, "mid-range" }' test.txt
ccawk '/London/ { print "City:", $3 } /^J/ { print "J-name:", $1 }' test.txt

The first should print names of people older than 25. The second should print “John”. The third should print lines where the first field starts with A or J. The fourth should print a header, all names with ages, then “Done”. The fifth should set the field separator to colon inside BEGIN and print names from the colon-separated file. The sixth should print people whose age is between 25 and 35 exclusive. The seventh demonstrates multiple rules -- John’s line matches both patterns. Compare against awk.

Step 4

In this step your goal is to support variables, arithmetic operators, and assignment operators.

Awk variables are dynamically typed -- they can hold strings or numbers and convert between the two as needed. Uninitialised variables default to 0 when used as numbers and "" when used as strings.

Implement arithmetic operators (+, -, *, /, %, ^), assignment operators (=, +=, -=, *=, /=, %=), and string concatenation (which in awk is done by placing values next to each other with no operator).

Testing:

ccawk '{ total += $2 } END { print "Total age:", total }' test.txt
ccawk '{ print $1, $2 * 2 }' test.txt
ccawk '{ name = $1 " from " $3; print name }' test.txt
ccawk 'BEGIN { x = 2; print x ^ 10 }'

The first should print the sum of all ages. The second should print names with doubled ages. The third should concatenate fields with text. The fourth should print 1024. Compare against awk.

Step 5

In this step your goal is to support control flow: if/else, while, for, do-while, and C-style for loops.

Also implement break and continue for loops, next to skip to the next input record, exit to stop processing entirely, and the ternary conditional operator (condition ? value_if_true : value_if_false).

Testing:

ccawk '{ if ($2 > 25) print $1, "senior"; else print $1, "junior" }' test.txt
ccawk '{ for (i = 1; i <= NF; i++) print $i }' test.txt
ccawk '$1 == "Bob" { next } { print }' test.txt
ccawk '{ print; if (NR == 3) exit }' test.txt
ccawk '{ print ($2 > 25) ? $1 " is senior" : $1 " is junior" }' test.txt

The first should label people as senior or junior based on age. The second should print every field on its own line. The third should skip Bob’s line and print everything else. The fourth should print only the first three lines. The fifth uses the ternary operator to produce the same senior/junior labelling in a different style. Compare against awk.

Step 6

In this step your goal is to support associative arrays and the for (key in array) construct.

Associative arrays are one of awk’s most powerful features. They’re indexed by strings (not just integers) and can be used to count, group, and aggregate data. Implement the in operator for testing membership, for (key in array) for iterating over keys, and the delete statement for removing elements.

Testing:

ccawk '{ count[$3]++ } END { for (city in count) print city, count[city] }' test.txt
ccawk '{ ages[$1] = $2 } END { if ("Bob" in ages) print "Bob is", ages["Bob"] }' test.txt
ccawk '{ a[$1] = $2 } END { delete a["Bob"]; for (k in a) print k, a[k] }' test.txt

The first should count how many people live in each city. The second should check if Bob exists and print his age. The third should delete Bob and print the rest. Note that for (key in array) iterates in an unspecified order, so don’t worry about the ordering of output lines -- just make sure the content matches.

Step 7

In this step your goal is to support the printf statement and built-in string functions.

Implement printf with C-style format strings (supporting at least %d, %f, %s, %c, and %x with width and precision specifiers).

Implement these built-in string functions: length, substr, index, split, sub, gsub, match, sprintf, tolower, and toupper.

Testing:

ccawk '{ printf "%-10s %3d %s\\n", $1, $2, $3 }' test.txt
ccawk '{ print length($1) }' test.txt
ccawk '{ print substr($1, 1, 3) }' test.txt
ccawk '{ gsub(/o/, "0", $1); print }' test.txt
ccawk '{ print toupper($1) }' test.txt

The first should print a neatly formatted table. The second should print the length of each name. The third should print the first three characters of each name. The fourth should replace all “o” characters with “0” in the first field. The fifth should print names in uppercase. Compare against awk.

Step 8

In this step your goal is to support user-defined functions and built-in arithmetic functions.

Implement user-defined functions with the syntax function name(params) { body }. Functions should support local variables (declared as extra parameters in the function signature, which is how awk handles local scope) and return values with return.

Implement the built-in arithmetic functions: int, sqrt, sin, cos, atan2, exp, log, rand, and srand.

Testing:

ccawk 'function max(a, b) { return a > b ? a : b } { print $1, max($2, 30) }' test.txt
ccawk 'BEGIN { srand(42); for (i = 0; i < 5; i++) printf "%.4f\\n", rand() }'
ccawk '{ print $1, int(sqrt($2)) }' test.txt

The first should print each name alongside the greater of their age or 30. The second should print 5 random numbers. The third should print names with the integer square root of their age. Compare against awk (except for random numbers where the seed behaviour may differ).

Step 9

In this step your goal is to support the remaining output and input features.

Implement the output built-in variables: OFS (output field separator), ORS (output record separator), and RS (record separator). When print outputs multiple fields separated by commas, it uses OFS between them. Each print statement ends with ORS.

Implement the -f flag to read the awk program from a file instead of the command line, -v var=value for setting variables before execution, and support for reading from multiple input files. Implement the FILENAME, ARGC, and ARGV built-in variables.

Testing:

ccawk 'BEGIN { OFS="-" } { print $1, $2, $3 }' test.txt
echo 'BEGIN { print "Start" } { print FILENAME, $0 }' > prog.awk
ccawk -f prog.awk test.txt
ccawk -v threshold=25 '$2 > threshold { print $1 }' test.txt
ccawk '{ print FILENAME, $0 }' test.txt test2.txt

The first should print fields separated by dashes. The second should read the program from a file and print each line with its filename. The third should use the command-line variable. The fourth should process both files and show which file each line came from. Compare against awk.

Step 10

In this step your goal is to support piping output to shell commands and the getline function.

Implement the pipe operator for print, which lets you send output to an external command: print "hello" | "sort". Awk keeps the pipe open across multiple print statements to the same command, so all the output goes to a single invocation of the command. Implement the close() function to close a pipe or file, which is needed when you want to reopen a pipe or when the external command needs to receive EOF to produce output.

Implement getline in its various forms: getline to read the next line from the current input, getline var to read into a specific variable, getline < "file" to read from a file, and "command" | getline to read from a command.

Testing:

ccawk '{ print $1 | "sort" }' test.txt
ccawk '{ while (("date" | getline line) > 0) print line; close("date") }'
ccawk 'BEGIN { while ((getline line < "test.txt") > 0) print line }'

The first should print names in sorted order. The second should print the current date. The third should read and print the test file from within a BEGIN block. The fourth is a classic awk one-liner that sums file sizes from ls output. Compare against awk.

ls -l | ccawk 'NR > 1 { total += $5 } END { print "Total bytes:", total }'

Going Further

Here are some ideas to take your awk implementation further:

• Add support for multi-character record separators (which gawk supports but POSIX awk does not)

• Implement the OFMT and CONVFMT variables for controlling numeric-to-string conversion

• Add support for range patterns (/start/,/stop/) which match all lines between two patterns

• Implement coprocess communication with |& (a gawk extension)

• Add support for @include to include other awk source files

• Build a bytecode compiler and virtual machine instead of a tree-walking interpreter for better performance

• Add support for the ENVIRON array for accessing environment variables

• Implement nextfile to skip to the next input file

• Try running your awk against real-world awk scripts (there are many collected online) and see how compatible your implementation is

P.S. If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️ twice a month, with the bonus that you also get 20% off any of my courses.

3. Buy one of my courses that walk you through a Coding Challenge.

4. Subscribe to the Coding Challenges YouTube channel!

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #115 - Code Sherpa

This challenge is to build your own semantic code exploration tool - a system that helps developers make sense of large, unfamiliar codebases using natural language questions instead of reading files top to bottom.

We’ve all been there. You join a new team, or pick up a legacy project, and you’re staring at thousands of files with no idea where the interesting bits live. You grep for keywords, open file after file, and slowly piece together how things fit. It works, but it’s slow and frustrating.

Code Sherpa takes a different approach. You point it at a codebase and ask questions in plain English: “where do we handle payment failures?” or “how does the authentication flow work?” The system finds the relevant code by meaning, not by keyword matching, explains what it does, and remembers what you’ve already explored so each session builds on the last.

Under the hood, the system parses code into semantic chunks, embeds them into vectors, and stores them in a vector database such as Oracle Database 26ai. When you ask a question, it retrieves the most relevant code using both vector similarity and full-text search, and uses an LLM to explain it in context. The agentic behaviour - memory, follow-up questions, multi-step exploration - is orchestrated using LangGraph’s state machine model, while LangChain handles the retrieval chains and LLM integration. It’s a practical introduction to vector search, full-text search, embeddings, code parsing, agent orchestration, and building a web interface to tie it all together.

If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️, with the bonus that you get 20% off any of my courses.

3. Buy one of my self-paced courses that walk you through a Coding Challenge.

4. Join one of my live courses where I personally teach you Go by building five of the coding challenges or systems software development by building a Redis clone.

The Challenge - Building Code Sherpa

You’re going to build a semantic code exploration tool. It starts by ingesting a codebase into a vector database, then lets you query it through a web interface using natural language. Step by step you’ll add code parsing, vector storage, semantic retrieval, LLM-powered explanations, project management, a web interface, agent memory, and intelligent navigation. By the end, you’ll have a tool that genuinely helps you understand unfamiliar code.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

You’ll need to make a few decisions and get some infrastructure running:

1. Set up your vector database. I suggest Oracle Database 26ai running in a local Docker container. Pull the container-registry.oracle.com/database/free:latest image, start the container, and set a password for the admin account. You can find full setup instructions in the Oracle Database Free Get Started guide. Once the container is running, connect using a SQL client and verify you can create a table. Store all credentials in an environment file, not hardcoded anywhere.

docker pull container-registry.oracle.com/database/free:latest
docker run -d -p 1521:1521 -e ORACLE_PWD=<your-password> container-registry.oracle.com/database/free:latest

1. Choose your embedding model. You need a code-aware embedding model - one that understands programming constructs, not just prose. Nomic’s CodeRankEmbed is open source (MIT), purpose-built for code retrieval, and lightweight enough to run locally on CPU without a GPU or API key. It produces 768-dimensional vectors. Install it via Hugging Face: pip install sentence-transformers and load it as nomic-ai/CodeRankEmbed. If you have a GPU available, Nomic’s larger nomic-embed-code (7B parameters, requires ~14GB VRAM) offers a few points better retrieval quality.

2. Set up your LLM provider. You’ll need a language model for generating explanations later. Any provider with a chat API will work - Anthropic, OpenAI, Google, Mistral, or a local model.

3. Set up LangChain and LangGraph. You’ll be using LangChain for retrieval chains and LLM integration, and LangGraph for orchestrating the agent’s behaviour as a state machine. Install both: pip install langchain langgraph. LangChain handles the plumbing of embedding, retrieval, and prompting. LangGraph handles the agentic flow - deciding when to search memory, when to retrieve code, when to ask follow-up questions, and how to route between these steps.

Prepare a test codebase to work with throughout the challenge. Pick an open source project you’re curious about but haven’t explored in depth - something with a few thousand lines across multiple files and directories. A project with clear structure (like a web framework, CLI tool, or library) works well.

Testing: Verify your Oracle Database container is running and you can connect to it. Load your embedding model and generate a test embedding to confirm it returns a vector of the expected dimensions. Make a test call to your LLM API to confirm it returns a valid response. Verify your environment file is being read correctly and no credentials are in your source code.

Step 1

In this step your goal is to build a code parsing pipeline that breaks a codebase into meaningful chunks.

The foundation of semantic code search is good chunking. Rather than splitting files at arbitrary character boundaries, you want to split at logical boundaries: functions, classes, and modules. A chunk should represent one coherent unit of code that makes sense on its own.

Point your parser at a local directory and have it walk the file tree, identify source files, and split each one into chunks. Each chunk should carry metadata: the file path it came from, what type of chunk it is (function, class, module), the programming language, and the character range within the original file.

Not every file will parse cleanly. Some might have syntax errors, use unusual language features, or be in a format your parser doesn’t support. When that happens, log the failure and keep going. A partial index is far more useful than no index at all.

Display progress in the terminal as ingestion runs: how many files have been processed, how many chunks have been created, and any failures encountered.

Testing:

• Run your parser against your test codebase and verify it produces chunks at function and class boundaries, not arbitrary splits.

• Inspect several chunks and confirm each one contains a complete, coherent unit of code.

• Check that the metadata on each chunk is correct - file path, chunk type, language, and character range should all match the source.

• Introduce a file with a deliberate syntax error and verify the parser logs the failure and continues processing the rest.

• Verify the terminal output shows meaningful progress: files processed, chunks created, and any errors.

Step 2

In this step your goal is to embed the code chunks and store them in your vector database.

Take each chunk from your parser, generate a vector embedding for it using your chosen embedding model, and store the embedding alongside the chunk’s text and metadata in Oracle Database. The metadata fields - file path, chunk type, language, and character range - should all be stored and indexed so you can filter on them later.

Think about how you structure your storage. You’ll want to be able to search by vector similarity, but also filter by metadata (e.g. “only show me Python files” or “only functions, not classes”). Set up your vector indexes accordingly. Also create an Oracle Text full-text index on the code text column - this will let you fall back to keyword search when vector similarity alone doesn’t find good matches, and is particularly useful when developers search for exact identifier names, error messages, or string literals.

Testing:

• Run the full pipeline - parse then embed and store - against your test codebase.

• Query Oracle Database directly to verify the data is there: check the total number of stored chunks matches what your parser reported.

• Inspect a few stored entries and confirm they contain the embedding vector, the original code text, and all metadata fields.

• Verify that both the vector index and the full-text index have been created on the appropriate columns.

• Run the pipeline again against the same codebase and verify it handles the re-run sensibly (either updating existing entries or skipping duplicates).

Step 3

In this step your goal is to implement semantic retrieval so you can ask natural language questions and get back the most relevant code.

This is where the tool starts to feel useful. Take a natural language question from the user, embed it using the same model you used for the code chunks, and search your vector database for the closest matches using cosine similarity. Also run the query through Oracle’s full-text search on the same table. Combine the results - vector search finds semantically related code even when the words don’t match, while full-text search catches exact identifier names and string literals that vector search might rank lower. Return the top results along with their file paths and line references.

Not every query will have good matches. Set a minimum cosine similarity threshold - start at around 0.3 for CodeRankEmbed embeddings, which is a reasonable baseline for filtering out unrelated results. You may need to adjust this based on your embedding model: if you’re getting too many irrelevant results, raise it; if you’re missing relevant code, lower it. When nothing exceeds the threshold and full-text search also returns no matches, the system should tell the user honestly rather than returning low-confidence results that waste their time.

Build this as a simple CLI interface for now - you’ll add the web interface later. The user types a question, and the system returns the matching code chunks with their locations. This is database search only so far.

Testing:

• Ask a question about something you know exists in your test codebase (e.g. “where is the main entry point?” or “how are errors handled?”). Verify the returned chunks are genuinely relevant.

• Ask the same question using different phrasing and verify you get similar results. This is the whole point of semantic search - it matches by meaning, not keywords.

• Search for an exact function or variable name. Verify the full-text search catches it even if the vector similarity score would be low.

• Ask a question about something that definitely isn’t in the codebase. Verify the system tells you no relevant code was found rather than returning irrelevant results.

• Check that every returned chunk includes its file path and line reference.

Step 4

In this step your goal is to add LLM-powered explanations so the system doesn’t just find code - it explains what the code does.

Raw code chunks are useful, but an explanation in plain language is far more helpful when you’re trying to understand an unfamiliar codebase. Wire up your LLM through LangChain to take the retrieved chunks and generate a clear explanation. Use LangChain’s retrieval chain to handle the prompt construction - passing the retrieved code as context alongside the user’s question.

The explanation should cite the specific functions and files involved. When the user asks how two parts of the codebase relate to each other, the system should retrieve both and explain the connection. Where multiple implementations of the same concept exist, it should surface all of them and explain the differences.

Rather than hardcoding the retrieval orchestration, bind tool definitions to the LLM so it can invoke tools during response generation. Define three tools: search_code(query) for hybrid vector and full-text search, read_file(file_path) for full file contents, and list_files(pattern) for glob-based file listing. The agent loop sends the user’s question, executes any tool calls the LLM requests, returns the results, and repeats until the LLM produces a final text response or hits a configurable iteration limit (default 10). Each tool invocation should be emitted as a progress event so the frontend can show what the agent is doing.

Crucially, the system should not speculate beyond what the retrieved code supports. If a question can’t be fully answered from what’s been ingested, it should say so explicitly rather than making things up.

Testing:

• Ask what a specific function does. The explanation should be accurate, in plain language, and cite the file and function name.

• Ask how two parts of the codebase relate (e.g. “how does the router connect to the request handlers?”). The system should retrieve relevant code from both areas and explain the relationship.

• Verify the agent uses tool calls to gather information iteratively - it should invoke search_code, read_file, or list_files as needed and you should see progress events for each tool invocation.

• Ask a question that requires multiple tool calls to answer fully (e.g. “trace the request lifecycle from entry point to response”). Verify the agent makes several tool calls before producing a final answer.

• Ask a question that the codebase only partially answers. Verify the system explains what it can and explicitly flags what it can’t determine from the code.

• Ask about a concept that has multiple implementations in the codebase. Verify the system surfaces all of them and explains how they differ.

Step 5

In this step your goal is to add project management so a developer can maintain separate knowledge bases for different codebases.

A developer working across multiple projects needs each one indexed and searchable independently. Add support for named projects. Store project metadata - name, source path, creation date, last ingestion timestamp, file count, and chunk count - in Oracle Database alongside your embeddings. Each project should store its embeddings, metadata, and any agent memory in isolation, so queries against one project never return results from another.

The user should be able to create a new project, list existing projects, and select which project to query. When a codebase is re-ingested into an existing project, only the changed files should be re-embedded - unchanged files should keep their existing embeddings. This makes re-ingestion fast even for large projects.

All project data should persist between runs in Oracle Database. When the user comes back tomorrow and selects a project, everything should be exactly as they left it.

Testing:

• Create two projects from two different codebases. Query each one and verify the results come only from the correct project.

• List your projects and verify both appear with the correct names and metadata (source path, file count, last ingestion time).

• Query the project metadata directly in Oracle Database and verify it matches what the system reports.

• Modify a single file in one of your test codebases, re-ingest, and verify that only the changed file’s chunks are re-embedded. Unchanged files should not be re-processed.

• Stop and restart your system. Verify all project data is still intact and queryable.

Step 6

In this step your goal is to build a web interface for browsing and querying your indexed codebases.

The web interface should launch automatically when the system starts, opening in the default browser or displaying the local URL clearly in the terminal. Build it as a three-panel layout: a file tree on the left, a chat panel in the centre, and a code viewer on the right.

The chat panel should present a chat-style conversation interface. User and assistant messages appear in a scrollable history as distinct message bubbles. When follow-up mode is enabled (the default), conversation history is sent with each request so the LLM can reference prior exchanges. A “New Chat” button resets the conversation. Assistant responses should render markdown and include source citations with expandable code snippet previews. Clicking a file reference in a citation or the file tree opens the code in the viewer panel alongside the chat.

The web interface should also include a dedicated Projects page, separate from the code exploration view, for managing projects. From this page, users can create new projects by specifying a name and source (local path or GitHub URL), trigger ingestion or re-ingestion, and monitor progress in real time. Progress updates should stream via Server-Sent Events (SSE), showing the current phase and batch progress (e.g. “Embedding chunks: batch 3 of 10”). On completion, display a summary of chunks stored and files skipped, updated, or removed. Concurrent ingestion on the same project should be prevented.

Add a browsable file tree of the ingested codebase structure, built from the stored metadata. This gives the user a visual overview of the project layout without needing to look at the actual file system.

While the agent is processing a query, show a loading state so the user knows something is happening.

Testing:

• Start the system and verify the web interface launches and is accessible in your browser.

• Select a project and ask a question through the chat panel. Verify the response appears as a message bubble with markdown rendering, source citations, and expandable code previews.

• Ask a follow-up question and verify the system uses conversation history to maintain context.

• Click “New Chat” and verify the conversation resets.

• Click a file reference in a citation and verify it opens in the code viewer panel.

• Switch between projects and verify the results update to reflect the selected project.

• Browse the file tree and verify it accurately reflects the structure of the ingested codebase.

• Create a new project from the Projects page by providing a name and source path. Trigger ingestion and verify progress streams in real time, showing the current phase and batch progress.

• Attempt to start a second ingestion on the same project while one is running. Verify it is prevented.

• Submit a query and verify a loading indicator appears while the response is being generated.

Step 7

In this step your goal is to add agent memory so the system remembers what’s been explored and what the developer has told it about the project.

Without memory, every session starts from scratch. The developer re-explains the same context, re-asks the same orientation questions, and the system re-explains things it’s already covered. Memory changes that.

Implement two types of memory, stored in Oracle Database alongside your code embeddings. Episodic memory tracks which areas of the codebase the developer has already explored, so the system can avoid re-explaining concepts that have already been covered. Semantic memory stores project-level context that the developer provides - things like “this service owns all payment logic” or “the legacy auth module is being deprecated” - and applies it to future responses within that project.

Use LangGraph to build a memory-aware query graph. When a question arrives, the graph should first check memory for relevant prior context, then decide how to handle the query: if the user has already explored this area, route to a node that builds on prior understanding rather than explaining from scratch; if it’s new territory, route to a full retrieval and explanation. This routing logic is where LangGraph’s state machine model pays off - each node in the graph handles one concern (check memory, retrieve code, generate explanation, update memory) and the edges encode the decision logic.

Since you already have Oracle Database storing your code embeddings, it’s a natural home for memory too. Store memory entries as vectors so they can be retrieved by semantic similarity - when a developer asks a question, the system can search its memory for relevant prior context the same way it searches the codebase for relevant code. Keep memory isolated per project, just like your code embeddings.

Both types of memory should persist across sessions in Oracle Database. When the user returns to a project, the system should pick up where it left off. When asked, the system should be able to provide a summary of what’s been explored so far and what remains unvisited.

Provide a dedicated Memory page in the web interface, accessible from the main navigation alongside the Explorer and Projects pages. The page should show two sections: episodic memory (exploration history with queries, files explored, summaries, and timestamps) and semantic memory (developer-provided context with content and timestamps). Users can search across both memory types using a unified search that combines text matching and vector similarity. Semantic memory entries can be added, edited, and deleted individually. Both memory types support individual deletion and bulk clear operations with confirmation dialogs.

Testing:

• Explore several areas of a codebase across a session. End the session, start a new one, and ask the system what you’ve explored so far. It should accurately summarise the areas you’ve already covered.

• Tell the system something about the project (e.g. “the payments module is the most critical part of this service”). In subsequent queries, verify the system uses this context to inform its responses.

• Ask about something you’ve already explored. The system should recognise this and build on prior understanding rather than explaining from scratch.

• Ask for a summary of what’s been explored versus what remains unvisited. Verify it gives a reasonable breakdown.

• Open the Memory page and verify it shows episodic and semantic memory entries with timestamps.

• Add a new semantic memory entry from the Memory page. Return to the Explorer and verify the system uses it in subsequent responses.

• Edit and delete individual memory entries. Verify changes take effect immediately.

• Use the search function on the Memory page and verify it finds entries by both keyword and meaning.

• Use the bulk clear operation on episodic memory and verify all entries are removed after confirmation.

Step 8

In this step your goal is to add intelligent navigation so developers can drill deeper into code and follow connections naturally.

A good exploration tool doesn’t just answer isolated questions - it helps you follow threads. Start by classifying query intent using a single LLM call rather than regex patterns. The classifier should determine whether a query is a map request, a follow-up to a previous exchange, a broad exploration, or a specific question. Include recent conversation history in the classification prompt so the LLM can detect follow-ups contextually rather than relying on keyword matching.

Use LangGraph to model multi-step retrieval as a state graph. When a user asks “what calls this?”, the graph should: retrieve the current function’s code, identify references to it across the codebase, retrieve those callers, and generate an explanation that ties them together. Each step is a node in the graph, with state passed between them.

Add support for follow-up questions that drill deeper into a previous result without the user needing to re-state context. If the system just explained a function, the user should be able to ask “what calls this?” or “where is the return value used?” and get a meaningful answer. LangGraph’s state carries the conversation context forward, so the agent knows which function is being discussed.

Dependency extraction should be language-aware. Use the language field stored on each code chunk to select appropriate import and require patterns for the chunk’s language. Support at minimum Python (import X, from X import Y), JavaScript/TypeScript (import ... from, require(), import()), Go (import "pkg", multi-line import (...)), and Java (import com.example.Foo) import styles, plus class inheritance patterns for each language. For unsupported languages, fall back to a generic regex that detects common import keywords.

When the system identifies a dependency or reference in retrieved code, it should offer to retrieve and explain the linked code. In the web interface, render these as clickable elements that trigger a follow-up retrieval.

Add an exploration planning capability using LangGraph. When a user asks a broad question like “how does the authentication system work?”, the agent should plan a multi-step exploration: find entry points, trace the authentication flow through the codebase, retrieve each step, and produce a coherent walkthrough. Model this as a graph where each retrieval step feeds into the next, building up a complete picture rather than returning a single set of search results.

Finally, add a “map” query that returns a high-level summary of the codebase structure: a breakdown of languages used, top-level modules, and entry points where identifiable. The map query should identify key files that signal project structure: build configuration (pyproject.toml, package.json, Cargo.toml, go.mod, pom.xml, build.gradle, Makefile, CMakeLists.txt), container definitions (Dockerfile, docker-compose.yml), documentation (readme, changelog, contributing, license), and entry points (main., app., index., server., cli.*). This gives the developer a bird’s-eye view before diving into specifics.

Testing:

• Ask about a function, then ask a follow-up like “what calls this?” without re-stating which function you mean. The system should understand from context and return relevant results.

• Look for linked dependencies in a response. If the system identifies an import or function call, verify it offers to explain the linked code.

• In the web interface, click a dependency link and verify it triggers a follow-up retrieval and explanation.

• Verify that query classification correctly identifies a map request, a follow-up, a broad exploration, and a specific question.

• Run the “map” query and verify you get a useful high-level summary: languages, modules, key files (build config, Dockerfiles, documentation), and entry points should all be represented.

• Verify dependency extraction works across languages: test with Python imports, JavaScript/TypeScript requires, Go imports, and Java imports if your test codebase includes them.

Going Further

You’ve built a working semantic code exploration tool. Here are some ways to take it further:

• Cloud database support: Add an option to connect to a cloud-hosted Oracle Database instance instead of the local Docker container. Read the connection string from configuration so teams can share a single index.

• Multi-language parsing: Extend your parser to handle a wider range of programming languages. Tree-sitter is a good foundation for this - it provides consistent parsing across dozens of languages.

• Collaboration features: Allow multiple developers to share a project index and see each other’s exploration history. This is particularly useful during onboarding, where a senior engineer’s exploration trail becomes a guided tour for newcomers. Think about how to share the insights they provide and create a shared memory.

• IDE integration: Build an extension for VS Code or another editor that lets developers query Code Sherpa directly from their IDE, with results that link back to the exact line in the editor.

• Export and share: Let users export their exploration session - the questions asked, the code found, and the explanations generated - as a shareable document. This turns an exploration session into reusable documentation.

This coding challenge was sponsored by Oracle.

P.S. If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️ twice a month, with the bonus that you also get 20% off any of my courses.

3. Buy one of my courses that walk you through a Coding Challenge.

4. Subscribe to the Coding Challenges YouTube channel!

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #114 - Gzip

This challenge is to build your own version of gzip, the widely used file compression utility.

Gzip has been a cornerstone of computing since 1992. It’s used everywhere, compressing files on the command line, serving web content over HTTP, packaging up tarballs for distribution, and much more. Under the hood, gzip uses the DEFLATE compression algorithm (a combination of LZ77 and Huffman coding) wrapped in a simple file format defined by RFC 1952. Building your own gzip will give you a deep understanding of how data compression works, how file formats are structured, and how command-line tools handle the many options users expect.

If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️, with the bonus that you get access to a monthly AMA and 20% off any of my courses.

3. Buy one of my self-paced courses that walk you through a Coding Challenge.

4. Join one of my live courses where I personally teach you Go by building five of the coding challenges or systems software development by building a Redis clone.

The Challenge - Building Gzip

In this challenge you’re going to build your own version of the gzip compression tool. Your tool will compress and decompress files using the DEFLATE algorithm, producing output that is fully compatible with the standard gzip and gunzip utilities.

You can use a library for the DEFLATE algorithm if you want, but I’d really encourage you to implement it yourself by reading the spec. Data compression is one of those topics that sounds intimidating but becomes surprisingly approachable once you break it down. You’ll learn far more about how compression actually works by building LZ77 and Huffman coding from scratch than by calling a library function. The RFCs are well written and this challenge is structured to walk you through it incrementally.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

Choose your target platform and programming language. I’d encourage you to pick a language that gives you access to low-level byte manipulation, as you’ll be working with binary data, checksums, and bit-level operations.

Before you start coding, read through RFC 1952 (the gzip file format) and RFC 1951 (the DEFLATE compressed data format). Don’t worry about understanding every detail right now, we’ll work through the key parts step by step. The important thing is to get a feel for how the pieces fit together: gzip is a wrapper format, and DEFLATE is the compression algorithm inside it.

Step 1

In this step your goal is to produce a valid .gz file using DEFLATE stored (uncompressed) blocks.

Before you tackle actual compression, get the gzip file format right first. Your tool should take a filename as an argument, wrap its contents in a valid .gz file with the correct header and trailer, and write it out with the .gz extension appended. After writing the compressed file, remove the original (this is the default gzip behaviour).

The gzip format (RFC 1952) requires a 10-byte header containing the magic number (1f 8b), the compression method (08 for DEFLATE), and flags. It also requires a trailer containing the CRC-32 checksum of the original data and the original file size (modulo 2^32).

For the DEFLATE payload, use stored blocks (block type 00 in RFC 1951, section 3.2.4). A stored block simply contains the raw data with a small header — no actual compression. This lets you get the gzip wrapper, CRC-32 calculation, and file handling working correctly before you add real compression. Your output will be valid gzip, just larger than the input.

Testing: Create a test file and compress it with your tool, then decompress it with the system gunzip to verify your format is correct:

echo "Hello, World!" > test.txt
ccgzip test.txt
gunzip test.txt.gz
cat test.txt

You should see Hello, World! and the file should be identical to the original. The .gz file will be slightly larger than the original since you’re not compressing yet — that’s fine, the important thing is that gunzip accepts it.

Step 2

In this step your goal is to implement LZ77, the first half of the DEFLATE algorithm.

LZ77 works by sliding a window over the input data and looking for sequences that have already appeared. When it finds a match, instead of storing the bytes again, it stores a (length, distance) pair — “copy 5 bytes from 12 positions back”. This is how DEFLATE eliminates repeated patterns.

Implement a sliding window (up to 32,768 bytes as per the spec) and a match-finding algorithm. For each position in the input, search the window for the longest match. If you find a match of 3 bytes or more, emit a (length, distance) pair. Otherwise, emit the literal byte.

At this point, encode your LZ77 output using DEFLATE fixed Huffman codes (RFC 1951, section 3.2.6). Fixed codes use a predefined Huffman table built into the spec, so you don’t need to build your own trees yet — you just need to emit the right bit sequences for literals, lengths, and distances.

Testing: Compress a file with your tool and decompress with gunzip:

echo "abcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabc" > test.txt
ccgzip test.txt
gunzip test.txt.gz
cat test.txt

The .gz file should now be smaller than the original for repetitive data. Compare the compressed size to what the system gzip produces — yours won’t be as good yet (we’ll get there), but it should be noticeably smaller than the stored blocks from Step 1. Try it on a larger text file to see a more dramatic difference.

Step 3

In this step your goal is to implement Huffman coding, the second half of DEFLATE.

Fixed Huffman codes work, but they’re not optimal for every input. Dynamic Huffman codes (RFC 1951, section 3.2.7) let you build custom Huffman trees tailored to the actual frequency of symbols in your data, which gives much better compression.

Build a Huffman tree from the frequencies of the literal/length and distance symbols in your LZ77 output. Encode the tree itself into the DEFLATE block header (the spec describes exactly how to do this using code length codes), then encode the data using your custom tree.

This is the trickiest part of the challenge, take it slow and test frequently. The encoding of the Huffman tree in the block header is fiddly, with its own mini-Huffman encoding for the code lengths. The RFC walks through it methodically; follow it closely.

Testing: Compress files with your tool and compare the sizes to the system gzip:

echo "abcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabc" > test.txt
cp test.txt test-original.txt
ccgzip test.txt
ls -la test.txt.gz
gunzip test.txt.gz
diff test.txt test-original.txt

Rename one of them to compare both. Your compressed output should now be close to what the system gzip produces. Test with a variety of files — text, source code, binary — and decompress each with gunzip to make sure everything round-trips correctly.

Step 4

In this step your goal is to decompress a .gz file using the -d flag.

When your tool is called with -d, it should read a .gz file, validate the gzip header, decompress the DEFLATE data, and write the original content to a new file with the .gz extension removed. After decompression, the .gz file should be removed.

You’ll need to reverse everything you built in the previous steps: parse the DEFLATE block headers, reconstruct the Huffman trees (for dynamic blocks), decode the symbols, and replay the LZ77 back-references to reconstruct the original data.

Your tool should validate the CRC-32 checksum and original file size stored in the gzip trailer. If the checksum doesn’t match the decompressed data, your tool should report an error and exit with a non-zero status code — this is how gzip detects corrupt files.

Testing: Compress a file with the system gzip, then decompress it with your tool:

echo "Testing decompression" > test.txt
gzip test.txt
ccgzip -d test.txt.gz
cat test.txt

You should see Testing decompression. Also test with a corrupted file — use a hex editor to change a byte in a .gz file and verify your tool reports a CRC error.

Step 5

In this step your goal is to support reading from standard input and writing to standard output, along with the -c, -k, and -f flags.

When no filename is provided, your tool should read data from stdin, compress it, and write the compressed output to stdout. When decompressing, it should read compressed data from stdin and write the decompressed output to stdout.

Implement the -c flag, which writes compressed or decompressed output to stdout instead of to a file, leaving the original file unchanged. Implement the -k flag, which keeps (does not delete) the original file after compression or decompression. And implement the -f flag, which forces overwriting of existing output files without prompting.

Testing: Test piping data through your tool:

echo "Piped data" | ccgzip | gunzip
echo "Piped data" | gzip | ccgzip -d

Both should output Piped data. Test -k and -f:

echo "Keep me" > test.txt
ccgzip -k test.txt
ls test.txt test.txt.gz
ccgzip -f test.txt

The -k flag should leave both files in place. The second compression should succeed because -f forces the overwrite of the existing .gz file.

Step 6

In this step your goal is to support compressing and decompressing multiple files and to implement compression levels.

Your tool should accept multiple filenames as arguments and process them one at a time. If any file fails, your tool should report the error and continue processing the remaining files.

Implement compression levels from -1 (fastest, least compression) to -9 (slowest, best compression), with -6 as the default. These levels control how aggressively the LZ77 stage searches for matches — at level 1, you might limit the search to short look-aheads and a smaller window, whilst at level 9, you search more thoroughly for the longest possible matches.

Testing: Test multiple files:

echo "File one" > a.txt
echo "File two" > b.txt
echo "File three" > c.txt
ccgzip a.txt b.txt c.txt
ls *.gz

You should see a.txt.gz, b.txt.gz, and c.txt.gz. Test compression levels on a larger file:

ccgzip -1 -k test.txt && mv test.txt.gz test-fast.gz
ccgzip -9 -k test.txt && mv test.txt.gz test-best.gz
ls -la test-fast.gz test-best.gz

For a sufficiently large file, the -9 version should be smaller than the -1 version.

Step 7

In this step your goal is to implement the -l, -t, and -v flags.

Implement the -l flag, which displays compression statistics for a .gz file without decompressing it. The output should include the compressed size, uncompressed size, compression ratio, and the original filename. The format should match the standard gzip output:

         compressed        uncompressed  ratio uncompressed_name
                 73                  26  42.3% test.txt

Implement the -t flag, which tests the integrity of a .gz file by decompressing it and validating the CRC-32 checksum without writing the output to disk. If the file is valid, it exits silently with status 0. If it’s corrupt, it reports an error.

Implement the -v flag, which displays the name and compression ratio for each file as it is processed. This is useful when compressing multiple files so you can see progress.

Testing: Test the -l flag:

gzip -k test.txt
ccgzip -l test.txt.gz
gzip -l test.txt.gz

Compare the output of both commands — they should show the same statistics. Test -t on a valid and a corrupted file. Test -v by compressing multiple files and checking that each one shows its name and ratio.

Step 8

In this step your goal is to implement recursive directory compression, preserve file metadata, and ensure full compatibility with the standard gzip tools.

Implement the -r flag, which recursively traverses directories and compresses (or decompresses) all files found within them.

The gzip header has optional fields for the original filename and the modification timestamp of the source file. Your tool should store these when compressing and restore the modification timestamp when decompressing.

Your output should be fully compatible with the system gzip and gunzip , any file compressed by your tool should decompress correctly with gunzip, and any file compressed by gzip should decompress correctly with your tool. The .gz extension should be added on compression and removed on decompression.

Testing: Test recursive compression:

mkdir -p testdir/subdir
echo "Root file" > testdir/file1.txt
echo "Sub file" > testdir/subdir/file2.txt
ccgzip -r testdir
find testdir -name "*.gz"

You should find testdir/file1.txt.gz and testdir/subdir/file2.txt.gz. Test timestamp preservation:

touch -t 202301151200.00 test.txt
ccgzip -k test.txt
ccgzip -d test.txt.gz
stat test.txt

The modification timestamp should match the original. Test full compatibility by compressing a variety of files (text, binary, empty) with both your tool and the system gzip, and cross-decompressing them to verify they produce identical output.

Going Further

Here are some ideas to take your gzip implementation further:

• Add support for concatenated gzip streams (multiple gzip members in a single file)

• Implement the -rsyncable option which makes the compressed output more friendly to rsync’s delta transfer algorithm

• Add support for the -suffix option to use a custom file extension

• Build a parallel compression mode (like pigz) that uses multiple CPU cores to compress data faster

• Add support for decompressing other formats that gzip can handle, such as compress (.Z) files

• Experiment with different match-finding strategies (hash chains, binary trees, optimal parsing) and measure the compression ratio and speed trade-offs

P.S. If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️ twice a month, with the bonus that you also get 20% off any of my courses.

3. Buy one of my courses that walk you through a Coding Challenge.

4. Subscribe to the Coding Challenges YouTube channel!

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John

🙏 Thank you for being a subscriber, I’m honoured to have you as a reader. 🎉

If there is a Coding Challenge you’d like to see, please let me know by replying to this email📧

Coding Challenge #113 - AI Writing Detector

This challenge is to build your own AI writing detector that analyses text and determines the likelihood it was written by an AI rather than a human.

Some people believe and argue that AI-generated text has recognisable patterns and characteristics. They argue that language models tend to favour certain vocabulary, use particular phrasing structures, and employ specific rhetorical techniques. To a certain extent they’re right, LLMs can indeed sometimes produce text that is often unnaturally uniform in sentence length, excessively formal in tone, and littered with vague attributions that sound authoritative but say nothing specific. By identifying these patterns, you can build a system that scores text and provides a detailed analysis of what makes it look AI-generated or human-written.

By building this project you’ll learn how to implement some complex text analysis and just how hard it truly is to detect AI written text. Once you have a working solution try it on some famous text that pre-dates LLMs (i.e. written before 2017). The reality is it is hard to detect AI writing, but a great learning exercise.

If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️, with the bonus that you get access to a monthly AMA and 20% off any of my courses.

3. Buy one of my self-paced courses that walk you through a Coding Challenge.

4. Join one of my live courses where I personally teach you Go by building five of the coding challenges or systems software development by building a Redis clone.

The Challenge - Building an AI Writing Detector

You’ll build a system that accepts text input and analyses it using a rule-based approach to detect linguistic patterns and characteristics commonly found in AI-generated writing. Your detector will evaluate text across multiple dimensions — vocabulary, sentence structure, rhetorical patterns, and statistical properties — and produce a detailed report showing the likelihood the text was AI-generated. And no, the em-dash ‘—‘ is not one of them!

This is a coding challenge that focuses on text analysis, pattern detection, and building a scoring system. You can implement it in any programming language you’re comfortable with.

Step Zero

In this introductory step you’re going to set your environment up ready to begin developing and testing your solution.

Choose a programming language you’re comfortable with for text analysis and string manipulation. You’ll be doing a fair amount of tokenisation, regular expression matching, and statistical calculation. Consider whether you’d like to build a command-line tool, a web-based interface, or both.

For testing, you’ll evaluate your detector against sample texts. Some AI-generated and some human-written. Prepare a small collection of texts to test against as you build. You can generate AI-written samples by asking any large language model to write an essay on a topic, and use your own writing or published articles for the human-written samples.

Step 1

In this step your goal is to accept text input for analysis and display basic statistics about it.

Your system should provide a way for users to submit text. Whether that’s reading from a file, accepting command-line input, or providing a text area in a web interface. As text is entered, display a character count and word count. If the input is empty, the analysis should not proceed.

Testing guidance: Load a sample text and verify that your character count and word count are correct. Try submitting empty input and confirm your system handles it gracefully by preventing the analysis from running.

Step 2

In this step your goal is to identify vocabulary and phrases that are commonly overused in AI-generated text.

Language models have characteristic vocabulary patterns, they favour words like “delve into”, “navigate”, “robust”, “innovative solutions”, “transformative”, “leverage”, “streamline”, and “ecosystem”. Build a detector that scans for these LLM-signature words and phrases and tracks how many distinct ones appear. You can find examples in Wikipedia’s Signs of AI writing article.

Score this detector based on the number of distinct AI vocabulary terms found, with each contributing a few points up to a maximum cap. This gives you your first pattern detection category.

Testing guidance: Write or generate a short AI-written essay and run your detector on it. You should find multiple hits. Then run it on a piece of your own writing. You should see far fewer. Verify that the score increases with more AI vocabulary terms but doesn’t exceed your cap.

Step 3

In this step your goal is to identify several common structural patterns in AI writing. Language models frequently use:

• The “rule of three” — describing things in groups of three adjectives, three nouns, or three parallel phrases. AI text is full of these.

• Negative parallelism — rigid constructions like “not only... but also” and similar overly formal parallel structures.

• Outline-style conclusions — formulaic endings that follow the pattern: “Despite [challenges], [subject] offers [benefits/opportunities].”

• False ranges — “from X to Y” constructions where the two endpoints don’t form a coherent or logical scale.

Build detectors for each of these patterns. When you find them, record where they occur in the text and contribute to the score. Each pattern category should have its own maximum cap so that no single category can dominate the overall score. Again you can find examples in Wikipedia’s Signs of AI writing article.

Testing guidance: Craft test sentences for each pattern type and verify your detectors find them. For the rule of three, try: “This approach is efficient, scalable, and maintainable.” For negative parallelism, try: “It not only improves performance but also enhances reliability.” Test that your scoring caps are working, text saturated with one pattern type should hit the cap and stop accumulating.

Step 4

In this step your goal is to identify patterns where text makes broad claims without concrete evidence. AI writing often lacks specificity, hiding behind phrases that sound authoritative but name no actual sources.

Build detectors for three categories:

• Vague attributions — phrases like “experts agree”, “studies show”, “research indicates”, and “industry insiders report” that appeal to unnamed sources.

• Superficial analysis — hedging and filler like “it is worth noting”, “significant developments”, “one could argue”, and “various sources indicate”.

• Overgeneralisation — patterns that frame limited information as universal, such as “everyone knows”, “it is well established”, and “universal consensus”.

Each category should contribute to the score independently, with its own cap.

Testing guidance: Write test sentences containing each type of vague language and verify your detectors find them. Then test with text that contains legitimate attributions (”According to a 2024 study by Smith et al. in Nature...”) and confirm your detector doesn’t flag specific, concrete citations.

Step 5

In this step your goal is to identify excessive emphasis and promotional language patterns. AI writing tends to oversell, using superlatives and marketing-style phrasing that reads more like advertising copy than natural writing.

Build detectors for three categories:

• Undue emphasis — excessive use of superlatives, intensifiers (”tremendous”, “remarkable”, “groundbreaking”), and emphatic punctuation.

• Promotional language — marketing phrasing like “game-changer”, “revolutionary”, “impressive features”, and “transformative potential”.

• Elegant variation — where the same concept or entity is repeatedly referred to by different but equivalent terms across sentences. AI text often cycles through synonyms for the same thing rather than simply repeating the word as a human would.

Testing guidance: Run your detectors on marketing copy or AI-generated product descriptions — these should score highly. Compare with straightforward human writing about the same topic. For elegant variation, look for text where “the company” becomes “the organisation” becomes “the firm” becomes “the enterprise” within a few sentences.

Step 6

In this step your goal is to move beyond pattern matching and analyse the statistical properties of the writing itself. AI text has measurable differences from human writing in several dimensions.

Build analysers for some or all of these linguistic factors:

• Lexical diversity — calculate the type-token ratio (unique words divided by total words). AI text often falls outside the normal range, either too uniform or artificially varied.

• Sentence length variation — measure the standard deviation and coefficient of variation of sentence lengths. Human writing naturally varies; AI text tends to be unnaturally uniform, with a coefficient of variation below 0.35.

• Passive voice frequency — identify passive voice constructions and calculate what percentage of sentences use them. Human writers typically use passive voice in 5-10% of sentences; AI text often exceeds 15%.

• Transition word density — detect formal discourse markers (”furthermore”, “moreover”, “consequently”, “additionally”) and calculate the percentage of sentences containing them. More than 20% suggests AI generation.

• Reading grade level — calculate the Flesch-Kincaid Grade Level. AI text often scores at an artificially high grade level (above 14), suggesting unnecessary complexity.

• Punctuation patterns — analyse the density of semicolons, em-dashes, colons, and ellipses. AI text tends to overuse semicolons and em-dashes whilst rarely using ellipses.

• Rare word usage — identify uncommon words and calculate their frequency. Human writers typically use rare words at 3-8% frequency; AI text often exceeds 12%.

Display each factor as a labelled indicator with a percentage value and a brief explanation of what the result means. Beware that some of these fail on short text.

Testing guidance: Run your analysers on both AI-written and human-written samples of similar length and topic. Compare the results — you should see measurable differences. Pay particular attention to sentence length variation and transition word density, which tend to be strong signals. Verify your Flesch-Kincaid calculation against an online readability calculator.

Step 7

In this step your goal is to aggregate all your pattern detections and linguistic analyses into an overall AI probability score from 0 to 100.

Combine the contributions from each detector. If the raw combined score exceeds 100, normalise the individual contributions proportionally so the final score is clamped to 100.

Then classify the text based on the score:

• Below 30: “Likely Human-Written”

• 30 to 59: “Possibly AI-Generated”

• 60 or above: “Likely AI-Generated”

Display the score with a colour-coded indicator, green below 30, yellow for 30-59, red for 60 and above.

Testing guidance: Run your complete detector on several sample texts. Verify that AI-written samples score above 60 and human-written samples score below 30. If your scores don’t separate well, experiment with the weighting of different detectors. Check that the individual detector contributions add up correctly to the total score, and that normalisation works when the raw total would exceed 100.

Step 8

In this step your goal is to produce a comprehensive report that shows what was detected and why the text received its classification.

Your report should include:

• The overall AI probability score and classification

• Text statistics (word count, character count, average word length)

• A breakdown of linguistic factors, each with its score and an explanation

• A breakdown of pattern detections, each showing the category, occurrence count, score contribution, and explanatory text

• The timestamp of when the analysis was performed

Present the linguistic factors and pattern detections in separate labelled sections so the report is easy to scan.

Testing guidance: Generate reports for both AI-written and human-written samples. The reports should tell a coherent story about why each text received its classification. Verify that the pattern breakdowns add up to the overall score. Check that the report is readable and the explanations make sense to someone who doesn’t know the internals of your system.

Going Further

Once you’ve built the core detector, here are ways to extend it:

• Text highlighting — mark detected patterns directly in the original text with category-specific colours. When highlights from different detectors overlap, keep the first one and discard subsequent overlaps. Display category badges showing which patterns were found.

• Advanced linguistic analysis — implement Zipf’s Law comparison (comparing word frequency distribution against the expected power-law distribution), named entity density analysis, and paragraph coherence measurement through inter-sentence similarity.

• Copy and share — add a “Copy Results” button that copies the score and analysis to the clipboard, and an “Analyse Another Text” option to return to the input.

• Accuracy measurement — collect a larger corpus of human-written and AI-written samples and measure your detector’s precision and recall.

• Weighting experiments — try different weightings for each detector. Some patterns are stronger signals than others — which ones matter most?

P.S. If You Enjoy Coding Challenges Here Are Four Ways You Can Help Support It

1. Refer a friend or colleague to the newsletter. 🙏

2. Sign up for a paid subscription - think of it as buying me a coffee ☕️ twice a month, with the bonus that you also get 20% off any of my courses.

3. Buy one of my courses that walk you through a Coding Challenge.

4. Subscribe to the Coding Challenges YouTube channel!

Share Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know via Bluesky or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo

Request for Feedback

I’m writing these challenges to help you develop your skills as a software engineer based on how I’ve approached my own personal learning and development. What works for me, might not be the best way for you - so if you have suggestions for how I can make these challenges more useful to you and others, please get in touch and let me know. All feedback is greatly appreciated.

You can reach me on Bluesky, LinkedIn or through SubStack

Thanks and happy coding!

John