Finding AI’s Niche

Local LLMs have been really interesting to me recently, so much so that I picked up an NVidia DGX Spark. While I still maintain that reading AI slop is utterly demoralizing, there are some parts of it that I find help me a lot. Partly as an exercise in self discovery, and partly to find where AI is overwhelmingly useful, I have explored using it on some personal projects. I hate the idea of letting a clanker have all the fun of programming, but not all programming is unique and exciting.

The biggest litmus test for “Should I use AI for this?” is “Would I do this myself?”. If you, the reader, wouldn’t personally do something, then don’t have an AI do it for you. Your agent acts as an extension of you. Having it do something you wouldn’t means it isn’t you, yet so many people allow it to use their name, their logins, and thus their personal brand.

Unit Tests

For example, writing unit tests is pretty dull work. The coding standard for tests is much lower, with increased grace for things like copy-pasting code, repetitive set up, barely any documentation, excess field and method visibility, no immutability rules, etc. We don’t hold test code to the same standard as regular code.

But, we still write tests. We all agree tests are useful to have. If we did have time, we would write them. Both at work and at home. Thus, AI is okay. It still needs a lot of hand-holding, but it has accelerated my coding allowed me to alt-tab to other things while it churns in the background. I get the most satisfaction out of writing the main code itself, and the (unit) tests are just insurance that the code does what I think it does. It doesn’t really make me a better programmer, and it takes time. It’s easy to zone out while writing unit tests: something that is hard to do while writing real code.

Caveat: Don’t have the AI write both the main code and the tests. It’s the fox watching the hen house. Do one, or the other, but not both.

Build Files

I have not, and probably will never, learn how Gradle works. Despite having used it for nearly a decade, and having hand written thousands of lines of it, I still don’t get it. I’ve tried. Gradle’s documentation is impenetrable. It seems like it’s written as a reference, rather than for people who just kinda know what they want. There is some obscenely complex model for projects, configurations, tasks, plugins, closures, and what not. However, knowledge of Gradle, in my experience, can never be boiled down to something that earns its keep in my working memory. Every few weeks, Gradle releases another minor version that breaks something one of my plugins depends on.
The huge amount of churn in APIs means there’s pretty much no point in learning it; it will be different next month.

Thus, my next litmus test: Do I want to get better at this skill? Am I okay with slowly getting worse at it?

For build files (Gradle, Make, Bazel, etc.), the answer is pretty much no. It is not my life’s purpose to master building projects. LLMs are extremely good at reading through all the build documentation for these tools. Hunting down where this documentation even is is something they excel at. Did I look at the Gradle blog? Did I make sure to check the git commit notes for the most recent Bazel release? Was there some note posted on a far off, newly created git repo that boldly clarifies the new best practices of Bazel? The answer is no, and it’s a waste of time to go on an expedition every time I encounter some build issue. LLMs have none of these problems. They are machines and love rote, methodical work. Best of all, they shed some human biases, such as assuming they have already checked something. Or misremembering important details.

Fast Reconnaissance

Recently, I debugged an OOM with a machine learning product I built. While spelunking through process boundaries and unfamiliar libraries is fun, I also need to be careful about yak shaving. The issue was how Java, XGBoost, and OpenMP manage threads. Java code, and the JVM, are both pretty familiar to me and I can quickly zip through where they might have bad behavior. However, I only have a novice understanding of how XGBoost and OpenMP interact.

To understand how these pieces work, and if the calendar read “2023”, I would have Google’d my questions. Hopefully some curious person would have asked the same question, and some grizzled pro would answer. I would find my answer on a badly HTML-rendered mailing list, or a Stack Overflow page, closed for having been useful-but-not-objective. It’s in places like these that small sparks of insight are found. Even if they aren’t precise matches to my query, they are enough to unwedge my mind.

Therefore, the last litmus test I’ll proffer is Would I have Google’d this?. Generally the answer is “yes, obviously!” I do greatly fear losing my skills, such as no long being able to diagnose and overcome technical problems. But, if I am being honest, using the LLM seems substantially equivalent to searching online for someone else having solved it.

I eventually found out that XGBoost uses OpenMP to create teams of threads, but has no way to globally limit the number of threads. OpenMP’s docs confirmed this, and XGBoost provides no way to close the thread pool down. I worked around the issue by managing the top-level Java threads better, seeing as neither XGboost nor OpenMP provided a way to fix it. Without being able to ask complex questions and using local copies of their git repos, it would have taken a very long time to track down the OOM. That said, I don’t take the LLMs response for granted. It’s just a machine. We still need to retain a hint of doubt, and verify what it says is true. This is especially critical when working in an area I am not familiar with, as I am more susceptible to subtle, but believable, lies.

Respectful AI Usage

Using AI day to day is still immature in the workplace today. I think we need to establish etiquette around code generation, commit and PR messages, and Slack thread chatter. I find it rude when someone’s bot spews a vomitrocious amount of text into the commit message. It’s de-humanizing how I asked people questions, and they respond with “Claude said…” (great, why do you think I couldn’t do that myself?)

Using the first litmus test above (Would I do this myself?) I can’t in good conscience say I would write a novela’s worth of text in my commit message, or respond to people’s inquiries with mechanically-separated drivel.

Humans come first. AI is so new, it’s like we are deer staring into headlights. We are amazed that it can do so much, but yet we forget how using it makes other people feel. The Golden Rule comes in to play here: how would I feel if someone hurled slop across the Internet into my face? If I wouldn’t read it myself, why would I write it? If I wouldn’t trust it myself, why would I expect others to?

Aside: unfortunately, some companies have proclaimed “Use as much AI as possible, or else…” I think this has pressured otherwise respectful people to use LLM slop in improper places.

Conclusion

AI and LLMs have been a marvelous addition to humanity, but we should be prudent about when and where they are appropriate.

Futures Should Be Composable

Recently, and for the first time in my career, I have the opportunity to implement a large, highly concurrent application. After seeing a lot of interest in Java’s new Virtual Threads, I decided to try them out. I’ve done a decent amount of concurrent programming, so I felt like it would be relatively easy to get the project started and grow it. From what I encountered, Java’s Future abstraction is not up to the task.

Without too much backstory of what I looked at, I made a bet: Futures and blocking are the right abstraction. This means:

• Avoid any sort of callback hell. Generally, blocking get() calls are the right way. This is the same bet that Golang makes, with blocking being the norm. Lean into the scheduler to make the code work.

• Skip flow control. Reactive Java, like Mono and Flux are work arounds for the Java problems of a decade ago. The stack traces are impossible to understand, and the exception handling in general doesn’t mesh with the rest of the language. It served its purpose, but avoid it.

• Avoid API dependence on CompletableFuture. This class is bloated to the max. Every time I want to call a method I need to look at the docs, then get mad the code has nearly zero Javadoc, Look up the CompletionStage for the specification, and finally scratch my head as to how it’s subtly different than nearly identically named methods nearby. Also, CompletionStage is practically impossible to implement, and is missing all the useful methods of Future.

Thus, I decided to make Future<T> the standard return type and interface of choice for my code.

Threads and Futures

In Java, Futures were designed around a thread pool. The idea being work would be scheduled onto an Executor(Service), and return a Future result which would eventually be populated by another thread.

Unlike Futures or Promises in other languages (notably Javascript), Futures in Java had the concept of being associated with a thread. To see why this is interesting, take a look at this method on the interface:

public interface Future<V> {

    /**
     * Attempts to cancel execution of this task.  ...
     */
    boolean cancel(boolean mayInterruptIfRunning);
}

Two things are interesting here:

1. Futures can be cancelled. Other languages and libraries often do not support cancellation. This is a special feature.
2. Interruption. Java has a special feature of Threads called interruption, which allows other threads to ask a sleeping thread to wakeup, and possibly stop waiting for some event.

This is a very useful thing to have, as it means we don’t have to commit to doing the work in the thread, should things change. As far as I have seen, almost no other programing model has this as a core part. Additionally, to implement this yourself, it would be challenging. As a quick thought experiment, I would ask the reader to think how they would implement this? Using something as basic as pthread_cond_signal and pthread_cond_wait require holding a lock, which Java’s implementation doesn’t! How did they do it? (See the link above for an explanation of the magic).

Thus, Futures, at least when introduced originally, strongly implied attachment to a thread.

Threadless Futures

As of Java 25, there are three main implementations of Futures in the JDK:

1. FutureTask. This is both a Runnable and a Future, and is intended to be extended. It holds a reference to the “runner” Thread, which is mounted and unmounted when the task runs and completes, respectively. Cancelling this Future can interrupt the runner Thread.

2. CompletableFuture. While I think the implementation is way overengineered, it is the more powerful of the implementations. It is full featured, and has a solid, reliable way to chain work together.

3. ForkJoinTask. This happens to be a Future, but I haven’t seen anyone seriously use it as one. I mention it here for completeness, but it’s more meant for Fork-Join style work, and less for complex, heterogeneous work items.

CompletableFuture is the main implementation of interest, since it is capable of building a general DAG of computation.

Consider the above. CompletableFuture, hereafter “CF”, is a general purpose computation tool. The dependency graph between Future stages is built dynamically, meaning the whole graph is not known ahead of time. Each CF can be used to notify multiple downstream CFs. Two CFs can be used to complete a single downstream CF. The key takeaway here is that any individual CF does not know what other CFs depend on it.

A consequence of this design decision is that cancellation doesn’t have a clear meaning for CFs. What does it mean for a CF to be cancelled, and the mayInterruptIfRunning bit is set? The CF may be a combination of many other CFs. There may be no thread at all attempting to fulfill a particular CF. The linkage between a CF and a Thread is weakened. As a result, CompletableFuture does not cancel the underlying thread. (because there may not even be a thread)

Cancellation and Bi-directionality

Is giving up cancellation that big of deal? Well, maybe. In the world that CF was born into, threads may not have played as big a role. CF is decidedly push based, despite its predecessor being pull based. As computations complete, they pop their Treiber Stack of dependent CFs and fulfill them. Each downstream CF in the DAG is completed, usually on the thread that is completing the current CF. (As an aside, this is one of the reasons there are a jillion overloads in CF; they needed a way to schedule the downstream “callback” work potentially on a different thread.) Keeping track of which thread is doing the async work may not have been that valuable. Since the idea of a thread working hard to fulfill a future is gone, where’s the need to interrupt the thread?

Enter Virtual Threads. It’s now possible to have as many Goroutines Green Threads M:N Threads Virtual Threads as you want. They can all block without consequence waiting for CPU or IO bound work to complete as they patiently await to fulfill a Future. The idea and value of cancellation now seems more tenable.

CompletableFuture and Chaining

Let’s look at how CFs chain together, in a simple, unidirectional chain.

When CF 1 completes, it notifies (and completes) CF2. When CF2 completes, it notifies CF3. The flow is from left to right. Control flow only goes one direction. Consider the following snippet of code:

// Build the HTTP Request
CompletableFuture<HttpRequest> requestFuture = 
    CompletableFuture.completedFuture(request);

// Issue the request
CompletableFuture<byte[]> httpResponse = 
    requestFuture.thenComposeAsync(
        req -> fetchHttp(req), executor);

// Validate and convert the response
CompletableFuture<MyObject> parsedResult =
    httpResponse.thenComposeAsync(
        rawJson -> validateAndConvert(rawJson), executor);

System.out.println(parsedResult.get());

Each stage depends on the previous one.

Why Cancellation Matters

Using the snippet above, instead of printing the result, suppose the parsedResult CF is returned to a caller. Also suppose that the caller is an RPC, and the RPC is cancelled for whatever reason. We want to cancel the work being done to avoid consuming memory and threads. How well does this work?

Despite CFs being chained together, they are only chained in one direction! Whoever cancels the CompletableFuture<MyObject> parsedResult object, it won’t stop the HTTP request. The parsing future, which has yet to be assigned a thread, has no way to indicate that the upstream result is no longer needed. In a sense, dependency is a singly-linked list, with no way to get back to the original CF.

You might suggest that this linkage be added, and the CF class could be made to propagate cancellation of a downstream CF to the upstream. However, this is where the DAG property bites us. Consider the following, legal, CF chain:

Cancelling one of the downstream CF’s doesn’t mean the otherones should be.

Without properly cancelling futures, it means that there is a risk of consuming limitted resources. While it may be okay to do a little extra work if the RPC client cancels their request, it’s not okay to consume all threads and connection pools on responses that will never be seen. (In my own work, we saw this result in an OOM due to a runaway executor that kept adding threads.) Cancellation matter for stability.

Bi-directionality

When thinking through a solution to this problem it becomes obvious that it can’t be solved by just added a cancel listener to each CF. Someone will eventually forget to add it and drop the link. The real problem is that the implementation of CF, and the general interface contract of Future, don’t afford it. Futures do one thing well: defer execution. However, this is not enough. The true problem is that only results and exceptions flow from one future to another but not the consumer’s interest in the result.

I have to say I unfairly judged Reactive Java here, with their fully featured cancellation and flow control mechanics. Originally I had written them off because flow control is only a seldom useful feature, and primarily between systems, rather than inside them. That said, flow control is another “consumer interest” signal like cancellation. I guess the implementers saw that cancellation and flow control nicely unified into a “subscription”, and added both. I still maintain that flow control is overkill with their request(n) call, but I can clearly see the value of cancellation propagation.

We do need bi-directionality.

Composability

Given the above history and problems, I now bring my full request: Futures should be composable. CompletableFuture did a decent job of composition for downstream dependence. However, it is not enough. We need a way to formally describe the cancellation semantics of asynchronous computation. It is an error-prone pain in the ass to write this every time:

CompletableFuture<HttpRequest> requestFuture =
    CompletableFuture.completedFuture(request);

CompletableFuture<byte[]> httpResponse =
    requestFuture.thenComposeAsync(
        req -> fetchHttp(req), executor);
httpResponse.whenComplete((_, _) -> {
    if (httpResponse.isCancelled()) {
        requestFuture.cancel(true);
    }
})

CompletableFuture<MyObject> parsedResult =
        httpResponse.thenComposeAsync(
            rawJson -> validateAndConvert(rawJson), executor);
parsedResult.whenComplete((_, _) -> {
    if (parsedResult.isCancelled()) {
        httpResponse.cancel(true);
    }
});

Manually wiring cancellation is not sustainable.

Execution Context

One additional concern is how execution context is propagated along. In my case, we are using gRPC. By default, gRPC Java propagates RPC cancellation and deadlines through a thread-local Context object. One idea for propagating cancellation is to just wire through the cancellation signal to the root of the dependency tree. For example, if the client RPC triggered the code above, but then went away, maybe only the end of the dependency chain needs to be cancelled. If the fetchHttp() call just checked the thread local gRPC context, all the chained futures between it and the final consumer parsedResult, could be ignored. The root would transitively cancel all the others.

The problem here is in how CF delegates work to the executor. Each dependent execution stage in CF only triggers on completion of the source CF. This means the original calling context has been lost by the time work is scheduled on the executor! To be specific, suppose that CompletableFuture<HttpRequest> requestFuture was not an immediate, but instead had to be asynchronously loaded. When it finishes and schedules the HTTP call work, it may do so on it’s thread, or it may do so on the caller thread. We don’t know. The original gRPC context won’t be propagated to other threads, since we don’t know how that work was scheduled. In other words, there is no reliable way to make sure that the calling context is propagated to the async work.

This is why we need full composition with Futures. Between cancellation, deadlines, and execution context, it’s verbose and error-prone to pass these along reliably.

For those of you designing your own languages and libraries, consider these problems carefully! CompletableFuture can be used, but it can’t be re-used. When you make your implementation, make it so that the right thing is the default usually, and custom or specialized behavior doesn’t become onerous.

The End Of Software Engineering: The Advent of Vibe

(Before we begin, I like the idea of robots, which we now call AI, enabling humanity’s progress. The people in this post are not the problem, it’s the incentive structure. This post is a lament, but not yet a eulogy for the software engineer’s way of life that is going away.)

My team at $currentCompany has been using Claude and other AI tools recently to build a project that is beyond my comprehension. As part of a team, I feel both a desire to help others, but also an accountability for when things go wrong. As I see my team lean more into AI generation of code, both of these team oriented feelings are evaporating. I feel a growing disconnect about what they are working on, and no stock in the success or failure of their project. (Agents project?) As a software romantic, I can’t help but feel emotionally torn about the consequences of AI, as it seems to offer both incredible ability, but at the cost of all our dearly earned practices.

The premise of the problem is simple. We must justify our salaries, and we must have something to work towards promotion, so a new big project it is. However, how can we show enormous and lightning fast impact? As our fearless leaders have pronounced, the answer is to use AI to write our code.

What does that mean?

In my case, the answer is “writing” lots of code. Code, whose authorship is not quite certain. Code which is by most humans’ reading is distasteful. Code which does, in fact, fulfill the desire of the human who asked for it. For those of you who haven’t seen this yet, think about all the AI art you have seen over the past 2 years, and then imagine it as code. Most people I know consider AI art to be somewhere in the Uncanny Valley. So we have lots (thousands of lines a day) of code written and being checked in.

Many companies expect their software engineers to engage in code review. It’s frequently a legal requirement. Many human beings would agree that the practice of having someone else read, review, and provide commentary on code is a good thing. But here is where I see our way of life beginning to change. Let’s check our assumptions. Why do we think code review is valuable?

Code review spreads the knowledge of what one human is working on, with the rest of the humans on the team. The other humans know, at least a little, what is changing. If something is hard to understand (for a human), they can provide that feedback to the author. The knowledge flows from the reviewer to the author too! A [senior] reviewer can provide feedback on possibly better ways to doing things, either through different structure or different APIs. Knowledge is spread between the humans, and everyone increases in skill as a result.

With AI generated code, that’s all gone. The author vibe codes 1,500 lines of something, sends the PR out for review, and then submits at the first sign of approval. Does the [human] author understand what it does? Well not really, but that’s okay. Our $fearlessLeaders said it was okay to do it. You’re not going to directly contradict them, arrrrreeee youuu?

So curmudgeonly me provides feedback on 1,500 lines of mystery meat. Another second reviewer comes in and approves the whole mess, and the code is submitted without any knowledge pollination. My feedback is, at best, ignored. The old way of doing things. Understanding. Providing. Learning. At worst, it’s being used to replace us.

In a subsequent PR, I read through countless lines of slop, I provide my thoughts. 14 years of hard earned battle experience. Surprisingly, the author takes my feedback seriously, and sends out the next commit in minutes, completely addressing the 45 minutes I took to reason through the mess. The “author” scrapes the GitHub comments, feeds it into their agent, applies the changes, and sends it right back out. No need to spend time learning, arguing, or disagreeing. I’m absolutely right!™

It’s at this point I realize my feedback is not valuable. I’m not working to help improve my teammates or improve the code. I’m being used to train my replacement. Any more words that I say are basically going to be used against me. Those 14 years of being a hard worker don’t seem so valuable now. The “author” of the code is merely a proxy to the agent.

There’s a deeper problem here though. Let’s check our assumptions. We assumed that writing good, easy to maintain, easy to understand code, is a good thing. But why? If humans are going to be maintaining and modifying the code, it is a good thing! But that’s not the future. The machine is capable of digesting all knowledge, all code, all things ever written. And it does not forget. So why is good code needed, when the machine can keep track of everything? A machine can remember all things, and keep an enormous working set in its digital brain. It doesn’t need to know “why did someone write the code this way?”, like a human would.

The conclusion is that “good” code, is really just good-for-meat-bags code. Since AI lacks our weaknesses of limited brainpower, it can re-absorb everything in a moment. Consider the case where you have joined a team with a 15 year old code base, and the code evolved from tens of amateur programmers, to the point it’s a hot mess of undebuggable garbage. And your manager wants you to add a big, complex feature, in the next 7 days, or else. You have no hope! You might as well take some vacation days because there is no way you’ll untangle the Gordian knot of pig-shit code with your puny, software engineer, brain.

But with AI, that isn’t a problem any more. Bad code is no challenge at all. There is no problem to make sweeping changes. The goal-oriented approach of agentic development means that we can verify that the new code delivers the feature. Why bother “reviewing” code, when it can be “fixed” with an utterance of agent prose?

Here lies the deeper problem. With AI, it can keep track of more details than you or I ever could. It can know all things. It will write code that exceeds both your and my ability to understand it. It meets the goals, but humans can no longer grok it. AI writes more code, it’s harder and harder to understand for humans, and thus humans become less and less involved with code progeny. Play this game out over hundreds of iterations: the only way we can interact with code from now on is through the agent. In effect, it becomes the only entity able to code. And the longer this goes on, the longer only it knows WTF is happening.

I have to return here to my central premise: that our way of life is going away. In the words of Scarlett O’Hara: “Where shall I go? What shall I do?” How we adapt to the new world isn’t clear. Even being experienced and wise does not seem to be enough. My experience is being used by my successor. But unlike a teacher or a parent, there is no relationship being formed. It’s hard to see how I provide value in the future. I don’t think our way of life, as experienced software engineers, is going to stick around much longer. We are going to be sucked dry by the machine, or left by the roadside as the sacrificial lambs re-purpose our work, one last time.

Footnotes:

1. I don’t mean to criticize the people. It’s the incentive structure that’s being setup that’s to blame.

2. Using AI, or any code generation, is fine, as long as it isn’t abused. The machine is subordinate to the human; it must not subsume the human. When people attach their name, and their face, and their brand, to work that isn’t theirs, their identity is diminished. The unique, individual, personhood becomes less than one, as they blend into another’s.

3. Either write the code yourself, or the test and validation code yourself, but not both. If the test taker and test administrator are the same, it creates real hazard. There is temptation to skim the logic and assume you know what and why it works.