I wrote this originally on LinkedIn, this post is an updated version of it.

An introduction…

Having used AI in programming for a while, I wanted to share some thoughts on it from an experienced developer’s point of view. The overall experience has been, well, profound. I have mostly used ChatGPT (3.x and lately 4) with my experiments, which lately have been transformed into new workflows which I use in my daily work. Here are some notes I took along the way. Obviously these are from one human’s perspective and as such probably inadequate. I thought: chuck yours out there with the rest of them, why not! 🙂

A conversation with an unfamiliar intelligence

One area where AI in my opinion excels, is process summarisation. By this I mean a situation where I have to approach a completely new topic, such as a new library or perhaps a new system integration for your application. This is something that can take considerable time, because often I must read through a lot of documentation to find the part of the library or the system I am interested in.

Authentication and Authorization, especially with OAuth, are good examples of process summarisation: the protocol is complex and each cloud provider does it a bit differently. Instead of wading through a swath of web pages, I ask ChatGPT: “Can you outline the steps I need to take in order to authenticate my Android game with Firebase using Google’s ID Tokens?”. The bot then lists me step by step instructions on how to do that with code examples.

Some of the code the chatbot spits out is old, and some of it has errors, but I have a working process to try out in less than a minute. The bot of course stops at step five, since the token limit is exceeded, but I ask it to continue from step five and omit anything that came before it. The bot is happy to oblige.

I read through the step-by-step list and ask more details about step 2, since the bot was a bit vague there. And here we go again. In an optimal world I could ask the bot to write the process in Terraform or Ansible, and run it in a Gitlab CI workflow on manual input. One can dream about making oneself obsolete.

It is true that most components, services and libraries offer “getting started” tutorial, but when your problem deals with multiple components, you still have to fit the getting started tutorials together, or find a Medium post about it -which might answer to your problem or answer another problem which is close to the one you have at hand, but does not quite help you. When the problem gets more fringe, less tutorials are available (looking at you Google Analytics 4 and your APIs and libraries). ChatGPT has access to a large data set: 10 % of English Wikipedia, a boatload of books and others if I remember correctly -and it shows. I probably will never do another integration case without an AI. 

You can be my wingman any time, I think?

Generating actual code is a bit trickier: it requires me to think about the coding process a bit differently, and there are definitely limits on what the AI can do. It helps to think about the bot as a very young human developer with a bit of amnesia. What surprises me is that the junior developer knows a lot of languages. I try to stomp it and ask it to write Godot code in GDScript: it replies “Certainly! Here is your code”.

Depending on what I ask, the code works more often than it does not. Syntax errors are pretty easy to fix, I just tell the bot that this line of code has a syntax error. The bot apologises and outputs a new chunk of code. Sometimes the fix works. I can even ask it to modify the code it generated earlier, and it can show me only the parts it changed. I can ask it to show the full code for the function it changed, and the bot is happy to oblige. Where we run off a cliff a bit, is when we move to more complex topics.

First of all, the bot forgets the function and variable names it used previously, but still generates somewhat functionally correct code, which is oddly funny and annoying at the same time. This happens regularly when it refactors its own code or adds features to it. This makes sense when we understand that ChatGPT does not actually know about the context of the topic it is talking about. Instead it tries to predict the correct sequence of strings that best fits to the question I was asking. But alarmingly often I forget this.

When the bot refactors its answer, it feels like it takes an extra left turn in its transformer network and ends up in a place which is semantically close to the place where it was when it generated the original answer, but not the same place. This means that most of the time, I have to map the new function and variable names to the ones it used before, which is doable as long as the context does not completely break down. This leads to another problem, which is actually understanding the bot’s code, because you as the senior developer are responsible for it. In this, the bot is like a new human developer: the code might work, but most of the time it cannot be maintained properly. I shudder to think what the senior developers at Nokia had to deal with during the late 90’s and early 2000’s when they had to debug my code.

A new developer often struggles with code formatting, design patterns and code’s testability, however the bot does not seem to have this concept at all. And in fact it does not. The bot does exactly what you ask it to do, trying to predict which sequence of words makes you most happy. It cannot reason or use the vast unwritten knowledge humans have. It does not think about all the other developers, quality engineers and operations people who have to deal with its code.

The bot does not have a coding style either. Even the youngest human developers have a coding style, which helps in time to understand the code they wrote. AI’s coding style often changes on each iteration it does to its code base. But, as soon as you understand all of this, you start to grasp its limits and can stay within those limits. After all, you do not put the most junior developer in your team to work solo on the most complex problem on your backlog.

I usually tend to “assign” the bot the tasks which are low on complexity but high on effort. You can ask the bot to generate a simple SQL schema for users, projects and organisations (with you explaining the relations) or give it a database table and ask the bot to write a CRUD NodeJS service for it. The bot will pick up express and use a reasonable SQL library for the service. Again, you have to read the code through and ask the bot to fix input sanitations and null scenarios. You did save some time, but more importantly the bot did something you really did not want to do, and allowed you to concentrate on the fun part: actually designing something.

Another case of the same is unit tests: an important part of working code, but hoo boy, the amount of mind-numbing “call this method with parameters x, y and z” work. The tests did not not catch all errors, but they did catch some, and you didn’t have to look up which format the assets had to be written in this language -which is absolutely an annoying hoop to jump through when you develop multiple projects in different languages in parallel. While AI as of now does not quite compare to humans, it is learning fast. I fully expect the forgetfulness and hallucinations to diminish, if not disappear completely, in the future. It is hard for me to see how the AI could utilise the unwritten knowledge humans have, but so it was hard for me to see myself using ChatGPT as a coding assistant in real-life projects.

Final thoughts

There are flaws and shortcomings with the current AI when it comes to supporting developers, however I find myself working with ChatGPT more and more. I am starting to be a bit worried how easy it is to rely on the bot to do the grunt work I really don’t want to do. I already sometimes feel annoyed if I need to write a unit test file from scratch, and I wonder how this will be when the AI can write near-flawless unit tests in the future. Then I am reminded how much we developers have relied on the internet and StackOverflow for years, and I feel better. A few years ago, who wanted to implement anything without Google and StackOverflow? I think in a few years, we can replace Google and StackOverflow with AI and multiply the effect it has on us by ten.

I do worry about young developers, who in the future will probably accept the bot’s code as is, either because they are in a hurry or they do not understand how it works. We humans need to learn too, and most of the time we learn from our own mistakes. If we accept as-is what the bot offers, this will not happen, and the junior developers will not become senior developers. This has been a problem with “Stack Overflow copy-paste” already before AI, and I fear it will become something we have to learn to live with more and more in the future.

On the other hand, who knows where we move as a society on the back of our AI workhorse. Maybe we no longer need to learn the basics of Java, but instead can focus on metaprogramming and designing things on a more abstract level. I am not sure if construction workers feel sad if they no longer need to use hammer and nails to build a wall, but instead tell a bot where the wall needs to go.

One thing is sure, we are moving at increasing speed on good ol’ information highway, and there most likely is no stopping it. I do not agree with Elon and others on trying to press the AI-brakes, not because it would not be beneficial, but because I think it is impossible. I have used AI for less than a year and already it is starting to grow on me. I think this particular train has left the station.

Bishop takes Knight’s Pawn.