DEV Community

Aleksei Zagoskin
Aleksei Zagoskin

Posted on • Originally published at zagosk.in on

ChatGPT-4 outperformed 90% of developers

The title may sound bold and clickbait-ish, but it is what it is...

A few days ago I stumbled upon a post on LinkedIn (unfortunately, I didn't save the link) in which a programmer demonstrated his solution to a LeetCode challenge. The code was fine and the post got dozens of likes, which it undoubtedly deserved. I decided to feed the requirements to ChatGPT-4 and see if it could understand and solve the problem. The outcome was... interesting and I spent some time experimenting with the AI, modifying the prompt to see how it affected the results.

The Challenge

First, I would like you to take a look at the LeetCode problem and attempt to solve it. Take your time, give it some thought. Then, google it. Yes, search the internet for the best solution.

Given two integer arrays pushed and popped each with distinct values, return true if this could have been the result of a sequence of push and pop operations on an initially empty stack, or false otherwise.

Example 1:

Input: pushed = [1,2,3,4,5], popped = [4,5,3,2,1]
Output: true
Explanation: We might do the following sequence:
push(1), push(2), push(3), push(4),
pop() -> 4,
push(5),
pop() -> 5, pop() -> 3, pop() -> 2, pop() -> 1

Example 2:

Input: pushed = [1,2,3,4,5], popped = [4,3,5,1,2]
Output: false
Explanation: 1 cannot be popped before 2.

Constraints:

  • 1 <= pushed.length <= 1000

  • 0 <= pushed[i] <= 1000

  • All the elements of pushed are unique.

  • popped.length == pushed.length

  • popped is a permutation of pushed.

Done?

Alright, remember I asked you to brainstorm and google it? So if the machine-generated solution happens to be more efficient, the argument "it's not fair because it was trained on data from the Internet" is not accepted πŸ˜‰

Release the Kraken

When I initially asked ChatGPT to tackle the challenge, I copy-pasted the task from LeetCode. However, in this case, it might indeed seem like the AI pulled the solution from the depths of its "memory" without much "thinking". So let's rephrase the requirements and see if the machine can handle it. Here's the prompt I used:

ChatGPT-4 prompt for LeetCode #946

The output:

ChatGPT-4 solution for LeetCode #946

Not bad, and according to LeetCode statistics, this is an average solution, more or less. Now let's push our beloved AI a bit further and ask it to enhance the code:

ChatGPT-4 solution v2 for LeetCode #946

This is where things become really interesting! Not only did it offer an elegant solution without allocating a stack, but it also explained a potential issue that could arise from this optimization. Let's check it out on LeetCode:

LeetCode assessment of ChatGPT-4 solution

The numbers speak for themselves, right? Of course, some might argue that LeetCode's benchmark is inconsistent, which is a valid point. Nevertheless, the fact remains that the AI instantly produced a decent solution and improved it in a way that not every software engineer can do.

What's Next?

So, are our jobs safe?

Yes πŸ˜… (unless you work for Elon Musk)

To draw an analogy, today's ChatGPT (and other AI assistants like GitHub Copilot) is like an electric tool that can drill holes, drive screws, saw and more. However, if you just place it on the ground, it won't build a house for you. What is more, unskilled individuals might even harm themselves with this tool. On the other hand, experienced engineers can use the power and potential of AI to significantly improve the quality of their work.

That's it for now. Cheers!

Top comments (9)

Collapse
 
ant_f_dev profile image
Anthony Fung

Thanks for the unbiased analysis of AI.

There are far too many articles that predict the elimination of junior developer roles without exploring how future senior developers (who would supposedly use AI in their place) would be trained, or how AI would address an organisation's support needs.

Collapse
 
powerz profile image
Aleksei Zagoskin

Thanks, @ant_f_dev! It's funny how we software developers are concerned about our job security, while guys like accountants, lawyers, and many others seem unaware of what's happening. If, for instance, I were a first-level support staff member or a technical/copywriter, I'd feel somewhat stressed as well.

Collapse
 
ant_f_dev profile image
Anthony Fung

It's funny actually - fields like accountancy and law seem like areas that AI could really affect. And yet the first applications for AI being showcased focus on creative arts (writing, visual arts, music) and programming; maybe because the consequences of getting things wrong are relatively trivial? After all, if an algorithm is badly wrong, it 'just' won't compile or a developer will review and reject it.

I don't think copywriters have that much to fear at this point. From the reactions I've seen, I get the feeling that AI will greatly help them, but the output so far isn't good enough to be truly compelling marketing material. (Disclaimer: I'm not a copywriter, so I don't know how true this sentiment is)

Thread Thread
 
powerz profile image
Aleksei Zagoskin

It's pretty hard to guess which areas will be significantly impacted by AI first. Time will tell. In the meantime, we'll keep an eye on it 🧐

Collapse
 
jeet_b profile image
Jeet Bhattacharjee

This is exactly how I think about using AI for coding. It’s quite similar to how excel and other tools shrunk the market for manual calculation jobs (like statistician and book keeper). People just trained up to use the tools and be more productive, or found new niches.
Constant learning is a fundamental for a career any STEM field.

Collapse
 
emily-flias profile image
Emily Flias • Edited

Think about GPT-5 and GPT-6, which will be released in the coming years and have trillions of parameters?
And this is baby google:
blog.google/technology/ai/code-wit...

Collapse
 
powerz profile image
Aleksei Zagoskin

True, more mind blowing things are coming!

Collapse
 
rolfstreefkerk profile image
Rolf Streefkerk

these are very specific computational tasks. Programming is a lot about context, to get a more accurate reading you'd need to give it a general task that it needs to solve which would require it to break them down into these kind of individual solutions.

Collapse
 
kubawu profile image
Kuba Wach

Interesting experiment, thanks for sharing!