I Used ChatGPT to Answer Questions on Stack Overflow

Hi dev.to! It’s been a while since I’ve written a blog post, and I’m really excited to share this one with you all. Recently, OpenAI opened (released?) their ChatGPT AI for a Free Research Preview, which has been all the buzz lately. I created an account, and just started asking simple questions, watched some videos of other people using it, and used it to help me write up some simple boilerplate code for a fun side project I had in mind. Then I thought, what would happen if I started giving it questions from Stack Overflow? Some questions I had mind were:

• How accurate would the answers be?
• How often would the answers I provided be chosen as the accepted answer?
• How often would the answers I provide not work at all?

In this post, I will answer those questions, as well as provide my experience in doing this, pros and cons, things I noticed along the way, and what I plan on doing on going forward

My Experience

Overall, I would say my experience in doing this has been quite positive. My intent was to help people and answer their question, or rather have ChatGPT answer their question. This has also been a bit of a double-edged sword though, because I would often find myself answering questions I would have little to no idea where to start or how to start troubleshooting the problem if I were doing it on my own. At first, I tried answering as many questions as possible, simply copy + paste the answer and submit. The idea here was to see the success / fail rate of just blindly copying the answer. Not surprisingly, the success rate was rather low. Once I started double checking the answers ChatGPT provided me, however, the rate at which my answers were accepted increased. Only a bit, but still, it was noticeable.

This took place over the last 3 days (Dec 01 - Dec 03). Let’s look at the data:

I answered 46 questions. Of those 46 questions:

• My answer was the selected answer for 10 of them (21.7%)
• Another answer was selected for 7 of them (15.2%)
• My answer was incorrect for 4 of them (8.6%)
• For the remaining 25 questions, no answer has been selected yet (54.3%).

I gained 392 rep, which was an increase of 247%. I started with 159 rep, and I am currently at 551 at time of writing. Also at the time of writing, this has ranked me 69th for rep gained this month. Nice.

My original plan was to do this for the entire month of December. I then re-assessed and decided to do it for 1 week. But after only 3 days, I decided I simply cannot continue running this experiment. Perhaps I could dial it back, but I was spending a large amount of time doing this. It was fun at first, and still is highly interesting to me, this concept of using AI to answer coding questions from humans, but I noticed I was developing some unhealthy habits, so I have decided to stop completely. Perhaps, if allowed and after some community discussion, I will return and answer only a handful of questions a day, somewhere in the range of 1-3 or so. Or give myself a time limit of an hour or so, and stop after that.

I can confidently say though, I was NOT the only one doing this. If you have used ChatGPT to answer coding questions, you may have noticed it uses the same terminology very often in its responses. I noticed that it after it was done explaining why the code wasn’t working, or explaining steps I should take for what I wanted to accomplish, right before the code snippet, it would say something like:

“Here is an example of how you would do X:”

Also, if the code it would send you would be too long, it would just abruptly stop and the code would be unfinished. This happened somewhat often with React code where the component was quite large, so the JSX would be incomplete. So if you’re browsing Stack Overflow, looking at people's answers, and they show either of these 2 traits, that’s a ChatGPT answer right there.

Furthermore, I also noticed quite a lot of new accounts answering questions using ChatGPT. And I know they were using ChatGPT because of the way ChatGPT words its answers. It looks like a lot of users were looking to also gain a bunch of rep. The issue here is that they would very often not format the answer properly. Code blocks were fine, but any variables, npm libraries, etc., would not be surrounded in backticks for better readability in the plain text parts of the answer. I think what is happening here is that people who (may) have little to no experience with coding are answering questions to solely gain rep.

Pros and Cons

Next, I want to go over some of the pros and cons that I experienced during these last few days.

One downside that comes to mind having done this, and a point I think people will argue, to which I agree, is that it doesn’t benefit me to just copy and paste answers to questions that ChatGPT gives me without understanding what the code is doing. Now, to be fair, ChatGPT does a pretty damn good job (at least in my opinion) of explaining what the code is doing. It typically provides at least a paragraph or 2 of explanations.

The upside to this though, is that it has opened to me many different areas of web development / programming I wouldn’t have thought to explore. For example, one question was working with the webcam with React. I found that so interesting, so I went and built a simple demo of simply turning my webcam on and off using React + TypeScript. I also ended up answering a question where someone was having an issue with their Discord bot. Now I want to go and build a simple Discord bot. There were a few questions regarding MongoDB, so now I am motivated to go back to learning more about the MERN stack, and try to put together some simple projects there.

So while I may not have fully understood the answer I was providing sometimes, it did motivate me to go out on my own and try some things out I likely wouldn’t have gotten to, and / or reigniting my interest in areas I was previously exploring.

Another downside I found was that I personally found it quite difficult to answer questions where the person provided, for example, the code for a React component or two, and explained the issue(s) they were having. I found this difficult because to me it’s hard to look at only that code in isolation, and not in the larger context of the entire project. Where is that component being used? What are the types of the data / props, etc. I am so used to using TypeScript now that reviewing React code written in JS was a bit difficult. This really made me appreciate those who have not only answered my questions in the past, but also continue answering these types of questions on a regular basis. It is definitely a skill I want to improve upon.

Going Forward

This is where things get tricky, because I am not sure if what I have been doing is against Stack Overflow’s ToS. I am definitely going to pause on this for a while now, and switch my focus on the areas I’m interested in that I listed above. I’ll just be upfront and share the link to my Stack Overflow account here. Feel free to take a look at the answers I provided, and let me know your thoughts. I am really interested in discussing this with the web dev community, so please leave your comments down below! Also, if anyone from Stack Overflow, or highly ranked members have anything to say, or want to get in touch with me directly, you can please do so as well.

Cheers!

Update

Stack Overflow has enacted a temporary policy of banning ChatGPT answers.

My account has also been suspended for 1 week for the use of ChatGPT. I'm not even a little bit mad, I had a good laugh.