DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

Cover image for I Used ChatGPT to Answer Questions on Stack Overflow
Andrew Shearer
Andrew Shearer

Posted on

I Used ChatGPT to Answer Questions on Stack Overflow

Hi dev.to! It’s been a while since I’ve written a blog post, and I’m really excited to share this one with you all. Recently, OpenAI opened (released?) their ChatGPT AI for a Free Research Preview, which has been all the buzz lately. I created an account, and just started asking simple questions, watched some videos of other people using it, and used it to help me write up some simple boilerplate code for a fun side project I had in mind. Then I thought, what would happen if I started giving it questions from Stack Overflow? Some questions I had mind were:

  • How accurate would the answers be?
  • How often would the answers I provided be chosen as the accepted answer?
  • How often would the answers I provide not work at all?

In this post, I will answer those questions, as well as provide my experience in doing this, pros and cons, things I noticed along the way, and what I plan on doing on going forward

My Experience

Overall, I would say my experience in doing this has been quite positive. My intent was to help people and answer their question, or rather have ChatGPT answer their question. This has also been a bit of a double-edged sword though, because I would often find myself answering questions I would have little to no idea where to start or how to start troubleshooting the problem if I were doing it on my own. At first, I tried answering as many questions as possible, simply copy + paste the answer and submit. The idea here was to see the success / fail rate of just blindly copying the answer. Not surprisingly, the success rate was rather low. Once I started double checking the answers ChatGPT provided me, however, the rate at which my answers were accepted increased. Only a bit, but still, it was noticeable.

This took place over the last 3 days (Dec 01 - Dec 03). Let’s look at the data:

I answered 46 questions. Of those 46 questions:

  • My answer was the selected answer for 10 of them (21.7%)
  • Another answer was selected for 7 of them (15.2%)
  • My answer was incorrect for 4 of them (8.6%)
  • For the remaining 25 questions, no answer has been selected yet (54.3%).

I gained 392 rep, which was an increase of 247%. I started with 159 rep, and I am currently at 551 at time of writing. Also at the time of writing, this has ranked me 69th for rep gained this month. Nice.

Image description

My original plan was to do this for the entire month of December. I then re-assessed and decided to do it for 1 week. But after only 3 days, I decided I simply cannot continue running this experiment. Perhaps I could dial it back, but I was spending a large amount of time doing this. It was fun at first, and still is highly interesting to me, this concept of using AI to answer coding questions from humans, but I noticed I was developing some unhealthy habits, so I have decided to stop completely. Perhaps, if allowed and after some community discussion, I will return and answer only a handful of questions a day, somewhere in the range of 1-3 or so. Or give myself a time limit of an hour or so, and stop after that.

I can confidently say though, I was NOT the only one doing this. If you have used ChatGPT to answer coding questions, you may have noticed it uses the same terminology very often in its responses. I noticed that it after it was done explaining why the code wasn’t working, or explaining steps I should take for what I wanted to accomplish, right before the code snippet, it would say something like:

β€œHere is an example of how you would do X:”

Also, if the code it would send you would be too long, it would just abruptly stop and the code would be unfinished. This happened somewhat often with React code where the component was quite large, so the JSX would be incomplete. So if you’re browsing Stack Overflow, looking at people's answers, and they show either of these 2 traits, that’s a ChatGPT answer right there.

Furthermore, I also noticed quite a lot of new accounts answering questions using ChatGPT. And I know they were using ChatGPT because of the way ChatGPT words its answers. It looks like a lot of users were looking to also gain a bunch of rep. The issue here is that they would very often not format the answer properly. Code blocks were fine, but any variables, npm libraries, etc., would not be surrounded in backticks for better readability in the plain text parts of the answer. I think what is happening here is that people who (may) have little to no experience with coding are answering questions to solely gain rep.

Pros and Cons

Next, I want to go over some of the pros and cons that I experienced during these last few days.

One downside that comes to mind having done this, and a point I think people will argue, to which I agree, is that it doesn’t benefit me to just copy and paste answers to questions that ChatGPT gives me without understanding what the code is doing. Now, to be fair, ChatGPT does a pretty damn good job (at least in my opinion) of explaining what the code is doing. It typically provides at least a paragraph or 2 of explanations.

The upside to this though, is that it has opened to me many different areas of web development / programming I wouldn’t have thought to explore. For example, one question was working with the webcam with React. I found that so interesting, so I went and built a simple demo of simply turning my webcam on and off using React + TypeScript. I also ended up answering a question where someone was having an issue with their Discord bot. Now I want to go and build a simple Discord bot. There were a few questions regarding MongoDB, so now I am motivated to go back to learning more about the MERN stack, and try to put together some simple projects there.

So while I may not have fully understood the answer I was providing sometimes, it did motivate me to go out on my own and try some things out I likely wouldn’t have gotten to, and / or reigniting my interest in areas I was previously exploring.

Another downside I found was that I personally found it quite difficult to answer questions where the person provided, for example, the code for a React component or two, and explained the issue(s) they were having. I found this difficult because to me it’s hard to look at only that code in isolation, and not in the larger context of the entire project. Where is that component being used? What are the types of the data / props, etc. I am so used to using TypeScript now that reviewing React code written in JS was a bit difficult. This really made me appreciate those who have not only answered my questions in the past, but also continue answering these types of questions on a regular basis. It is definitely a skill I want to improve upon.

Going Forward

This is where things get tricky, because I am not sure if what I have been doing is against Stack Overflow’s ToS. I am definitely going to pause on this for a while now, and switch my focus on the areas I’m interested in that I listed above. I’ll just be upfront and share the link to my Stack Overflow account here. Feel free to take a look at the answers I provided, and let me know your thoughts. I am really interested in discussing this with the web dev community, so please leave your comments down below! Also, if anyone from Stack Overflow, or highly ranked members have anything to say, or want to get in touch with me directly, you can please do so as well.

Cheers!

Update

Stack Overflow has enacted a temporary policy of banning ChatGPT answers.

My account has also been suspended for 1 week for the use of ChatGPT. I'm not even a little bit mad, I had a good laugh.

Image description

Top comments (10)

Collapse
 
codenameone profile image
Shai Almog

So you're the guy Hacker News is complaining about πŸ˜‚

Collapse
 
andrews1022 profile image
Andrew Shearer

I guess so lol πŸ˜…

Collapse
 
styren profile image
Buster Styren

And now it's banned

Collapse
 
andrews1022 profile image
Andrew Shearer

My account has also been suspended for 1 week lol

Collapse
 
jjokah profile image
John Johnson Okah

This is great. For now, I believe ChatGPT is supposed to work with us, not for us.

Collapse
 
integerman profile image
Matt Eland

I absolutely love t his idea. Thank you.

Collapse
 
spaceofmiah profile image
Osazuwa Agbonze

Nice experiment and interesting how this has expand your explore terrain

Collapse
 
tiagorangel2011 profile image
Tiago Rangel de Sousa

Me: Can I use this bot to answer stackoverflow questions?

chatGPT: It is not advisable to use a bot to answer questions on Stack Overflow or any other online forum or platform. Stack Overflow is a community-driven website where users ask and answer questions about programming and software development. The community expects users to provide their own answers and insights, based on their own knowledge and experience.

Using a bot to answer questions on Stack Overflow would not be fair to other users, as the bot's answers are not based on the knowledge and experience of a human user. Additionally, the bot's answers may not always be accurate or relevant, which could lead to confusion or misinformation among users.

Furthermore, Stack Overflow has policies in place that prohibit users from using automated tools to post answers or otherwise manipulate the site. This includes the use of bots, so using a bot to answer questions on Stack Overflow would be against the site's policies.

In general, it is not a good idea to use a bot to answer questions on Stack Overflow or any other online forum or platform. It is better to use your own knowledge and experience to provide valuable answers and insights that can help other users.

Collapse
 
mcsee profile image
Maxi Contieri ⭐⭐⭐

Hahaha!

Great idea.

BTW, if you need some answer from stack overflow the hack is easy.
You post the question, then post a very wrong answer with another account and , after that you will get some hate and a lot of accurate answers

Collapse
 
rouilj profile image
John P. Rouillard

This seems like the hack to prevent getting lost in the wilderness.

1) take a typewriter and paper with you.
2) start typing and make a typo.
3) hiker finds you and points out your typo.
4) you're found.

Tired of sifting through your feed?

Find the content you want to see.

Change your feed algorithm by adjusting your experience level and give weights to the tags you follow.