In mid-2024, Tidelift fielded its third survey of open source maintainers. More than 400 maintainers responded and shared details about their work, including how they fund it, who pays for it, and what kinds of security, maintenance, and documentation practices they have in place today or would consider in the future. They also shared their thoughts about some “in the headlines” issues like the recent xz utils hack and the impact of AI-based coding tools. In this post, we share the eleventh of twelve key findings. If you don’t want to wait for the final post, you can download the full survey report right now.
In our previous finding, we learned quite a bit about open source maintainers’ perceptions of AI-based coding tools and how these tools are impacting their work as an open source maintainer today. But we didn’t stop there. We also thought it would be interesting to better understand how maintainers are using AI in their own work.
Are maintainers using AI-based coding tools today?
We asked maintainers to tell us how often they are using AI-based coding tools in their work today. About one-half (48%) are using these tools already, with 20% using them occasionally (less than once per week), 13% using them more than once a week, and 15% using them more than once a day.
Those who are not using AI-based coding tools mostly have no plans to use them in the future either, with 45% selecting that option and only 6% not using them today, but planning to in the future.
Interestingly, use of AI-based coding tools is highly correlated with the age of the maintainer. Splitting the same data by age, we find that 71% of maintainers under age 26 are using these tools at least occasionally (+23% over the full sample). This percentage drops to 58% for maintainers between 26-35 years old, and then to 39% for maintainers 36-45 years old, and finally increasing slightly to 42% for maintainers 46 years old or older.
How maintainers are using AI-based coding tools, in their own words
We asked those maintainers who are currently using AI-based coding tools to tell us in their own words what they were using the tools for. The highest percentage of respondents said they were using AI-based coding tools for code completion and suggestions, and the vast majority of these respondents are using either GitHub Copilot or ChatGPT.
A few example comments:
“I use Copilot pretty much every day for very basic tasks like remembering how to implement common traits like std::fmt::Display. I don't use Copilot for the overall structure, algorithms, or architecture.”
“I use Copilot for auto-completion of code, useful for the boilerplate I like to establish in my projects. I also use a custom discord bot that leverages GPT-4o for brainstorming and light delegation of simpler coding tasks.”
“I use ChatGPT for my day-to-day coding. I may ask it to help write new features, write CSS, diagnose errors, or generate boilerplate code to start a new feature/project.”
Many respondents report using AI-based coding tools, but do so with some concerns. For example:
“I use GitHub Copilot integrated with VS Code and RStudio. I occasionally use the code it suggests, but I nearly always have to modify it to make it correct.”
“They still make mistakes and can be dumb, but I use them daily for ideas, chore work like generating tests and types, etc. It has made me more productive as an open source maintainer.”
“While I know first-hand that AI can act as a good ‘research assistant’ especially for newcomers—full of ideas but perhaps not full of the required skills yet—I have yet to see AI be sufficiently helpful in matters where context is key, as is often the case in the more nitty gritty parts of open source projects.”
Another common way maintainers use AI-based coding tools is for documentation and testing.
“I'll often use AI to help build test cases, or to help me refactor difficult-to-understand code.”
“I use them sometimes to write tests. I can show it existing tests and ask it to write some new ones based on those.”
“I have started recently using it to provide better and more detailed commit messages. Those commit messages will end up in the changelog when releasing a new version. So it can help provide a more detailed changelog in the future.”
What kinds of open source software-related problems could be solved by AI?
We ended the AI section of the survey by asking maintainers to share with us any final thoughts about the types of open source problems that might be solved by AI.
We were able to tease out quite a few interesting ideas from maintainers for how AI could help improve open source. Here are the main categories of ideas, along with a few example quotes for each.
Documentation: help improving documentation, automating documentation tasks, and making documentation more accessible.
“Non-technical problems like changelog summaries or other similar boring tasks about presenting the content of technical actions to lay people. Perhaps some documentation related text, auto-extracted from the source code.”
“Creating release notes.... Automatically turn README files and Wiki entries into a chatbot.”
“GitHub Community questions could have an ‘AI’ proposed solution given to me that I can approve if it's correct to show as an answer. This could speed up answering similar questions multiple times with similar code.”
Issue triage: help automating issue triage, identifying duplicate issues, and prioritizing issues.
“Sometimes I receive vague bug reports or feature requests. I think having a chatbot that assists reporters and contributors in creating these could help reduce such cases.”
“Ensure issues have all the necessary context. Provide first answers for queries (especially for first-time users), with the context of the docs and tests. Attempt to fix trivial issues when people submit PRs (e.g. lint errors, breaking changes in Dependabot version updates)—most fixes are trivial, especially when looking at error messages and (in case of version upgrades) changelogs.”
Code quality and review: help automating code review and improving code quality.
“Resolve imports to dependencies needed to satisfy those imports. Provide intelligent refactoring. Assess safety of a given change. Generate tests and PRs to capture and resolve a reported issue.”
“Better fuzzing to detect and perhaps even auto-fix classes of bugs. For example, CPython reference counting issues.”
Dependency management and security: help automating dependency management, identifying security vulnerabilities, and updating dependencies.
“Given a changelog for a new release of one of my dependencies, and the way the dependency is actually used in my codebase, what changes in the dependency do I need to investigate further than my tests will cover?”
“Filter vulnerability reports so they are about dependencies my project actually uses and not just part of the build tools.”
And with that, we end our tour of maintainers’ thoughts and perceptions of AI. We now have a good sense for what maintainers think about AI-based coding tools and how they expect these tools will impact their work. We also learned more about how many maintainers are using AI-based coding tools today, and what they are using them to accomplish. And we ended by getting some specific ideas from maintainers for how they think AI could be used to solve open source software related problems.
Top comments (0)