ast-grep: 5000 stars and beyond!

We are thrilled to announce that ast-grep has reached 5000 stars on GitHub! This is a huge milestone for our project and we are very grateful for your feedback, contributions, and encouragement.

Why ast-grep?

ast-grep is a tool that allows you to search and transform code using abstract syntax trees (ASTs). ASTs are tree-like representations of the structure and meaning of source code. By using ASTs, ast-grep can perform more accurate and powerful operations than regular expressions or plain text search.

We have introduced a lot of new features in the past few months, and we want to share them with you. We hope that you will find them useful and that they will help you write better code.

What's new in ast-grep?

Core

• We have redesigned and implemented a new pattern engine inspired by difftastic. Now, patterns use Rust structures to represent the syntax of code, instead of tree-sitter objects. This improves performance by minimizing tree traversal and allows for more reliable and user-friendly pattern-matching.

CLI

• You can now use --inline-rules to run rules without creating any files on your disk! You can pass everything, pattern/rule/input, as a string. This is great for scripting!
• --stdin will always wait for your input so you can match some code written in your terminal.
• You can also select custom languages in sg new.

Language Support

• We have added support for three new languages: bash, php and elixir.
• We have updated our language support to include golang's generic syntax and python's pattern matching syntax.
• You can try out kotlin on our playground!

Rule

• You can now use expandStart and expandEnd to adjust the fix range selection for more precise code transformations.
• You can also use languageGlob to register alias languages for extension override, which gives you more flexibility in handling different file types.

Node/Python API

• We have added to napi a new function parseAsync, which allows you to leverage multiple cores in Node.js for faster code parsing.
• We have also added language globs to findInFiles in napi, which makes it easier to search for code patterns in non-standard files (like searching HTML in .vue file).
• You can now use getTransformed in napi to get the transformed code as a string.

Doc

• We have improved our napi/pyo3 documentation and added sandbox/colab links for you to try out ast-grep online!
• We have also updated our transformation and code fix documentation with more examples and explanations.
• We have added new language examples for go and python, which show you how to use ast-grep with these popular languages.
• We have created an ast-grep bot on discord, which can answer your questions and provide tips and tricks on using ast-grep.

Community

• We are excited to see that some awesome projects are using ast-grep for their code transformations, such as:
  • vue-macro cli helps you migrate your Vue projects to the latest version of Vue
  • a new unocss engine transforms JSX attributes into CSS classes
• We are also happy to see that some innovative platforms are using ast-grep as one of their tools to help developers understand and improve their codebases, such as:
  • coderabbit uses ast-grep to help AI analyzing your code and provide insights and recommendations
  • codemod is considering ast-grep as a new underlying tool in their code transformation studio

What's next in ast-grep?

Applying sub-rules to sub-nodes

Currently, ast-grep can only apply rules/transformations to the whole node that matches the pattern. This limits the flexibility and expressiveness of ast-grep, compared to other tools like babel or libcst.

We want to make ast-grep more powerful by allowing it to apply sub-rules to the metavariable nodes within the matching node. This will enable ast-grep to handle more complex and diverse use cases in code transformation.

For example, we can merge multiple decorators into one mega decorator in Python. This is impossible without API in the current version of ast-grep.

The basic workflow of ast-grep is "Find and Patch":

1. Find a target node based on rule/pattern.
2. Generate a new string based on the matched node.
3. Replace the node text with the generated fix.

However, this workflow does not allow us to generate different text for different sub-nodes in a rule. (This is like not being able to write if statements.)
Nor does it allow us to apply a rule to multiple sub-nodes of a node. (This is like not being able to write for loops.)

To overcome these limitations, we will add three new steps between step 1 and step 2:

a. Find a list of different sub-nodes that match different sub-rules.
b. Generate a different fix for each sub-node based on the matched sub-rule.
c. Join the fixes together and store the string in a new metavariable for later use.

The new steps are similar to the existing "Find and Patch" workflow, but with more granularity and control.

This is like doing syntax tree oriented programming. We can apply different rules to different sub-nodes, just like using conditional statements. We can also apply rules to multiple sub-nodes, just like using loops. "Find and Patch" is kind of a specialized "Functional Programming" over the AST!

That said, applying sub-rules is an advanced feature that requires a lot of learning and practice. When in doubt, you can always use the existing N-API/PyO3 workflow!

Thank you!

We want to thank all the ast-grep users and supporters for your feedback, contributions, and encouragement.

And we want to especially thank ast-grep's sponsors!

We hope that you enjoy the new features and improvements in ast-grep. We are always working to make ast-grep better and we look forward to hearing from you.

Happy coding!