Disclaimer: I don't work for JetBrains, so there may be and most likely there will be inaccuracies and errors in the article and code.

In the previous article, I showed the process of creating a framework for a language plugin. Well-known plugins for Java, go, Frege were used as examples. There are also examples from the plugin for the Monkey language, which I developed while I was figuring out how everything works. Since my goal was not to cover everything, the plugin covers a limited subset of the language. The interpreter for Monkey can be found here.

Formatting

Documentation, examples from go-plugin, monkey.
As always, everything starts from an extension point.

<lang.formatter language="Monkey"
implementationClass="com.github.pyltsin.monkeyplugin.formatter.MonkeyFormattingModelBuilder"/>

A class must implement the next interface

public interface FormattingModelBuilder {
@NotNull 
FormattingModel createModel(@NotNull FormattingContext formattingContext);

@Nullable 
TextRange getRangeAffectingIndent(PsiFile file, int offset,
                                  ASTNode elementAtOffset) ;
}

The most important method is the first one. It returns a formatting model, which is built on formatting blocks. To simplify it, you can use FormattingModelProvider.createFormattingModelForPsiFile method

Let's take a look at what a formatting block is. In IDEA, a formatting block is represented as an interface com.intellij.formatting.Block. It is some range of text (often corresponding to some PSI element) which is formatted according to some rules. The formatting blocks are nested into each other and create a tree.

public interface Block {

  @NotNull
  TextRange getTextRange();

  @NotNull
  List<Block> getSubBlocks();

  @Nullable
  Wrap getWrap();

  @Nullable
  Indent getIndent();

  @Nullable
  Alignment getAlignment();

  @Nullable
  Spacing getSpacing(@Nullable Block child1, @NotNull Block child2);

  @NotNull
  ChildAttributes getChildAttributes(final int newChildIndex);

/// some methods are skipped
}

If you want to visualize this tree you can use PSI-Viewer
(Tools->View PSI Structure, tab "Block Structure")

To simplify the implementation, you can use the AbstractBlock class.

getSpacing - determines the number of spaces or line breaks between the specified child elements. To simplify the implementation of this logic, you can use the com.intellij.formatting.SpacingBuilder class, which provides a convenient API to describe rules.

//an extraction from plugin for Golang
public Spacing getSpacing(@Nullable Block child1, @NotNull Block child2) {
return new SpacingBuilder(settings, GoLanguage.INSTANCE)
      .before(COMMA).spaceIf(false)
      .after(COMMA).spaceIf(true)
      .betweenInside(SEMICOLON, SEMICOLON, FOR_CLAUSE).spaces(1)
      .before(SEMICOLON).spaceIf(false)
      .after(SEMICOLON).spaceIf(true)
      .beforeInside(DOT, IMPORT_SPEC).none()
      .afterInside(DOT, IMPORT_SPEC).spaces(1)
      //and more rules
      .getSpacing(this, child1, child2);
}

Structure view

Documentation. Examples of implementation in plugin-go, Monkey.

In this section we will talk about filling this panel:

The next extension point is responsible for that

<lang.psiStructureViewFactory language="Monkey"
implementationClass="com.github.pyltsin.monkeyplugin.editor.MonkeyStructureViewFactory"/>

Your class must implement the next interface:

@FunctionalInterface
public interface PsiStructureViewFactory {
  @Nullable
  StructureViewBuilder getStructureViewBuilder(@NotNull PsiFile psiFile);
}

For implementation StructureViewBuilder you can also use prepared classes: com.intellij.ide.structureView.TreeBasedStructureViewBuilder, com.intellij.ide.structureView.StructureViewModelBase and com.intellij.ide.structureView.impl.common.PsiTreeElementBase.

Working with PsiTreeElementBase is similar to working with formatting blocks.

Caches, indexes, stub and goto

Caches

Let's start with the caches. IDEA asks extensions several times. If the extension is doing some time-consuming work, then the best solution is to cache the result of this work. IDEA provides many wrappers for this. For example, this is how the type of Go expression is calculated in the plugin-go:

  @Nullable
  public static GoType getGoType(@NotNull GoExpression o, @Nullable ResolveState context) {
    return RecursionManager.doPreventingRecursion(o, true, () -> {
      if (context != null) return unwrapParType(o, context);
      return CachedValuesManager.getCachedValue(o, () -> CachedValueProvider.Result
        .create(unwrapParType(o, createContextOnElement(o)), PsiModificationTracker.MODIFICATION_COUNT));
    });
  }

2 different managers are used here. The first is the CachedValuesManager, which caches the result for the psi element, and the second is the Recursion Manager, which helps to prevent infinite recursion and StackOverflowError. There is also a specialized cache com.intellij.psi.impl.source.resolve.ResolveCache, which is used for resolving elements (this will be specified below).

Indexes

Documentation.

We all know how IDEA likes to index everything. Let's see what it is and how it can be used.

Indexes in IDEA provide the opportunity to quickly find a necessary file or a psi element (for example, by the name of the annotation, you can find all the places where it is used).

You can view existing indexes using the Index viewer plugin.

IDEA supports two types of indexes - File-based and Stub. File-based indexes work with the contents of a file, Stub indexes work with Stub-tree, which is built on the basis of PSI-tree.

File-based indexes

An example of use can be found in the hackforce plugin

We, as usual, start from the extension point

<fileBasedIndex implementation="com.haskforce.index.HaskellModuleIndex"/>

Class must extend FileBasedIndexExtension. To get result of indexing you can use FileBasedIndex.getInstance()

The hack force plugin, for example, uses this type of index to get all the files within the module:

@NotNull
public static Collection<VirtualFile> getVirtualFilesByModuleName(@NotNull String moduleName, @NotNull GlobalSearchScope searchScope) {
  return FileBasedIndex.getInstance()
    .getContainingFiles(HASKELL_MODULE_INDEX, moduleName, searchScope);
}

Stub indexes

It seems to me that this type of indexes is used more often because it allows you to search through PSI elements (or rather through stub, which represents the required part of the psi tree). Stubs are used only for named psi elements (which implement the PsiNamedElement interface). They will be described in more detail in the Reference section

To declare a new index, the following extension point is used:

<stubIndex implementation=
"com.github.pyltsin.monkeyplugin.stubs.MonkeyVarNameIndex"/>

Your class must implement StubIndexExtension.
Example from the plugin for Monkey:

class MonkeyVarNameIndex : StringStubIndexExtension<MonkeyNamedElement>() {
    override fun getVersion(): Int {
        return super.getVersion() + VERSION
    }

    override fun getKey(): StubIndexKey<String, MonkeyNamedElement> {
        return KEY
    }

    companion object {
        val KEY: StubIndexKey<String, MonkeyNamedElement> =
            StubIndexKey.createIndexKey("monkey.var.name")
        const val VERSION = 0
    }
}

Examples from go-plugin, frege.

Now we need to teach IDEA how to create a tree of Stubs and save the necessary elements under the desired index.

For each element type that we want to save as a Stub, we create a Stub definition. In this case, the root of all Stubs should be FileStub.

Example from the plugin for Monkey:

class MonkeyFileStub(file: MonkeyFile?) : PsiFileStubImpl<MonkeyFile>(file)

class MonkeyVarDefinitionStub : NamedStubBase<MonkeyVarDefinition> {
    constructor(parent: StubElement<*>?, elementType: IStubElementType<*, *>, name: StringRef?) : super(
        parent,
        elementType,
        name
    )

    constructor(parent: StubElement<*>?, elementType: IStubElementType<*, *>, name: String?) : super(
        parent,
        elementType,
        name
    )
}

Examples from go-plugin (file, element), Frege (file, element)

The next step is to create a description of the element type for each Stub. (For automatically generated PSI elements with Grammar-Kit plugin, descriptions of each type of element are created automatically following elementTypeHolderClass and elementTypeClass parameters). The ElementType for the file must extend IStubFileElementType, for the element - IStubElementType.

IStubElementType requires the implementation of the following methods:

@NotNull
String getExternalId();

void serialize(@NotNull T stub, @NotNull StubOutputStream dataStream) throws IOException;

@NotNull
T deserialize(@NotNull StubInputStream dataStream, P parentStub) throws IOException;

void indexStub(@NotNull T stub, @NotNull IndexSink sink);

PsiT createPsi(@NotNull StubT stub);

@NotNull StubT createStub(@NotNull PsiT psi, StubElement<?> parentStub);

shouldCreateStub(ASTNode node)

How to index Stubs is specified in the index Sub method. For example, in Monkey I used this implementation:

    override fun indexStub(stub: S, sink: IndexSink) {
        val name = stub.name
        if (name != null) {
            sink.occurrence(MonkeyVarNameIndex.KEY, name)
        }
    }

The implementation of other methods can be found in the examples - plugin for Monkey, go-plugin, Frege

Now our stubs need to be connected to the parser. This can be done in 2 steps.

• Step 1: define your element type factory for which we have made IStubElementType, for example as

object MonkeyElementTypeFactory {
    @JvmStatic
    fun factory(name: String): IElementType {
        if (name == "VAR_DEFINITION") return MonkeyVarDefinitionStubElementType(name)
        throw RuntimeException("Unknown element type: $name")
    }
}

and specify in bnf file that it should be used for some PSI elements:

elementTypeFactory("var_definition")=
"com.github.pyltsin.monkeyplugin.psi.impl.MonkeyElementTypeFactory.factory"

• Step 2. Specify that these PSI elements should extend StubBasedPsiElementBase

Let's consider everything together. During indexing, StubBasedPsiElementBase is created, then a Stub is created using IStubElementType.createStub. This stub can be serialized and a reference to it is saved in the index(indexStub).

Client code which works with Stubs should call only those methods that have enough stored information to execute. Therefore, it is necessary to include in a stub all the information that may be needed later in the analysis. To get the PSI element, you can call the getNode() method, but it is expensive because it requires file parsing.

An example of saving information can be found in com.intellij.psi.impl.java.stubbs.impl.PsiAnnotationStubImpl#getPsiElement, which uses text from ASTNode.

Using stub indexes

Indexes are widely used for go-to functions. For example, they are used for this panel:

Simplified go-to implementation for symbols based on stub indexes:

• plugin.xml

<gotoSymbolContributor implementation=
"com.github.pyltsin.monkeyplugin.usages.MonkeySymbolContributor"/>

• MonkeySymbolContributor:

class MonkeySymbolContributor : ChooseByNameContributorEx {
    private val myIndexKeys = listOf(MonkeyVarNameIndex.KEY)
    override fun processNames(
        processor: Processor<in String>,
        scope: GlobalSearchScope,
        filter: IdFilter?
    ) {
        for (key in myIndexKeys) {
            ProgressManager.checkCanceled()
            StubIndex.getInstance().processAllKeys(
                key,
                processor,
                scope,
                filter
            )
        }
    }

    override fun processElementsWithName(
        name: String,
        processor: Processor<in NavigationItem>,
        parameters: FindSymbolParameters
    ) {
        for (key in myIndexKeys) {
            ProgressManager.checkCanceled()
            StubIndex.getInstance().processElements(
                key,
                name,
                parameters.project,
                parameters.searchScope,
                parameters.idFilter,
                MonkeyNamedElement::class.java,
                processor
            )
        }
    }
}

Many custom plugins also use indexes widely. For example, Request mapper (currently not supported, since the same functionality appeared in IDEA), which helps to search for REST method declaration points

Request Mapper uses this code under the hood:

//JavaAnnotationIndex - the usual Subindexbindex for Java for annotations
JavaAnnotationIndex
            .getInstance()
            .get(annotationName, project, GlobalSearchScope.projectScope(project))
            .asSequence()

References

Documentation

At the moment, the API is being changed, so there may be inaccuracies.

References create links between elements. When you press Ctrl+B, you will go to the element to which this link refers. By pressing Ctrl+B again, you will see all the elements that link to this element.

Only the element that defines a name should implement PsiNamedElement (or better PsiNameIdentifierOwner, you can see their use in the Rename section)

@JvmStatic
fun setName(expr: MonkeySimpleRefExpr, name: String): PsiElement {
    val e: PsiElement = MonkeyElementTextFactory.createStatementFromText(expr.project, "$name + 1")
    //newLetExpr must implement PsiNamedElement
  val newLetExpr = PsiTreeUtil.findChildOfType(e, MonkeySimpleRefExpr::class.java)
    if (newLetExpr != null) {
        expr.replace(newLetExpr) 
    }
    return expr
}

In order for the PSI element to provide a link, you need to implement the methods

PsiReference getReference();

PsiReference @NotNull [] getReferences();

@Experimental
default @NotNull Iterable<? extends @NotNull PsiSymbolReference> getOwnReferences() {
    return Collections.emptyList();
}

To simplify the implementation of the PsiReference interface, you can use the PsiReferenceBase. You have to implement the method PsiElement resolve() or ResolveResult [] multiResolve(boolean incompleteCode), which return the referenced elements. When implementing this method, it makes sense to use a specialized cache:

override fun multiResolve(incompleteCode: Boolean): Array<ResolveResult> {
    return ResolveCache.getInstance(psiElement.project).resolveWithCaching(
        this, { referenceBase, _ ->
            referenceBase.resolveInner(false)
                .map { PsiElementResolveResult(it) }
                .toTypedArray()
        },
        true, false
    )
}

After the implementation of this part, IDEA will already be able to provide navigation to the referenced element and back.

Rename

Documentation

Renaming is one of the most popular refactorings (Shift+F6)

2 extension points are responsible for that:

<lang.refactoringSupport language="Monkey"
                         implementationClass="com.github.pyltsin.monkeyplugin.refactor.MonkeyRefactoringSupportProvider"/>
<renameInputValidator implementation="com.github.pyltsin.monkeyplugin.refactor.MonkeyRenameInputValidator"/>

renameInputValidator must implement RenameInputValidator.

refactoringSupport must extend RefactoringSupportProvider.

This class also contains methods to support other types of refactorings. At the moment we are interested in a method that indicates whether in-place editing is supported.

  public boolean isMemberInplaceRenameAvailable(@NotNull PsiElement element, @Nullable PsiElement context) {
    return false;
  }

Now it is required to implement renaming methods. As mentioned before, the PSI element that defines the name should implement the PsiNamedElement interface.

public interface PsiNamedElement extends PsiElement {
  String getName();
  PsiElement setName(@NlsSafe @NotNull String name)
  throws IncorrectOperationException;
}

We are interested in setName method here. One of the easiest ways to implement this method is to create a new PSI element from text, like this

private fun createFileFromText(project: Project, text: String): MonkeyFile {
    return PsiFileFactory.getInstance(project)
        .createFileFromText("A.monkey", MonkeyLanguage.INSTANCE, text) as MonkeyFile
}

And replace the element

expr.replace(newLetExpr)

It remains to implement the renaming of elements that refer to our named PSI element. To do this, you need to implement the method from PsiReference.

PsiElement handleElementRename(@NotNull String newElementName) 
throws IncorrectOperationException;

or you can use com.intellij.psi.PsiReferenceBase.

Markers

IDEA widely uses hints in the form of markers

Examples from go-plugin, Frege, monkey

The required extension point:

<codeInsight.lineMarkerProvider language="Monkey"
implementationClass="com.github.pyltsin.monkeyplugin.editor.MonkeyFunctionLineMarkerProvider"/>

The file must implement LineMarkerProvider interface, whose methods return LineMarkerInfo object. Please note that markers should be linked only to the leaves of the PSI-tree.

Autocompletion

I think everyone who uses IDEA likes how it works with hints. It is very difficult to write a good autocompletion mechanism. But at the same time, you can implement some autocompletion relatively quickly, for example, these:

This completion is implemented with PsiReference.getVariants method, which returns all visible suitable values. Filtering by characters is performed by IDEA itself.

For more complex cases, you can use the extension point:

<completion.contributor language="Monkey"                         implementationClass="com.github.pyltsin.monkeyplugin.completion.MonkeyKeywordCompletionContributor"/>

Your class must implement CompletionContributor abstract class.

Documentation. Examples of implementation from go-plugin, frege, monkey

Testing

Documentation

In IDEA, tests are often several files with the source code of the language that show the state BEFORE and AFTER the action. BasePlatformTestCase class is usually used for it. Usually tests look like this:

myFixture.configureByFiles("RenameTestData.monkey")
myFixture.renameElementAtCaret("test")
myFixture.checkResultByFile("RenameTestData.monkey", "RenameTestDataAfter.monkey", false)

where RenameTestData.monkey and RenameTestDataAfter.monkey are source code files before and after renaming.

The conclusion

This is the end of the story about creating a language plugin for IDEA. Good luck in setting up your IDE for yourself!