DEV Community

Karl Heinz Marbaise
Karl Heinz Marbaise

Posted on • Edited on

Automate It - Be Lazy - Part II

#devops #maven #opensource #git

As I wrote in my previous post about automation I would continue
my post. So here it is.

The following parts are left from my process:

The process

  1. Create an issue for myself (if already existing move to step 2)
  2. Create a branch within the appropriate project based on the issue
  3. Assign the issue to myself
  4. Migrate the issue into the state of IN PROGRESS
  5. Done.
  6. Done.
  7. Done.
  8. Done.
  9. Close the appropriate jira issue with a reference to the commit in comment.

If I work on updates of dependencies or parents in our builds, it is required to make a JIRA ticket for each of those updates (ok not really required but it is useful). It makes sense to test each of those updates separately in Jenkins. In the end it is an important information for the end-users, so they can see what has been changed.

Automation Step 4

The first step is to create a ticket with some information like a summary, a description and giving Affected versions and Fix Version/s useful values. This is only needed if we do not have a ticket yet for the work.

So far I was going into my browser opened the appropriate project and pressed the Create button and typed in the appopriate values. I already
mastered the copy and paste parts from browser url to command line etc. After some time, I thought about this and came to the conclusioni, there must be a more convenient way.

I tried to use the REST API of JIRA with some JavaScript which I used earlier to create some release notes but it was too complex to handle that via JavaScript. After some research, I found a command line client which looks promising.

I started to play around with the command line client. There are some things to mention about it:

  1. Single binary artifact; simply download it and put on the path. Name it as you like (I decided to name it jira-cli)
  2. Templating mechanism to create custom commands
  3. Simple and powerful default commands already existing
  4. Easy configuration via yaml file.

The first thing is, you need to login into the JIRA instance which can be handled by the following:

$ jira-cli login --endpoint=https://issues.apache.org/jira
Enter fullscreen mode Exit fullscreen mode

You will be prompted for your password. The user which is used to login is derived by the user you are working with. For me it was wrong. So I needed to add a parameter:

$ jira-cli login --endpoint=https://issues.apache.org/jira --user=khmarbaise
Enter fullscreen mode Exit fullscreen mode

So now I can view a ticket on command line very easy:

$ jira-cli view MEJB-118
issue: MEJB-118
created: 339 days ago
status: Closed
summary: Remove "J2EE" from plugin description and site
project: MEJB
issuetype: Task
assignee: khmarbaise
reporter: afloom
fixVersions: 3.0.1
priority: Trivial
votes: 0
description: |
  The term "J2EE" was repalced by "Java EE" by Sun/oracle logn time ago and we shouldn't use "J2EE". It's currently used at least in the plugin's description (pom.xml) and on the intro page of the plugin's site.
  Replace "J2EE Enterprise Javabean (EJB)" with "Java Enterprise JavaBean (EJB)".

comments:
  - | # khmarbaise, 338 days ago
    Done in [r1805332|https://svn.apache.org/r1805332]
  - | # hudson, 338 days ago
    SUCCESS: Integrated in Jenkins build maven-plugins #9069 (See [https://builds.apache.org/job/maven-plugins/9069/])
    [MEJB-118] Remove "J2EE" from plugin description and site (khmarbaise: [http://svn.apache.org/viewvc/?view=rev&rev=1805332])
    * (edit) maven-ejb-plugin/pom.xml
    * (edit) maven-ejb-plugin/src/site/apt/index.apt.vm
Enter fullscreen mode Exit fullscreen mode

Just to take a short look on a particular issue it is much faster than going via the browser.

After more detailed reading the documentation I found that I can put the information about the end point and the user which should be used for jira into a configuration file (within your home directory) .jira.d/config.yml which looks like this:

endpoint: https://issues.apache.org/jira/
user: khmarbaise
login: khmarbaise
Enter fullscreen mode Exit fullscreen mode

With this the login command can be reduced to jira-cli login. Easy? The jira-cli will safe session information in .jira.d/ directory which is stored in cookies.js file. For my convenience I have created a Git repository in .jira.d and added cookies.js to the .gitignores file to prevent me to commit the cookies.js file accidently.

Now I have the ability to create JIRA tickets via command line.

So far so good. But how do I know for which project I need to create a ticket?

But wait a second. The pom.xml contains the information project.issueManagement.url which is the URL for the JIRA tracker. So I needed to find a way to extract it from the pom.xml.

Luckily the maven-help-plugin is here the way to go which looks like this (Just for brevity I have added \; usually this is a single line):

ISSUE_URL=$(mvn org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate \
  -Dexpression=project.issueManagement.url \
  -q \
  -DforceStdout)
Enter fullscreen mode Exit fullscreen mode

This will extract the URL for JIRA out of the pom file. So you get something like this:

https://issues.apache.org/jira/browse/MDEPLOY
Enter fullscreen mode Exit fullscreen mode

but unfortunately this is not the project you need for jira-cli. What I need is the MDEPLOY. This can easily achieved by:

PROJECT=$(basename $ISSUE_URL)
Enter fullscreen mode Exit fullscreen mode

These are the information which are needed to create a JIRA ticket. Hm...really? No. What about the version for Fixed versions/s and Affected versions ? This can be done by:

PLAIN_VERSION=$(mvn build-helper:parse-version \
  -Dx=\${parsedVersion.majorVersion}.\${parsedVersion.minorVersion}.\${parsedVersion.incrementalVersion} \
  org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate \
  -Dexpression=x \
  -q \
  -DforceStdout)
Enter fullscreen mode Exit fullscreen mode

This will extract the version out of the pom file without the trailing -SNAPSHOT. It assumes that the version number has three digits.

So now comes a favorite feature of jira-cli. You can create your own commands very easy. You simply define things like this in your config.yml file:

custom-commands:
  - name: mine
    help: display issues assigned to me
    script: |-
      {{jira}} list --template table --query "resolution = unresolved and assignee=currentuser() ORDER BY priority asc, created"
Enter fullscreen mode Exit fullscreen mode

So now you can simply call jira-cli like this:

$ jira-cli mine
Enter fullscreen mode Exit fullscreen mode

and you will see the list of issues which are assigned to yourself. See more details in the documentation of jira-cli. Based on the above, I have created a custom command which looks like this:

custom-commands:
  - name: createdependencyupgrade
    help: creates an Dependency Upgrade ticket.
    options:
      - name: affected
        help: The affected version
      - name: fix
        help: The fix versions.
      - name: project
        default: $JIRA_PROJECT
      - name: description
        default: ""
      - name: priority
        default: "Minor"
    args:
      - name: summary
        required: true
    script: |-
      {{jira}} create --project={{options.project}} --issuetype="Dependency upgrade" -o summary="{{args.summary}}" -o description="{{options.description}}" -o fixVersions="{{options.fix}}" -o versions="{{options.affected}}" -o priority="{{options.priority}}" --noedit
Enter fullscreen mode Exit fullscreen mode

So as the result of this, I can create a JIRA ticket via this:

$ jira-cli createdependencyupgrade --project MWAR --fix="3.0.2" --affected="3.0.2" "Summary Text"
Enter fullscreen mode Exit fullscreen mode

These have been the prerequisite to finally create a JIRA ticket in a very convenient way.
The result of previous things and integrating some other parts is now my
createdependencyupgradeissue.sh script which I can simply call like this:

$ createdependencyupgradeissue.sh "Upgrade maven-plugins parent to version 32"
Enter fullscreen mode Exit fullscreen mode

The output of that script looks at the moment like this:

$ createdependencyupgradeissue.sh "Upgrade maven-plugins parent to version 32"
Checking if we are on a branch...done.
Check that we are on master...done.
Extracting issue url from pom.xml file...done.
Extracting version from pom.xml file...done.
Creating JIRA issue...done.
OK MWAR-419 https://issues.apache.org/jira/browse/MWAR-419
M    pom.xml
Switched to a new branch 'MWAR-419'
Enter fullscreen mode Exit fullscreen mode

As you might already have realized, I have added things to create a branch in Git based on the JIRA issue name and switched to that branch.

The fourth step of automation accomplished which means more improvement but not enough.

Automation Step 5

If I like to assign a ticket to myself so far I had to go to the ticket page and click on assign to me link which prerequisites to know the issue number and type in the correct URL etc.

So based on my assumptions that the branch name is the JIRA issue number the only thing I need to do, is to extract the number from the branch. This can simply be achieved by:

BRANCH=$(git symbolic-ref --short HEAD)
Enter fullscreen mode Exit fullscreen mode

So next the thing is to assign the ticket to myself. This
can be done with jira-cli easily like this:

jira-cli take $BRANCH
Enter fullscreen mode Exit fullscreen mode

I have to add some sanity checks in case of errors and the result is the takeissue.sh script which I can use within the branch like this:

$ takeissue.sh
Enter fullscreen mode Exit fullscreen mode

The fifth step of automation is accomplished which means becoming comfortable but I think, we could do more.

Automation Step 6

This step is more or less identical to the previous one cause I would like to change the state of an issue into IN PROGRESS. This can easily be achieved by using jira-cli like this:

$ jira-cli start MDEP-234
Enter fullscreen mode Exit fullscreen mode

Based on the ideas before, this can be baked into a script startissue.sh which results in such a simple call

$ startissue.sh
Enter fullscreen mode Exit fullscreen mode

The sixth step of automation accomplished which means more improvement is possible.

Automation Step 7

Ok I can create an issue in JIRA including the creation of the branch in Git so what about to commit here? Usually I update files or add some new ones which is done by git add . and afterwards I can commit the state.

So based on the issue the commit message should look like this:

[XXX-123] - Summary 
Optional description
Enter fullscreen mode Exit fullscreen mode

In the majority of the cases I need only the first line. In other words this is copy and paste from the issue I created before. Let us make that easier. I created, of course you know already a script commitonissue.sh which exactly does that.

It identifies the branch based on the name of the branch in Git and gets the information about the summary via jira-cli from JIRA which results in such line:

SUMMARY=$(jira-cli view $BRANCH | grep "^summary: " | cut -d " " -f2-)
Enter fullscreen mode Exit fullscreen mode

and finally I need to do the real commit which can be handled by this:

git commit -a -m"[$BRANCH] - $SUMMARY"
Enter fullscreen mode Exit fullscreen mode

The seventh step of automation accomplished which means we are on the home straight. But not yet crossed the line.

Automation Step 8

This step is nothing really new, it is just improving something already existing. The gitmergeclean script of the first post. This is needed to be tweaked. It should also close the JIRA ticket after the branch has been merged to master with an appropriate comment on it.

I have added the following lines to the script to handle this.

#
# Get the latest commit HASH
#
COMMITHASH=$(git rev-parse HEAD)
#
# Get the GIT URL from pom file:
# TODO: Can we do some sanity checks? Yes: scm:git:..  if not FAIL!
echo -n "Get the git url from pom file..."
GITURL=$(mvn help:evaluate -Dexpression=project.scm.connection -q -DforceStdout | cut -d":" -f3-)
echo " URL: $GITURL"
GITPROJECT=$(basename $GITURL)
GITBASE=$(dirname $GITURL)
#
echo "Closing JIRA issue $BRANCH"
jira-cli close -m"Done in [$COMMITHASH|$GITBASE?p=$GITPROJECT;a=commitdiff;h=$COMMITHASH]" --resolution=Done $BRANCH
## Error handling?
echo "Closing finished."
Enter fullscreen mode Exit fullscreen mode

The BRANCH information is already extracted at the beginning of the script. The final lines of the script are to call jira-cli to close the issue with an appropriate comment and set the resolution to a useful value. In our case it is simply Done. The GITURL etc. is extacted to create a useful comment which contains a link to the real git repository with showing the diff of the commit which has been made. You can take a look at the MCLEAN-87 issue which shows a link to the git repository.

A full output of the new gitmergeandclean script looks like this:

~/ws-git-maven/plugins/maven-rar-plugin (MRAR-79)$ gitmergeandclean.sh
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
Updating 1ffdfba..055bbda
Fast-forward
 pom.xml | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
Total 0 (delta 0), reused 0 (delta 0)
remote: Sending notification emails to: ['"commits@maven.apache.org" <commits@maven.apache.org>']
remote: To git@github:apache/maven-rar-plugin.git
remote:    1ffdfba..055bbda  055bbda0907661aa6515cc8e79ba66cee43aec12 -> master
remote: Syncing refs/heads/master...
To https://gitbox.apache.org/repos/asf/maven-rar-plugin.git
   1ffdfba..055bbda  master -> master
remote: Sending notification emails to: ['"commits@maven.apache.org" <commits@maven.apache.org>']
remote: To git@github:apache/maven-rar-plugin.git
remote:  - [deleted]         MRAR-79
remote: Syncing refs/heads/MRAR-79 (FORCED)...
To https://gitbox.apache.org/repos/asf/maven-rar-plugin.git
 - [deleted]         MRAR-79
Deleted branch MRAR-79 (was 055bbda).
Get the git url from pom file... URL: https://gitbox.apache.org/repos/asf/maven-rar-plugin.git
Closing JIRA issue MRAR-79
OK MRAR-79 https://issues.apache.org/jira/browse/MRAR-79
Closing finished.
Enter fullscreen mode Exit fullscreen mode

Conclusion

So finally for a dependency upgrade the process looks like this:

  1. Change the appropriate information in pom.xml file
  2. createdependencyupgradeissue.sh "Upgrade maven-plugins parent to version 32"
  3. commitonissue.sh
  4. gitpushwithlease.sh
  5. takeissue.sh
  6. startissue.sh After that I have to wait for the result of Jenkins. If the build is successful I can merge back to master otherwise I need find the cause of the problem. In this case the result was successful so I can do it like this:
  7. gitmergeandclean.sh

The output looks in total like this:

~/ws-git-maven/plugins/maven-linkcheck-plugin (master *)$ createdependencyupgradeissue.sh "Upgrade maven-plugins parent to version 32"
Checking if we are on a branch...done.
Check that we are on master...done.
Extracting issue url from pom.xml file...done.
Extracting version from pom.xml file...done.
Creating JIRA issue...done.
OK MLINKCHECK-32 https://issues.apache.org/jira/browse/MLINKCHECK-32
M    pom.xml
Switched to a new branch 'MLINKCHECK-32'
~/ws-git-maven/plugins/maven-linkcheck-plugin (MLINKCHECK-32 *)$ commitonissue.sh
[MLINKCHECK-32 cd34b97] [MLINKCHECK-32] - Upgrade maven-plugins parent to version 32
 1 file changed, 1 insertion(+), 1 deletion(-)
~/ws-git-maven/plugins/maven-linkcheck-plugin (MLINKCHECK-32)$ gitpushwithlease.sh
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 1.53 KiB | 1.53 MiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Sending notification emails to: ['"commits@maven.apache.org" <commits@maven.apache.org>']
remote: To git@github:apache/maven-linkcheck-plugin.git
remote:  * [new branch]      cd34b97ddb661d5881bf7fc01e3291a9b0f67041 -> MLINKCHECK-32
remote: Syncing refs/heads/MLINKCHECK-32...
To https://gitbox.apache.org/repos/asf/maven-linkcheck-plugin.git
 * [new branch]      MLINKCHECK-32 -> MLINKCHECK-32
~/ws-git-maven/plugins/maven-linkcheck-plugin (MLINKCHECK-32)$ takeissue.sh
~/ws-git-maven/plugins/maven-linkcheck-plugin (MLINKCHECK-32)$ startissue.sh
Enter fullscreen mode Exit fullscreen mode

Here is the time where I have to wait for the results of Jenkins. So in this case the result was successful so I can continue like this:

~/ws-git-maven/plugins/maven-linkcheck-plugin (MLINKCHECK-32)$ gitmergeandclean.sh
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
Updating ff5dfab..cd34b97
Fast-forward
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Total 0 (delta 0), reused 0 (delta 0)
remote: Sending notification emails to: ['"commits@maven.apache.org" <commits@maven.apache.org>']
remote: To git@github:apache/maven-linkcheck-plugin.git
remote:    ff5dfab..cd34b97  cd34b97ddb661d5881bf7fc01e3291a9b0f67041 -> master
remote: Syncing refs/heads/master...
To https://gitbox.apache.org/repos/asf/maven-linkcheck-plugin.git
   ff5dfab..cd34b97  master -> master
remote: Sending notification emails to: ['"commits@maven.apache.org" <commits@maven.apache.org>']
remote: To git@github:apache/maven-linkcheck-plugin.git
remote:  - [deleted]         MLINKCHECK-32
remote: Syncing refs/heads/MLINKCHECK-32 (FORCED)...
To https://gitbox.apache.org/repos/asf/maven-linkcheck-plugin.git
 - [deleted]         MLINKCHECK-32
Deleted branch MLINKCHECK-32 (was cd34b97).
Get the git url from pom file... URL: https://gitbox.apache.org/repos/asf/maven-linkcheck-plugin.git
Closing JIRA issue MLINKCHECK-32
OK MLINKCHECK-32 https://issues.apache.org/jira/browse/MLINKCHECK-32
Closing finished.
Enter fullscreen mode Exit fullscreen mode

So if you look in my previous post you might be astonished that the number process
steps has not been reduced dramatically? Those scripts are replacing a large number of manual steps (git command line typing, jira in browser, clicking typing etc.) which is now simply handled by scripts which means the steps are done always in the same way and I am sure I do not forget steps. Furthermore many of scripts can be used in other situations as well which makes my life easier for example in handling pull request etc.

So this was a much longer post. I hope you enjoyed it.

Top comments (2)

Collapse
 
dmfay profile image
Dian Fay •

Have you thought about having Jenkins close the issue once a commit message including its id lands on the master branch? Especially in concert with a pull request workflow, that would let the build server handle just about everything gitmergeandclean.sh does (Jenkins would need permissions to automatically delete the remote branch and wouldn't be able to clean your local repository). But you might have some specific process requirements that necessitate kicking off the merge manually.

Collapse
 
khmarbaise profile image
Karl Heinz Marbaise •

Hi Dian,
first thanks for your comment.
And yes this could be done but unfortunately this can't be done on the Apache Software Foundation side, cause there we do not use pull request (or something which something like PR's). A PR must be "authorized" by a committer which means we have to create a branch for it and commit it later which means a name of a committer is given here (difference between committer/author)..cause the main source of truth is not GitHub but it is gitbox.apache.org/repos instead which does not directly support PR's in the way GitHub does. That's the reason why we need to go that way.

I'm currently experimenting with scripts to handle this also making this easier for me to support the contributions from other people in more simpler and faster way...

Read next

karthickdkk profile image

Dangerous Linux Configurations You Must Avoid for Secure Systems (Deep Dive)

karthick-dkk -

amandeol063 profile image

Week 2 in DevOps: From Linux Basics to Shell Scripting

Aman Deol -

nag5000 profile image

Discover JSREPL.io – A JavaScript REPL & Playground

Aleksey Nagovitsyn -

dazfuller profile image

Managing LLM prompts in DotNet with DotPrompt

Darren Fuller -