DEV Community

Mario García
Mario García

Posted on • Updated on

Automate your posting process on DEV with Python, Bash and GitLab CI

#linux #tutorial #bash #gitlab

The common process for creating a new post on DEV is:

  • Login in
  • Click on "Create Post"
  • Write the content of your post using Markdown
  • Click on "Publish"

DEV API

You can also use the DEV API for creating a new post. According to the documentation, you can create a new post using curl:

curl -X POST -H "Content-Type: application/json" \
 -H "api-key: API_KEY" \
 -d '{"article":{"title":"Title","body_markdown":"Body","published":false,"tags":["discuss", "javascript"]}}' \
 https://dev.to/api/articles
Enter fullscreen mode Exit fullscreen mode

In the above command you must replace the value of:

  • title
  • body_markdown
  • published
  • tags

according to the details of your post.

title is the name of your post.

body_markdown is equal to the content of your post. You can create a Markdown file using an editor like ghostwriter and pass the content of that file to this variable. Create a folder named articles to store your Markdown files.

published set to false or true, when set to false your post will be saved as draft.

tags can be any value here and you can assign more than one tag.

You need to generate an API Key following the instructions here and replace that value in the script.

Bash

If you're passing the content of your post from a Markdown file you must format it first, you can create a Bash script, publish.sh, to do it:

md="articles/article.md" # Markdown file

markdown=$(sed 's/$/\\n/' $md | tr -d '\n')
dev_api_key="API KEY"

curl -X POST -H "Content-Type: application/json" \
 -H "api-key: $dev_api_key" \
 -d '{"article":{"title":"Title","body_markdown":"'"$markdown"'","published":true,"tags":["discuss", "javascript"]}}' \
 https://dev.to/api/articles
Enter fullscreen mode Exit fullscreen mode

In your script, you can define a variable with the tags of your post and replace the value of tags in the curl command, you can modify your script as follows:

md="articles/article.md" # Markdown file
t=("linux tutorial") # Article tags

markdown=$(sed 's/$/\\n/' $md | tr -d '\n')
dev_api_key="API KEY"

tags="["
for i in "${t[@]}"; do
 tags+="\"$i\""
 if [[ $i != ${t[-1]} ]]; then
 tags+=","
 fi
done
tags+="]"

json='{"article":{"title":"Title","body_markdown": "'"$markdown"'","published":true,"tags":["tag1", "tag2"]}}'
json=$(echo $json | jq --argjson tags $tags '.article.tags |= $tags')
Enter fullscreen mode Exit fullscreen mode

The script will get the tags from the t variable and create a string similar to "linux", "tutorial", this value is assigned to the tags variable.

Then a json string is defined and tags variable is assigned with temporary value, ["tag1", "tag2"], that will be replaced using jq. Install jq on your system if it isn't installed.

The curl command is updated and will look like this:

curl -X POST -H "Content-Type: application/json" \
 -H "api-key: $dev_api_key" \
 -d "$json" \
 https://dev.to/api/articles
Enter fullscreen mode Exit fullscreen mode

Now you can publish a new post by running the Bash script you created before. Using only an offline editor and the command line.

Python

A YAML file, articles.yml, containing details of your post, should be created in the articles folder:

recent_articles:
 - title: "Title"
   body: "articles/article.md"
   tags: "linux tutorial"
Enter fullscreen mode Exit fullscreen mode

title is the title of your post.

body is the name of the Markdown file

tags is the tags assigned to your post

To get these values, a Python script is created, publish.py, then this script will run the Bash script previously created and pass those values.

sed command is used to escape characters in the Markdown file but some of them are not properly escaped. You can use json.dumps() in the Python script to get the JSON string required to pass to the curl command.

import yaml
import subprocess
import json

def json_escape(md):
 markdown=json.dumps(md)
 return markdown

with open('articles/articles.yml') as f:
 # use safe_load instead load
 dict = yaml.safe_load(f)

title = dict['recent_articles'][0]['title']
body = dict['recent_articles'][0]['body']
tags = dict['recent_articles'][0]['tags']

f = open('articles/article.md', 'r')
md = f.read()
markdown = json_escape(md)

subprocess.run(["/bin/bash", "script.sh", title, body, tags, markdown])
Enter fullscreen mode Exit fullscreen mode

Before running this scipt, you must install the PyYAML library by running:

pip install PyYAML
Enter fullscreen mode Exit fullscreen mode

The Bash script must be updated:

title=$1 # Article title
md=$2 # Markdown file
t=($3) # Article tags
markdown=$4

dev_api_key="API KEY"

tags="["
for i in "${t[@]}"; do
 tags+="\"$i\""
 if [[ $i != ${t[-1]} ]]; then
 tags+=","
 fi
done
tags+="]"

json='{"article":{"title":"'"$title"'","body_markdown":'$markdown',"published":false,"tags":["tag1", "tag2"]}}'
json=$(echo $json | jq --argjson tags $tags '.article.tags |= $tags')

curl -X POST -H "Content-Type: application/json" \
 -H "api-key: $dev_api_key" \
 -d "$json" \
 https://dev.to/api/articles
Enter fullscreen mode Exit fullscreen mode

GitLab CI

Now the process has changed to this:

  • Create your post using a Markdown editor
  • Edit articles.yml and add details of your post
  • Run publish.py from the command line

You still have to run the Python script manually and as it would be great to also announce your post on Twitter, you will use GitLab CI to automate part of the process.

Create a GitLab repository and upload the scripts you previously created. Replace dev_api_key="API KEY" with dev_api_key=sys.argv[1] in your publish.sh. And change your publish.py script to:

import yaml
import subprocess
import sys
import json

def json_escape(md):
 markdown=json.dumps(md)
 return markdown

dev_api_key = sys.argv[1]

with open('articles/articles.yml') as f:
 dict = yaml.safe_load(f)

title = dict['recent_articles'][0]['title']
body = dict['recent_articles'][0]['body']
tags = dict['recent_articles'][0]['tags']

f = open(body, 'r')
md = f.read()
markdown = json_escape(md)

subprocess.run(["/bin/bash", "publish.sh", title, body, tags, dev_api_key, markdown])
Enter fullscreen mode Exit fullscreen mode

Also, replace the Bash script as shown below:

title=$1 # Article title
md=$2 # Markdown file
t=($3) # Article tags
dev_api_key=$4
markdown=$5

tags="["
for i in "${t[@]}"; do
 tags+="\"$i\""
 if [[ $i != ${t[-1]} ]]; then
 tags+=","
 fi
done 
tags+="]"

json='{"article":{"title":"'"$title"'","body_markdown": '$markdown',"published":true,"tags":["tag1", "tag2"]}}'
json=$(echo $json | jq --argjson tags $tags '.article.tags |= $tags')

url=$(curl -X POST -H "Content-Type: application/json" \
 -H "api-key: $dev_api_key" \
 -d "$json" \
 https://dev.to/api/articles | jq '.url')

echo $url
Enter fullscreen mode Exit fullscreen mode

Before configuring GitLab CI, you have to create a Python script that will be run after your post is published for announcing on Twitter your new content.

Twitter

If you don't have a developer account, you must create one, in order to use TWitter API. You have to apply for getting access.

Once you're developer account is approved, go to developer.twitter.com/en/portal/dashboard and create a new app, then generate API Key and Secret, and Access Token and Secret for your app.

Create a Python script:

from twitter import *
import sys
import yaml

with open('articles/articles.yml') as f:
 # use safe_load instead load
 dict = yaml.safe_load(f)

title = dict['recent_articles'][0]['title']
url = sys.argv[5]

t = Twitter(auth=OAuth(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]))

update = "New Blog Post: " + title + "\n" + url + "\n #DEVCommunity"

# Update your status
t.statuses.update(status=update)
Enter fullscreen mode Exit fullscreen mode

twitter and PyYAML libraries are required. This script will get details of your post from articles/articles.yml and will be run right after your post is created.

To access Twitter API, values of Access Token and Secret, and API Key and Secret will be passed as parameters to this script.

Configuring GitLab

Before configuring GitLab CI, go to Settings → CI/CD and add the following masked variables:

  • ACCESS_TOKEN
  • TOKEN_SECRET
  • API_KEY
  • API_SECRET
  • DEV_API_KEY

GitLab CI pipeline will have two stages, build and deploy, and two jobs, publish and tweet.

Create .gitlab-ci.yml file:

stages:
 - build
 - deploy

publish:
 image: python:3.9.7
 stage: build
 script:
 - apt update && apt install -y jq
 - pip install PyYAML 
 - URL=$(python publish.py $DEV_API_KEY)
 - echo "URL=${URL}" >> build.env
 artifacts:
 reports:
 dotenv: build.env
 allow_failure: false

tweet:
 image: python:3.9.7
 stage: deploy
 script:
 - pip install twitter PyYAML
 - python tweet.py $ACCESS_TOKEN $TOKEN_SECRET $API_KEY $API_SECRET $URL
 needs:
 - job: publish
 artifacts: true
Enter fullscreen mode Exit fullscreen mode

In the publish job, GitLab CI will install jq and PyYAML, and run publish.py with the value of DEV_API_KEY as parameter, this variable is defined in the CI/CD configuration.

This is the output you get after creating a new post using curl:

{
 "type_of": "article",
 "id": 150589,
 "title": "Byte Sized Episode 2: The Creation of Graph Theory ",
 "description": "The full story of Leonhard Euler and the creation of this fundamental computer science principle, delivered in a few minutes.",
 "cover_image": "https://res.cloudinary.com/practicaldev/image/fetch/s--qgutBUrH--/c_imagga_scale,f_auto,fl_progressive,h_420,q_auto,w_1000/https://thepracticaldev.s3.amazonaws.com/i/88e62fzblbluz1dm7xjf.png",
 "readable_publish_date": "Aug 1",
 "social_image": "https://res.cloudinary.com/practicaldev/image/fetch/s--6wSHHfwd--/c_imagga_scale,f_auto,fl_progressive,h_500,q_auto,w_1000/https://thepracticaldev.s3.amazonaws.com/i/88e62fzblbluz1dm7xjf.png",
 "tag_list": "computerscience, graphtheory, bytesized, history",
 "tags": [
 "computerscience",
 "graphtheory",
 "bytesized",
 "history"
 ],
 "slug": "byte-sized-episode-2-the-creation-of-graph-theory-34g1",
 "path": "/bytesized/byte-sized-episode-2-the-creation-of-graph-theory-34g1",
 "url": "https://dev.to/bytesized/byte-sized-episode-2-the-creation-of-graph-theory-34g1",
 "canonical_url": "https://dev.to/bytesized/byte-sized-episode-2-the-creation-of-graph-theory-34g1",
 "comments_count": 21,
 "positive_reactions_count": 122,
 "public_reactions_count": 322,
 "collection_id": 1693,
 "created_at": "2019-07-31T11:15:06Z",
 "edited_at": null,
 "crossposted_at": null,
 "published_at": "2019-08-01T15:47:54Z",
 "last_comment_at": "2019-08-06T16:48:10Z",
 "published_timestamp": "2019-08-01T15:47:54Z",
 "reading_time_minutes": 15,
 "body_html": "<p>Today's episode of Byte Sized is about Leonhard Euler and the creation of <a href=\"https://en.wikipedia.org/wiki/Graph_theory\">Graph Theory</a>.</p>\n\n<p>For more about how Graph Theory works, check out this video from BaseCS!</p>...\n",
 "body_markdown": "---\r\ntitle: Byte Sized Episode 2: The Creation of Graph Theory \r\npublished: true\r\ndescription: The full story of Leonhard Euler and the creation of this fundamental computer science principle, delivered in a few minutes.\r\ntags: computerscience, graphtheory, bytesized, history\r\ncover_image: https://thepracticaldev.s3.amazonaws.com/i/88e62fzblbluz1dm7xjf.png\r\nseries: Byte Sized Season 1\r\n---\r\n\r\nToday's episode of Byte Sized is about Leonhard Euler and the creation of [Graph Theory](https://en.wikipedia.org/wiki/Graph_theory).\r\n\r\nFor more about how Graph Theory works, check out this video from BaseCS!...",
 "user": {
 "name": "Vaidehi Joshi",
 "username": "vaidehijoshi",
 "twitter_username": "vaidehijoshi",
 "github_username": "vaidehijoshi",
 "website_url": "http://www.vaidehi.com",
 "profile_image": "https://res.cloudinary.com/practicaldev/image/fetch/s--eDGAYAoK--/c_fill,f_auto,fl_progressive,h_640,q_auto,w_640/https://thepracticaldev.s3.amazonaws.com/uploads/user/profile_image/2882/K2evUksb.jpg",
 "profile_image_90": "https://res.cloudinary.com/practicaldev/image/fetch/s--htZnqMn3--/c_fill,f_auto,fl_progressive,h_90,q_auto,w_90/https://thepracticaldev.s3.amazonaws.com/uploads/user/profile_image/2882/K2evUksb.jpg"
 },
 "organization": {
 "name": "Byte Sized",
 "username": "bytesized",
 "slug": "bytesized",
 "profile_image": "https://res.cloudinary.com/practicaldev/image/fetch/s--sq0DrZfn--/c_fill,f_auto,fl_progressive,h_640,q_auto,w_640/https://thepracticaldev.s3.amazonaws.com/uploads/organization/profile_image/865/652f7998-32a8-4fd9-85ca-dd697d2a9ee9.png",
 "profile_image_90": "https://res.cloudinary.com/practicaldev/image/fetch/s--1Pt_ICL---/c_fill,f_auto,fl_progressive,h_90,q_auto,w_90/https://thepracticaldev.s3.amazonaws.com/uploads/organization/profile_image/865/652f7998-32a8-4fd9-85ca-dd697d2a9ee9.png"
 }
}
Enter fullscreen mode Exit fullscreen mode

From above JSON, GitLab CI is getting the value of the url variable that contains the URL of your new post, and passing that value to next job.

In the tweet job, GitLab CI installs twitter and PyYAML, Python libraries required, and run tweet.py.

Your repository is now configured. When you're ready to publish your post, upload the Markdown file and edit articles/articles.yml with details of your post, and GitLab CI will publish your post on DEV and tweet about it.

You can check a repository already configured here.

Latest comments (0)

Read next

lilxyzz profile image

Incredible JavaScript Animation Libraries

Travis -

frivolouis profile image

Building for sustainability: Dashboards

Louis Leung -

sanx profile image

Java Memory Management Demystified: Ensuring Stability and Efficiency 🚀💡

Sanjay R -

muhammadazfaraslam profile image

Managing Global State with useReducer and Context API in Next JS 14

Muhammad Azfar Aslam -