Recently I have encountered a very cool site with cooking recipes, which had extremely poor UI, especially when using a mobile. There was no official API, and so I have decided to build a web service that would web scrape the content out of it, and publish it using RESTful API. In this post I will show you how to use Scala, Play Framework and Jsoup to build such service. Enjoy!
You will need
- Scala - I use the version 2.13.1
- Sbt - install from official website
- IDE of choice - IntelliJ or VS Code, I use latter
- Around 15 minutes of your time
Creating empty Play project
We will start off by creating a Play project. You can of course start from the scratch and add the code iteratively, but for the scope of this tutorial, I will suggest using on of the Play's example projects as a starter.
The Scala Rest API project have almost everything we need. This project serves as a nice backbone to our use case, which we will be able to extend with ease.
Let's start by cloning the Play Framework samples repository, and opening the Scala Rest API example like this:
git clone https://github.com/playframework/play-samples.git cd play-scala-rest-api-example
After this, try running this command - the Play application should shoot off, and you should be able to access it at http://localhost:9000
sbt run
If the application builds and runs correctly, we are done with initializing the project. Now, you can open this folder with your IDE of choice - I will use VS Code
Adding Jsoup dependency
Next, we will add the Jsoup - an open source HTML parser for Java and JVM. To do this, open a build.sbt file and add the following in the libraryDependencies block:
"org.jsoup" % "jsoup" % "1.13.1"
Of course, make sure to use the latest version possible - at the time of authoring this article, 1.13.1 was the latest one.
Now, wait for sbt to refresh the dependencies, and we are ready to write some code!
Defining an endpoint
The first coding step is to define an endpoint to Play framework. This is done, by adding route definitions to conf/routes file. In our case, I want to have a v1/recipes endpoint that is solely handled by RecipeRouter class. We can do this by adding the following:
-> /v1/recipes v1.recipe.RecipeRouter
Now, any request that hits that URL, will be redirected to our router.
Next of, I am creating a package for my recipe related classes app/v1/recipe. In here, let's create an entry point that will handle the requests: RecipeRouter
package v1.recipe
import javax.inject.Inject
import play.api.routing.Router.Routes
import play.api.routing.SimpleRouter
import play.api.routing.sird._
class RecipeRouter @Inject()(controller: RecipeController) extends SimpleRouter {
val prefix = "/v1/recipes"
def link(id: String): String = {
import io.lemonlabs.uri.dsl._
val url = prefix / id.toString
url.toString()
}
override def routes: Routes = {
case GET(p"/") =>
controller.index
case GET(p"/all/$id") =>
controller.showAll(id)
}
}
Connecting to website
Going further, it's time to finally use Jsoup to get some content out of the website. In the previous section, we have defined showAll function, and that's the one I will be implementing here. We will be making use of the id parameter, which come in handy in many situations.
So, let's move to RecipeController class, and define the first element: sourceUrl. I like to have it defined as a class field, so that it is accessible by all methods:
val sourceUrl: String = "https://kwestiasmaku.com"
Then, inside our method, let's get the content of this website. It can be done like this:
def showAll(pageId: String): Action[AnyContent] = Action { implicit request =>
val htmlDocument = Jsoup.connect(s"${sourceUrl}/home-przepisy?page=${pageId}").get()
val r: Result = Ok(htmlDocument)
r
}
In the snippet above, we are connecting to our website, b using Jsoup.connect method, while passing the correct URL as a string. What we get, is an HTML document, that we just return for now. We will do some finer extraction in the next section.
To test the endpoint, call the following URL in your browser, or any http client, like curl:
http://localhost:9000/v1/recipes/all/1
You should see a response, which is raw HTML document. Now, let's parse it and extract some meaningful data.
Extracting the content
Now, let's extract some specific elements from the raw HTML. For this, we can use the select method of Jsoup. This method requires a CSS like selector syntax - if you have used CSS before, then it is as easy specifying the correct string - in our case, to extract single recipe HTML object, use the following:
import scala.jdk.CollectionConverters._
val recipesDomElements = htmlDocument.select("section#block-system-main .col").asScala
This query returns, collection-like structure, on which we can easily iterate. To extract specific elements from a single recipe, use the following:
final case class Recipe(title: String, href: String, img: String)
val recipeData = for(recipeElement <- recipesDomElements)
yield Recipe(
recipeElement.select(".views-field-title a").html(),
sourceUrl + recipeElement.select(".views-field-title a").attr("href"),
recipeElement.select("img").attr("src")
)
In here, we use a simple for loop, which yields the concrete data, by wrapping it into Recipe case class. Now, we have nicely cleaned data, and we can finally return it as JSON.
Returning a JSON
Last step is to return our collection of Recipe case classes as a JSON. We can do this using Play's built in Json class, and its methods:
implicit val recipeWrites: OWrites[Recipe] = Json.writes[Recipe] val r: Result = Ok(Json.toJson(recipeData)) r
An important thing to have in mind, is that before converting a case class into its JSON representation, we must declare implicit writer, so that the Json class knows how to write this specific data.
And that's it! The complete RecipeController class implementation looks like the following:
package v1.recipe
import javax.inject.Inject
import org.jsoup.Jsoup
import play.api.mvc._
import play.api.libs.json.{Json, OWrites}
import scala.jdk.CollectionConverters._
class RecipeController @Inject()(val controllerComponents: ControllerComponents) extends BaseController {
implicit val recipeWrites: OWrites[Recipe] = Json.writes[Recipe]
val sourceUrl: String = "https://kwestiasmaku.com"
def index: Action[AnyContent] = Action { implicit request =>
val r: Result = Ok("hello world")
r
}
def showAll(pageId: String): Action[AnyContent] = Action { implicit request =>
val htmlDocument = Jsoup.connect(s"${sourceUrl}/home-przepisy?page=${pageId}").get()
val recipesDomElements = htmlDocument.select("section#block-system-main .col").asScala
val recipeData = for(recipeElement
Summary
I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so 🙂
Oldest comments (0)