DEV Community

Cover image for [Beginner] How to build Youtube video crawler web application with Rails 6, Devise, Nokogiri and Bootstrap 4?
Derek Nguyen
Derek Nguyen

Posted on

[Beginner] How to build Youtube video crawler web application with Rails 6, Devise, Nokogiri and Bootstrap 4?

#rails #bootstrap #crawler #youtube

In this tutorial, you will get a Funny Youtube Videos web application with Ruby on Rails up and running on your local server and will push your work into Github. How to make it alive? If you are casually browsing and are maybe interested in doing a Ruby on Rails tutorial, stop what you are doing RIGHT NOW and give this podcast a listen! It will give you the fuel you need to continue down the web developer path; it definitely fueled me.

Reading and executing tutorials is an amazing way to learn, so roll up your sleeves, take a slug of coffee, and let’s dive in! This tutorial assumes you have Ruby on Rails installed and all other pre-requisites, like github and a text editor and such.

I have put my github of tutorial at the end of the article.

1. Design our web application

Here, I have been design our app with 2 main layouts:

  • List youtubes videos
  • Share youtube video url
  1. Not signed in page

Alt Text

  1. Singed in page Alt Text
  2. Share youtube video page Alt Text

2. Building our web application

2.1 Setup Rails

As most Ruby on Rails fans might be aware, Rails 6 is coming soon, and bringing a number of eagerly awaited features and changes. For starters, remember that Rails 6 requires Ruby 2.5+ . So, make sure you have a plan to upgrade your systems accordingly, in case you have not done so already.

We will use Postgres for our project. So let write:

rails new funny-youtube-videos --database=postgresql
Enter fullscreen mode Exit fullscreen mode

A whole bunch of cool stuff happens in the terminal:
Alt Text

And wait amoment until it run out. Then:

cd funny-youtube-videos
rails s
Enter fullscreen mode Exit fullscreen mode

Now, navigate to localhost/3000 to make sure everything is working:

Yay! You’re on Rails!

Alt Text

Next, we will use Devise gem for authentication. If you don't know Devise, please go to https://github.com/plataformatec/devise, and read. This gem is very helpful,it help you save a lot time with how to control authentication for web application.

Add the following line to your Gemfile:

gem 'devise'
Enter fullscreen mode Exit fullscreen mode

Then run bundle install

Next, you need to run the generator:

$ rails generate devise:install
Enter fullscreen mode Exit fullscreen mode

In our application, User is model contains user's information. Next, we will setup devise with User:

 rails generate devise User
Enter fullscreen mode Exit fullscreen mode

Then runrails db:migrate

Now, check our folder models and db/schema.rb we got User table with: email, password, etc... Restart application server again.

Next step, we need table to storage youtube video shared by users. So we will create Post model with fields under:

  • Id : integer
  • Title: string
  • Share Url: string
  • Description: text
  • User Id : who share youtube video

That's Ok! We run rails g model Post title:string share_url:string description:text user:references . We got migrate file look likes:

class CreatePosts < ActiveRecord::Migration[6.0]
  def change
    create_table :posts do |t|
      t.string :share_url, null: false
      t.string :title
      t.text :description
      t.references :user, null: false, foreign_key: true

      t.timestamps
    end
  end
end
Enter fullscreen mode Exit fullscreen mode
2.2 Building controllers, views.

In our application, we need PostsController & UsersController. Open our config/routes.rb and adding bellow:

Rails.application.routes.draw do
  root 'posts#index'

  resources :posts
  devise_for :users

  post 'access' => 'users#access', as: :create_session
  get 'share' => 'posts#new', as: :share_movie
end
Enter fullscreen mode Exit fullscreen mode

In our UsersController we will control session with Devise. Help anonymous can sign up & sign in our application. At access , when anonymous enter correct their & password match with our users, it will create session and they signed in successfully. Opposite, it will create new user with email & password, then signed in.

def access
    if valid_user_params?
      user = User.find_for_authentication(email: user_params[:email])
      valid_auth = user && user&.valid_password?(user_params[:password])
      user = User.create(user_params) unless valid_auth
    end
    if user
      sign_in(user)
      flash[:success] = "You have been access successfully!"
    else
      flash[:error] = "Invalid email or password. Please try again!"
    end
    redirect_to root_path
  end
Enter fullscreen mode Exit fullscreen mode

At PostsController we have

a. index list youtube videos

@posts = Post.order(id: :desc)
Enter fullscreen mode Exit fullscreen mode

We will pagination for our list with gem will_paginate. Add our Gemfile gem 'will_paginate', '~> 3.1.0' then bundle install. Now:

@posts = Post.order(id: :desc).paginate(page: params[:page], per_page: 10)
Enter fullscreen mode Exit fullscreen mode

b. create post when cuser submit their youtube url at share page. I have been write script help we can get youtube video information such as: title, description. I use nokogiri to do it. nokogiri is a Rubygem providing HTML, XML, SAX, and Reader parsers with XPath and CSS selector support. We can use it to crawling websites, ... etc.

I write my script in a service object app/services/post_service.rb

require 'nokogiri'
require 'open-uri'
class PostService
  def initialize(current_user, params)
    @params = params
    @current_user = current_user
  end

  def create
    post = Post.new(@params)
    post.user = @current_user
    if valid_share_url?
      doc = get_doc(@params[:share_url]) if @params[:share_url].present?
      post.title = doc.css('title')&.first&.content&.strip
      post.description = doc.search('#watch-description-text')&.first&.content
    else
      post.errors.add(:share_url, 'Invalid Youtube url')
    end
    post
  end

  private

  def valid_share_url?
    @params[:share_url].match(/^(?:https?:\/\/)?(?:www\.)?youtu(?:\.be|be\.com)\/(?:watch\?v=)?([\w-]{10,})/).present?
  end

  def get_doc(path)
        user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.854.0 Safari/535.2"
        Nokogiri::HTML.parse(open(path, {'User-Agent' => user_agent}), nil, 'UTF-8')
    end
end
Enter fullscreen mode Exit fullscreen mode

So, now our create of PostsController will look like:

@post = PostService.new(current_user, post_params).create
respond_to do |format|
    if @post.errors.blank? && @post.save
        format.html { redirect_to root_url, notice: 'Your moive was successfully shared. Thank you!' }
        format.json { render :show, status: :created, location: @post }
    else
        flash[:error] = @post.errors.full_messages.first
        format.html { render :new }
        format.json { render json: @post.errors, status: :unprocessable_entity }
    end
end
Enter fullscreen mode Exit fullscreen mode

That's all logic. Now we need implement our web site with front end. We will use Bootstrap for front end. It too easy to use. I have been use slim to write html.

In views/layouts/application.html.slim

doctype html
html
  head
    meta content=("text/html; charset=UTF-8") http-equiv="Content-Type" /
    title Funny Movies Site
    = csrf_meta_tags
    = csp_meta_tag
    = stylesheet_pack_tag 'application', media: 'all', 'data-turbolinks-track': 'reload'
    = javascript_pack_tag 'application', 'data-turbolinks-track': 'reload'
  body
    = render 'shared/nav'
    .container
      = yield
      = render 'shared/footer'
Enter fullscreen mode Exit fullscreen mode

In views/shared/_nav.html.slim

.d-flex.flex-column.flex-md-row.align-items-center.p-3.px-md-4.mb-3.bg-white.border-bottom.box-shadow
  .col-sm-5
    = link_to root_path do
      h3.font-weight-normal Funny Movies Site
  .col-sm-7 style="display: flex; justify-content: flex-end;"
    - if current_user
      button.btn-light Welcome #{current_user.email}
      | &nbsp;  
      = link_to "Share a movie", share_movie_path, class: 'btn btn-success', role: "button"
      | &nbsp;
      = link_to "Logout", destroy_user_session_path, method: :delete, class: 'btn btn-danger', role: "button"
    - else
      = form_for :user, method: :post, url: create_session_path, html: { class: 'form-inline' } do |f|
        .form-group.mb-2
          = f.email_field :email, class: 'form-control', placeholder: 'email'
        .form-group.mx-sm-3.mb-2
          = f.password_field :password, class: 'form-control', placeholder: 'password'
        = f.submit 'Login / Register', class: 'btn btn-primary mb-2'
Enter fullscreen mode Exit fullscreen mode

In home page, also index of PostsController.

views/posts/index.html.slim

.container
  - if @posts.empty?
    h5 No movies
  - @posts.each do |post|
    .card.mb-3
      .row.no-gutters
        .col-md-6
          iframe width="420" height="315" src=(post.share_url.gsub('watch?v=', 'embed/'))
        .col-md-6
          .card-body
            h5.card-title #{post.title&.gsub('- YouTube', '')}
            p.share Share by: #{post.user.email}
            p Description:
            p.card-text #{post.description&.truncate(250)}

  = will_paginate @posts
Enter fullscreen mode Exit fullscreen mode

In share a youtube page.
views/posts/new.html.slim

.card style=("width: 40rem;margin: auto") 
  .card-header.text-center
    h5 Share a Youtube movie
  .card-body
    = render 'form'
Enter fullscreen mode Exit fullscreen mode

And views/posts/_form.html.slim

= form_for @post do |f|
  .form-group.row
    .col-sm-2
      = f.label :share_url, class: 'col-form-label'
    .col-sm-10
      = f.text_field :share_url, class: 'form-control', required: true
  .form-group.row
    .col-sm-2
    .col-sm-10
      = f.submit 'Submit', class: 'btn btn-primary btn-lg btn-block'
Enter fullscreen mode Exit fullscreen mode
2.3 Make it alive!

I recommend guys using Heroku for hostingRails application. Heroku is a platform as a service (PaaS) that enables developers to build, run, and operate applications entirely in the cloud. Register now!

This artice will help you deploy our website to Heroku.

Alt Text

Our website alive now: https://funny-movies-site.herokuapp.com

We have awesome funny youtube videos with Rails 6.

Alt TextAlt TextAlt Text

Conclusion

In this tutorial, we can build Sharing Youtube Videos with Rails 6 and gems: Devise, nokogiri, will_paginate and Bootstrap 4. Learn how about crawling website. And deploy our website to heroku. Make it alive!

Github Funny youtube videos

Top comments (0)

Read next

fedatu profile image

Mi primer programa.

Fernando -

bobby_chao profile image

Opensource and low-code platform for building configurable Gen-AI chatbots

Bobby -

krinskumar profile image

Proper Contributions - 1

Krinskumar Vaghasia -

xun_wang_6384a403f9817c2 profile image

鸿蒙案例实践：图像处理应用中多线程任务调度与性能优化

SameX -