When I was building my Covid-19 Dashboard app, I needed a way to automatically query the Covid Tracking Project API on a daily basis and update my database. Accomplishing this with Laravel consists of the following steps:
- Create an Artisan console command to start the task
- Create a job that performs the work
- Tell Laravel when the task needs to run
- Update the server to work with Laravel
All of this assumes that you have already created the database and tables that you need. Building the database is outside the scope of this article. I will be using my Covid-19 Dashboard as an example. The code for this project can be found here.
Step 1 - Create an Artisan console command
An Artisan console command is like a switch or a button that you can press whenever you want Laravel to start performing a specific task. You can "activate the switch" or "press the button" to initiate your task, and this can be done in different ways. One way is to type the Artisan command into the command line of your console. Another way is to include the command inside your application code so that your task will be executed at a specific date and time.
Here's something nice. You can use one Artisan command to make another! To create your new command, use the Laravel supplied Artisan make:command. For the Covid-19 Dashboard project, I needed a job that would query the Covid Tracking Project API and update my database. But first, I needed a command that would start this job. I decided to call this command LoadApiData, so I entered the following in the console.
php artisan make:command LoadApiData
Laravel then generated the following class, named LoadApiData, and placed it in the app/Console/Commands folder.
<?php
namespace App\Console\Commands;
use Illuminate\Console\Command;
class LoadApiData extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'command:name';
/**
* The console command description.
*
* @var string
*/
protected $description = 'Command description';
/**
* Create a new command instance.
*
* @return void
*/
public function __construct()
{
parent::__construct();
}
/**
* Execute the console command.
*
* @return mixed
*/
public function handle()
{
//
}
}
The class generated by Laravel is not complete. We finish it by filling in the signature and description properties of the class, as well as the handle() method.
The signature will be used to run our command. When we type it into the console, the command will be executed. In my example it looks like this.
protected $signature = 'app:load_api_data';
The description should be very brief, just a few words that get displayed in the console whenever we list our console commands. In my example it looks like this.
protected $description = 'Load data from the Covid Tracking Project API';
The handle() method gets called when the command is executed. We will use this function to kick off the job that performs the actual work that needs to be done. In my example, that job is the LoadApiDataJob class. As you can see, the name of my job class is the same as the name as my command class, except for the word "Job" appended to it.
public function handle()
{
LoadApiDataJob::dispatch();
}
The finished LoadApiData class is shown here.
<?php
namespace App\Console\Commands;
use Illuminate\Console\Command;
use App\Jobs\LoadApiDataJob;
class LoadApiData extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'app:load_api_data';
/**
* The console command description.
*
* @var string
*/
protected $description = 'Load data from the Covid Tracking Project API';
/**
* Create a new command instance.
*
* @return void
*/
public function __construct()
{
parent::__construct();
}
/**
* Execute the console command.
*
* @return mixed
*/
public function handle()
{
LoadApiDataJob::dispatch();
}
}
Step 2 - Create a Job
Next, we need a job that will do the heavy lifting and performs the tasks we want to accomplish. This job will be initiated by the command we just built. Once again we can use an Artisan command to generate a skeleton class that will help us get started. In the command line I typed the following.
php artisan make:job LoadApiDataJob
Laravel then generated the following class, named LoadApiDataJob, and placed it in the app/Jobs folder.
<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
class LoadApiDataJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
/**
* Create a new job instance.
*
* @return void
*/
public function __construct()
{
//
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
//
}
}
When the job is called, the handle() method will be executed, so the logic for completing our automated task is placed in this function. For the Covid-19 Dashboard, this is a two part process. First, state specific data is retrieved from the API and then used to update the state_historical_data table. Then United States national level data is retrieved from the API, and this data is used to update the us_historical_data table. The finished code for this is shown below.
<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Http;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\DB;
use Carbon\Carbon;
use App\Helpers\Misc;
class LoadApiDataJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
/**
* Create a new job instance.
*
* @return void
*/
public function __construct()
{
//
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
$job = Misc::jobStart('LoadApiDataJob');
// Retrieve state data from dev api
$query_string = 'https://covidtracking.com/api/states/daily';
$response = Http::get($query_string)->getBody();
$apiRecords = json_decode((string) $response);
$count = count($apiRecords);
for ($i = 0; $i < $count; $i++) {
$r = $apiRecords[$i];
$date = Misc::formatDate($r->date);
if ($date['carbonDate']->lessThan($job['earliestDate'])) {
continue;
}
if ($date['carbonDate']->greaterThan($job['latestDate'])) {
continue;
}
$death = property_exists($r, 'death') ? $r->death : 0;
$death = $death === NULL ? 0 : $death;
$deathIncrease = property_exists($r, 'deathIncrease') ? $r->deathIncrease : 0;
$deathIncrease = $deathIncrease === NULL ? 0 : $deathIncrease;
$totalTestResults = property_exists($r, 'totalTestResults') ? $r->totalTestResults : 0;
$totalTestResults = $totalTestResults === NULL ? 0 : $totalTestResults;
$totalTestResultsIncrease = property_exists($r, 'totalTestResultsIncrease') ? $r->totalTestResultsIncrease : 0;
$totalTestResultsIncrease = $totalTestResultsIncrease === NULL ? 0 : $totalTestResultsIncrease;
$positive = property_exists($r, 'positive') ? $r->positive : 0;
$positive = $positive === NULL ? 0 : $positive;
$positiveIncrease = property_exists($r, 'positiveIncrease') ? $r->positiveIncrease : 0;
$positiveIncrease = $positiveIncrease === NULL ? 0 : $positiveIncrease;
DB::table('state_historical_data')
->updateOrInsert(
['day' => $date['formattedDate'], 'state' => $r->state],
[
'total_deaths' => $death, 'daily_deaths' => $deathIncrease,
'total_tests' => $totalTestResults, 'daily_tests' => $totalTestResultsIncrease,
'total_cases' => $positive, 'daily_cases' => $positiveIncrease
]
);
}
// Retrieve US data from dev api
$query_string = 'https://covidtracking.com/api/us/daily';
$response = Http::get($query_string)->getBody();
$apiRecords = json_decode((string) $response);
$count = count($apiRecords);
for ($i = 0; $i < $count; $i++) {
$r = $apiRecords[$i];
$date = Misc::formatDate($r->date);
if ($date['carbonDate']->lessThan($job['earliestDate'])) {
continue;
}
if ($date['carbonDate']->greaterThan($job['latestDate'])) {
continue;
}
$death = property_exists($r, 'death') ? $r->death : 0;
$death = $death === NULL ? 0 : $death;
$deathIncrease = property_exists($r, 'deathIncrease') ? $r->deathIncrease : 0;
$deathIncrease = $deathIncrease === NULL ? 0 : $deathIncrease;
$totalTestResults = property_exists($r, 'totalTestResults') ? $r->totalTestResults : 0;
$totalTestResults = $totalTestResults === NULL ? 0 : $totalTestResults;
$totalTestResultsIncrease = property_exists($r, 'totalTestResultsIncrease') ? $r->totalTestResultsIncrease : 0;
$totalTestResultsIncrease = $totalTestResultsIncrease === NULL ? 0 : $totalTestResultsIncrease;
$positive = property_exists($r, 'positive') ? $r->positive : 0;
$positive = $positive === NULL ? 0 : $positive;
$positiveIncrease = property_exists($r, 'positiveIncrease') ? $r->positiveIncrease : 0;
$positiveIncrease = $positiveIncrease === NULL ? 0 : $positiveIncrease;
DB::table('us_historical_data')
->updateOrInsert(
['day' => $date['formattedDate'], 'state' => 'US'],
[
'total_deaths' => $death, 'daily_deaths' => $deathIncrease,
'total_tests' => $totalTestResults, 'daily_tests' => $totalTestResultsIncrease,
'total_cases' => $positive, 'daily_cases' => $positiveIncrease
]
);
}
Misc::jobEnd('LoadApiDataJob');
}
}
At this point, we can test our work. All we have to do is enter our new command in the console, using the same signature we defined in the LoadApiData class. We enter the command we created in step 1, and the command then executes the job we created in step 2.
php Artisan app:load_api_data
Step 3 - Schedule the Job
Once we've determined that our logic is working as planned, we can schedule our task to run whenever we want. We accomplish this inside of class app/Console/Kernel.php. Once again we use the command signature that we defined in step 1 when we built the LoadApiData class. To update the Covid-19 database three times every day, at 2:00, 10:00 and 18:30, we could modify the schedule() function in Kernel.php ****to look like this.
/**
* Define the application's command schedule.
*
* @param \Illuminate\Console\Scheduling\Schedule $schedule
* @return void
*/
protected function schedule(Schedule $schedule)
{
$schedule->command('app:load_api_data')->dailyAt('2:00');
$schedule->command('app:load_api_data')->dailyAt('10:00');
$schedule->command('app:load_api_data')->dailyAt('18:30');
}
You can schedule your commands to run at any frequency you want. The Laravel documentation has all the details.
Step 4 - Update the server cron table
Unix-like servers contain a software utility called cron that is used for scheduling jobs. Using cron can be a difficult and error prone undertaking, but Laravel has found a way to simplify the process, so that you only need to have the most minimal interaction with the utility. For your application, you just add one entry to the cron table, and it's always the same entry for any Laravel application. This entry will tell your server to call the Laravel scheduler every minute of every day. Then, instead of cron, Laravel can assume the responsibility for keeping track of what needs to run, and when it needs to run, as described in step 3.
The Covid-19 Dashboard application is set up on my server like this, with the public files in the html directory and the private files in the covid19 directory.
var
|---www
|---covid19.bob-humphrey.com
|----html
|----covid19
I updated the cron table on my server, which is running Ubuntu 18.04, by entering
crontab -e
This brought up a text editing application that I could use to edit the cron table. I added the following entry to the bottom of the table and then saved the file.
* * * * * cd /var/www/covid19.bob-humphrey.com/covid19 && php artisan schedule:run >> /dev/null 2>&1
The only thing that changes from one project to the next is the folder being pointed to on the server. Just make sure it's the one that contains your project's private files.
Top comments (4)
Nice work!
Although when I try the
php artisan app:load_api_datastep I get the following error :
i have a question, what's the logic behind creating command and creating a job with that command. you can directly write the requirements in the command and both command and job runs in background and you can schedule the command directly.
can you please provide any good logic over this.
You're right, you can put your logic in the command class without creating a job. That makes a lot of sense to me.
The Laravel documentation has this suggestion:
"It is good practice to keep your console commands light and let them defer to application services to accomplish their tasks."
I based my approach on this.
ohhk. I personally prefer running commands as independent unit. no dependency on queue n all.