✨ In this article, we are going to discuss how to install a low price SEQ server in an AWS Linux EC2 instance and use it to search and analysis of application data logs.

What is SEQ ?

✔ Seq is a real-time search and analysis server for structured application log data. Its carefully-designed user interface, JSON event store, and familiar query language make it an efficient platform for detecting and diagnosing issues in complex applications and microservices.

• SEQ can easily be installed on a windows machine, but we are going to use docker to install SEQ on Linux instance.

• So in this will be a hands-on we will discuss:
  👉 Create Spot instance EC2 instance.
  👉 Create EBS Volume and mount it, then use it to store logs.
  👉 Install docker and use it to install SEQ.

  ---

  Create EC2 Spot instance:

• From EC2 dashboard, click on Launch instances
  

• From Quick start, click on Ubuntu AMI, select the latest available version of Ubuntu Ubuntu Server 22.04 LTS (HVM), SSD Volume Type.

• Select Architecture of 64-bit(Arm) to be able to select t4g.micro instance type which have an hourly cost of about 0.0094 USD and 2 vCPUs and 1 GB of memory.

• Then select a key-pair or create a new one for your instance and save it to be able to access it using SSH connection later.

• Select an existing security group or create a new one

  • Allow SSH access to your IP to SSH connect from your PC later or Allow SSH from one of AWS IP ranges like 197.61.178.208/32 if you want to connect via EC2 instance connect.
  • You can also allow all SSH traffic (0.0.0.0/0) for the sake of installation then edit your security group and remove all SSH inbound rules later.

• Then, for storage, we will go with a root volume of 8 GB General Purpose SSD

• Then, we will add a new Volume to store our logs in it, let's say we will start with a 10 GB General Purpose SD, we can extend it later if we need, Click Add New Volume

• Make sure to change Delete on Termination to false, that will allow you to keep your logs in the future and attach them to another instance in the future in case if you needed a larger instance.

• Now, Let's extend our saving, from Advanced Details, Check Request Spot Instances, this option will request Spot Instances at the Spot price, capped at the On-Demand price.

• After that click on Launch instance.

• Until your instance launches, let's check what is the effect of selecting a Spot instance.

• Spot instances are spare EC2 capacity that is available for less than the On-Demand price, for more info about it, check the official AWS documentation here

• Let's quickly check how much can we save when we use a spot instance, let's open a new tab and from EC2 Dashboard, let's select Spot Requests.

• Now, click on Pricing History, then select the instance we chose to host out application, which is t4g.micro ...

• As you can see, using spot instance will save you about 70% from the on-demand price, of course this saving changes from one region to another and from time to another.

• Let's get back to our instance setup again and let's connect to it.

• Select your instance, click on Actions and select Connect.

• Now select the preferred way for you to connect to your new EC2 instance, I'll go in this demo with EC2 Instance Connect.

• Now, we are connected to our instance.

Mount EBS Volume to EC2 instance:

• We should now mount the second volume we created [ 10 GB ]. After you make the EBS volume available for use, we can access it in the same ways that you access any other volume.

1- Let's view your available disk devices and their mount points (if applicable) to help you determine the correct device name to use by using lsblk

ubuntu@ip-10-0-0-142:~$ lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0          7:0    0 21.2M  1 loop /snap/amazon-ssm-agent/6313
loop1          7:1    0   49M  1 loop /snap/core18/2636
loop2          7:2    0   59M  1 loop /snap/core20/1699
loop3          7:3    0  102M  1 loop /snap/lxd/23545
loop4          7:4    0   43M  1 loop /snap/snapd/17885
nvme0n1      259:0    0    8G  0 disk 
├─nvme0n1p1  259:1    0  7.9G  0 part /
└─nvme0n1p15 259:2    0   99M  0 part /boot/efi
nvme1n1      259:3    0   10G  0 disk 

• As you can see, our first partition /dev/sd1 (nvme01n1) is mounted with 2 partitions, and out second partition /dev/sdb (nvme1n1) is not mounted yet ...

2- Format the volume to the xfs filesystem using command sudo mkfs -t xfs /dev/nvme1n1.

ubuntu@ip-10-0-0-142:~$ sudo mkfs -t xfs /dev/nvme1n1
meta-data=/dev/nvme1n1           isize=512    agcount=16, agsize=163840 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=2621440, imaxpct=25
         =                       sunit=1      swidth=1 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

3- Use the mkdir command to create a mount point directory for the volume. The mount point is where the volume is located in the file system tree and where you read and write files to after you mount the volume. The following example creates a directory named /data

sudo mkdir /data

4- Use the following command to mount the volume at the directory you created in the previous step.

sudo mount /dev/nvme1n1 /data

Now we are done mounting, you can check it by running lsblk again and you will notice that the MOUNTPOINT value for /dev/nvme1n1 is now updated to /data ...

Automatically mount an attached volume after reboot:

5-Create a backup of your /etc/fstab file that you can use if you accidentally destroy or delete this file while editing it.

sudo cp /etc/fstab /etc/fstab.orig

6- Use the blkid command to find the UUID of the device. Make a note of the UUID of the device that you want to mount after reboot.

sudo blkid

let's assume that our id is aaaa-bbbb-cccc-dddd-cccc

7- Open the /etc/fstab file using any text editor, such as nano or vim.

sudo vim /etc/fstab

8- After vim opens, click I to start inserting, then add the following line

UUID=aaaa-bbbb-cccc-dddd-cccc /data  xfs  defaults,nofail  0  2

Then click escape and write :wq to exit vim.

The final result will be like the image below

9- The final step is to verify that your entry works, run the following commands to unmount the device and then mount all file systems in /etc/fstab. If there are no errors, the /etc/fstab file is OK and your file system will mount automatically after it is rebooted.

sudo umount /data
sudo mount -a

Installing Docker and SEQ :

1- Update the apt package index:

sudo apt-get update

2- Install Docker Engine:

sudo apt install docker.io

3- Run the following command to install seq in using docker

sudo docker run --name seq-server -d --restart unless-stopped -e ACCEPT_EULA=Y --mount type=bind,source=/data,target=/data -p 80:80 -p 5341:5341 datalust/seq

• Command explanation:
  • we pass -d to say that our container will run in the background.
  • --name seq-dev this part gives a name to our SEQ instance of seq-dev
  • then we specify the restart policy as shown --restart unless-stopped, so our container will restart with our machine and continue accepting logs.
  • then we pass port -p 80:80 -p 5341:5341 that will be mapped to container port 80.
  • Then we specify a volume for logs to be saved in, which will be our previously created and mounted EBS volume --mount type=bind,source=/data,target=/data.
  • Then we need to accept the license agreement by passing environment variable -e ACCEPT_EULS=Y.
  • Then give the name of our container datalust/seq

4- You can now check your containers using docker container ls, and you will find SEQ container ready ...

ubuntu@ip-10-0-0-142:~$ sudo docker container ls
CONTAINER ID   IMAGE              COMMAND     CREATED         STATUS                          PORTS     NAMES
a3dac5c88e73   datalust/seq   "/run.sh"   3 minutes ago   Restarting (1) 35 seconds ago             seq-dev

• After that check the Public IPv4 address of your EC2 instance, you can find it in the instance details, click it to open it in a new tab ( make sure you are connecting as http not https ).

• You will now be able to see your SEQ logging panel.

• Let's now make sure that our 10 GB EBS volume is the one attached with SEQ, not the image store ...

• From SEQ Panel, from the top menu, click Settings

• Click on Diagnostics in the left menu, and check that the free disk space is actually your EBS volume we previously created.

• And that's it ! You now have your SEQ Server ready to receive your logs and analyze them. __________ ##A couple of notes to take care of:

1- This way is efficient if you need a SEQ server for a low cost logging server (about 4 USD monthly) for a small application with a low traffic, you can log whatever data you need, visualize them in a great way and you may also request these data in your application an use them for analysis or charting purposes using SEQ API end points.

2- If your application is bigger, then this simple structure should be of course improved, and the first things that you need to increase is the instance being a spot instance, it should be an on-demand or reserved instance whatever suits your plans and budget.

3- Of course the other thing you need to change is the size of your EBS volume, as your logs increases everyday, you will need more space, btw, you can delay this step by setting a retention period for your logs if it's possible for you to deleted of course, or back them up and delete them, anyway, increasing the size will of course lead to increasing costs.

• And at the end, consider this a hands-on where we had some fun with EC2 instances, EBS Volumes, Docker and SEQ.

• Please let me know in the comments below if you would prefer an article taking about using SEQ in a .NET application for logging or analyzing data, I'd be more that happy to write one.

• If you liked the article, please leave a ❤ or pin it, or leave a comment below 😊, your opinion is so much appreciated.

Follow me on:
Linkedin.