DEV Community

Cong Li
Cong Li

Posted on

GBase 8a MPP Cluster FTP File Server Configuration Guide

#database

Since version V9.5.2.X, GBase 8a MPP Cluster supports loading data from general data servers through various protocols such as ftp, http, hdfs, and sftp. It also supports Kafka clusters as a data source for loading data. Today, we will take Red Hat Enterprise Linux 6.2 as an example to introduce how to configure an FTP file server in GBase databases.

1. FTP Server Configuration

We will use vsftp to set up the FTP server.

1) Check if vsftpd is installed 

# rpm -qa vsftpd
vsftpd-2.2.2-6.el6_0.1.x86_64
Enter fullscreen mode Exit fullscreen mode

2) Install vsftpd 

# rpm -ivh vsftpd-2.2.2-6.el6_0.1.x86_64.rpm
Enter fullscreen mode Exit fullscreen mode

3) Modify the default configuration of the FTP server 

# vim /etc/vsftpd/vsftpd.conf
Enter fullscreen mode Exit fullscreen mode
  • Allow anonymous user login (default is YES): 
  anonymous_enable=YES
Enter fullscreen mode Exit fullscreen mode
  • Allow local users to log in (default is YES): 
  local_enable=YES
Enter fullscreen mode Exit fullscreen mode
  • Open write permissions for local users (default is YES; if only used as a file loading server, you can set this to NO): 
  write_enable=NO
Enter fullscreen mode Exit fullscreen mode
  • Set the file creation mask for local users (default mask is 077, but you can change it to 022): 
  local_umask=022
Enter fullscreen mode Exit fullscreen mode
  • Allow anonymous FTP users to upload files (default is NO): 
  #anon_upload_enable=YES
Enter fullscreen mode Exit fullscreen mode
  • Allow anonymous FTP users to create directories (default is NO): 
  #anon_mkdir_write_enable=YES
Enter fullscreen mode Exit fullscreen mode
  • Enable connection requests on FTP data ports (default is YES): 
  connect_from_port_20=YES
Enter fullscreen mode Exit fullscreen mode
  • PAM authentication configuration file, located in the /etc/pam.d directory: 
  pam_service_name=vsftpd
Enter fullscreen mode Exit fullscreen mode
  • Control access to the FTP server with the userlist file: 
  userlist_enable=YES
Enter fullscreen mode Exit fullscreen mode
  • Deny access to specific files or directories: 
  #deny_file={*.mp3,*.mov,.private}
Enter fullscreen mode Exit fullscreen mode
  • Hide specific files or directories: 
  #hide_file={*.mp3,.hidden,hide*,h?}
Enter fullscreen mode Exit fullscreen mode
  • Set the port range for passive mode (default is 0, meaning any available port): 
  pasv_min_port=20001
  pasv_max_port=21000
Enter fullscreen mode Exit fullscreen mode
  • Set the maximum number of client connections (default is 2000): 
  max_clients=2000
Enter fullscreen mode Exit fullscreen mode
  • Set the maximum number of client connections per IP (default is 50): 
  max_per_ip=50
Enter fullscreen mode Exit fullscreen mode
  • Timeout for passive transmission mode connections (default is 60 seconds): 
  accept_timeout=60
Enter fullscreen mode Exit fullscreen mode
  • Timeout for active transmission mode connections (default is 60 seconds): 
  connect_timeout=60
Enter fullscreen mode Exit fullscreen mode
  • Timeout for data transfer in progress with no activity (default is 300 seconds): 
  data_connection_timeout=300
Enter fullscreen mode Exit fullscreen mode
  • Idle session timeout (default is 300 seconds): 
  idle_session_timeout=300
Enter fullscreen mode Exit fullscreen mode
  • Whether to use system calls sendfile to optimize transmission (default is YES; if using NFS or other network disks, it should be set to NO): 
  use_sendfile=YES
Enter fullscreen mode Exit fullscreen mode
  • Set the home directory for non-anonymous users: 
  #local_root=/var/ftp/pub
Enter fullscreen mode Exit fullscreen mode

For more configuration options, refer to the vsftpd.conf documentation:

# man vsftpd.conf
Enter fullscreen mode Exit fullscreen mode

When the maximum number of concurrent load tasks in the cluster is N, and the maximum number of machines per load task (max_data_processors) is M, the minimum and recommended values for some parameters are as follows:

Parameter Name Default Value Minimum Value Recommended Value
max_clients 2000 M*N M*N*2
max_per_ip 50 N N*2
pasv_min_port 0 M*N M*N*2

4) Configure the user list that allows or denies access to the FTP server (optional) 

# vim /etc/vsftpd/user_list
Enter fullscreen mode Exit fullscreen mode

When the following is configured in /etc/vsftpd/vsftpd.conf, all users in /etc/vsftpd/user_list are denied access to the FTP server:

userlist_enable=YES
userlist_deny=YES (default is YES)
Enter fullscreen mode Exit fullscreen mode

When configured as below, all users in /etc/vsftpd/user_list are allowed access to the FTP server:

userlist_enable=YES
userlist_deny=NO
Enter fullscreen mode Exit fullscreen mode

5) Configure the user list to deny access to the FTP server (optional) 

# vim /etc/vsftpd/ftpusers
Enter fullscreen mode Exit fullscreen mode

6) Disable or modify SELINUX configuration (choose one of the two methods)

Disable SELINUX 

# vim /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disable
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
Enter fullscreen mode Exit fullscreen mode

Restart or execute:

# setenforce 0
Enter fullscreen mode Exit fullscreen mode

Modify SELINUX configuration 

# setsebool ftp_home_dir 1
Enter fullscreen mode Exit fullscreen mode

Note: When accessing the FTP server using a browser, if you encounter "500 OOPS: cannot change directory:/home/...", this might be the issue.

7) Disable or configure the firewall

Disable firewall

Stop the firewall service:

# service iptables stop
Enter fullscreen mode Exit fullscreen mode

Check if the firewall starts automatically at boot:

# chkconfig --list iptables
Enter fullscreen mode Exit fullscreen mode

Prevent the firewall from starting automatically at boot:

# chkconfig iptables off
Enter fullscreen mode Exit fullscreen mode

Configure firewall

Set default rules:

# iptables -A INPUT -j DROP
Enter fullscreen mode Exit fullscreen mode

Open FTP ports:

# iptables -I INPUT -p tcp --dport 21 -j ACCEPT
# iptables -I OUTPUT -p tcp --sport 21 -j ACCEPT
# iptables -I INPUT -p tcp --dport 20001:21000 -j ACCEPT
# iptables -I OUTPUT -p tcp --sport 20001:21000 -j ACCEPT
Enter fullscreen mode Exit fullscreen mode

Save firewall settings:

# iptables-save > /etc/sysconfig/iptables
Enter fullscreen mode Exit fullscreen mode

8) Start the vsftpd service and set it to start on boot 

# service vsftpd start
# chkconfig vsftpd on
Enter fullscreen mode Exit fullscreen mode

9) Copy files to the FTP directory

If local_root=/var/ftp/pub is not set, copy files to /home/xxxx (the user's home directory).

If local_root=/var/ftp/pub is set, copy files to /var/ftp/pub.

If anonymous_enable=YES is set, copy files to /var/ftp or /var/ftp/pub (the home directory for anonymous login).

Top comments (0)

Read next

alexmercedcoder profile image

Understanding Apache Iceberg's metadata.json file

Alex Merced -

dolphindb profile image

DolphinDB to Offer Access to ICE Consolidated Feed and Historical Tick Data

DolphinDB -

danielweinmann profile image

Introducing Flashboard: Instant Admin Panels for PostgreSQL

Daniel Weinmann -

minhduc159 profile image

Understanding Data Normalization and Database Design Principles

Duc Dang -