Ingestion Methods—SFTP, API, and Object Storage
Learn three different ingestion methods: SFTP, API, and object storage.
To handle various data sources, data engineers should know at least a few ingestion methods to adapt to different scenarios. Let’s look at three common ingestion methods: SFTP, API, and object storage.
SFTP
SSH File Transfer Protocol (SFTP) is a network protocol that provides file access, file transfer, and file management over any reliable data stream. The SSH protocol supports encryption and other security methods to better protect file transfers. Many financial institutions use SFTP to send customer transaction files for security reasons.
Fetch files from a local SFTP server
Follow the steps below to establish a connection and perform basic file operations:
Connect to the local SFTP server: Initiate a connection to the local SFTP server by executing the following commands:
sudo passwd sftpusersudo service ssh startsftp sftpuser@127.0.0.1
Enter the new password when prompted. We are now connected to our local SFTP server.
Download a sample file: Fetch a sample file from the server using the following command:
get -r readme.txt
Disconnect: When finished or download a sample file, exit the SFTP session by typing the
!
symbol.Verify the downloaded file: Confirm that the file has been successfully downloaded by using the following command:
cat readme.txt
This command will display the content of the downloaded file.
Run the above commands ...
Get hands-on with 1400+ tech skills courses.