Ingestion Methods—SFTP, API, and Object Storage

Learn three different ingestion methods: SFTP, API, and object storage.

To handle various data sources, data engineers should know at least a few ingestion methods to adapt to different scenarios. Let’s look at three common ingestion methods: SFTP, API, and object storage.

SFTP

SSH File Transfer Protocol (SFTP) is a network protocol that provides file access, file transfer, and file management over any reliable data stream. The SSH protocol supports encryption and other security methods to better protect file transfers. Many financial institutions use SFTP to send customer transaction files for security reasons.

Fetch files from a local SFTP server

Follow the steps below to establish a connection and perform basic file operations:

  • Connect to the local SFTP server: Initiate a connection to the local SFTP server by executing the following commands:

sudo passwd sftpuser
sudo service ssh start
sftp sftpuser@127.0.0.1

Enter the new password when prompted. We are now connected to our local SFTP server.

  • Download a sample file: Fetch a sample file from the server using the following command:

get -r readme.txt
  • Disconnect: When finished or download a sample file, exit the SFTP session by typing the ! symbol.

  • Verify the downloaded file: Confirm that the file has been successfully downloaded by using the following command:

cat readme.txt

This command will display the content of the downloaded file.

Run the above commands ...

Get hands-on with 1400+ tech skills courses.