🛣️ Week 02 - Running commands on a remote computer

Lab Roadmap (90 min)

Author

Published

06 October 2023

Welcome to your second DS105 lab.

Last week we explored how to navigate your computer using the bash shell. Now, it is time to go beyond your machine and get to the cloud! As you would have seen in the lecture, accessing the cloud opens up various possibilities for us as data scientists. This week, you will access a remote machine without leaving your terminal.

🥅 Learning Objectives

Connect to a remote machine using SSH.
Identify when you are inside your own computer and when you are inside a remote machine.
Navigate the remote machine.
Transfer files between your local machine and the remote machine.

📋 Lab Tasks

Here are the instructions for this lab:

Do you know your CANDIDATE NUMBER? You will need it.

“Your candidate number is a unique five digit number that ensures that your work is marked anonymously. It is different to your student number and will change every year. Candidate numbers can be accessed using LSE for You.”

Source: LSE

Part I: Acquire your credentials (20 min)

Click here to see the 🎯 ACTION POINTS

You will need a username to connect to the cloud. Your class teacher will ask you to inform them of your candidate number
Your class teacher will input your student number in our internal bash script to create your cloud credentials. Your student number will not be shown to anyone else.
After the class teacher runs the script, you can now access the cloud with your newly created username, student_<CANDIDATE_NUMBER> where <CANDIDATE_NUMBER> is your LSE STUDENT NUMBER. For instance, if your ID is 78451, your username will be student_78451.

Part II: Connect to the cloud (20 min)

The first thing we will do is we will connect to the cloud. You will do this using an SSH connection. There are differences between UNIX systems and others in their work with SSH.

Click here if you are using bash or the Windows Powershell

If you were able to follow our suggestion from last week and install the bash shell, use this same shell you’ve been practising with to connect to the cloud. Just follow the instructions below:

🎯 ACTION POINTS

Launch your bash shell.
Type the following command after replacing <username> with your username. Your class teacher will display the host name of the cloud machine on the screen.
```
 ssh <username>@<host_address>
```
Enter your password (which is, at this point, your username).

The first time you log in, you need to give a new password for your account. Type your current password and then make a unique password, as instructed. Note that it may fail to update the password because it is too simple. Try to create a more complex pattern in this case. We recommend at least eight characters with a combination of letters and numbers.

You should now be connected to a machine in the cloud!

If you wish to close your connection to the cloud, write exit and hit enter.

Click here if you are Windows and the above didn’t work

There are various SSH clients that you can use for Windows. Here we explore the one called PuTTY. You will need to follow the steps below to establish an SSH connection.

🎯 ACTION POINTS

Go to this website and download the installation file.
Install PuTTY using the installation file.
Launch PuTTY.
Your class teacher will inform you of the hostname of the cloud machine. Navigate to the Host Name (or IP address) box and enter the host address of our cloud.
Click Open. If you see a security alert, click Accept.
Enter your username and your password (which is, at this point, your username).

The first time you log in, you need to give a new password for your account. Type your current password and then make a unique password, as instructed. Note that it may fail to update the password because it is too simple. Try to create a more complex pattern in this case. We recommend at least eight characters with a combination of letters and numbers.

Once the password is updated, the connection will be closed, and you need to reopen a connection using PuTTY. If your connection has been successful, you will see a message starting with “Welcome to Ubuntu”. Ask your teacher to check if it is done correctly.

Voilà! Now you are connected to the cloud!

If you wish to close your connection to the cloud, write exit and hit enter.

Part III: Explore the cloud (20 min)

You are now inside a computer but on a cloud server, and you can see what is in the machine! If you follow the instructions below, you will see that the cloud machine is similar to your local one. It means we can run the same commands as we ran locally last week.

Click here to see the 🎯 ACTION POINTS

🎯 ACTION POINTS

Use the pwd and $HOME commands to understand whether you are in your home directory.
Does your cloud machine have any files stored? Maybe any hidden files?
If you forgot how to answer those questions feel free to use the materials from last week.
Can you access the root directory of the machine? If so, what are the folders inside the root directory?
Come back to your home directory to continue the exercises.
If you experience any difficulties, ask your class teacher for help.

Now, you can see that the cloud machine is very similar to your local one. It means that we can also create files and directories there!

Create files

Make sure you are in your home directory using cd and pwd.
Create a folder called <username>_files. Make sure to replace username with your actual username.
Go to this folder and create a file called secret_name.txt.
Save the name of your favourite place to eat in the file. It must be kept very secret. We don’t need this place to be too crowded, do we?
Create another file called secret_address.txt.
Save the address of the same place there.

Perfect! Now we have two (very secret) files in our system. In the next step, we will explore how to send these files to your machine.

Part IV: A bridge between the worlds (20 min)

In the first steps of this lab, we have established a secure connection between our computer and the cloud machine. We can use a secure connection for more than just sending commands; we can also send and receive files. Let’s see how it works:

Generally, the scp commands work in the following way:

 scp location_1/file location_2/file

It means we are copying a file from location_1 to location_2. Let’s practice it together.

Click here to see the 🎯 ACTION POINTS

🎯 ACTION POINTS

Exit the cloud machine using the following code:

    exit

To copy the secret_name.txt file from your cloud machine, use the following command. Make sure you have replaced all the <username>s with your username.
```
 scp <username>@<server-url>:/home/<username>/<username>_files/secret_name.txt .
```
Important
We would encourage you to pay attention to several things here:
1. There is a period symbol at the end of the code. This period stands for your current location on your local machine. You remember that the scp command takes two locations - the one we copy the file from and the one we copy the file to. A period here basically says, “Copy it right here”.
2. You can see that we specify the actual path to the file on your virtual machine after the hostname, followed by a colon. First, we set the username, then the hostname and then the path inside of the machine.
Make sure you have copied to the chosen directory.
But what if we wanted to copy all the files from the folder or the folder itself? For that, we can avoid specifying the whole path to a concrete file and replace the name of the file with an asterisk:
```
 scp <username>@<server-url>/* .
```
This way, we copy all the files from the <username>_files folder.

Should you wish to copy the folder itself, use the path to the folder. You will also add the -r (for recursive) option in the code, as shown below.
```
 scp -r <username>@<server-url>:/home/<username>/<username>_files .
```
Make sure both commands work for you.

Sending it back

We have just learned how to copy files from the cloud. The last task for us today is to send files to the cloud machine.

Choose any directory you want on your local machine and cd there.
Create a file called secret_dish.txt and save the name of your favourite dish there.
🤔 Stop for a second. Do you think you can guess how to copy this file to your cloud machine using your already-acquired knowledge? Try it out before looking at the solution.

Solution

Use the code below to copy your file to the cloud machine. Make sure to replace the <username>s with your actual username.

```bash
 scp secret_dish.txt <username>@<server-url>:/home/<username>/<username>_files
```

8 Log in to your virtual machine and check if the file is there.

🏡 Take-home Tasks

It can wait

You are now able to connect to the cloud machine, navigate it, and exchange files. Now it’s time to get to the most exciting part! Running code on the cloud! Suppose you need to process millions of rows of data — it could take ages to do that on your computer. Instead of taking your computer’s resources, you can run the scripts on the cloud.

Let’s do it, but before that, we will learn how to do an exciting trick.

Click here to see the 🎯 ACTION POINTS

🎯 ACTION POINTS

Launch a shell on your local machine.
Navigate to a directory of your choice or create one.
Create a new file called waiting.py and include the following code inside:
```
import time
time.sleep(10)

print('The waiting is complete.')
```
This file launches a script that waits for 10 seconds and then prints ‘The waiting is complete.’. You might think that it doesn’t make sense. However, let us show you something…
Run this Python script in the following way:
```
python waiting.py
```
What did it do? Hopefully, exactly what we expected. It waited for 10 seconds and then printed one sentence. You might have noticed that you could not execute commands while it was running. But what if we could?
Try running the following code:
```
python waiting.py &
```
Do you see the difference? Can you now run whoami or ls while we wait for the code to run?

Well done! Now you have learned how to create and run Python scripts in your terminal and do things in parallel. Shall we try it in the cloud?

More principled script running

Let’s now explore the same operations in the cloud.

Click here to see the 🎯 ACTION POINTS

🎯 ACTION POINTS

Connect to the virtual machine.
Create a folder called test_code.
Create a file called test.py or test.R depending on what language you want to use (for Python and R, respectively).
In the file
- create a variable called age
- assign it with your age
- make the machine wait for 5 seconds (time.sleep(5) in Python or Sys.sleep(5) in R)
- use print() function to print your age
Use the following code to execute your script on the cloud:
```
 python3 test.py
```
Does it print your age?
Go ahead and experiment with using the & operator. It comes in handy if you want your cloud machine to continue running without you constantly monitoring it.

You can check out the tutorial on how to run Python scripts or R scripts to help you.