You can also use this guide for any Linux Ubuntu machine.
In this post I document the most common practices I have used in order to remember myself and be able to share this knowledge with others.
This post is written as an actual (executable) Linux Bash shell script in Jupyter Notebook to make executing some of the commands easier. This assumes I run this notebook on the actual AWS instance.
Most of the time I run t2.micro, it is very underpowered and slow (think Raspberry Pi), but it is OK for the following tasks:
When training machine learning deep neural networks, I stop t2.micro and change the type to the following GPU instance:
This EC2 instance is exivalent to very powerul scientific work station.
If I feel frisky (or reckless with money) I choose even more powerful instance:
It is a true monster machine.
I have an static (AWS elastic IP) which never changes which is assigned to my primary instance.
I have a domain name address that forwards to that IP so I do not have to remember the numbers.
I also created a script on my laptop in the ~/.bash_profile to ssh connect with this simple command.
Originally, I created "Amazon Elastic Block Store" (a.k.a. an SSD-drive) to be 8 GiB, which run out immediately.
I had to change it to 30 GiB which should be in the free tier level. The 30 gigs does not sound like a lot, but for running machine learning models it should be enough.
To check the disk usage I run the following, currently I am at about 50%:
ps aux | grep python # EXAMPLE # ubuntu 1690 0.0 0.1 378096 68036 pts/0 Sl 16:00 0:12 /home/ubuntu/anaconda3/bin/python /home/ubuntu/anaconda3/bin/jupyte # ubuntu 2522 0.0 0.0 556184 44604 ? Ssl 20:43 0:00 /home/ubuntu/anaconda3/bin/python -m bash_kernel -f /run/user/1000/ # ubuntu 2531 0.0 0.0 21284 4996 pts/1 Ss+ 20:43 0:00 /bin/bash --rcfile /home/ubuntu/anaconda3/lib/python3.5/site-packag # ubuntu 3123 0.0 0.0 12944 1084 pts/3 S+ 22:50 0:00 grep --color=auto python # kill 1690
I try to install most of my Python packages with in Anaconda (conda), if I cannot find one I use pip (Python package index, Pip Installs Packages, go figure).
Sometimes I do experiment with different enviroments when there is a cataclysmic change e.g. Python 2.7 -> 3.5 -> 3.6 that breaks everything.
I am making sure I am in the conda environment I want to use:
conda info --envs
# print out installed packages and their version numbers conda list | grep -E "keras|teano|tensorflow" # expected tensorflow-gpu 1.0.1 # Example of OLD: tensorflow 0.10.0rc0 np111py35_0 (see uninstall Tensorflow below)
I am writing 95% of my code in Python, Bash, Markdown or HTML in Jupyter Notebook. For last 15 years I used Java, but I program Java on my local laptop in Android Studio / IntelliJ IDEA.
Starting Jupyter Notebook (disconnected from terminal)
pip install keras
Install TensorFlow GPU for Python 3.5
Please note that TensorFlow changes rapidly, so when you started in ancient times (say 6 months ago), you probably have some old garbage versions installed that do not work.
pip show tensorflow # EAMPLE OUTPUT with OLD VERSION: # Name: tensorflow # Version: 0.10.0rc0 # Summary: TensorFlow helps the tensors flow
pip install --upgrade --ignore-installed tensorflow-gpu
$ conda install tensorflow
The following NEW packages will be INSTALLED:
# once again make sure you have everything you want conda list | grep -E "keras|teano|tensorflow" # expected tensorflow-gpu 1.0.1 or newer
uname -m && cat /etc/*release
lspci | grep -i nvidia # expected Tesla K80 on p2.xlarge
gcc --version # expected 5.4 on Ubuntu
# Version of your kernel # expected 4.4.x uname -r
# Install Linux headers sudo apt-get install linux-headers-$(uname -r)
cd ~/Downloads/ pwd
# do not copy the URL below, find the newest wget https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64-deb
# do not copy, match your download sudo dpkg -i cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64-deb
# don't be shy to run this every day sudo apt-get update
sudo apt-get install cuda
# verify version cat /proc/driver/nvidia/version # expected: NVIDIA UNIX x86_64 Kernel Module 375.26 or newer
You can have a yaml script to create a whole enviroment, but it works well only in brand new setup.
conda install opencv
pip install eventlet
pip install python-socketio
conda update --all
Since this page is created with Bash Jupyter Notebook, when I change it I have to commit them.
I decided to add this because these "little" things are usually omited elsewhere, but they are essential in my workflow.
Remember to CLEAR OUTPUTS before commiting!
git add .
git commit -m "changes to AWS post"