Tag Archives: ubuntu

OpenCL Cookbook: Running remote multi-gpu OpenCL computations over SSH on Ubuntu or Debian using AMD Catalyst drivers

Continuing on in the OpenCL cookbook series here I present a post not about code but about environmental setup further diversifying the scope of the cookbook. Here is a no-nonsense guide to running OpenCL programs via ssh remote login into linux. This is intended as a follow up to my previous guide to compiling OpenCL on linux so have a look at that if you haven’t done so already but this account should be fairly self contained nevertheless.

Important: Note that this is not a guide to setting OpenCL on linux in a headless fashion. It requires GUI linux, it requires X and it requires that X be running on screen. Although there are reports of people getting headless setups to work I haven’t yet had time to try these things out so I cannot comment on them.

Install Linux

I went for Ubuntu and Debian for their easy and well known package management. I installed them in adjacent partitions so that I could test my setup on both and compare them easily. Eventually I settled on Debian due to strongly disliking unity and the presence of several known issues with raid on boot in Ubuntu. Regarding desktop environments/window managers I installed an assortment of them because I wanted fully featured ones for development and lightweight ones for benchmarking. I also wanted to test OpenCL in a variety of environments. I installed GNome, KDE, Blackbox, Fluxbox, XFCE and E17 on each distro. It’s good to have options. 🙂 Note that you should NOT install any drivers automatically through the ubuntu/debian package management tools!

Install AMD Catalyst 12.11 beta drivers and AMD APP SDK 2.7

Installing the above is as simple as downloading the linux tarballs, extracting them and running the executable script within each as root. I don’t bother building distro specific packages. Just install the damn things. They work. If you want a bit more elaboration on installing and setting these up look at my previous guide. The reason I’m suggesting the 12.11 beta drivers is because the 12.10 stable driver doesn’t work with X on ubuntu or debian whereas 12.11 beta does. Plus I haven’t noticed any instability with 12.11 beta at all. It seems to work just fine.

Use amdconfig to generate a new Xorg configuration using all adapters

Amdconfig (also aliased as aticonfig) is a powerful command line tool provided by the AMD catalyst driver that you can use for an assortment of tasks such as listing your adapters, querying clocks and temperatures of your gpus, setting overclocks and most importantly for our purposes generating a variety of Xorg configurations with different settings. Here we need to generate an X configuration that will enable all our adapters in Xorg.

sudo amdconfig --initial -f --adapter=all

This will back up your existing xorg.conf in /etc/X11/xorg.conf and create a new one in its place containing device and screen sections for all your gpus. Though if you’ve just installed ubuntu or debian you won’t have an xorg config to begin with anyway.

Reboot to start using the new AMD fglrx driver and your new X configuration

A reboot is required to unload the old open source driver and load fresh the new proprietary amd driver fglrx which cannot be done cleanly by simply using modprobe -r to remove old drivers and modprobe to load new drivers into the kernel and then restarting X. A reboot is recommended and essential. Once you’ve rebooted you’ll also start to use your new X configuration with all adapters enabled. When you see a login manager screen (gdm/xdm/kdm/slim) log into it.

Test your new local OpenCL environment

Start up a terminal. First check that amdconfig sees all your adapters. If you have multiple gpus installed you should see all of them with one of them (typically the first) selected as default.

dhruba@debian:~$ amdconfig --list-adapters
* 0. 05:00.0 AMD Radeon HD 7900 Series  
  1. 06:00.0 AMD Radeon HD 7900 Series  
  2. 09:00.0 AMD Radeon HD 7900 Series  
  3. 0a:00.0 AMD Radeon HD 7900 Series  
  4. 85:00.0 AMD Radeon HD 7900 Series  
  5. 86:00.0 AMD Radeon HD 7900 Series  

* - Default adapter

Next, check that all your devices are being picked up for computation by the OpenCL runtime.As you can see below on my workstation it shows 7 devices – 6 of them being gpus and 1 being the cpu. I’m using 3 7990s each of which are a dual 7970.

dhruba@debian:~$ clinfo | grep 'Number of devices'
Number of devices:				 7

If you see all your devices in the count above your environment should be ready to compile and run OpenCL programs. The next step is get it ready for remote ssh multi-gpu OpenCL computations.

Enable multiple GPUs in the OpenCL runtime

At this point you may find that although amdconfig lists all your adapters the OpenCL runtime only sees one gpu or sees fewer gpus than you have installed. You may also find that some window managers/desktop environments see all devices and others see only one. To enable multiple gpus you may need to do two things.

First you need to disable crossfire. You can do this either from within the catalyst control panel which you can start by running sudo amdcccle or you can use the powerful amdconfig tool to do it by running sudo amdconfig --crossfire=off. See my post about amdconfig for more detail on how to use this amazing tool.

Secondly you may need to pass a hint to the OpenCL runtime to tell it to use the current display for computation as follows.

export COMPUTE=:0

As an alternative to this you can also do the following.

unset DISPLAY

However I’d recommend the first as it is an override flag that does not require any change to the existing DISPLAY variable. Implementing the above two tweaks should enable multi-gpu support in your OpenCL programs. Check either by using clinfo | grep 'Number of devices' or by using a C program as in my previous article.

Install SSH and set up your server for remote computations

Install SSH on ubuntu/debian by running sudo apt-get install openssh-server. This should install the ssh server and start it automatically. Login to your OpenCL server from a remote client and run clinfo again to check number of devices. If this is showing all your gpus then you’re done. However chances are that it won’t yet show all your devices. It will show only your cpu and no gpus at all. The reason for this is that the remote client has not yet been granted access to the X runtime. You can grant access as follows.

dhruba@debian:~$ xhost +
access control disabled, clients can connect from any host

This disables access control altogether. However if you are concerned about security you can be more restrictive and enable it only for specific usernames, hostnames, ip addresses, networks or protocols. See man xhost for details. Basically you append the hostname, username or ip to the plus sign. Once you’ve done the above run clinfo again from your ssh session and now you should see all your gpus and your system is ready to remote OpenCL computations.

Automating setup of remote computation environment

In a true datacentre environment you don’t really want to physically login to X and set environment variables particularly because your server may get rebooted. What you really want is for it to be ready to run OpenCL via remote sessions on a fresh boot without any manual intervention. If you wish to do this there is a guide available on the amd forum for ubuntu specifically which I suppose could be adapted to Debian. I’ve tried this on ubuntu and it does work though it’s a bit hackish. I’ll leave this guide at your discretion.

AMD OpenCL Limitations

Note that this is not a headless setup. In this guide we installed a GUI linux, ran X, logged into X and kept X displayed on screen. All of these are essential. If you either do not run X or run X but switch away from it to one of the other virtual terminals you will not be able to run OpenCL either from the virtual terminals or remotely. If you try to run OpenCL without X running and showing on screen the execution will just hang indefinitely until you start X and display it on screen at which point that hanging computation will continue and complete.

This seems to be an inherent limitation of the AMD driver. AMD say they’re working on separating X from the OpenCL runtime but who knows when this will actually be done. There are quite a few reports of people succeeding at running truly headless OpenCL setups – without any peripherals attached – but I have not had time to research these methods yet so I cannot comment on them.

OpenCL Cookbook: Compiling OpenCL with Ubuntu 12.10, Unity, AMD 12.11 beta drivers & AMD APP SDK 2.7

Continuing on in the OpenCL cookbook series here I present a post not about code but about environmental setup further diversifying the scope of the cookbook. It can be a real challenge for the uninitiated to install all the above and compile an opencl c or c++ program on linux. Here’s a short guide. First download and install ubuntu (duh!).

Install ubuntu build tools and linux kernel extras

Then install the following packages which are a prerequisite to the amd installers and the subsequent c/c++ compilation.

sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install linux-source
sudo apt-get install linux-headers-generic

Then download AMD 12.11 beta drivers (amd-driver-installer-catalyst-12.11-beta-x86.x86_64.zip) and AMD APP SDK 2.7 (AMD-APP-SDK-v2.7-lnx64.tgz). Obviously download either 32bit or 64bit based on what your system supports.

AMD 12.11 beta drivers installation

Once you’ve done that install the AMD 12.11 beta drivers as root first. Installation is as simple as extracting the tarball, marking the script inside as executable and running the script as root. Reboot. After the reboot unity should start using the new AMD 12.11 beta driver and you’ll know it’s the beta because you’ll see a watermark at the bottom left of the screen saying ‘AMD Testing use only’. Note that the reason we’re using the beta here is because unity does not work with earlier versions of the driver. You get a problem where you see the desktop background and a mouse pointer but there’s no toolbar or status bar. But the 12.11 beta driver works which is great.

AMD APP SDK 2.7 installation

Then install the AMD APP SDK 2.7 also as root. Again installation is very simple and exactly the same as for the beta driver above. The AMD beta drivers install a video driver and the OpenCL runtime. The AMD APP SDK install the SDK and also OpenCL and OpenGL runtimes. However if you’ve already installed the video driver first you’ll already have the OpenCL runtime on your system in /usr/lib/libamdocl64.so so the APP SDK won’t install another copy in its location of /opt/AMDAPP/lib/x86_64/libOpenCL.so. You’ll see some messages during installation that it’s skipping the opencl runtime and that’s absolutely fine for now.

Test your OpenCL environment

Now you should test your OpenCL environment by compiling and running an example c opencl program. Get my C file to list all devices on your system as an example calling it devices.c and compile as follows.

gcc -L/usr/lib -I/opt/AMDAPP/include devices.c -lamdocl64 -o devices.o # for c
g++ -L/usr/lib -I/opt/AMDAPP/include devices.c -lamdocl64 -o devices.o # for c++

Once compiled run the output file (devices.o) and if it works then you should output similar to that below.

1. Device: Tahiti
 1.1 Hardware version: OpenCL 1.2 AMD-APP (923.1)
 1.2 Software version: CAL 1.4.1741 (VM)
 1.3 OpenCL C version: OpenCL C 1.2 
 1.4 Parallel compute units: 32
2. Device: Intel(R) Xeon(R) CPU E5-2687W 0 @ 3.10GHz
 2.1 Hardware version: OpenCL 1.2 AMD-APP (923.1)
 2.2 Software version: 2.0 (sse2,avx)
 2.3 OpenCL C version: OpenCL C 1.2 
 2.4 Parallel compute units: 32

Enabling multiple gpus for OpenCL

You may find that you are only seeing one gpu in your opencl programs. There are two things you need to do to enable multiple gpus in the OpenCL runtime. The first is to disable all crossfire. You can do this either in the amd catalyst control centre > performance which you start by running amdcccle or you can do it using the awesome amdconfig tool by running amdconfig --crossfire=off. See my post on amdconfig to find out more about this incredibly powerful tool.

The second thing you may or may not need to do is to enable COMPUTE mode as follows.

export COMPUTE=:0

Once you’ve done the above you should see program output from the program above similar to below.

dhruba@debian:~$ ./source/devices.o 
1. Device: Tahiti
 1.1 Hardware version: OpenCL 1.2 AMD-APP (1084.2)
 1.2 Software version: 1084.2 (VM)
 1.3 OpenCL C version: OpenCL C 1.2 
 1.4 Parallel compute units: 32
2. Device: Tahiti
 2.1 Hardware version: OpenCL 1.2 AMD-APP (1084.2)
 2.2 Software version: 1084.2 (VM)
 2.3 OpenCL C version: OpenCL C 1.2 
 2.4 Parallel compute units: 32
3. Device: Tahiti
 3.1 Hardware version: OpenCL 1.2 AMD-APP (1084.2)
 3.2 Software version: 1084.2 (VM)
 3.3 OpenCL C version: OpenCL C 1.2 
 3.4 Parallel compute units: 32
4. Device: Tahiti
 4.1 Hardware version: OpenCL 1.2 AMD-APP (1084.2)
 4.2 Software version: 1084.2 (VM)
 4.3 OpenCL C version: OpenCL C 1.2 
 4.4 Parallel compute units: 32
5. Device: Tahiti
 5.1 Hardware version: OpenCL 1.2 AMD-APP (1084.2)
 5.2 Software version: 1084.2 (VM)
 5.3 OpenCL C version: OpenCL C 1.2 
 5.4 Parallel compute units: 32
6. Device: Tahiti
 6.1 Hardware version: OpenCL 1.2 AMD-APP (1084.2)
 6.2 Software version: 1084.2 (VM)
 6.3 OpenCL C version: OpenCL C 1.2 
 6.4 Parallel compute units: 32
7. Device: Intel(R) Xeon(R) CPU E5-2687W 0 @ 3.10GHz
 7.1 Hardware version: OpenCL 1.2 AMD-APP (1084.2)
 7.2 Software version: 1084.2 (sse2,avx)
 7.3 OpenCL C version: OpenCL C 1.2 
 7.4 Parallel compute units: 32

Standardising the OpenCL runtime library path

Now – it may be that you wish for the OpenCL runtime library to be installed in the standard AMD APP SDK location of /opt/AMDAPP/lib/x86_64/libOpenCL.so as opposed to the non-standard location of /usr/lib/libamdocl64.so which is where the beta driver installation puts it. The proper way to do this would probably be to install the AMD APP SDK first and then the video driver or simply skip the video driver installation (I haven’t tried either of these options so they may need verification).

However, I used a little trick to make this easier since I’d already installed the video driver followed by the APP SDK. I renamed /usr/lib/libamdocl64.so to /usr/lib/libamdocl64.so.x and reinstalled the APP SDK. This time it detected that the runtime wasn’t present and installed another runtime in /opt/AMDAPP/lib/x86_64/libOpenCL.so – the standard SDK runtime path. With the new APP SDK OpenCL runtime in place I was able to compile the same program using the new runtime as below depending on whether you want the c or c++ compiler.

gcc -L/opt/AMDAPP/lib/x86_64/ -I/opt/AMDAPP/include devices.c -lOpenCL -o devices.o # for c
g++ -L/opt/AMDAPP/lib/x86_64/ -I/opt/AMDAPP/include devices.c -lOpenCL -o devices.o # for c++

Summary

And there you have it – an opencl compiler working on ubuntu 12.10 using the AMD 12.11 beta drivers and the AMD APP 2.7 SDK. Sometimes you just need someone else to have done it first and written a guide and I hope this serves to help someone out there.

VirtualBox: Sharing folders between host and guest

Recently, having finally refused to surrender to windows, I installed Ubuntu virtualised as a guest on Windows as a host using Oracle’s recently released VirtualBox. Here’s a tip on how to share folders between guest and host in the official way.

On the guest VM virtualbox menu open ‘Shared folders’.

Open Shared Folders

Open Shared Folders

On the top right of the dialogue box that comes up click the ‘+’ icon. Fill in the dialogue by adding a name and location.

Add share

Add share

After that you should have a shares dialogue as below.

Shares Dialogue

Shares Dialogue

Next, as root, mount manually.

mkdir /mnt/share
mount -t vboxsf virtual-box-ubuntu-share /mnt/share/

And, finally, add the following entry into /etc/fstab for future boots.

virtual-box-ubuntu-share /mnt/share vboxsf defaults 0 0

Done. Ubuntu on VirtualBox running as guest on a Windows host is by far the best and most compelling complement to your development environment if you are forced into using Windows as a host. VirtualBox even supports seamless mode which means that you can have Linux and Windows windows intermingled on the windows desktop. Superb. And best of all – both VirtualBox and Ubuntu being completely free.

Update: Great news. VirtualBox 4.0 is out. Here’s what’s new.

Installing Java, Scala and Vim support on Linux

Here’s a quick guide on how to install Scala on linux (in my case Ubuntu 9.04). A prerequisite obviously is to have the sun jdk installed and properly integrated into your linux environment. For completeness I detail how I normally tend to set up the jdk on my linux environment before moving onto the installation of Scala. Note that here /opt/ is used as the destination directory for installation as that is what I prefer but this can be any directory that you have write permissions on. You’ll also note that I don’t use any automated linux installation tool like apt-get and that is deliberate as the following methods of installation allow you not only complete control of installation of the packages but greater flexibility when upgrading. Finally we’ll look at adding Scala support to my favourite command line editor – Vim.

Install Sun JDK

Download the latest Sun JDK (JDK 6 Update 14 at time of writing).

Make executable and extract.

$ chmod u+x jdk-6u14-linux-i586.bin
$ ./jdk-6u14-linux-i586.bin

Relocate.

$ mv jdk1.6.0_14/ /opt/

Symlink.

$ cd /opt/
$ ln -s jdk1.6.0_14/ java

In /etc/profile set JAVA_HOME variable and add JDK bin directory to system path.

export JAVA_HOME="/opt/java"
export PATH="${JAVA_HOME}/bin:${PATH}"

Import your newly modified profile

$ source /etc/profile

Test Java.

$ java -version
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) Server VM (build 14.0-b16, mixed mode)
$ javac -version
javac 1.6.0_14

Install Scala

Download the latest Scala (2.7.5 at time of writing).

Extract.

$ tar -xvzf scala-2.7.5.final.tgz

Move.

$ mv scala-2.7.5.final/ /opt/

Symlink.

$ cd /opt/
$ ln -s scala-2.7.5.final/ scala

In /etc/profile set SCALA_HOME environment variable and add the Scala bin directory to your system path.

export SCALA_HOME="/opt/scala"
export PATH="${SCALA_HOME}/bin:${JAVA_HOME}/bin:${PATH}"

Import your newly modified profile

$ source /etc/profile

Test Scala.

$ scala
Welcome to Scala version 2.7.5.final (Java HotSpot(TM) Server VM, Java 1.6.0_14).
Type in expressions to have them evaluated.
Type :help for more information.

scala> 1+1
res0: Int = 2

scala> println("Hello World!")
Hello World!

Add Scala support into Vim

If you’re like me and you can’t live without Vim or you just need Vim to get started with Scala prior to moving onto your favourite IDE then here’s how you can add Scala support into Vim.

Create required vim directories.

$ mkdir -pv ~/.vim/ftdetect
$ mkdir -pv ~/.vim/indent
$ mkdir -pv ~/.vim/syntax

Download Scala support into vim directories.

$ wget --no-check-certificate https://lampsvn.epfl.ch/trac/scala/export/18260/scala-tool-support/trunk/src/vim/ftdetect/scala.vim -O ~/.vim/ftdetect/scala.vim
$ wget --no-check-certificate https://lampsvn.epfl.ch/trac/scala/export/18260/scala-tool-support/trunk/src/vim/indent/scala.vim -O ~/.vim/indent/scala.vim
$ wget --no-check-certificate https://lampsvn.epfl.ch/trac/scala/export/18260/scala-tool-support/trunk/src/vim/syntax/scala.vim -O ~/.vim/syntax/scala.vim

Create a basic ~/.vimrc configuration if you haven’t got one.

set nocompatible
set nu  
syntax on
filetype indent on
set autoindent
set ic
set hls 
set lbr 
colorscheme delek

Try editing a Scala file. It should appear in colour.

1 for {i <- 1 to 10
2 j <- 1 to 10}
3 println(i*j)

Rejoice. Get coffee. Develop. Lose sleep. The usual lifecycle.

Acknowledgements (Scala Vim support, Vim colorised output).

Fixing Java UnknownHostException on Ubuntu

On Ubuntu, on fresh installs, I frequently tend to get UnknownHostException(s) while running Java such as below.

INFO: I/O exception (java.net.UnknownHostException) caught when processing request: repository.springsource.com.s3.amazonaws.com

This normally happens for me when running 32-bit Java on 64-bit Ubuntu. The solution is simply to install lib32nss-mdns as follows and then retry.

sudo apt-get install lib32nss-mdns

Sources (ubuntu bug, ehow).

K(u)buntu 8.10 released

I’m a little late in reporting this but on 30 Oct (3 days ago) Ubuntu and Kubuntu 8.10 were released. Currently I use Debian for server and Kubuntu for desktop. Previously I was a big fan of Gentoo and prior to that I tried every distribution under the sun while at university. Kubuntu 8.10, it seems, includes KDE4 which I really hope has stabilised sufficiently to become the default distributed version and reached the 4.1 version. The last remix cd was, from what I’ve heard, unusable.

While previously Ubuntu was a little new on the scene and was being treated with skepticism as yet another operating system based on Debian it has now matured into a stable, trusted and widely adopted linux distribution and for me, as with most people, it simply works and stays out of the way allowing me to get my work done – the work being in my case not only being my primary operating system but also being my preferred environment for Java development. A dual boot between Vista and Linux is of course always necessary – Windows being for games, IE and other Windows specific requirements.