> For the complete documentation index, see [llms.txt](https://docs.nuvolos.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nuvolos.com/how-to-guides/workflows-for-researchers/gpu-computation.md).

# GPU Computation

<mark style="color:$primary;">**Outcome**</mark>\
You configure an application to use a GPU, install the right libraries, and verify the setup works.

<mark style="color:$primary;">**Before you start**</mark>

* Your space has credit-based application sizes enabled. See [Administration › Space management](/administration/space-management.md).
* You have enough Credits in the resource pool mapped to the space.
* You know which framework you will use - PyTorch, TensorFlow, XGBoost, or another.

GPU acceleration on Nuvolos requires two things:

* A GPU-enabled size for the application. By default, applications run on nodes without GPUs. You scale the application to a GPU-enabled size - all of which are [credit-based](/billing/pricing-structure.md#application-resource-accounting).
* Properly configured libraries. The remainder of this guide covers per-framework setup so your Application actually uses the available GPU.

### Library versions

The NVIDIA device drivers are automatically loaded in all GPU-enabled sizes. However, depending on the software you use, additional components (e.g., CUDA toolkit) might need to be installed via conda.

If you launch an app in a GPU-enabled size on Nuvolos, the `nvidia-smi` [tool](https://developer.nvidia.com/nvidia-system-management-interface) will be available from the command line/terminal. You can use this to check the driver version and monitor memory usage of the card.

```
$ nvidia-smi
Thu Jun  1 08:39:06 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.08    Driver Version: 510.73.08    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A10-4Q       On   | 00000002:00:00.0 Off |                    0 |
| N/A   N/A    P0    N/A /  N/A |    333MiB /  4096MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
```

{% hint style="info" %}
Due to the underlying virtualization technology in Nuvolos, the `nvidia-smi` tool is currently unable to list processes using the GPU
{% endhint %}

Note that [nvidia-smi reports](https://stackoverflow.com/questions/53422407/different-cuda-versions-shown-by-nvcc-and-nvidia-smi) the CUDA Driver API version in its output (11.6). However, most high-level machine learning frameworks use the CUDA Runtime API as well, which is provided by the CUDA Runtime library. Most frameworks can automatically install the required version of the runtime, so if you're starting from scratch, this should be straightforward to set up.

Please find examples below on how to get started with GPU computations on Nuvolos, or consult the relevant machine learning library documentation directly. If you need additional support, reach out to our team.

### GPU Monitoring

For interactive GPU monitoring, install the nvitop package:

```sql
pip install nvitop
nvitop  # run from a terminal
```

{% hint style="warning" %}
Due to the underlying virtualization technology in Nuvolos, the `nvitop` tool cannot load the details of the processes using the GPU.
{% endhint %}

### Large Language Models

Two practical guidelines for running LLMs on Nuvolos:

* Assess your VRAM requirements first. The [Hugging Face model size estimator](https://huggingface.co/docs/accelerate/main/en/usage_guides/model_size_estimator) is a useful tool.
* Try loading models with quantised parameters before reaching for larger GPU sizes. The Hugging Face transformers library has built-in support for automatic weight quantisation via [PreTrainedModel.from\_pretrained](https://huggingface.co/docs/transformers/main_classes/model#transformers.PreTrainedModel.from_pretrained).

## Python

Installing the right CUDA version for Python packages can be intricate. Start with a clean Application and install your high-level ML library (PyTorch, TensorFlow) first - these libraries pull in the exact CUDA they need. Install other libraries afterwards.

### PyTorch

For PyTorch, pip works better than conda - pip does not try to overwrite system libraries:

```sql
pip3 install torch torchvision torchaudio
```

{% hint style="info" %}
The standard PyTorch pip command installs PyTorch with the latest major CUDA Runtime version (12). On Nuvolos, all GPUs currently support version 12 except the A10 card. If you need to run on an A10, install PyTorch with the older CUDA Runtime version 11.
{% endhint %}

You do not need a GPU available in your application to install PyTorch with GPU support - install on any size, then scale up to a GPU-enabled size. To verify the installation works on a GPU-enabled size:

```sql
import torch
print(torch.cuda.is_available())  # should print True
```

If it completes without an error, your configuration is correct.

{% hint style="info" %}
Note that pip will install the runtime libraries needed by PyTorch, but will **not** set up a complete developer environment that you could use outside Python (see [official notes](https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html#pip-wheels-linux)). To use tools like nvcc from the command line, please install the [CUDA Toolkit](#cuda-toolkit) via conda instead.
{% endhint %}

### **NVCC (CUDA compiler)**

To compile CUDA executables with [nvcc](https://anaconda.org/nvidia/cuda-nvcc), install the compiler binaries and the C runtime headers:

```sql
conda install -c nvidia cuda-nvcc cuda-cudart-dev
```

Both packages are available in CUDA 11 and 12 versions. For the full toolkit:

{% code overflow="wrap" %}

```sql
conda install -c nvidia cuda-toolkit
```

{% endcode %}

### TensorFlow

To install TensorFlow, we recommend using conda as TensorFlow requires the cudatoolkit package.

```sql
conda install -c conda-forge tensorflow-gpu "cudatoolkit<=CUDA_VERSION"
```

Substitute `CUDA_VERSION` with the version reported by `nvidia-smi`. If you do not need the latest CUDA, start with an older version (such as 11.6) for compatibility with older GPU cards.

{% hint style="info" %}
Install TensorFlow and cudatoolkit from the same conda channel when possible.
{% endhint %}

You don't need a GPU available in your running app to install TensorFlow with GPU support. It's sufficient to scale up to a GPU-enabled size after installation is complete. To test if your installation was successful, execute the following code snippet while on a GPU-enabled size:

```sql
import tensorflow as tf

a = tf.constant([1, 2, 3])
print(a.device)
```

If you see an output similar to

```sql
/job:localhost/replica:0/task:0/device:GPU:0
```

that ends with `GPU:0`, your configuration is correct.

## RStudio

With Machine Learning (CUDA-enabled) RStudio images, you can run GPU computations on GPU-accelerated nodes. These images have the CUDA runtime/toolkit installed as well.

### XGBoost

We recommend using the pre-built experimental binary to get started with XGBoost and R. In a terminal on a GPU node:

```sql
# define version used - update if needed
XGBOOST_VERSION=1.4.1
# download binary
wget https://github.com/dmlc/xgboost/releases/download/v${XGBOOST_VERSION}/xgboost_r_gpu_linux_${XGBOOST_VERSION}.tar.gz
# Install dependencies
R -q -e "install.packages(c('data.table', 'jsonlite'))"
# Install XGBoost
R CMD INSTALL ./xgboost_r_gpu_linux_${XGBOOST_VERSION}.tar.gz
```

Test with the [XGBoost GPU acceleration demo](https://github.com/dmlc/xgboost/tree/master/demo/gpu_acceleration).

### TensorFlow and Keras

You can use TensorFlow with GPU acceleration by following our TensorFlow installation guide and selecting to install version = "gpu" when installing TensorFlow.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.nuvolos.com/how-to-guides/workflows-for-researchers/gpu-computation.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.