, you will need to set this tracking server in your code with:
`import mlflow`\
`mlflow.set_tracking_uri("http://localhost:8080")`
{% endhint %}
The following tutorial, adapted from the MLFlow documentation, shows how to track model training and register the trained model with MLFlow on Nuvolos:
# The data set used in this example is from http://archive.ics.uci.edu/ml/datasets/Wine+Quality
# P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis.
# Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547-553, 2009.
import os
import warnings
import sys
import pandas as pd
import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.linear_model import ElasticNet
from urllib.parse import urlparse
import mlflow
import mlflow.sklearn
import logging
logging.basicConfig(level=logging.WARN)
logger = logging.getLogger(__name__)
def eval_metrics(actual, pred):
rmse = np.sqrt(mean_squared_error(actual, pred))
mae = mean_absolute_error(actual, pred)
r2 = r2_score(actual, pred)
return rmse, mae, r2
if __name__ == "__main__":
mlflow.set_tracking_uri("http://localhost:8080")
mlflow.set_experiment("Wine Quality")
warnings.filterwarnings("ignore")
np.random.seed(40)
# Read the wine-quality csv file from the URL
csv_url = (
"http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv"
)
try:
data = pd.read_csv(csv_url, sep=";")
except Exception as e:
logger.exception(
"Unable to download training & test CSV, check your internet connection. Error: %s", e
)
# Split the data into training and test sets. (0.75, 0.25) split.
train, test = train_test_split(data)
# The predicted column is "quality" which is a scalar from [3, 9]
train_x = train.drop(["quality"], axis=1)
test_x = test.drop(["quality"], axis=1)
train_y = train[["quality"]]
test_y = test[["quality"]]
alpha = float(sys.argv[1]) if len(sys.argv) > 1 else 0.5
l1_ratio = float(sys.argv[2]) if len(sys.argv) > 2 else 0.5
with mlflow.start_run():
lr = ElasticNet(alpha=alpha, l1_ratio=l1_ratio, random_state=42)
lr.fit(train_x, train_y)
predicted_qualities = lr.predict(test_x)
(rmse, mae, r2) = eval_metrics(test_y, predicted_qualities)
print("Elasticnet model (alpha=%f, l1_ratio=%f):" % (alpha, l1_ratio))
print(" RMSE: %s" % rmse)
print(" MAE: %s" % mae)
print(" R2: %s" % r2)
mlflow.log_param("alpha", alpha)
mlflow.log_param("l1_ratio", l1_ratio)
mlflow.log_metric("rmse", rmse)
mlflow.log_metric("r2", r2)
mlflow.log_metric("mae", mae)
tracking_url_type_store = urlparse(mlflow.get_tracking_uri()).scheme
# Model registry does not work with file store
if tracking_url_type_store != "file":
# Register the model
# There are other ways to use the Model Registry, which depends on the use case,
# please refer to the doc for more information:
# https://mlflow.org/docs/latest/model-registry.html#api-workflow
mlflow.sklearn.log_model(lr, "model", registered_model_name="ElasticnetWineModel")
else:
mlflow.sklearn.log_model(lr, "model")
# Open WebUI
[Open WebUI](https://docs.openwebui.com/) is an extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It supports various LLM runners like Ollama and OpenAI-compatible APIs, with built-in inference engine for RAG, making it a powerful AI deployment solution.
### Open WebUI and MCP
Open WebUI is suitable for supporting [Model Context Protocol (MCP)](https://modelcontextprotocol.io/docs/getting-started/intro) which is Anthropic's popular framework for connecting AI agents to external tools. Thus, the Open WebUI application offers a user-friendly interface just like that of the web-accessable ChatGPT, Gemini...etc., but with two major advantages. First, the user can access any kind of AI model (OpenAI, Google, Anthropic...) within the same interface. Second, it can be directly connected to external sources that the user wants the agent to see.
### Setting Up Agents in Open WebUI
To add model providers, open Open WebUI, select your user menu in the lower-left corner, and open Admin Panel. Then go to Settings > Connections. Add the API endpoint and API key for the provider you want to use. Open WebUI supports OpenAI-compatible API connections, so OpenAI models can be added directly, while other providers such as Anthropic or Google may require OpenAI-compatible endpoints and explicit Model IDs. For OpenAI models, specifying Model IDs is not required
The specific models can be selected from the LLM providers' sites.\
**Anthropic:**\
Google Gemini:\
### Setting Up MCP Servers in Open WebUI
Adding MCP servers to Open WebUI has two steps:
1. Create the JSON configuration file that defines the MCP servers, including their source code paths, required packages, and optional port values. When Open WebUI starts, a startup script reads this JSON file and configures the servers from it. If you switch to a different JSON file, update the script accordingly.
2. In Open WebUI, open Settings > Tools and add each MCP server as a tool. For each one:
* set the URL to the server address and port, for example `http://0.0.0.0:8002`
* enter a clear Name
* set Visibility to Public
### Using Open WebUI
#### Turning on Tools
Tools can be enabled or disabled separately for each chat. To choose which MCP servers are available in a conversation, select the + button in the lower-left corner of the chat window and choose the tools you want to use. This is especially useful when multiple tools provide overlapping functionality. For best results, open Chat Controls near the profile icon in the upper-right corner of the chat and set Function *Calling* to *Native*.
#### System Prompts
System prompts provide persistent instructions that influence agent behavior. In Open WebUI, chats can be organized into folders, and each folder can have its own system prompt. Any chat placed in that folder inherits the folder’s system prompt behavior.
### Error Handling, Debugging
If Open WebUI cannot connect to MCP servers or fails to configure agents correctly, review the application logs. Open the Sessions view and select SEE LOGS to inspect startup and runtime messages and identify the root cause.
# Databricks Connect
Nuvolos now offers a VSCode application with Python 3.9 and R 4.2 and Databricks Connect (`databrics-connect`) pre-installed. From this application, you can submit Spark jobs to Databrics-hosted Spark clusters.
[PySpark](https://spark.apache.org/docs/latest/api/python/) and [sparklyR](https://spark.rstudio.com/) are both installed in the application.
### Prerequisites
{% hint style="info" %}
[Databricks Connect](https://docs.databricks.com/dev-tools/databricks-connect.html) supports Databricks clusters up to 10.4 LTS.
{% endhint %}
To configure the connection, you need:
* the URL of your Databricks cluster, and
* a [personal access token](https://docs.databricks.com/dev-tools/api/latest/authentication.html#token-management).\
Personal access tokens are not available in Databricks Community Edition.
To connect from Nuvolos:
1. Create a **Databricks 10.4 LTS + Py39 + R 4.2** application.
2. Start the application.
3. Open a terminal and run:
```
databricks-connect configure
```
4. Enter the [Databricks cluster URL](https://docs.databricks.com/dev-tools/databricks-connect.html#step-2-configure-connection-properties) and your [personal access token](https://docs.databricks.com/dev-tools/api/latest/authentication.html#token-management) when prompted.
5. Verify the setup with:
```
databricks-connect test
```
### Python example
To run the example, please install the `slugify` Python package with the following command:
`conda install -y -c conda-forge python-slugify`
Once you have configured the Databricks connection, you can try the following simple example to create a Databricks table and run a SQL query on the table:
```python
import pandas as pd
from pyspark.sql import SparkSession
from slugify import slugify
spark = SparkSession.builder.getOrCreate()
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled","true")
# Use NYC Squirrel Census data
df = pd.read_csv("https://data.wa.gov/api/views/f6w7-q2d2/rows.csv?accessType=DOWNLOAD")
df.columns = [slugify(c, separator="_") for c in df.columns]
df = df.drop(columns=["vehicle_location", "electric_utility"])
df = spark.createDataFrame(df)
df.write.mode('overwrite').saveAsTable('ev_data')
spark.sql('select make, model, count(*) as registered from ev_data group by make, model order by registered desc').show(10)
```
### R example
The `sparklyr` package is pre-installed in the application which allows you to connect to Databricks Spark clusters, configured with `databricks-connect.`
You can run the following R script example to run a simple job on your Databricks cluster:
```r
library(sparklyr)
library(dplyr)
databricks_connect_spark_home <- system("databricks-connect get-spark-home", intern = TRUE)
sc <- spark_connect(method = "databricks", spark_home = databricks_connect_spark_home)
cars_tbl <- copy_to(sc, mtcars, overwrite = TRUE)
cars_tbl %>%
group_by(cyl) %>%
summarise(
mean_mpg = mean(mpg, na.rm = TRUE),
mean_hp = mean(hp, na.rm = TRUE)
)
print(as_tibble(cars_tbl), n = 10)
spark_disconnect(sc)
```
# Dynare.jl
### Prerequisites
Nuvolos provides a pre-built application with **Dynare 5.3**, **Julia 1.8**, and the **Dynare.jl** package pre-installed. To get started, add a **Dynare 5.3 + Julia 1.8 + VSCode** application to your working instance.
### Dynare.jl example
Start the application and create a new Julia file `dynare_demo.jl` with the following contents:
```julia
cp("/usr/local/lib/dynare/examples/example2.mod","/files/example2.mod", force=true)
using Dynare
context = @dynare "example2.mod";
using JLD2
DD = load("example2/output/example2.jld2")
```
This example will copy one of the example models provided with Dynare to the Files area (`/files` ) parse and run the calculations on the model. You can run the example either from the command line with the `julia /files/dynare_demo.jl` command, or using VSCode's debugger by pressing the `F5` key.
The output will be saved under `/files/example2/output/example2.jld2` in JLD2 format, which can be processed with the [JLD2 Julia library](https://github.com/JuliaIO/JLD2.jl).
Plots are available under the `/files/example2/output/graphs` folder.
{% hint style="info" %}
This application does not provide a Linux X window display. The plots will not be automatically opened. You can open them with VSCode by double-clicking on the .png files in the file navigator tree view.
{% endhint %}
# CloudBeaver
CloudBeaver is a lightweight web application designed for comprehensive data management. It allows you to work with various data sources, including SQL, NoSQL, and cloud databases, all through a single secure cloud solution accessible via a browser.
## Saving connection details
CloudBeaver runs on Nuvolos in anonymous mode because authentication is handled by Nuvolos. As a result, connections created in the Private project do not persist after the application restarts. To save a connection for reuse, create it in the Shared project instead.
{% hint style="info" %}
If you create a new connection in the Shared project, any user having access to the application (Instance Editors and Space Admins) will be able to see it.
{% endhint %}
# InveLab
The Nuvolos team created InveLab to apply advanced finance models with real financial data to allow students to learn finance concepts intuitively and practically. Among other things, this platform lets students develop the following skills:
* Visualising real financial data.
* Understanding the process of portfolio construction, in-sample & out-of-sample back-testing, international diversification and currency hedging.
* Analysing investable assets, including hedge funds, to assess their ability to generate abnormal performance (alpha).
* Modelling and measuring the dependence among risks, such as Value-at-Risk and Expected Shortfall.
* Practicing various quantitative investment strategies, including factor investments, factor-tilting, etc.
### InveLab editions
The fundamental building blocks in InveLab are *modules*, which represent different operations on financial data. One module might enable visualising a dataset, while another might use the same dataset to compute portfolio weights. Since they generally depend on each other, you don't directly work with individual modules in InveLab. Rather, we have created bundles of modules called *editions.* Editions tell a complete story about some aspect of financial analysis, and guide you through the various modules intuitively.
Each InveLab edition can be installed as an independent Nuvolos application. Currently Nuvolos offers the following editions:
* Alternative Investments
* Global Asset Allocation and Management
* Risk Management in Private Banking
* Data Analysis & Advanced Modelling
* Unified course application
* Regression application
### Workflows
To work with InveLab modules within a given edition, first you need to start a *workflow*. A workflow is a named container for all the objects you work with during your analysis. Whenever you add/modify data in any of the modules, the new state of your workflow will be saved. For this reason, there is no explicit save button in InveLab - anytime you change some objects, a new save is created automatically.
The next time you start InveLab, you can resume your work from where you left off. You can also start multiple workflows in parallel, and run different analysis on them in the same edition.
#### Distributing workflows
In a teaching setting, it can be useful for instructions to prepare workflows for students that already have some prepared data in them, so the students only need to focus on the modules that consume the prepared data.
This can be achieved on Nuvolos in just a few steps:
1. The instructor prepares the workflow(s) in the InveLab edition running in the Master instance of the course space;
2. The instructor distributes the application for all students;
3. Students see the prepared workflows and can work with them.
When distributing an InveLab application, all saved workflows in the Master instance are distributed to student instances, with the following features:
* Workflows can be distributed throughout the course, but students might need to restart their applications to register new incoming workflows
* It is safe for instructors to modify workflows that have already been distributed to students. The next time the application is distributed, the instructor’s changes will not overwrite any changes students have made in their own instances.
* It is not possible to distribute updates to a workflow that has already been distributed once. However, you can duplicate an existing workflow and distribute it as a brand new workflow.
### InveLab modules
Within editions, the structure of InveLab is divided into *objects*, representing financial data types, and *modules*, representing different operations of financial analysis. Modules operate on objects. Some modules have dependencies on others.
Here are some general tips for how to work with InveLab modules:
* Each module requires an input to activate. Certain operations generate and save an output, which can then be used for other modules or functions as an input.
* A new workflow always starts with selecting a dataset (except the valuation module). If an input is missing, the module will show in grey and cannot be entered.
* In any input or output field, you can click the object for operations to **show object information**, **show related objects**, or **delete the object**.
* Deleting the parent object will also delete all child objects.
This table describes the allowed input objects for each module (for object names, please read the **object name** section), where "Y" means that this object category can activate the module as the input:
| Module | D | RD/RR | URD | ERD/ERR | PTS | RP/RPP | M | R | MC | SRD | Special Reqs |
| ------------------------- | - | ----- | --- | ------- | --- | ------ | - | - | -- | --- | -------------------------------------------------------------------------------- |
| Time-Series Visualisation | Y | Y | Y | Y | | Y | | Y | Y | Y | |
| Moment | | Y | Y | Y | | | | | | | |
| Black-Litterman | | | | | | | Y | | Y | | Both M and MC needed |
| Dynamic Strategy | | Y | Y | | | | | | | | |
| Portfolio Analysis | | | | | Y | | | | | | |
| Carry Trade Strategy | Y | | | | | | | | | | Must have both "Interest Rate 1- Month Monthly" and "Spot Exchange Rate monthly" |
| Conditional Volatility | | Y | | | | | | | | | |
| Risk Measures | | Y | | | | | | | | Y | |
| Frontiers | | Y | | | | | | | | Y | |
| Mean-Variance Frontiers | | | | | | | Y | | | | |
| Performance Analysis | Y | Y | Y | Y | | | | | | | |
| Replication | | Y | | | | | | | | | |
| Factor Tilting | | Y | | | | | | | | | |
| Factor Factory | | | Y | | | | | | | | |
| Benchmarking | | Y | | | | | | | | | |
Note that some modules have special requirements for their inputs.
### InveLab objects
The object abbreviations given above correspond to the following meanings:
| Abbreviation | Data type/object category |
| ------------ | --------------------------------------------------------- |
| D | non-return variables (balanced dataset) |
| RD | total return; return without dividends (balanced dataset) |
| RR | return reference (one RD) |
| URD | total return (unbalanced dataset) |
| ERD | excess return |
| ERR | excess return reference (one ER) |
| PTS | portfolio |
| RP | random portfolio |
| RPP | random portfolio plot |
| M | moment |
| R | non-return reference (one D) |
| MC | market capitalization |
| SRD | scaled return |
| FR | frontier (generated via return) |
| MFR | frontier (generated via moments) |
| MR | moving risk |
| CD | characteristics (e.g. hedge fund's strategy) |
| UD | non-total-return variables (unbalanced dataset) |
| FM | factor model |
### Basic operations
Below is some additional info for basic operations in InveLab.
#### Date:
**Date range selection** allows you to select the calendar day/month/year to input. You can also write the date directly in the date field with the format yyyy-mm-dd.
**Restrict to periods** can limit the dataset into certain pre-defined date ranges, including:
* Dotcom Bubble (2001)
* Euro-Sovereign Debt Crisis (2010-2012)
* The Great Recession (2007-2009)
* All NBER Recessions
#### Save:
When you can save an object, it is optional to provide a description of the saved object and then save. This will show in the object information.
#### Compare:
When comparing objects, you can choose the same or different objects on the left and right sides for different operations.
### Further information
For a more detailed module-by-module guide to InveLab, please see the [application reference](broken://pages/Qnnr1vzeY1o8HBGz8l0X) section of the documentation.
# Dataset selection
InveLab works by selecting a dataset and a dataset-specific time range (the input), in order to perform operations on it. The result can then be used for further operations (the output). The list of available datasets and valid time ranges is as follows, where **asterisks indicate default values**:
| | | | | |
| ----------------------------------- | ---------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Dataset** | **Resources** | **Time Range (Frequency)** | **Currency** | **Data Type** |
| Chinese Equity\* | RESSET & WIND | - Daily
- Weekly
- Monthly\*
- Quarterly
- Yearly
| | - Total return\*
- Market cap
- Price adjusted
- Price unadjusted
- Return w/o dividends
- Shares outstanding
- Trading volume
|
| Chinese Index | RESSET & WIND | - Daily
- Weekly
- Monthly\*
- Quarterly
- Yearly
| | - Total return\*
- Price adjusted
|
| Chinese Industry Portfolios | RESSET & WIND | - Daily
- Weekly
- Monthly\*
- Quarterly
- Yearly
| | - Total return\*
- Price adjusted
|
| CN Equity Unbalanced | RESSET & WIND | | | - Total return\*
- Average return T-2-12
- Book-to-Market ratio
- Market Cap T-1
- Market Cap
- Price adjusted
- Price unadjusted
- Return without dividends
- Shares outstanding
- Trading volume
|
| CN Factors | RESSET & WIND | | | - Total return\*
- Excess return
|
| CN Hedge Fund | SUNTIME & WIND | | | - Total return\*
- Net asset value
- Strategy
|
| CN Hedge Fund Index | SUNTIME | - Weekly
- Monthly\*
- Quarterly
- Yearly
| | - Total return\*
- Price adjusted
|
| CN Replication | Multiple | - Daily
- Weekly
- Monthly\*
- Quarterly
- Yearly
| | |
| Exchange Rate | Refinitiv | - Daily
- Weekly
- Monthly\*
- Quarterly
- Yearly
| | - Carry Trade Return\*
- CIP Excess Return
- Forward 1-Month
- Forward 1-Week
- Forward 3-Month
- Spot Exchange Rate
- Spot Total Return
|
| Global Equity Index | Refinitiv | - Daily
- Weekly
- Monthly\*
- Quarterly
- Yearly
| | - Total return\*
- Index level
|
| Global Sovereign Bonds | Refinitiv | - Daily
- Weekly
- Monthly\*
- Quarterly
- Yearly
| | - Total return\*
- Index level
|
| Interest Rate | Refinitiv | - Daily
- Weekly
- Monthly\*
- Yearly
| | - Interest rate 1-month\*
- Interest rate 1-week
- Interest rate 2-month
- Interest rate 2-week
- Interest rate 3-month
- Interest rate 6-month
- Interest rate overnight
|
| US Book-to-Market Sorted Portfolios | Fama-French | | | |
| US Equity | Compustat & CRSP | - Daily
- Weekly
- Monthly\*
- Quarterly
- Yearly
| | - Total return\*
- Market capitalization
- Price adjusted
- Price unadjusted
- Return without dividends
- Shares outstanding
- Trading volume
|
| US Equity Unbalanced | Compustat & CRSP | | | - Total return\*
- Average return T-2-12
- Book-to-market ratio
- Market cap T-1
- Market capitalization
- Price adjusted
- Price unadjusted
- Return without dividends
- Shares outstanding
- Trading volume
|
| US ETF | Yahoo Finance | - Daily
- Weekly
- Monthly\*
- Quarterly
- Yearly
| | - Total return\*
- Price adjusted
|
| US Factors | Fama-French | | | - Total return\*
- Price adjusted
|
| US Hedge Fund | TASS | | | - Total return\*
- Asset under management
- Net asset value
- Strategy
|
| US Hedge Fund Index | Credit Suisse HF | | | - Total return\*
- Assets under management
|
| US Index | Compustat & CRSP | - Daily
- Weekly
- Monthly\*
- Quarterly
- Yearly
| | - Total return\*
- Price adjusted
|
| US Industry Portfolios | Fama-French | | | |
| US Replication | Multiple | - Daily
- Weekly
- Monthly\*
- Quarterly
- Yearly
| | |
# Modules
In this section you find module-specific instructions for InveLab.
# Time-series visualisation
1. **Time-series**. The user needs to first select the overall dataset in the header section as the asset pool, and then the user can select specific data series in this section. It can plot one or multiple assets:
* Summary Table: at the bottom of the page, this section calculates the summary statistics of selected assets.
* Boxplot: this section draws the box plot with min, lower fence, 25%, median, 75%, upper fence and max.
2. **Autocorrelation functions**. This section plots the autocorrelation between selected data series.
3. **Rolling functions**. This section needs to select one benchmark series and the comparison series first, then it will show the **length of the rolling window** for calculation. It supports two calculation methods:
* Correlation: simple Pearson correlation for each rolling window
* CoVar: calculate the risk dependency between assets by filtering the return series lower (higher) than or equal to the target quantile number value.
4. **Scatter plots**. This section plots the multiple assets return series jointly
* Joint Distribution: the shadow field represents the loess regression fitting (refer to: )
* Joint Risk Metrics: it supports only two assets. The visualization provides VaR, CoVaR, ES, and MES.
5. **Time-series differential**. This section only uses for currency-related visualization with **\[D\_]** type objects:
* *Select series a* from a Forward exchange rate (e.g. currency X)
* *Select series b* from a Spot exchange rate (same currency X)
* *Select series c* from a USD interest rate (USD)
* *Select series d* from a Foreign currency interest rate (same currency X)
* All series should use the same duration (e.g. 1-Month or 3-Month, etc.)
* The *Difference plot* section plots the 1-month forward spread in log **(named e)**, e.g. ``e=log(`AUD:[D_9]`) - log(`AUD:[D_10]`); and the 1-month interest rate differential in log``` `**`(named f)`**``, e.g. f= log(1+`USD:[R_1]`/12) - log(1+`AUD:[D_7]`/12)\`\`. It also provides summary statistics and the box plots of the spread and the differential series.
* The *Difference regression* section regresses f on e with their joint distribution.
* The *Residual plot* section shows the time series plot of the regression residual.
6. **Compare random portfolios**: it supports only **\[RP\_]** type objects. This section plots the number of portfolio constituents in each random portfolio on the corresponding standard deviations. The shadow field represents the loess regression fitting.
7. **Dependency**. This section calculates the overall correlation and tail correlation between one asset and other assets, then plots and compares the difference between the overall correlation and the tail correlation:
* It supports two methods of correlation calculation: Pearson correlation and Spearman correlation.
* It also supports Value-at-Risk and CoVaR.
---
[Next Page](/llms-full.txt/1)