CalypsoAI is designed to provide a comprehensive solution that addresses the security risks inherent in the widespread use of Large Language Models (LLMs.) The following guide is designed to help you install and configure the CalypsoAI solution in your environment.

There are two (2) common methods of deploying CalypsoAI:

CalypsoAI instance installed on a single compute instance

CalypsoAI installed into a highly available Kubernetes Cluster

CalypsoAI is built and deployed as an on-premises solution. CalypsoAI is packaged as a single container application, which also includes Keycloak (https://www.keycloak.org/) for user authentication. A second container is used with a PostgreSQL (https://www.postgresql.org/) database to store CalypsoAI data. There is a separate installation for the custom scanner functionality. There are a number of different methods of deploying containers into environments and CalypsoAI leaves it up to the customer to choose whichever method makes sense to the organization. Because this is an on-premises solution, there is no data that ever comes back to CalypsoAI.

As an example at CalypsoAI, we deploy into a highly available Kubernetes cluster in a cai-moderator namespace via Helm. Helm is a tool that automates the creation, packaging, configuration, and deployment of Kubernetes applications by combining the configuration files into a single reusable package. If this is your chosen deployment option, CalypsoAI can supply you with the Helm charts.

The diagram below shows the different components of the CalypsoAI solution:

The following list presents the hardware requirements when implementing a small Proof of Concept for CalypsoAI Moderator and the recommended AWS specific instance type.

Moderator can be run on a CPU only or on a GPU instance. If using custom scanners, that application will need to be run on a GPU instance.

CPU only Instance requirements:

- vCPUs - 16

- Memory - 32.0 GiB

- Clock Speed - 3.5 (GHz)

- CPU Architecture - x86_64

- Persistent Storage - 100.0 GiB

GPU instance requirements:

- vCPUs - 4

- Memory - 16.0 GiB

- CPU Architecture - x86_64

- Persistent Storage - 100.0 GiB

- GPU - Nvidia A10G (CUDA Supported)

AWS EC2 instance type recommendation if no GPU's are available: c6i.4xlarge

GPU AWS EC2 instance type recommendation: g5.xlarge

If deploying into a Kubernetes cluster, CalypsoAI recommends a single node for the initial implementation and testing of Moderator and another single GPU node for the cai-scanner.

The compute instance must have a security group or something similar for network access using the following rules:

Inbound

Outbound (used to retrieve software, updates as well as communicate with the LLM provider)

If you are using custom scanners, you will need to allow inbound access to the instance running the scanners via port 8000.

Keycloak is the mechanism that CalypsoAI uses to authenticate users of the solution. Keycloak requires the use of HTTPS. In order for HTTPS to function properly, SSL certificates are required.

Using a certificate from a trusted Certificate Authority is highly recommended for any production website or application. However, for testing purposes you can use self-signed certificates.

These certificates will need to be added to the Load Balancer of choice. If following the AWS example below, you can import the certificates using Amazon Certificate Manager (ACM). Please see the following as reference:

https://docs.aws.amazon.com/acm/latest/userguide/import-certificate-api-cli.html

Once imported, you will have an opportunity to select these certificates during the creation of the Application Load Balancer.

Self-signed certificates are not issued by a trusted Certificate Authority (CA), so browsers like Google Chrome cannot verify the authenticity of the website. When a user tries to access a website with a self-signed certificate, Chrome displays a security warning, indicating that the connection might not be secure. In order to avoid this error, you will need to make sure that whatever Computer/Browser you are using to access the CalypsoAI Solution, has the self signed certificates installed and trusted.

There are lots of resources that discuss and explain both the concepts and the creation of Self Signed Certificates. One of these articles is referenced here:

https://deliciousbrains.com/ssl-certificate-authority-for-local-https-development/

CalypsoAI requires the use of a load balancer in order to facilitate the proper http/https redirects to both the CalypsoAI software and Keycloak for authentication. CalypsoAI uses two ports, one for keycloak authentication (to authenticate the users) and the other for the application itself. The load balancer is used to route the /auth uri to port 8080 and then / to port 5500 on the instance.

The following example walks through the instructions for setting up an Application Load Balancer in AWS. The same concepts can be applied to other cloud providers or local servers such as NGINX.

For reference: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-application-load-balancer.html

Step 1: Configure Target Groups

CalypsoAI will require the creation of two (2) target groups. Each target group will have a target type of “Instances.”

Target Group #1:

Target Group #2:

Step 2: Create the Application Load Balancer

For this example the ALB will be an Internal Facing ALB. This assumes there are private subnets created within your selected VPC.

Security Group

Inbound

Outbound

Listeners and Routing

You will need to create 2 listeners along with associated rules.

Listener #1

Listener #2

Unless deploying on a localhost, CalypsoAI requires a qualified Domain Name. This qualified domain name should have an A record that points to either the Application Load Balancer that is setup in the previous step, or to another Load Balancer that has been configured similarly.

Optionally, Domain names or IP addresses on a local computer can be resolved by adding entries in the local hosts file on the compute instances. Entries in the local hosts file have the added advantage that the system can run the application server, even when disconnected from the network. If you are using a hosts file to resolve IP addresses, the file must be configured correctly, and on each computer that will be connecting to the CalypsoAI solution.

The most important thing is the name chosen and configured in either method needs to match exactly with the variable placed inside the .env file created in later steps.

CalypsoAI does not provide access to any LLM providers by default. The customer must have at least one provider available. The following information is required to configure CalypsoAI:

For Azure OpenAI, the following is required:

As stated in the Networking Requirements for the instance, network access over HTTPS:443 is required, and access to the provider API endpoint resource is needed.

CalypsoAI also supports application requests using a full RESTful API. Please see the API section later in this document.

CalypsoAI supports the use of SAML authentication and authorization for Single Sign-On (SSO). Single Sign-On is a feature that allows your users to authenticate their identity once within CalypsoAI using existing credentials. Keycloak, an open-source Identity and Access Management tool developed by Red Hat, handles user authentication. Keycloak supports popular social networks, SAML 2.0 Identity Providers (IdP), an existing OpenID Connect, or Active Directory / LDAP servers.

Before starting, please ensure you have been provided the following:

Harbor is used as a private registry where you will be able to pull the CalypsoAI Moderator Docker image or Helm chart from the repositories depending on the preferred installation method.

The instructions below enable you to run CalypsoAI Moderator using Docker Compose. This simplifies the deployment by defining the group of services and configuration in a single YAML file for moderator application and Postgresql. And a single YAML file for the custom scanner functionality. Environment variables are passed in from a .env file and a volume is used to provide persistent storage, which saves data between runs.

If using custom scanners, be sure that these prerequisite steps are completed on both the instance being used for Moderator as well as the instance being used for the custom scanners application.

Prerequisite: Install Docker Engine, Docker CLI, and Docker Compose

The steps below are for an Ubuntu Linux distribution. This is only required if you don't have docker installed. See the Docker help documentation (https://docs.docker.com/engine/install/) for setup on other systems. Docker Desktop is not required.

Step 1. Setup the Docker repository.

Step 2. Install Docker Engine, CLI, containerd, and Docker Compose.

Step 3. Optional: Post-installation setup.

https://docs.docker.com/engine/install/linux-postinstall/

Step 4. Verify Docker and Docker Compose installations.

GPU

If you want to run CalypsoAI with Nvidia GPU, first install the Nvidia driver on the underlying instance. This is a requirement for the cai-scanner (custom scanner functionality).

Then install the Nvidia Container Toolkit

Installing the NVIDIA Container Toolkit -- NVIDIA Container Toolkit 1.16.0 documentation

Install CalypsoAI Moderator

Step 1. Log in to the Harbor registry using the provided access credentials. You may omit the --password option to be prompted for entering a hidden value.

Step 2. Create a folder called moderator and enter it.

Step 3. Copy the code shown below into a docker-compose.yaml file.

If running on a GPU instance you are required to add the following under the moderator service.

With this being the case the docker-compose.yaml will look like this:

Step 4. Copy the code shown below into a .env file.

Step 5. Pull the Moderator and PostgreSQL Docker images and run the containers. A full Moderator image can take 10 minutes to pull. A `postgres` subdirectory is created in the `moderator` parent directory, which contains the database to persist data between runs.

Step 6. Browse to the URL set in CAI_MODERATOR_BASE_URL and verify a login prompt is displayed.

Install CalypsoAI Scanner

Step 1. Log into the Harbor registry using the provided access credentials. You may omit the --password option to be prompted for entering a hidden value.

Step 2. Create a folder called scanner and enter it.

Step 3. Copy the code shown below into a docker-compose.yaml file.

Step 4. Copy the code shown below into a .env file.

Step 5. Run the container via docker compose with the below command.

Troubleshooting

The following instructions assume:

An ingress controller is a load balancer used for routing external traffic to your Kubernetes cluster and is responsible for L4-L7 Network services. The ingress controller or load balancer operates at Layer 7. Ingresses or the object route HTTP and HTTPS traffic.

Cloud-based ingress controllers include:

Open-source ingress controllers include:

Moderator uses two ports, one for Keycloak user authentication and the other for the Moderator application itself. The ingress controller (load balancer) is used to route the /auth URI to port 8080 on the Kubernetes service and then / (root) to port 5500 on the Kubernetes service. The following diagram shows how this works with the Istio ingress controller.

The configuration steps below show an example how to do this with nginx. CalypsoAI can supply you with examples for both nginx, Istio ingress controllers and Nginx Ingress. Whichever ingress controller (load balancer) is chosen, the same principles apply.

If using the custom scanner functionality, the installation should look like below.

Pod scheduling is one of the most important aspects of Kubernetes cluster management. How pods are distributed across nodes directly impacts performance and resource utilization. Kubernetes node affinity is an advanced scheduling feature that helps administrators optimize the distribution of pods across a cluster. Node affinity enables administrators to match pods according to the labels on nodes.

GPU Operator

If you choose to use GPU's with Kubernetes, you will need to Install the Nvidia GPU operator in order to provide Kubernetes access to the GPU. You can install this with a simple helm command:

Install NVIDIA GPU Operator — gpu-operator 1.8 documentation

CalypsoAI Moderator uses two Helm charts, a PostgreSQL chart and the Moderator application chart. If using custom scanners, there is a third Helm chart for the custom scanners.

Install the PostgreSQL Helm Chart

Install the CAI-CalypsoAI Helm Chart

Step 2. Login to Harbor with your login credentials.

Step 3. Create a values.yaml to use with the Helm chart.

Step 4. In your local working directory, run the cai-moderator helm chart installation with the values file filled in.

Install the CAI-Scanner Helm Chart

Step 2. Login to Harbor with your login credentials.

Step 3. Create a values.yaml to use with the Helm chart.

Step 4. In your local working directory, run the cai-scanner helm chart installation with the values file filled in.

To enable the custom scanners feature, you will need to use the Moderator API. First create an API key if you do not have one as described in the "Generate Moderator API Token below.

In a terminal, run the below command after replacing the placeholder values {token} with your API key and {moderator_url} with the url that points to your Moderator installation.

After everything has been installed, your very first task is to add a user. Adding a user to your account is simple and straightforward. As indicated earlier in the guide, Keycloak is the mechanism that CalypsoAI uses to authenticate users of the solution. By default, Keycloak is running as a service on port 8080.

Step 1: Navigate and Log into the Keycloak service, by opening your browser and navigating to the base URL. If you have not setup a Load Balancer with port mappings as suggested earlier in the guide, you will need to make sure you input port 8080 directly into the URL. i.e. https://{url}:8080 If everything is working, you should see a login screen similar to this:

The default username is Keycloak, and the default password is Keycloak, unless you changed it in the initial setup above.

Step 2: Change Realms to CalypsoAI by clicking on the drop down box in the upper left corner of the screen.

Step 3: Change the Admin password. By default, CalypsoAI creates a default user called “admin”. You will need to change the password for this account. Click on “Users” in the navigation menu on the left.

Select the admin user.

Click on the “Credentials” tab and click the “Reset password” button

Change the password and click “Save”

Step 4: Next, let’s verify that you can log in to the platform with the admin user. Open up a second browser window or tab and navigate to the root url of the installed platform. If you have not setup a Load Balancer with port mappings as suggested earlier in the guide, you will need to make sure you input port 5500 directly into the URL. i.e. https://{url}:5500 If everything is working, you should see a login screen similar to this:

Step 5: Login with the “admin” user and the newly created credentials from the earlier step.

Step 6: Next, let’s make sure the “admin” user is displayed in the Users screen. Click on “Settings” button on the bottom left navigation menu.

Step 7: Next, click on the “Users” tab. You should see something like this:

Step 8: There are 2 ways to add in additional users to the platform. If you have an email server configured and accessible from the platform, you could invite users. Most on-premises deployments do not have this accessible, so the most effective way is to add individual users through the keycloak service described earlier. Navigate back to the Keycloak browser window/tab you had opened in the earlier step.

Step 9: Click on the Users button in the menu on the left side and it will bring you into the users screen.

Step 10: Click the “Add user” button. This will bring you to a “Create User” screen.

Step 11: ill in all the appropriate information as seen in the screenshot above and click the “Create” button

Step 12: Click on the newly created user and click the “Credentials’’ tab in order to set the password.

Step 13: Click “Set password” and fill in the information and bit “Save”

Step 14: Navigate back to the browser window/tab that you have open with the platform and navigate back to the Users tab from settings. You should see the newly created user listed.

Step 15: From here, you can do things like create Groups, assign users groups, and make users the admin of a group.

The CalypsoAI platform is built on a RestFUL api. You can find our API documentation inside the platform.

Step 1: Click on the “Support” item in the navigation pane on the bottom left of the screen.

Step 2: Click on the “API Docs” button