Docker Deployment

If you are using GPU based AI Guardrails, directly download and deploy the Netskope LLM Docker image.
As a standard image, it can run in any environment supporting containers as follows:

• SageMaker
• Bare-metal or VM with GPU
• Kubernetes
• Custom infrastructure

Authentication Options:

• JWT authentication is optional for deployments.
• If you choose no authentication during deployment, AISecurityService will skip token generation and communicate directly with the LLM endpoint.

AISecurityService requires the following configurations:

• (Required) Host URL – the LLM’s base HTTP endpoint.
• Optional (if JWT is enabled by the customer)
  • JWT Token URL
  • Client ID
  • Client Secret
  • Scope

You can configure the above values in the AI Gateway CLI.

Hardware Requirement

Hardware	Requirements
GPU	1
GPU Architecture	NVDA (CUDA)
GPU Memory	16 GB
CUDA Support	Yes
vCPU	2
RAM	16 GB
Storage	50 GB

Example Docker Run Command

docker run -d -p 8080:8080 <netskope-llm-docker-image>

Example for customer deployment (Sagemaker endpoint):

You can deploy the docker image in a different environment.

To deploy a Sagemaker endpoint, follow the steps below (JWT is optional).

1. Create an ECR repository and push the LLM Docker image.

   ECR Push commands are available on ECR page. This image will be referenced by SageMaker during model creation.

2. Deploy SageMaker Endpoint.

   1. Create the Model using the following commands.

      aws sagemaker create-model \
        --model-name <model-name> \
        --primary-container Image=<ECR-image-URI> \
        --execution-role-arn <sagemaker-execution-role>

   2. Create the Endpoint Configuration using the following commands:

      aws sagemaker create-endpoint-config \
        --endpoint-config-name <endpoint-config-name> \
        --production-variants VariantName=AllTraffic,ModelName=<model-name>,InitialInstanceCount=1,InstanceType=ml.g5.xlarge
      

      ml.g5.xlarge is the current GPU instance type but can be substituted with other GPU families.

   3. Create the Endpoint using the following command.

      aws sagemaker create-endpoint \
        --endpoint-name <endpoint-name> \
        --endpoint-config-name <endpoint-config-name>
      

When the GPU based LLM for guardrails service is ready, follow the steps below.

Selecting LLM Guardrails Service <link>

Configure LLM Service Certificate <link>

Configure LLM Service Host <link>

Delete LLM  Service Host Configuration <link>