Configuring the LLM in Podman Desktop

The Podman AI lab extension enables you to use an open-source model from a curated list of models and use it locally in your system.

The code fix suggestions generated by a model depends on the model’s capabilities. Models deployed through the Podman AI Lab were found to be insufficient for the complexity of code changes required to fix issues discovered by Konveyor. You must not use such models in a production environment.

Prerequisites

You installed Podman Desktop in your system.
You completed initial configurations in Konveyor required for the analysis.

Procedure

Go to the Podman AI Lab extension and click Catalog under Models.
Download one or more models.
Go to Services and click New Model Service.
Select a model that you downloaded in the Model drop-down menu and click Create Service.
Click the deployed model service to open the Service Details page.
Note the server URL and the model name. You must configure these specifications in the Konveyor extension.
Export the inference server URL as follows:
```
export OPENAI_API_BASE=<server-url>
```
In the Konveyor extension, type Open the GenAI model provider configuration file in the Command Palette to open the provider-settings.yaml file.
Enter the model details from Podman Desktop. For example, use the following configuration for a Mistral model.

  podman_mistral: &active
    provider: "ChatOpenAI"
    environment:
      OPENAI_API_KEY: "unused value"
    args:
      model: "ibm-granite/granite-3.3-8b-instruct-GGUF"
      configuration:
        baseURL: "http://localhost:56885/v1"

Note: The Podman Desktop service endpoint does not need a password but the OpenAI library expects the OPENAI_API_KEY to be set. In this case, the value of the OPENAI_API_KEY variable does not matter.