Emerging Patterns in Building GenAI Products

Scoring and Judging

Necessary arguments are fed through a scorer, which is a component or function that assigns numerical scores to generated outputs, reflecting evaluation metrics like relevance, coherence, factuality, or semantic similarity between the model's output and the expected answer.

Different evaluation techniques exist based on who computes the score, raising the question: who, ultimately, will act as the judge?

• Self evaluation: Self-evaluation lets LLMs self-assess and enhance their own responses. Although some LLMs can do this better than others, there is a critical risk with this approach. If the model’s internal self-assessment process is flawed, it may produce outputs that appear more confident or refined than they truly are, leading to reinforcement of errors or biases in subsequent evaluations. While self-evaluation exists as a technique, we strongly recommend exploring other strategies.
• LLM as a judge: The output of the LLM is evaluated by scoring it with another model, which can either be a more capable LLM or a specialized Small Language Model (SLM). While this approach involves evaluating with an LLM, using a different LLM helps address some of the issues of self-evaluation. Since the likelihood of both models sharing the same errors or biases is low, this technique has become a popular choice for automating the evaluation process.
• Human evaluation: Vibe checking is a technique to evaluate if the LLM responses match the desired tone, style, and intent. It is an informal way to assess if the model “gets it” and responds in a way that feels right for the situation. In this technique, humans manually write prompts and evaluate the responses. While challenging to scale, it’s the most effective method for checking qualitative elements that automated methods typically miss.

In our experience, combining LLM as a judge with human evaluation works better for gaining an overall sense of how LLM is performing on key aspects of your Gen AI product. This combination enhances the evaluation process by leveraging both automated judgment and human insight, ensuring a more comprehensive understanding of LLM performance.

Example

Here is how we can use DeepEval to test the relevancy of LLM responses from our nutrition app

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric

def test_answer_relevancy():
  answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.5)
  test_case = LLMTestCase(
    input="What is the recommended daily protein intake for adults?",
    actual_output="The recommended daily protein intake for adults is 0.8 grams per kilogram of body weight.",
    retrieval_context=["""Protein is an essential macronutrient that plays crucial roles in building and 
      repairing tissues.Good sources include lean meats, fish, eggs, and legumes. The recommended 
      daily allowance (RDA) for protein is 0.8 grams per kilogram of body weight for adults. 
      Athletes and active individuals may need more, ranging from 1.2 to 2.0 
      grams per kilogram of body weight."""]
  )
  assert_test(test_case, [answer_relevancy_metric])

In this test, we evaluate the LLM response by embedding it directly and measuring its relevance score. We can also consider adding integration tests that generate live LLM outputs and measure it across a number of pre-defined metrics.

Running the Evals

As with testing, we run evals as part of the build pipeline for a Gen-AI system. Unlike tests, they aren't simple binary pass/fail results, instead we have to set thresholds, together with checks to ensure performance doesn't decline. In many ways we treat evals similarly to how we work with performance testing.

Our use of evals isn't confined to pre-deployment. A live gen-AI system may change its performance while in production. So we need to carry out regular evaluations of the deployed production system, again looking for any decline in our scores.

Evaluations can be used against the whole system, and against any components that have an LLM. Guardrails and Query Rewriting contain logically distinct LLMs, and can be evaluated individually, as well as part of the total request flow.

Evals and Benchmarking

Benchmarking is the process of establishing a baseline for comparing the output of LLMs for a well defined set of tasks. In benchmarking, the goal is to minimize variability as much as possible. This is achieved by using standardized datasets, clearly defined tasks, and established metrics to consistently track model performance over time. So when a new version of the model is released you can compare different metrics and take an informed decision to upgrade or stay with the current version.

LLM creators typically handle benchmarking to assess overall model quality. As a Gen AI product owner, we can use these benchmarks to gauge how well the model performs in general. However, to determine if it’s suitable for our specific problem, we need to perform targeted evaluations.

Unlike generic benchmarking, evals are used to measure the output of LLM for our specific task. There is no industry established dataset for evals, we have to create one that best suits our use case.

Example Image Embedding

Deep learning models create more effective image embeddings than hand-crafted approaches. Therefore, we'll use a CLIP (Contrastive Language-Image Pre-Training) model, specifically clip-ViT-L-14, to generate them.

# python
from sentence_transformers import SentenceTransformer, util
from PIL import Image
import numpy as np

model = SentenceTransformer('clip-ViT-L-14')
apple_embeddings = model.encode(Image.open('images/Apple/Apple_1.jpeg'))

print(len(apple_embeddings)) # Dimension of embeddings 768
print(np.round(apple_embeddings, decimals=2))

If we run this, it will print out how long the embedding vector is, followed by the vector itself

768

[ 0.3   0.25  0.83  0.33 -0.05  0.39 -0.67  0.13  0.39  0.5  # and so on...

768 numbers are a lot less data to work with than the original 3.6 million. Now that we have compact representation, let's also test the hypothesis that similar images should be located close to each other in vector space. There are several approaches to determine the distance between two embeddings, including cosine similarity and Euclidean distance.

For our nutrition app we will use cosine similarity. The cosine value ranges from -1 to 1:

Given two embeddings, we can compute cosine similarity score as:

def cosine_similarity(embedding1, embedding2):
  embedding1 = embedding1 / np.linalg.norm(embedding1)
  embedding2 = embedding2 / np.linalg.norm(embedding2)
  cosine_sim = np.dot(embedding1, embedding2)
  return cosine_sim

Let’s now use the following images to test our hypothesis with the following four images.

apple 1

apple 2

apple 3

burger

Here's the results of comparing apple 1 to the four iamges

In reality there could be a number of variations - What if the apples are cut? What if you have them on a plate? What if you have green apples? What if you take a top view of the apple? The embedding model should encode meaningful relationships and represent them efficiently so that similar images are placed in close proximity.

It would be ideal if we can somehow visualize the embeddings and verify the clusters of similar images. Even though ML models can comfortably work with 100s of dimensions, to visualize them we may have to further reduce the dimensions ,using techniques like T-SNE or UMAP , so that we can plot embeddings in two or three dimensional space.

Here is a handy T-SNE method to do just that

from sklearn.manifold import TSNE
tsne = TSNE(random_state = 0, metric = 'cosine',perplexity=2,n_components = 3)
embeddings_3d = tsne.fit_transform(array_of_embeddings)

Now that we have a 3 dimensional array, we can visualize embeddings of images from Kaggle’s fruit classification dataset

Click on any data point for preview

The embeddings model does a pretty good job of clustering embeddings of similar images close to each other.

So this is all very well for images, but how does this apply to documents? Essentially there isn't much to change, a chunk of text, or pages of text, images, and tables - these are just data. An embeddings model can take several pages of text, and convert them into a vector space for comparison. Ideally it doesn't just take raw words, instead it understands the context of the prose. After all “Mary had a little lamb” means one thing to a teller of nursery rhymes, and something entirely different to a restaurateur. Models like text-embedding-3-large and all-MiniLM-L6-v2 can capture complex semantic relationships between words and phrases.

Embeddings in LLM

LLMs are specialized neural networks known as Transformers. While their internal structure is intricate, they can be conceptually divided into an input layer, multiple hidden layers, and an output layer.

A significant part of the input layer consists of embeddings for the vocabulary of the LLM. These are called internal, parametric, or static embeddings of the LLM.

Back to our nutrition app, when you snap a picture of your meal and ask the model

“Is this meal healthy?”

The LLM does the following logical steps to generate the response

• At the input layer, the tokenizer converts the input prompt texts and images to embeddings.
• Then these embeddings are passed to the LLM’s internal hidden layers, also called attention layers, that extracts relevant features present in the input. Assuming our model is trained on nutritional data, different attention layers analyze the input from health and nutritional aspects
• Finally, the output from the last hidden state, which is the last attention layer, is used to predict the output.

RAG Template

Once we have document fragments from the retriever, we then combine the users prompt with these fragments using a prompt template. We also add instructions to explicitly direct the LLM to use this context and to recognize when it lacks sufficient data.

Such a prompt template may look like this

Guardrails using LLMs

If we don’t want our users of the nutrition app to respond to queries about topics other than nutrition then we can implement the self_check_input rails of NeMo Guardrails framework.

We wrap the user's prompt inside a special template, such as this.

Under the hood, the guardrail framework will use a prompt like the one above to decide if we need to block or allow user query.

Embeddings based guardrails

Guardrails may not rely solely on calls to LLMs. We can also use embeddings to enforce safety, topic constraints, or ethical guidelines in Gen AI products. By leveraging embeddings, these guardrails can analyze the meaning of user inputs and apply controls based on semantic similarity, rather than relying solely on explicit keyword matches or rigid rules.

Our teams have used Semantic Router to safely direct user queries to the LLM or reject any off-topic requests.

Rule based guardrails

Another common approach is to implement guardrails using predefined rules. For example, to protect sensitive personal information we can integrate with tools like Presidio to filter personally identifiable information from the knowledge base. 3