AI Workload Management

AI Workload Management is an LLM observability dashboard that helps you monitor and optimize AI workload costs. It surfaces key metrics about prompt caching, error rates, and retry overhead, and provides actionable recommendations to reduce wasted spend across your RAG pipelines.

Access the dashboard

To access the AI Workload Management dashboard:

1. Sign in to the DoiT console.

2. From the top navigation mega menu, select Workload optimization > AI Workload Management.

3. Use the time-range toggle in the upper-right corner to specify the period to view: the last 24 hours, 7 days, or 30 days.

   

KPI summary

The four KPI summary cards at the top of the page provide an at-a-glance view of your AI workload health:

• AI/LLM spend: Total spend on LLM calls for the selected time range, with a period-over-period trend indicator and total call count.

• Cache hit rate: The percentage of LLM calls served from cache, along with the estimated savings compared to uncached calls.

• Error rate: The percentage of LLM calls that resulted in errors, the top error type, and the total number of failed calls.

• Retry overhead: The cost of retried LLM calls, shown as both an absolute value and as a share of total AI spend, with the number of retried calls.

Cache hit rate vs retry cost

The Cache hit rate vs retry cost chart plots cache hit rate alongside retry cost over the selected time range. It helps you identify periods of poor cache performance that correlate with increased retry costs.

Prompt and response log

The Prompt and response log lists individual LLM calls with details including timestamp, model, service, token counts, cache status, retry count, latency, status, and cost. Use the filter chips above the log to narrow the view:

• All: Shows every log entry.

• Cache misses: Shows only calls where the prompt was not served from cache.

• Errors: Shows only calls that returned an error status.

• Retried: Shows only calls that required one or more retries.

Select a log entry to expand it for additional detail:

• Prompt preview: A preview of the prompt text sent to the model.

• Attempt timeline: A step-by-step timeline of each attempt. When retries occurred, the wasted-cost breakdown indicates how much spend was consumed by failed attempts.

RAG pipeline optimization

The RAG pipeline optimization section analyzes your RAG (Retrieval-Augmented Generation) pipeline across four stages and surfaces recommendations to reduce overhead.

Pipeline stages

Each stage displays a health indicator and key metrics:

• Embed: Converts input text into vector embeddings.

• Retrieve: Fetches relevant documents from your vector store.

• Rerank: Re-scores retrieved documents for relevance before generation.

• Generate: Sends the final prompt with context to the LLM for response generation.

Retry and regeneration overhead

The Retry & regeneration overhead shows your current retry and regeneration overhead as a percentage.

When you apply recommendations, a projected overhead bar appears with the estimated reduction and projected monthly savings.

Recommendations

Actionable recommendations are listed with their estimated monthly savings. Select Apply on a recommendation to include it in the projected overhead calculation.