When LLM Output Stalls, We Fall Back to Skeleton Screens

The Incident: A 12-Second Stall That Broke Our Chatbot

The Client Constraint

During my internship at a fintech startup in Jaipur, we shipped a customer support chatbot powered by Anthropic's Claude 3 Haiku. The product team insisted on a clean, minimal UI with no loading indicators. They wanted the experience to feel instant. That decision came back to bite us during a live demo to enterprise clients in Mumbai.

What We Tried First

We implemented a basic spinner that appeared after 500ms of no response. It looked fine in development, where Claude responded in under 2 seconds. But during the demo, a complex query about loan amortization schedules triggered a 12-second stall. The spinner spun, users clicked it repeatedly, and the entire interface appeared frozen. The client walked away.

What Actually Failed

The spinner approach failed because it gave no spatial context. Users had no idea what shape the response would take, so when it finally arrived, the layout shifted dramatically. Worse, the spinner implied a short wait. When it kept spinning past 5 seconds, users assumed the system had crashed. We also failed to account for Claude's variable latency: simple greetings returned in 800ms, but multi-turn financial queries took 8-15 seconds.

The Working Approach: Skeleton Screens with Tiered Fallbacks

Real Implementation Details

We replaced the spinner with a skeleton screen that matched the expected chat bubble layout. Two grey rounded lines of varying width inside the assistant bubble. The implementation used a CSS animation with a 600ms shimmer cycle:

.skeleton-bubble {
  background: linear-gradient(90deg, #e0e0e0 25%, #f0f0f0 50%, #e0e0e0 75%);
  background-size: 200% 100%;
  animation: shimmer 1.5s infinite;
}
@keyframes shimmer {
  0% { background-position: 200%; }
  100% { background-position: -200%; }
}

The skeleton appeared immediately on request submission, before the first token arrived. We measured the real DOM height of typical responses and derived skeleton dimensions from those numbers. Never guessing.

Tiered Fallback After 3 Seconds

After 3 seconds of no token streaming, we swapped the skeleton for honest microcopy: 'Still working on your response...'. This prevented the 'infinite shimmer' problem where users on slow networks watched a pulsing placeholder for 9+ seconds.

Real API Integration

We used Anthropic's streaming API with Server-Sent Events (SSE):

const response = await fetch('https://api.anthropic.com/v1/messages', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'x-api-key': ANTHROPIC_API_KEY
  },
  body: JSON.stringify({
    model: 'claude-3-haiku-20240307',
    stream: true,
    messages: conversation
  })
});

const reader = response.body.getReader();
let receivedText = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  const chunk = new TextDecoder().decode(value);
  // Parse SSE event and append to UI
}

Pitfalls We Would Warn an Intern About

Skeleton Mismatch Creates Jarring Snaps

Never guess skeleton dimensions. We initially used fixed 200px heights for all bubbles, which caused a visible 'snap' when the real content arrived at 350px. The fix: render the populated component first, log its computed height per breakpoint, then derive skeleton dimensions from those numbers.

Gating on Fonts and Images

We revealed the skeleton before fonts and images settled, causing stutter. The fix: use the Font Loading API and decode() so the swap happens once, not in stages.

No Minimum Display Floor

A skeleton that flashes for 80ms reads as a glitch. We added a 200ms minimum display floor so the eye registers intent.

No Escape Hatch

After 8 seconds, we added a 'Cancel and try rephrasing' button. Users appreciated having control.

What We Would Do Differently Next Time

Implement Thought Tracing Instead of Generic Skeletons

Following Pranjal Rastogi's approach, we would stream status events from the backend: 'Searching knowledge base...', 'Reading documents...', 'Generating response...'. This turns 'Waiting' into 'Watching' and builds trust. Users tolerate 2x longer waits when they see what the AI is doing.

Cache Common Responses Aggressively

About 30% of support queries are variations of the same question. We would implement a Redis cache keyed on normalized query text to serve instant responses for common cases.

Test on Slow Networks, Not Just Dev Machines

We would throttle DevTools to 'Slow 3G' and watch the skeleton behavior. If it outlives its welcome, the tiered fallback kicks in. This network discipline applies to cold start time too. Optimize for the worst device on the worst connection.

Consider Model Size vs. Latency Tradeoffs

For simple queries, we would route to a smaller, faster model. A 2-second accurate response beats a 12-second perfect one when the user just wants to know their account balance.

Takeaways for Interns Building AI Features

Skeleton screens are not magic. They are contracts. You promise the user that something shaped like the skeleton is about to appear. If the dimensions mismatch, the layout snaps and trust evaporates. If the shimmer runs too long, users assume the app crashed.

The real lesson from our 12-second stall: never ship an LLM feature without a fallback plan. Skeletons for the first 3 seconds, honest microcopy after that, and a cancel button after 8. Test on Slow 3G. Measure real response heights. Stream tokens. And always, always give users a way out.

The Deeper Pattern: Why Skeletons Win Over Spinners

The fundamental difference between a spinner and a skeleton is that a spinner says 'wait' while a skeleton says 'almost ready'. This psychological shift matters enormously when dealing with LLM latency. A spinner is a black box. It gives no information about what is coming, how big it will be, or whether the system is still alive. Users interpret prolonged spinning as a crash, which is exactly what happened during our Mumbai demo.

A skeleton, by contrast, is a promise. It shows the user the shape of what is coming. When the real content loads and matches that shape, the transition feels seamless. The brain registers completion rather than interruption. This is why e-commerce sites use skeletons for product cards and why chat interfaces should use them for response bubbles.

But skeletons are not a silver bullet. They only work when they are accurate. A skeleton that is too short causes a jarring snap when the real content expands the container. A skeleton that is too tall leaves empty space that looks broken. The key is measurement. We spent two days instrumenting our chatbot to log the computed height of every response bubble across desktop, tablet, and mobile breakpoints. Those numbers became our skeleton dimensions.

Building the Fallback Chain

The tiered approach we settled on has three stages:

1. 0-3 seconds: Skeleton screen with shimmer animation. This covers the majority of responses and feels responsive.

2. 3-8 seconds: Honest microcopy. 'Still working on your response...' replaces the shimmer. This prevents the infinite pulse problem and acknowledges the delay.

3. 8+ seconds: Cancel button appears. 'Cancel and try rephrasing' gives users agency. No one likes feeling trapped.

This chain is driven by a simple timer that starts when the request is sent. We also listen for the first token from the streaming response. If a token arrives within 3 seconds, we skip the microcopy stage and go straight to rendering the streamed content.

Testing Under Real Conditions

After the Mumbai incident, we made network testing part of our deployment checklist. Every feature branch gets tested on Chrome DevTools throttled to 'Slow 3G' with a 4x CPU slowdown. We watch the skeleton behavior and verify that the fallback chain triggers at the right times.

We also added a synthetic monitoring test that sends a complex financial query through our chatbot every hour. If the response takes longer than 10 seconds, we get a Slack alert. This catches regressions before they reach production.

The Cost of Getting It Wrong

The Mumbai demo cost us the client. But it also taught us a lesson that has stuck with me through every AI project since. LLM latency is not a bug to be optimized away. It is a feature of the technology. The best UX does not hide latency. It manages expectations and gives users a sense of control.

Skeleton screens, tiered fallbacks, and escape hatches are not just technical patterns. They are empathy patterns. They acknowledge that the user's time is valuable and that waiting is frustrating. When done right, they turn a moment of frustration into a moment of trust.

Sources:

• Designing for LLM Latency: Streaming UX Patterns
• Skeleton Screen Deception
• Pranjal Rastogi on Thought Tracing
• Payloop Community Discussion
• Skeleton screens mask LLM latency
• Piyush Singh on AI-native UI
• AI UX Patterns for Design Leads
• arablex/llm-ux-patterns