AI Report Summaries
Designing the platform's first AI feature: a self-serve report summary that cut manual requests — 21.1% adoption in 90 days.
Report viewers, including external partners who receive campaign reports, often requested a concise summary alongside the full report. Creating these summaries manually fell to internal teams. By automating the process with AI, we enabled viewers to generate summaries directly within the platform while reducing the internal team's workload.
As the sole product designer on this project, I led user flows, wireframes, prototyping, and usability testing. Working closely with the PM and engineers, I completed the design within the two-week timeline, and the feature launched in October 2024.
Problem
Report viewers couldn't understand their data without asking our team.
Report summaries weren't self-serve. The internal team handled 200+ requests a month, spending an estimated 3 hours interpreting the data and writing each summary while report viewers waited.
Goals
Help report viewers understand performance data without having to ask.
Our goal was to deliver AI-generated summaries directly within the reporting workflow, eliminating manual requests and putting performance highlights within reach without leaving the report.
Solution
Generate a summary in one click, then copy or regenerate it.
Report viewers trigger the summary from the report they're already looking at. Once it's there they can copy it to use elsewhere, regenerate it if the report has changed, and rate whether it was useful.
Outcome
21.1% of report viewers generated their own summary within 90 days, and manual requests dropped.
21.1%
of unique report viewers generated an AI summary within 90 days of launch.
Against a 20% target.
67.2%
of the summaries generated were copied out for use elsewhere.
~50%
estimated drop in summary requests to the internal team, as reported by the SVP of Operations at a weekly company stand-up.
Understanding
How might we introduce AI summaries without disrupting the reporting experience?
Four requirements, and two open questions
Because this was a new AI feature within an existing reporting workflow, I began by clarifying the requirements and how the summary should fit into the experience. The PM's PRD defined four requirements:
📊 All Report Levels
Generate summaries at every reporting level.
🔄 Copy & Regenerate
Copy or regenerate summaries anytime.
👀 Continuous Report Access
Keep using the report with the summary open.
👍 Summary Feedback
Rate whether a summary is useful.
With these priorities established, I mapped the end-to-end user flow and identified two central design questions:
- How should report viewers initiate summary generation?
- Where should the AI-generated summary appear within the report?
Mapping the End-to-End User Flow
Exploration
Comparing interaction patterns for generating and viewing AI summaries.
I explored four approaches, building on existing product patterns where possible. I evaluated them against three criteria, in priority order: Can viewers keep using the report? Can they find the trigger? Can it ship in the required timeline? I added discoverability because the PRD defined the feature, but not how viewers would find it. The full scoring is in the table below.
Engineering review reshaped the shortlist
What engineering ruled out
The FAB was being retired, and a persistent drawer required major report-layout changes.
What I added — Sidebar
The engineering lead suggested using the existing side navigation, so I prototyped a sidebar concept.
Trigger labeled item in side nav
Surface panel inside the side-nav column
Switching the drawer to an overlay reduced the layout work but weakened its main strength: an overlay covers part of the report, so it's harder to keep using with the summary open.
| Rank | Approach | Report access | Discoverability | Ease of build | Rationale |
|---|---|---|---|---|---|
| 1new | Side navigation button → Sidebar | High | Medium | High | Uses an existing, labeled pattern and keeps report content accessible, but the entry point sits below the filters. |
| 2was 3rd | Icon button → Floating window | Medium | Low | Medium | Supports report navigation, but overlaps content and requires adapting the existing dialog component to support moving and resizing. |
| 3was 4th | Inline summary | Low | High | Medium | Easy to find and moderately complex to build, but scrolls out of view with the page. |
| 4was 1st | Icon button → |
Low | Low | High | Difficult to discover and limits access to the underlying report, though the overlay avoids the drawer's layout work. |
| Removed | FAB → Floating window | Medium | High | Medium | Removed because FABs were being phased out across the app. |
The sidebar ranked first until I tried to fit a text-heavy summary into a narrow column.
Collapsed
Benefit Every filter reachable, report untouched.
Tradeoff The summary cuts off mid-sentence.
Expanded
Benefit Summary readable, report still untouched.
Tradeoff Filters replaced — the report can't be changed.
Giving the summary room meant covering the filters, which undercut the reason the sidebar ranked first. I took both concepts into usability testing, each with a question: Was the sidebar readable enough to justify blocking the filters? Did the floating window's overlap meaningfully disrupt report use?
Testing
Testing the two strongest concepts: Sidebar vs. Floating window.
Due to the project's tight timeline, we were unable to recruit external users. Instead, I conducted a one-day, unmoderated Maze test with 35 employees who also use reports and review summaries in their work. Each participant completed the same task in both prototypes before rating ease and selecting a preference. I used their feedback to compare the two strongest concepts and treated the findings as directional rather than validation of external viewers' needs.
Floating-window Option
An icon in the toolbar opens a movable window over the report.
Sidebar Option
A labeled button under the filters opens the summary in the left column.
The floating window concept won
The floating window scored 6.5/7 on the Single Ease Question (SEQ), 0.9 points above the sidebar, and won the preference vote 20 to 14 (one participant had no preference). Feedback also revealed discoverability and rating-context issues, which informed two final changes below.
Before
The icon-only trigger was easy to miss.
After
Added a text label and introduced a new color palette for AI-powered features to improve discoverability.
An optional field adds context to ratings, helping the team improve future AI-generated summaries.
I had ranked the floating window below the sidebar. Two of my three objections to it didn't hold — a text label fixed discoverability, and participants cared more about readability than about what the window covered. Only the build cost survived.
Ranking got me to the right two. Testing told me which one.
Designing for the Model
I couldn't make the model right. I could make it checkable.
Placement was only half the problem. The PM's example summaries made claims about data elsewhere in the report. While the PM and engineering owned model quality, I designed interface responses for slow, stale, and failed output, and made cited metrics easier to check against the report.
Slow
The summary takes time to write.
Stale
The report changes underneath it.
Absent
It fails to arrive at all.
Wrong
It arrives looking perfect and is incorrect.
The interface could not guarantee that a confidently written statement was correct, so I focused on making its claims easier to check, in three places.
Problem
Report viewers may not know which claims they can check.
My approach
Bold every metric cited in the summary
Every cited metric follows this pattern, so bold marks what to check against the report.
I initially organized the summary into labeled cards for scannability. The PM pointed out that summaries were often copied into emails and presentations, where the structure would not transfer cleanly. We therefore shipped a text-only summary with key metrics in bold, giving the verification cues a better chance of carrying into those workflows.
Problem
The supporting data may be hidden behind the summary.
My approach
Let viewers move and size the summary off the data it cites
Problem
The report may change after the summary is written, so the two no longer match.
My approach
Flag outdated summaries when the report changes
Show which report the summary describes
The banner names both reports: the one the summary was generated from, and the one now open.
Keep the old text instead of clearing it
Clearing the summary might have seemed safer, but the summary remained accurate for the report it described, and viewers still needed to copy it. So we kept it available.
Waiting and failure
Like the outdated state, these keep the window's size and position stable so it doesn't shift over the report.
Generating
- Placeholder text previews the summary while the window and controls remain in place.
Fail to generate
- The window keeps its shape, explains the error, and offers a single Try again action.
Final Design
A summary viewers can find, keep in view, and check against the report.
The summary is one labeled button away, and stays in reach in a movable, resizable window. Bold metrics and clear generating, outdated, and failed states make it checkable, while one-click copy and optional feedback make it reusable.
Reflection
Rank the options after talking to engineering, not before.
I ranked four concepts before engineering review revealed a layout cost, ruled out a retiring pattern, and suggested a new option. Next time, I'd involve engineering before ranking, not after.
Design the moments around the output, not just the output.
At the start, I thought my job was to design how an AI summary would look once generated. As I worked through the experience, I realized the moments around the result mattered just as much: viewers would wait for it, sometimes encounter an error, and might keep reading after the report had changed. Addressing those states shifted my focus from presenting the output to making its status and context clear. Next time, I'd map those states before exploring the layout.
Next Steps
Validate with a broader user base and understand what drives or limits adoption.
Next, I'd test with external report viewers, since the initial study included only 35 employees. I'd also follow up on the 67.2% of summaries copied to learn whether the text-only format and bold metrics work in emails and decks or require editing. Finally, I'd investigate whether discoverability, trust, relevance, or lack of need kept other viewers from generating a summary.