StudioAI agents for data onboarding

Bringing a new client's data into the platform took engineers about three weeks of manual work. AI agents could do most of it. The design problem was making that work something people could check, overrule and undo.

Result

3 wks → 2–3 days

to onboard a new client's data

  • RoleDesign Lead
  • Timeline10 weeks
  • WhenNov 2025 – Jan 2026
  • Team5 disciplines
  • UsersData & security engineers, platform admins
  • PlatformWeb app · internal + admin
Studio/Onboarding/Finance team · Qualys
Waiting for your review

DisagreementsThe agents disagree about where “Owner” belongs. Waiting for you.

Disagreement 1 of 1

Does “Owner” mean a person or a team?

What will change

Resolve 1 disagreement first
Added
+47
Changed
12
Connected
—
Removed
0
Every change can be undone.

Working recreation. Press L for the decision behind each part.Open “Design notes” in the footer for the decision behind each part.

NDA Interactive recreation with sample data. These are not production screens or customer data.

In 30 seconds

  1. 1

    The problem

    Bringing a new client's data into the platform took engineers about three weeks of manual work. When a source tool changed, things broke without anyone noticing.

  2. 2

    What I did

    I designed how people supervise AI agents doing that work: how sure each agent is, what happens when agents disagree, a preview before anything changes, and one-click undo. A person always makes the final call.

  3. 3

    The outcome

    Onboarding went from about 3 weeks to 2–3 days, with 25+ data sources supported at launch and 4 hidden breakages caught in the first month.

Jump toKey decisions Results

01The problem

Every product depends on this data

Prevalent's products all read from one shared map of a company's security data. Every tool a client uses (Qualys, Okta, CrowdStrike, ServiceNow) describes things in its own way, and each one has to be translated into Prevalent's model before any product can use it.

The ML team was already building agents that could do that translation. What nobody had designed was how people would supervise them. I owned the experience end to end: research with engineers, how people work alongside the agents, what happens when agents disagree, and how changes are previewed, approved and undone.

I worked out the agents' roles with the ML team before drawing any screens, so the interface only promised what the agents could really explain.

Constraints

Everyone shares the data
One onboarding can change thousands of records that every other product relies on.
AI isn't always sure
The same input can give different answers. The design had to be honest about uncertainty, not hide it.
Two audiences, one tool
Data engineers want every detail; security engineers want a decision they can make in seconds.
Client privacy
What the agents learn from one client can't be used for another.

Onboarding took weeks, broke silently, and only engineers could do it

Every new data source meant an engineer reading through it, writing the translation by hand and checking it several times. Onboarding a client took about three weeks. Security teams waited in a queue for data they needed now.

Worse, when a source tool changed how it sent data, things broke without warning. The damage showed up weeks later as a dashboard that looked wrong, and undoing it meant an engineering fix.

Agents could make the work fast. But fast, unchecked changes to data every product relies on are a bigger risk than slow ones. The brief had to change.

The briefAutomate the onboarding work.

The brief, rewrittenMake the agents' work easy to check, so people can trust it at the speed it runs.

02Research

Checking took longer than doing

Research ran alongside the ML team's early agents, so I could design around what the agents could actually explain, not what we wished they could.

How

Shadowing engineers
Sat with data engineers while they onboarded real data sources, noting where the time actually went.
Reviewing past breakages
Went through old tickets to see how things broke and how late people found out.
Sessions with the ML team
Worked out what each agent could show people: how sure it is, why it decided, what else it considered.
Concept tests
Tested versions of the disagreement card with engineers, using real cases from past onboardings.

What we learned

  1. 01
    Engineers spent their time checking

    Engineers spent more time checking the work than doing it. A faster agent alone wouldn't save the time.

    Which led to Confidence on everything the agents do, and a preview before anything changes.

  2. 02
    Silent failures cost the most

    The expensive breakages were the ones nobody noticed until a dashboard looked wrong weeks later.

    Which led to Ongoing checks that flag when a source tool changes, long after onboarding.

  3. 03
    One big check was too much

    Checking everything from an onboarding in one sitting was heavy, and it was the part most likely to be rushed.

    Which led to Review split into smaller sessions: data sources, onboarding and versions.

03Approach

Principles for this product

Show the work
If an agent can't say what it did and how sure it was, its work doesn't go through.
Disagreement is useful
When agents disagree, it tells you something about the data. It isn't an error to hide.
Everything can be undone
Speed is only safe when every change can be previewed before and undone after.

I designed how the agents work together before the screens

My first artefact was a diagram of the agents, before any wireframe: what each one does, what it hands on, and what it can explain. Product and ML agreed it with me.

Once that was clear, the interface got simpler: each screen shows one step of the work, and each button maps to something real a person can do: check, choose, approve or undo.

Tried and set aside

One "AI is working" status
Simple to show, but impossible to tell which part to trust.
Pass / fail checks
Hides the difference between something the agent is 65% sure of and 97% sure of.
Reports after the fact
Telling people what changed after it had already changed.

Sketches

Redrawn for this portfolio. The original work is under NDA.

  • Sketch of a disagreement card with two equal options, neither selected, and a locked Approve button.
    When agents disagree, both options get equal weight and nothing is pre-selected. Approve stays locked until a person decides.

04Decisions

  1. Decision 01Agents do the work, people decideWhat I chose Agents do the preparation. The final step, making the change, is always a person's.
    What I turned down
    Letting agents make changes on their own when they're confident enough.
    Why
    Every product relies on this data. One bad automatic change spreads everywhere before anyone sees it. A person approving each change is the simplest place to catch it.
    What it cost
    A person in every step caps the speed. Onboarding takes days rather than hours.
  2. Decision 02Always show how sure the agent isWhat I chose Everything an agent does shows how confident it is. Low-confidence items are checked one by one; the rest are approved together.
    What I turned down
    A simple pass or fail.
    Why
    65% sure isn't the same as 97% sure. Showing it points people at what actually needs attention; pass / fail gives false comfort.
    What it cost
    People can over-read numbers, so the cut-off line, not the number itself, decides what needs a closer look.
  3. Decision 03When agents disagree, a person choosesWhat I chose Disagreements appear as a card showing both options, each agent's reasoning and what would change. Nothing is pre-selected.
    What I turned down
    Quietly picking the more confident option.
    Why
    Disagreement means the data really is unclear. Hiding it behind a best guess just moves the mistake somewhere harder to find.
    What it cost
    Every disagreement takes a person's attention.
  4. Decision 04Preview before anything changesWhat I chose Before approving, people see exactly what will be added, changed and connected, and every change can be undone in one step.
    What I turned down
    A report after the change is made.
    Why
    Every product uses this data, so people need to see the impact before they approve, not after.
    What it cost
    One more step in every review, in exchange for never finding a bad change later.

Agents do the work; people keep the final say

  1. User needs & defining success
    Where the line sits is a design decision

    The confidence cut-off decides how many items people check one by one and how many they approve together. It's a trade between care and effort.

  2. Mental models
    Review follows how the work is really done

    Engineers already think in data sources, onboarding and versions. Review is split the same way, so each check is a focused task.

  3. Explainability & trust
    Every decision comes with its reasoning

    Everything shows how sure the agent was. Every disagreement shows both options and why. Every record shows where it came from and who approved it.

  4. Feedback & control
    Approve, overrule, undo

    People approve every change, choose when agents disagree, set the confidence cut-off, and undo any change with a preview of what will be reversed.

  5. Errors & graceful failure
    Problems are shown, not hidden

    Disagreements become a card instead of a silent guess, and changes in a source tool are flagged before they break anything.

  6. Data & evaluation
    Designed around what the agents can really explain

    Sessions with the ML team mapped what each agent could actually show before any screen promised it.

Grouped by the themes of Google’s People + AI Guidebook.

Deliberately left out

Agents acting on their own
The agents can do the work, but they don't make the final change. No automation without human approval is a product rule, not a launch compromise.
Learning across clients
Easy to build, hard to do responsibly. Held back for client privacy.
Free-text instructions to agents
Powerful but unpredictable. Replaced with structured choices.

Where data came from, and undo

Studio/History/Finance team onboarding
Approved · 47 records
Where each record came from
RecordCame fromPrepared byHow sureApproved
DEVICE Payments gatewayQualysClean-up97%○ Approved together
RISK Remote access flawTenableClean-up91%○ Approved together
POLICY Patch within 30 daysServiceNowMatching74%◆ Checked one by one
ACCOUNT Database service accountOktaMatching88%○ Approved together
DEVICE Customer databaseCrowdStrikeMatching83%○ Approved together

Every record shows where it came from, which agent prepared it, how sure it was and who approved it. Any change can be undone.

05Testing

One big check was too much, so review was split up

Because every change needs a person, the review itself became the bottleneck. The iteration was about making that human step manageable, not removing it.

  1. How review is structured

    1. First version

      One complete check of everything an onboarding produced.

    2. What testing showed

      Checking everything at once was too much to do well in one sitting.

    3. What shipped

      Review split into smaller sessions (data sources, onboarding and versions), each a focused check that ends with a person approving.

    First version: One complete check of everything an onboarding produced.What shipped: Review split into smaller sessions (data sources, onboarding and versions), each a focused check that ends with a person approving.
    Drag to compare. Sketches redrawn for this portfolio; the original work is under NDA.

06Results

From a three-week queue to a few days

  • 3 wks → 2–3 daysClient onboarding

    Time to bring a new client's data into the platform, down from about three weeks.

  • 25+Data sources at launch

    Supported through the same agent flow from day one.

  • 4Hidden breakages caught

    Changes in source tools caught in the first month that would have broken data silently before.

What I’d measure next

How often people overrule agents
Shows where the agents need to get better.
Quality of review
Whether people are really checking, or just clicking approve.

Keeping people in the loop is a design problem

Design the system before the screens
Agreeing how the agents work with ML first made the interface simpler.
Design the human step, don't just require it
"No automation without approval" is easy to write as a rule. Making it work meant splitting review into sessions people could actually finish.
Measure review from day one
I'd track how people review from launch, so the confidence cut-off is set with data, not guesswork.
Next: fix breakages in one step
When a source tool changes, suggest the fix and show its impact together, so it's one review instead of a ticket.