Redesigning an internal survey authoring tool ·3m

fig. 01 — redesigned survey editor

project no. 02

Redesigning an internal survey authoring tool

I redesigned Pilotly’s internal survey authoring tool from a configuration-heavy interface into a document-style editor, closing the gap between how researchers draft and how they program.

Role

Sole Product Designer

Team

Joseph Audette, VP of Product and Engineering; James Norman, CEO & Co-founder; Najeeb Chowdhury, Director of Research Ops; 3-person engineering team

Timeline

December 2022 - present (ongoing migration)

Platform

Web-based survey authoring tool

Users

9-person research team building studies weekly, 6 person supporting research team, clients

Skills

User research, competitive analysis, UX design, system architecture, QA, phased launch

problem

Pilotly’s internal survey authoring tool was built around a configuration-heavy interface — modals, hidden detail views, and a container-driven architecture that didn’t reflect how researchers actually worked. Researchers drafted questionnaires with clients in Google Docs, then re-translated the document into the tool to program it, doubling their work and forcing them to context-switch between two completely different mental models.

solution

I redesigned the tool from the ground up into a document-style editor — think Notion for survey programming — and validated the direction with researchers across our four core user segments.

impact

new requested features

42% increase

in confidence from Q1 → Q2

71.4%

users rated a 3/5 or higher on performance

· summary

what is pilotly?

Pilotly is a SaaS market research platform that tests various forms of media, like TV shows, movies, and podcasts for major content studios like Amazon, Disney, Netflix. Our work spans title testing, rough cuts, episode binges, and more. Each format has its own survey type and reporting needs, and our researchers program every survey using the tool I designed.

The quality and speed of survey creation directly impact how quickly insights are delivered to clients, making it one of the most important features in our dashboard.

fig. 02 — survey editor product marketing video

· the problem

a tool researchers had outgrown

At Pilotly, researchers draft questionnaires with clients in Google Docs, then translate the draft into our internal tool, Survey Create, to program it. Collaboration lived outside the tool, programming lived inside, and every edit required reconciling both.

The interface didn't help — critical question details hidden in modals, container-heavy architecture, and several features that required engineering intervention to ship. The tool wasn't failing. It just wasn't aligned with how researchers actually worked.

fig. 03 — legacy “Classic” survey create

fig. 04 — several pain points & requests

· how might we?

framing the approach

How might we design a survey authoring experience that aligns with how researchers naturally draft while handling programming complexity in the background?

fig. 05 — why should we redesign the survey experience?

· false start

polishing the wrong model

To get a sense of how survey authoring tools function, I pulled apart Qualtrics, Typeform, and SurveyMonkey, analyzing user flows, information architecture, and feature sets.

My first iteration looked a lot like Qualtrics and SurveyMonkey — panels and containers replacing modals and hidden actions. After 1:1 reviews with researchers across our four core user segments, the synthesis was clarifying: yes, easier to use. No, it didn't fix the underlying workflow.

fig. 06 — first, safe redesign attempt

· the reframe

what our users told us

In an in-person designathon with the CEO and Head of Product, we landed on a different question:

What if researchers could upload a questionnaire document and have it automatically converted into a programmed survey that also looks like a doc?

I tested both the upload concept and the doc-like interface with the same researchers. The affinity map clustered around one theme:

Researchers didn't want AI to write surveys. They just wanted programming to feel flexible and familiar.

The real friction wasn't a missing automation. It was the gap between how researchers draft surveys and how they're forced to program them.

75% of our four core user segments preferred the document-style experience.

fig. 07 — hypotheses vs findings, post-UXR

fig. 08 — various user insights from UXR

· solution

programming as writing

We deprioritized the doc-upload flow and committed fully to the document-like interface. Think Notion for survey programming.

fig. 09 — redesigned survey editor experience

· the conflict

the autosave we shouldn’t have shipped

Engineering wanted to keep the save-button-per-question pattern from Survey Create. I was in document-world, where autosave felt essential. We shipped autosave on an architecture that wasn't fully stable and started seeing bugs — work not saving, errors appearing, editing behaving strangely.

We should have launched with manual save and layered autosave once we'd validated. The lesson wasn't about save patterns. It was that conviction has to come with humility — especially when the people closest to the system are telling you something.

fig. 10 — once upon a time, there was an autosave

· the current state

where it stands

Survey Editor is the new platform for every Pilotly survey. It unlocked methodologies the old tool couldn't support — MaxDiff, MaxDiff pathing, a new Highlighter question — and removed engineering as the gatekeeper for features researchers used to file tickets for, including a question-linking feature first requested in 2022.

fig. 11 — survey editor impact

· reflections

what i took with me

Workflow alignment is more powerful than fancy feature additions.

Stability must precede elegance.

The most transformative change wasn't the visual design or product decisions — it was respecting how researchers already think.

the end.

see my work →

read project no. 3