Behind the Design: StaffPad
https://devimages-cdn.apple.com/wwdc...D1D4/1024.jpeg
Great composers bring forth melody from the silence of their imaginations. Long before the orchestra strikes its first chord, an intermixed whisper of invisible strings, brass and percussion lives within a composer’s mind. Transforming those melodic machinations into sheet music, however, is its own special challenge.
The classical greats relied on the tried-and-true method of writing on sheet paper, letting their creativity flow unabashed onto the page. Once written, however, each part had to be painstakingly copied and reproduced for each musician — and revisions were far from easy.
In contrast, modern digital composition software provides an incredible array of tools for composers and musicians. Despite the many benefits of these apps, however, they can come across as cold and complicated, creating unnecessary barriers between the music in someone’s mind and the notes that end up on a music stand.
“I was spending more time learning these notation programs... than I was actually writing the music,” composer and designer David William Hearn tells us. This dissonance provided him with the beginnings of a different sort of musical idea: an app that provided the inspirational and inviting structure of pen and paper with the powerful features of a digital notation app.
“[It] started out as quite a selfish endeavor,” Hearn shares with a smile. “The dream app that I always wanted.” He envisioned something that would not only allow him to write music quickly and with ease, interfacing and interacting with other musicians, but also presented the opportunity to record and playback.
https://devimages-cdn.apple.com/wwdc...4A5E/1024.jpeg
With the help of cofounder and collaborator Dr. Matthew Tesch, Hearn’s dream app became StaffPad, an innovative approach to musical notation. Since its 2015 launch, StaffPad has continually evolved to take advantage of the latest technology, including a full rebuild and relaunch for iPad and Apple Pencil in early 2020.
“Musicians and composers generally appreciate power through simplicity,” says Hearn. “They want to focus on one thing only — the music. Every aspect of StaffPad’s design, both invisible and visible, was built with a no-compromise approach to enable that focus.” Subtle design choices create a feeling of unity between the composer’s environment and the interface as they transition into the app. The main screen’s background even changes color based on the weather and time of day to provide a welcoming atmosphere for creativity.
The paradox of choice is fascinating, but personally I feel being offered a couple of great choices beats having loads of potentially poor ones. You do need bravery in the design to believe the few choices you offer are the right ones.
David William Hearn, cofounder of StaffPad
The composition interface is free-form and inviting. The team specifically decided against the idea of a fixed page layout, knowing that people could use StaffPad on a device in any orientation or zoomed at any size. Instead, the app presents composers with an infinitely scrolling digital canvas of sheet music. Rather than ask the creator to pick specific styles or sizes before writing a new composition, they can just start notating, then share the correct size on export.
“There's certainly a place for having perfect pixel, perfect layout control over every layer,” Hearn says. “But whilst you're in a creative flow, you really don't want to be thinking about page breaks or line breaks.”
https://devimages-cdn.apple.com/wwdc...CAAD/1024.jpeg
While StaffPad is quite powerful in the options it provides composers, the app intentionally uses adaptive toolsets to help declutter its interface and guide composers to the right tool at the right moment. “We designed the most common things you do to be the most natural,” Hearn says. If a composer is writing for piano, for example, the app won’t display tools designed for percussion instruments.
That includes notational writing itself. StaffPad requires Apple Pencil — its precision is instrumental to interacting within the app. While the team considered alternative tools, they ultimately went with the methods that mapped accurately to how someone might write on paper. Pencil gives the composer differentiation, whereas touch is a complement to the overall process. “Touch to move. Pencil to draw. We make it very nuanced, but the dual concept is the key,” Hearn says.
That exploration also helped identify one of StaffPad’s defining features: toggling the erase tool through drawing pressure. They knew it would break the creative flow to add an extra button to toggle the erase tool. Instead, they relied upon an otherwise unused feature in the app — Apple Pencil’s pressure sensitivity, which isn’t needed for writing notation — and designed an erase toggle when pushing firmly on the Pencil. “I knew it had to be light enough that it didn’t feel dangerous, but also strong enough that it didn’t trigger accidentally while writing,” Hearn says. “We kept trying until it felt right."
Another key feature of the app is its text recognition, which translates handwritten strokes or shapes into digital notes, rests, accidentals, articulations, and dynamics. "It's actually consulting its own sort of music-theory brain,” Hearn says. Even with a scattering of scrawled notes, StaffPad is able to draw upon Core ML frameworks and models to decipher and craft digital orchestrations.
Despite the app’s ‘selfish’ origins, StaffPad has found support from people across the musical world. “One of my favorite things about the app is that it's fostered this sense of kind of community ownership,” Hearn says. “People are really quite emotionally attached to it in a lot of ways.”
Hearn and team receive regular suggestions, feedback, and stories from their community. His favorite emails are those from teachers and students: “That raw inspiration that comes from the education sector is really amazing.”
https://devimages-cdn.apple.com/wwdc...8CDE/1024.jpeg
StaffPad continues to explore new ways to support its audience, including its companion app, StaffPad Reader. The app allows musicians to digitally view and play scores from StaffPad, and even automatically turns pages in unison as they rehearse or perform. If the composer edits or transposes a score on StaffPad, the app also receives changes over the air, reformatting the parts in realtime.
Hearn and Tesch have even begun the process of introducing another natural input interface into the app: voice. “I believe voice will become a primary input method for most apps in the near future,” says Hearn. “Voice provides a vector into a nearly unlimited command set — and one that we already know how to use.”
Winning an Apple Design Award is just one note on the team’s musical journey. “There’s always more to do,” Hearn says. But over the last few years, he’s found joy in helping craft both the app and working with the team behind it. “A composer is a solitary animal,” he notes. An app-building team, less so. “[My] main learning has been how best to work to create a cohesive design and then explain that to a team of people.”
Learn more about StaffPad
Learn more about StaffPad on the App Store
Download StaffPad on the App Store
More...
