What is machine-learning and how does it help players learn guitar with Notey’s World? As an NYU student currently studying game design, I sat down with co-creator Adithya Bellathur to learn more about the innovative audio engine created for music education app Notey’s World. 

According to Adithya, the premise is simple: "Our game uses the microphone to detect the fundamental frequency of the notes being played and translates them into game actions.” 

The “game actions” refer to the way players can control the titular character, Notey. As you play notes on your real world instrument, Notey jumps across the platforms on each level. It’s satisfying to play - ramping up with difficulty as the game teaches you more and more notes.

As easy as it is to play as a user, it gets more complicated behind the scenes. The audio engine is an attachment that works seamlessly with Unity (the game development platform that Notey is built upon). It utilizes Unity’s physics engine to enhance visual and audio feedback. Adithya goes on to explain that the system works by combining machine learning techniques llike convolutional neural networks (CNNs) , and traditional audio signal processing like spectogram analysis. The goal of the audio engine is to identify when a new note is played and what the note is. It can both detect chords and individual notes, as a player plays it 

During our discussion, Adithya reflects on the development journey. "Back when we first started I was still in college. So this was a very rough  concept between Mak (author’s note: Notey’s CEO and other co-founder) and I. I did research for about 3 or 4 months to get an idea of how this would work. It took me about a month and a half to build the first version.” He built the audio engine entirely by himself. Since then, Team Notey continues to improve their technology,and the team is looking to expand further with a custom level builder, allowing users to make levels from any song.

Adithya emphasized its importance stating the engine is "at the core of everything we do. It's how we can create games based on audio, and it serves as the primary medium for user interaction."

He expressed excitement about the most recent update to the app, "We are particularly thrilled about this build's potential to teach beginners effectively, offering fully tutorialized level progressions. I can't wait to see absolute beginners try it out and receive the support they need as they learn."

Looking ahead, the team has ambitious plans. A roll out to Android is on the horizon. They plan to integrate new instruments. Additionally, Adithya shared a plan for multi-user input capabilities, “using the phone camera  to identify multiple players for group settings.” 

To check out the machine-learning audio engine and the latest update, you can try out Notey's World on the App Store for free.