The term samba, originally used to describe generic musical gatherings of enslaved and freed Black Brazilians during the nineteenth century, became associated with a specific commercial genre of popular music in the 1930s. Samba, the genre, became one of the essential symbols of national identity and was celebrated for its ability to represent the fraternity between white and Black Brazilians. Historians have documented the constant musical exchanges between Rio de Janeiro's white and Black communities, as well as the development of non-segregated neighborhoods where self-manumitted Africans and poor European immigrants lived in shared spaces. Musical styles that later became associated with samba were shaped in such spaces, and white and Black musicians often performed together in events or ensembles. Yet, Rio de Janeiro's periodicals shows that audiences continued to perceive distinct forms of the genre based on race. Drawing from archival research that includes dozens of periodicals from Rio de Janeiro and commercial recordings of music from 1900-1930, this paper examines how audiences and musicians in the city reinforced or challenged racialized notions of sound and music. I identify practices of white musicians, critics, and audiences that narrowly defined samba as a strategy aimed at racializing music and its black practitioners. The work of whiteness was characterized by discourses and practices that reinforced notions of white superiority, not only by rejecting specific aural elements that were perceived as black, but also appropriating those elements while attributing its "improved" character to white musicians. Black musicians, on the other hand, made constant artistic interventions that asserted samba's historical connections to Black aesthetics. These interventions did not result in an essentialized expression of blackness. Rather, Black music constantly defied models, and maintaining a commitment to improvisatory practices, allowed Black musicians and communities to explore the many expressions of their lived experiences. This paper builds on Nina Eidsheim's concept of the Race of Sound to argue that "race-ing" sound as "black" is a historically contingent act, that can be used either as a strategy to maintain racial hierarchies, or as a way to reinforce connections with African diasporic practices.