Some hints from my approach to the problem
Of all conventions that define a musical genre – that is, of all regularities in the behaviour related to music events to which members of a community conform – naming conventions deserve special attention. In many historical cases I can think of, it seems that the naming of a genre is a kind of sanction, of ratification that other conventions exist and have been acknowledged. In such cases practice anticipated naming: that is, a general acceptance of styles, social practices, functions, etc., under a specific name, followed years, decades, maybe even centuries of similar music activities, as with fado, flamenco, tango, the blues, jazz, rebetiko, up to rock 'n' roll. Often, music historians wonder if a certain genre existed before any reference to its name could be found: I would suggest that in many cases some of the most relevant conventions defining a genre tend to operate before a name for the genre is agreed upon, but that the 'act' of naming makes other conventions more 'visible' and helps to create new ones. Though apparently pointing in the opposite direction, the way film historian Rick Altman describes the early development of cinema could also be interpreted in the light of the above-mentioned theoretical scheme:
During the last years of the nineteenth century and the first of the twentieth, it simply cannot be said that there was such a thing as 'cinema', clearly separate from other phenomena. On the contrary, what we now retrospectively think of as cinema was at the time recognized as several quite different phenomena, each overlapping with an already existing medium. This multiple identity is made manifest by contemporary terminology, which applied to moving picture projection such shared terms as 'views', 'pictures', 'electric theater', 'living photographs', and 'advanced pictorial vaudeville', each self-evidently identifying the new technology with an already existing medium (Altman 2004, 19).
In the case of cinema, names based on existing media polarized practices around each of a number of established conventions prevented the main common element (moving picture projection, i.e. 'cinematography') to emerge both as a name and as the activity around which new conventions could be formed. From this example we may concede that rather than identifying previously unnamed practices (and conventions), a new naming convention can, so to speak, rearrange the whole field, by deleting previous names. That is how 'cinema' worked (and, maybe, 'jazz'). Names are crucial (see Marino 2013).
However, while I would insist that a name may be a sufficient condition to define a genre (though a named genre that keeps together any kind of music events, without any kind of codified regularity among them, is hard to imagine), it is not a necessary one. A genre with no name may exist: that is, a set of music events that conform to some conventions established within a community, while the community itself has not given it (and/or does not want to give it) a name at all; unless we decide that 'music that does not belong to any genre' is actually a genre name, such a genre exists (a few festivals celebrate it every year in some European countries [1]). And, of course, many genres with a name probably existed without one (for a short or long period), before getting one.