We explore the nonlinear dynamics of a cavity optomechanical system. Our realization consisting of a drumhead nanoelectromechanical resonator (NEMS) coupled to a microwave cavity allows for a nearly ideal platform to study the nonlinearities arising purely due to radiation-pressure physics. Experiments are performed under a strong microwave Stokes pumping which triggers mechanical self-sustained oscillations. We analyze the results in the framework of an extended nonlinear optomechanical theory and demonstrate that quadratic and cubic coupling terms in the opto-mechanical Hamiltonian have to be considered. Quantitative agreement with the measurements is obtained considering only genuine geometrical nonlinearities: no thermo-optical instabilities are observed, in contrast with laser-driven systems. Based on these results, we describe a method to quantify nonlinear properties of microwave optomechanical devices. Such a technique, now available in the quantum electromechanics toolbox, but completely generic, is mandatory for the development of schemes where higher-order coupling terms are proposed as a resource, like quantum nondemolition measurements or in the search for new fundamental quantum signatures, like quantum gravity. We also find that the motion imprints a wide comb of extremely narrow peaks in the microwave output field, which could also be exploited in specific microwave-based measurements, potentially limited only by the quantum noise of the optical and the mechanical fields for a ground-state-cooled NEMS device.