A striking feature of vascular plants is the regular arrangement of lateral organs on the stem, known as phyllotaxis. The most common phyllotactic patterns can be described using spirals, numbers from the Fibonacci sequence and the golden angle. This rich mathematical structure, along with the experimental reproduction of phyllotactic spirals in physical systems, has led to a view of phyllotaxis focusing on regularity. However all organisms are affected by natural stochastic variability, raising questions about the effect of this variability on phyllotaxis and the achievement of such regular patterns. Here we address these questions theoretically using a dynamical system of interacting sources of inhibitory field. Previous work has shown that phyllotaxis can emerge deterministically from the self-organization of such sources and that inhibition is primarily mediated by the depletion of the plant hormone auxin through polarized transport. We incorporated stochasticity in the model and found three main classes of defects in spiral phyllotaxis - the reversal of the handedness of spirals, the concomitant initiation of organs and the occurrence of distichous angles - and we investigated whether a secondary inhibitory field filters out defects. Our results are consistent with available experimental data and yield a prediction of the main source of stochasticity during organogenesis. Our model can be related to cellular parameters and thus provides a framework for the analysis of phyllotactic mutants at both cellular and tissular levels. We propose that secondary fields associated with organogenesis, such as other biochemical signals or mechanical forces, are important for the robustness of phyllotaxis. More generally, our work sheds light on how a target pattern can be achieved within a noisy background.