We extend the stochastic formalism to the setup of the models containing U(1) gauge fields, such as anisotropic inflation and Primordial magnetogenesis during inflation. In this talk, we show that the stochastic noises of electromagnetic perturbations significantly affect the system dynamics. These effects can not be neglected even in the leading order in contrast with the stochastic noises of the scalar fields. We revisit the model of axion inflation in the context of stochastic inflation and investigate the effects of the stochastic noises associated with the electromagnetic fields.