Those washers are the Caster washers. They come in different sizes. They are required to be there otherwise your caster alignement will be off. With the caster not set correctly this will put extra loading on the ball joints during turning. Caster is the alignment of the upper ball joint with relation to the lower ball joint.
You can take it to an alignment shop and have the caster re-aligned. The washers come in 1, 3, 6, 9, and 12mm thicknesses. So not only do you need the washers, But you need the correct size in the correct location.
