Desert soil has poor topsoil and hardly any organic material in it.
This is because regularly watered soil provides the perfect environment for certain bacteria, bugs and other organisms to grow which in turn provide quality topsoil.
In a desert with very little rain, the perfect environment is never produced, which results in a lot of minerals but poor topsoil for any quality agriculture to take place.
