They can be, in NZ there are is a huge amount of land that has been converted to dairy through massive irrigation schemes which has caused massive problems for the rivers that flow naturally through these places... I imagine there are other places in the world used for dairy that wouldn't be suitable if not for irrigation?
The almond example is frequently brought up, but this is still half of what dairy milk requires, without taking into account the difference in land use too