Over a third of crops are grown to feed livestock, and that's if you're not counting pasture as a crop, which it absolutely is - arguably our first solar powered factory floor. Even areas that were grazed in the past have had the relative proportion of native flora and fauna severely reduced to minimal levels through introduced grasses and overgrazing. To get a feel for land use against calorie production, you could have a browse through https://www.nationalgeographic.com/foodfeatures/feeding-9-billion/ for an overview.
Food is grown specifically to feed livestock though, it would be a pretty weird trophic pyramid for them to survive on our waste unless you went back to a time where people killed their one pig for the year and salted it away. In our country, the land degradation from clearing hill country for grazing has led to enormous biodiversity loss and a self-fufilling prophecy of eroded weak topsoil that people claim isn't good for anything else (though it could still be rewilded and in other cultures and times would be terraced and swaled to support plant crops).