Yes it can be done on a budget, although the more you get into it, the more alluring the expensive stuff is.
Even without a license, you can get started with some listening only stuff. For me, I picked up an rtl-sdr off Amazon for pretty cheap and started listening to NOAA weather satellites.
Before you start transmitting, you'll need to get your license. Hamstudy.org is a great place to start practicing for the exam.
Not sure how much money that is, but I would start a "quiet landscaping" company that used reel mowers instead of gas mowers, rakes instead of leaf blowers, and I would undercut every landscaping company in my town just to promote the idea of a quiet neighborhood.