I had that as well. Didn't feel any real benefits/effects from it and also no abstinence symptoms but drank coffee occasionally because of taste.
Then my high blood pressure was discovered, I was put on medication and since then I feel the rush of caffeine and unfortunately the headache after withdrawal as well.
I have the same coffee drinking habits, and someone challenged me to give it up for a week to prove I'm not addicted. I didn't have any symptoms but I was surprised to find I genuinely craved a hot drink several times a day. Hot water with lemon filled the gap until the week was over.
I love coffee for the taste! I do often get decaf.
In my case I simply don't see the need to quit. I don't see it having a negative effect on my life and I don't believe it to be particulary unhealthy either.