Very high caffeine tolerance (weirdly: sometimes not) as a non-regular coffee drinker

If I just had high caffeine tolerance I'd guess it's either because of fast caffeine metabolism or my ADHD. Sometimes a regular cup of coffee is more than enough to wake me up and improve my ability to concentrate and sometimes not even a 4x Espresso is enough, with no discernible reason why. I have tried drinking coffee on an empty / full stomach, which doesn't seem to make a difference. Any idea why this might happen?

Edit: Things that might matter:

• I am getting enough sleep
• I have normal body weight (BMI slightly below 24)
• I would consider my diet balanced, although I do like eating fastfood sometimes