The site started to go downhill around that period because of power users and some started to move to reddit, but it was still pretty niche.
I stayed on digg until v4, then I moved with the masses over to reddit. They lost over 30% of their users that month!
The main migration was actually in 2010 after the v4 redesign.