Net connection drops a lot.

First off my system specs:
CPU: Intel i5-750
RAM: 4GB RAM
Integrated Ethernet: Realtek 8111DL
OS: Windows 7 Ultimate 64-bit
Cable Modem: Cisco Scientific Atlanta DPC2100R2
Router: Linksys WRT54G
ISP: Roadrunner

So my net connection drops a lot. Usually 3 or more times a day and sometimes as often as every 2 hours. To get it to come back on again I have to unplug the modem & router and plug it back in. Sometimes I have to restart my computer too but not all the time.

When I get a dropped connection and run a Diagnose/Troubleshoot problems
in Windows 7's Network and Sharing Center, I get a message that says that there is an interruption between the modem, router and PC. And to unplug the modem and network devices and plug it back in, essentially what I've been doing.

At first I thought the problem was because of the cable modem, but I just got a new one today and I'm still having the same problem, so now I'm thinking its the router, which is almost 4 years old now. But before I go out and buy a new router I want a second opinion to confirm its the router and not some other issue I've overlooked.

Also When the connection drops it affects the host and client PCs the same. The interesting thing is that if i play an online video game and my connection "drops" while im playing, I'm still able to stay connected to the game server and play. If I alt+tab and open my browser, no pages will load. if i disconnect from that server and try to search for another server to join, it won't work.

Another example: There is a drop in the connection. The Host Computer can't load any web pages in the browser. Client Computer A can't load any web pages in the browser. Client Computer B was already connected to a game server before the drop and still is connected after the drop, but can't load any web pages in the browser.

This leads me to believe it could also be a DNS problem and if it is, how do I fix it?

If it is the router instead is there someway I could fix it maybe, tweaking some of the configurations or a firmware update?