You need to get the USB device permissions right. There's been a bit of confusion about this. Here's what I do.
I created /etc/udev/rules.d/51-android.rules. It contains:
SUBSYSTEM=="usb", ATTRS{idVendor}=="0bb4", SYMLINK+="android_adb", GROUP="plugdev"
After that, a quick /etc/init.d/udev restart as root and plug the device in.
0bb4 identifies my physical device, a HTC G1. It's the vendor ID you see when you lsusb:
Bus 001 Device 010: ID 0bb4:0c02 High Tech Computer Corp. Dream / ADP1 / G1 Phone (Debug)
This gives all users in group plugdev carte blanche to do stuff with the phone.
Previous advice had the udev rule set MODE=0666, giving access to everyone. It appears this rule no longer works.