After lots of hacking at FISL 7.0 in Porto Alegre, I've managed to get the PXA27x USB device controller to work in USB Ethernet emulation to work. I can now actually ping and telnet to the 2.6.16.5-running E680, using a debootstrapped Debian/ARM on SD-Card.
I'll publish the patches in one or two days, when everything has stabilized a bit, and the debugging code has been removed.
Also, at the event here, I've managed to convince quite a number of Free Software people that those Linux smartphones are actually quite interesting toys. Most notably, Keith Packard of Xorg fame has indicated he would probably be getting one and working on a lightweight UI. This motivates me even more to have a stable and fully working kernel environment finished soon.