Installing Linux on a G5 ClusterNode XServe
Now that I got this decent new dual G5 box, I wanted to install Linux. This turned out to be an extremely difficult job, as apparently nobody has ever tried to install Linux on any of the new XServe G5 Series machines, neither 32bit nor 64bit kernels.
There are a number of challenges:
- No internal IDE or SCSI CD-ROM
- Only serial console
- A very new hardware with little Linux support
First I tried a number of ready-built installation ISO images, including the current sarge Debian-installer image for PPC, and the 32bit and 64bit live images of Gentoo.
The first thing I had to do is to disable autoboot and enable the serial console. Luckily, the box actually ships with a manual that instructs you how to put the OF boot console on the serial port. You have to press the admin (!) Button at the front of the box a magic number of times.
To permanently make the serial console work, use the following OF commands:
> setenv input-device scca > setenv output-device scca
Next I had to figure out how to boot from the external firewire cdrom.. apparently this depends on your OF device tree and the GUID of your firewire device. On my particular box it works with
> devalias cd /ht/pci@5/firewire@e/node@00d04b3c50090210/sbp-2@c000/disk@0Using Commands like
> dir cd:,\I was then able to list files on the CD-ROM. To boot the yaboot loader on a Debian installer cd image, you can use
> boot cd:,\install\yaboot sbp2:Open ->login? speed=ffffffff 2 2 load-size=239a4 adler32=a5cf5aa0 Loading ELF sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 Config file read, 2907 bytes sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 \ sbp2:Open ->login? speed=ffffffff 2 2 Welcome to Debian GNU/Linux sarge! This is a Debian installation CDROM, built on 20040729. The default option is 'install'. For maximum control, you can use the 'expert' option. If the system fails to boot at all (the typical symptom is a white screen which doesn't go away), use 'install video=ofonly' or 'expert video=ofonly'. The plain options are for the powerpc family of processors (from 601 to G4). The *-power3 options are for IBM Power3 boxes, and the *-power4 options are for IBM Power4 and Apple G5 boxes. Press the tab key for a list of options, or type 'help' for help. ************************************ If in doubt, just choose 'install', and if that doesn't work, try 'install video=ofonly'. ************************************ Welcome to yaboot version 1.3.12 Enter "help" to get some basic usage information sbp2:Open ->login? speed=ffffffff 2 2 boot:I tried all of the provided images, with different options - no success. A common option to be used because of the serial port is "console=ttyS0,57600". All I got was:
boot: expert-power4 Please wait, loading kernel... sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 Elf32 kernel loaded... copying OF device tree...done starting cpu /cpus/PowerPC,G5...failed: 00000000 Calling quiesce ... erasing fff06000 of Micron B1 part flashing fff06000 of Micron B1 part swapping blocks DO-QUIESCE finishedreturning 0x01400000 from prom_init
Playing with the Gentoo live cd images didn't bring me any further at all.
I then tried to compile a current 32bit ppc 2.6.8-rc2 kernel by hand (for G5 CPU's). Putting this kernel on the debian installer ISO didn't get me any further. So apparently either the serial port is not working, or the kernel crashes somewhere.
Using a cross-compiler running on my dual G4 PowerMac, I compiled the same 2.6.8-rc2 kernel for ppc64 target platform. Putting this on the debian boot cd helped a lot, I now got it as far as:
boot: expert-g5-64 console=ttyS0,57600 Please wait, loading kernel... sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 sbp2:Open ->login? speed=ffffffff 2 2 Elf64 kernel loaded... Looking for displays OF stdout is : /ht@0,f2000000/pci@3/mac-io@7/escc@13000/ch-a@13020 Opening displays... Calling quiesce ... DO-QUIESCE finishedreturning from prom_init Found U3 memory controller & host bridge, revision: 53 Mapped at 0xe000000080000000 Found a K2 mac-io controller, rev: 96, mapped at 0xe000000080041000 PowerMac motherboard: XServe G5 Starting Linux PPC64 2.6.8-rc1 ----------------------------------------------------- naca = 0xc000000000004000 naca->pftSize = 0x17 naca->debug_switch = 0x0 naca->interrupt_controller = 0x1 systemcfg = 0xc000000000005000 systemcfg->processorCount = 0x2 systemcfg->physicalMemorySize = 0x20000000 systemcfg->dCacheL1LineSize = 0x80 systemcfg->iCacheL1LineSize = 0x80 htab_data.htab = 0xc00000001f800000 htab_data.num_ptegs = 0x10000 ----------------------------------------------------- [boot]0100 MM Init [boot]0100 MM Init Done idle = native_idle Linux version 2.6.8-rc1 (laforge@dathomir) (gcc version 3.4.1) #4 SMP Sat Jul 31 16:12:42 CEST 2004 [boot]0012 Setup Arch via-pmu: Server Mode is disabled PMU driver 2 initialized for Core99, firmware: 0c nvram: Checking bank 0... nvram: gen0=204, gen1=205 nvram: Active bank is: 1 Adding PCI host bridge /pci@0,f0000000 Found U3-AGP PCI host bridge. Firmware bus number: 240->255 Adding PCI host bridge /ht@0,f2000000 Can't get bus-range for /ht@0,f2000000, assume bus 0 U3/HT: hole, 0 end at 9fffffff, 1 start at b0000000 Found U3-HT PCI host bridge. Firmware bus number: 0->239 Can't get bus-range for /ht@0,f2000000 PCI Host 0, io start: fffffffffd800000; io end: fffffffffdffffff PCI Host 1, io start: 0; io end: 3fffff Top of RAM: 0x20000000, Total RAM: 0x20000000 Memory hole size: 0MB On node 0 totalpages: 131072 DMA zone: 131072 pages, LIFO batch:16 Normal zone: 0 pages, LIFO batch:1 HighMem zone: 0 pages, LIFO batch:1 [boot]0015 Setup Done Built 1 zonelists Kernel command line: ro debconf_priority=low devfs=mount,dall init=/linuxrc console=ttyS0,57600 PowerMac using OpenPIC irq controller at 0x80040000 [boot]0020 OpenPic Init OpenPIC Version 1.2 (4 CPUs and 120 IRQ sources) at e000000082ccd000 OpenPIC timer frequency is 25.000000 MHz [boot]0021 OpenPic Timer [boot]0022 OpenPic IPI [boot]0023 OpenPic Ext [boot]0024 OpenPic Spurious [boot]0025 OpenPic Done Slave OpenPIC at 0xf8040000 hooked on IRQ 56 [boot]0020 OpenPic U3 Init OpenPIC (U3) Version 1.2 [boot]0025 OpenPic2 Done PID hash table entries: 16 (order 4: 256 bytes) time_init: decrementer frequency = 33.333333 MHz Console: colour dummy device 80x25 Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) Inode-cache hash table entries: 65536 (order: 7, 524288 bytes) Memory: 498688k available (3840k kernel code, 4120k data, 212k init) [c000000000000000,c000000020000000] Calibrating delay loop... 66.56 BogoMIPS Mount-cache hash table entries: 256 (order: 0, 4096 bytes) PowerMac SMP probe found 2 cpus Processor 1 found. Synchronizing timebase Got ack score 299, offset 1000 score 299, offset 500 score 299, offset 250 score 299, offset 125 score 299, offset 62 score 299, offset 31 score 239, offset 15 score -107, offset 7 score 101, offset 11 score -5, offset 9 score 63, offset 10 score -51, offset 9 Min 9 (score 5), Max 10 (score 87) Final offset: 9 (61/300) Brought up 2 CPUs NET: Registered protocol family 16 PCI: Probing PCI hardware U3-DART: table not allocated, using direct DMA PCI: Probing PCI hardware done PCI: no pci dn found for dev=0001:04:0f.0 Apple Computer Inc. K2 GMAC (Sun GEM) PCI: no pci dn found for dev=0001:05:0c.1 PCI device 1166:0240 (ServerWorks) SCSI subsystem initialized usbcore: registered new driver usbfs usbcore: registered new driver hub nvram_init: Could not find nvram partition for nvram buffered error logging. rtasd: no RTAS on system VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) devfs: 2004-01-31 Richard Gooch (rgooch@atnf.csiro.au) devfs: boot_options: 0x1 Installing knfsd (copyright (C) 1996 okir@monad.swb.de). Initializing Cryptographic API pmac_zilog: 0.6 (Benjamin HerrenschmidtSo apparently, there were some issues finding the OpenFirmware dn (distinguished name) for the Ethernet Chips and the ServerWorks chips. I tried to put some printk's into the arch/ppc64/pci_dn.c file to see what's going on. This then led me to the earlier error messages about the U3-DART. After reading some more code, it appeared like the DART is Apple's IOMMU, and it is supposed to be needed only when running with >2GB RAM. My box had 512MB, but I tried to force usage of the DART by putting "iommu=force" into the kernel commandline.) ttyS0 at MMIO 0x80013020 (irq = 22) is a Z85c30 ESCC - Serial port ttyS1 at MMIO 0x80013000 (irq = 23) is a Z85c30 ESCC - Serial port RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize loop: loaded (max 8 devices) sungem.c:v0.98 8/24/03 David S. Miller (davem@redhat.com)
Great, this was apparently the problem, since now I got up to the point where it wanted to mount the root filesystem. I thought I didn't really need an initrd, since the kernel contained all drivers statically linked in. However, Debian installer seems to be running inside initrd only.
First try was just using one of the pre-supplied initrd.gz images. Yes, they have the wrong versions of the modules - but I don't want/need those modules anyway.
Of course this wouldn't work either:
RAMDISK: Compressed image found at block 0 Kernel panic: VFS: Unable to mount root fs on unknown-block(0,0) <0>Rebooting in 180 seconds..No errror message, nothing. So I thought the problem is with devfs, and I tried passing several different root parameters ('root=/dev/ram', 'root=/dev/rd/0') without any success.
In the end I found out that the structure sizes of the cramfs superblock (include/linux/cram_fs_sb.h) are arch-dependent, so I cannot use an initrd that was built on a ppc32 machine. Unfortunately it is also endian-dependent, and at this time I only have 32bit big endian and 64bit little endian boxes at home.
Next step was to use an ext2 initrd, since reasonable filesystems don't have any strange host/byteorder/wordsize dependencies.
Now it is able to load the initrd, and mount it... although then some other stuff goes terribly wrong. No time yet to investigate this.