{en} Please leave this page in english.

The Tyan Transport Server GX28 (B2881) - uses a Tyan K8SR (S2881) mainboard and can optionally be equipped with a SMDC 3289 server management daughter card. This page is about running Linux on it. It also applies to any system using just the 2881 motherboard in any case or blade, except for info specific to the case.

There are 2 versions of it, one has 4x SCA-SCSI (and a AIC7902 onboard SCSI controller, all HDDs attached to one channel), the other 4x S-ATA bays.

Hardware

Working:

• dual Gbit Ethernet, module: tg3
• lm sensors, (TODO: attach config from Tyan's web site here)
• highmem with 6GB of RAM (4x1GB, 4x512MB, PC3200 ECC DDR. No problems are expected with more, though)
• add-on card: 3ware 9500S-LP4 quad S-ATA
• onboard: Silicon Image SIL3114 quad S-ATA - with kernel 2.6.7+ all 4 channels seem to work (although the gentoo notes state that only first 2 channels work). Hotplug does not work due to missing code in kernel 2.6.9 driver.
• onboard (only SCSI version): Adaptec AIC7902 controller (should work), with optional ZCR DIMM module for RAID (unknown)
• PXE
• WOL. From soft-off only. Enable "wake on PME#" in BIOS.
• USB 1.1 front and rear ports. OK for USB storage. ohci_hcd module. Boot from USB CDROM
  • Does it have any INTERNAL usb ports?
  • Good support for booting from USB storage, as a floppy or hard drive image. Can even boot from a USB card reader with a card in device #4 (/dev/sdd in Linux).
• Rear standard RS232 9pin port. A bit hard to plug in as some metal slot part is in the way. There is also in internal only COM2.
• Slimline CDROM is connected to hdc P-ATA port (hda port is there, but unused). See nice trick below.
• The server also has a standard internal parallel port, no connector on the rear though. So maybe nice for some hw hacks.

BIOS

• Tyan's v2.04 BIOS puts the IOMMU apperture above 4GB, making it useless.
• With a 32bit kernel, you'll have to disable ACPI2.0 or the "multimedia timer" in the BIOS, because 2.6.8-1-k7-smp will hang once it finds the HPET.
• Tyan and AMD recommend leaving the HT link speeds at 800MHz between CPUs, and 600MHz to PCI for stability.
• USB can not be used for keyboard as emulation in BIOS is missing.
• Fan regulation in BIOS / hw tends to chaotic oscillations sometimes when set to do 0..100%. Better with 15..100%.
• BIOS serial port redirection works, blanks show as _. The misc. redirection options that can be chosen after BIOS behave strange, but we got LILO also working over serial console.

Booting

BIOS, LILO and kernel SW RAID

BIOS and LILO are very happy with booting from any hard disc. The leftmost disc is sda, the rightmost sdd.

We made a 2 disk linux kernel software RAID1 config with LILO's RAID mode and it boots from:

• good array, using the sda mbr (of course)
• first disc, second failed (also not special as it uses sda mbr in that case, too)
• first disc failed (in our case: simply unplugged), second OK, uses the seconds disc mbr but scsi maps it to sda (as the real sda isn't there)
• boots from any disc in any bay when the discs left of it are unplugged - the BIOS seem to boot from the leftmost disc it finds and LILO is also happy with that.

Couldn't be better!

Ejecting mounted CD media

Booted from CD media, installed linux and now want to boot from hard disk? No problem, just use:

eject -a1 /dev/hdc

At least with kernel 2.6.7..9 this remembers to eject the medium when the device is closed. And as it is a slimline (notebook type) kind of CD drive, it won't pull it back when resetting the EIDE hardware on BIOS POST (just not possible with that hardware). So it will boot from hard disc...

Kernel 2.6

• ...

Tips & Tricks

• Strange effect: when plugging a Powerware 5125 UPS into the serial port, kacpid takes 100% CPU on a 32bit 2.6 kernel. 64bit 2.6 kernels crash, or oops in kacpid during boot.
• PCI-X cards: the board has two independent PCI-X busses. The gigE is on the same bus as the slot marked PCIX2, so it might be better not to put a 33MHz-only card in that slot. OTOH, it's still a 64bit bus, and a 3Ware 9500S-LP4 can read at 160MB/s on a (33MHz) bus shared with an eepro100.
  • not all PCI cards fit into those slots:
    • the left slot needs the PCI card plugged in component side down - which makes it impossible to plug in cards with unusual component height (like e.g. the LSI Megaraid 320-2X controller), because they collide with mainboard components (like the unused left PCI slot).
    • the right slot is better regarding component height, but the slot hole in the case is not large enough for unusual cards like the LSI Megaraid 320-2X controller, which has 2 VHDCI external SCSI ports. This maybe can be fixed by a case modification.

Gentoo

• Gentoo AMD64 booted out of the box in SMP mode. tg3 module got the network working and except some small hickup on iputils (just retried and it worked), all went ok. -- ThomasWaldmann 2004-10-28 21:26:27

SMDC 3289

If you get this thing working under Linux, please add to this section. Or maybe just add what you tried (and whether it worked or not).

What we all want is a reliable serial console emulation, and remote power off or reboot. Using the windows client software is acceptable as long as there is no linux client solution.

What do we need on the server side? Kernel modules? Daemons?

What do we need on the client side?

Setup

Card can be screwed on the metal plate in the front area of the PC. The too short serial cable will make one card orientation possible ONLY. The serial cable connects to COM2 header at back left of S2881, the other end connects to the gnd9 connector on the SMDC (near Li battery), not to the gnd5 type connectors. The 2x25 flat ribbon cable connects to a connector well hidden under the power supply / SATA cables.

A major annoyance is that ARP on the SMDC is switched off by default (and nobody tells you). If you use the latest stuff, which is of course not on their public ftp and also not on the CDROM that did not come with the card. But after nagging them, one gets a running SMDC by:

• using S2881 BIOS 2.04b10 or 2.04d

• use b57diag -b57eng to:

  • seprg -fb336a228.bin (flash the broadcom firmware)

  • power cycle

  • asfcfg to set IP of "client" (that means the SMDC in that case!), netmask and IP of management PC

  • power cycle

• using util.exe /all to:

  • flash the SMDC with a firmware 1.74 (made to be compatible S2881 board)

  • set IP (should be already correct by default as the SMDC gets it from Broadcom NIC)

  • set MAC to be identical with the left NIC (when looking from front of machine), should also be correct

  • set gateway

  • switch broadcast ARP on (1) (on SB2881~1.mhx Version 1.74 doesnt seem to appear such a function)

  • set passwords for the SMDC users

W2K setup (just to get it working, later more on linux)

• current Java 1.4.02_06 works as good as the 1.4.1 stuff they offer on their CD

• install v1.6 TSO server and TSO console, use PC's ethernet port's IP for both external and internal

  • I had lots of trouble with older 1.5x version of that software. The TSO console never found the TSO server, although it was running on the same PC.

• when you start the TSO console, also give that same IP, Administrator and the Administrator password of Windows!

• then TSO console logs you in, opens a new window
• add your server manually, see menu (and use same IP for both hostname and IP address of your SMDC)
• now double click that new server (see lists on left panel)
• the login data it wants now is for the SMDC (you configured them with util.exe hopefully)

  • it happened to me that that login password somewhat got lost, so if it doesn't work as expected, just try with empty name and empty password

  • now, either it logs you in and you can power on/off or reset the machine,

  • or it says something about failed with just means that something unspecific didn't work (can be anything)

• working ok: power on/off/cycle, reset. warm start of smdc. didnt try cold starting it.
• working (f*cking slow (just wait some minutes....), but working): reading sensor data and events
• not working (yet): BIOS or DOS or linux console redirection (key presses get sent to machine, but BIOS screen not visible at TSO console, just black)
  • looks like SMDC is sending screen data to IP 0.0.0.0 8(

Ethereal

This should help anyone trying to extend ipmitool to do the Tyan flavour of serial over Lan

I don't know how to stop it or change the serial settings or anything similar.

The card creates some broken packets addressed to 0.0.0.0 in addition to the correct ones.

Note: I have only tried this using the Tyan Console on the registered Manager IP (from broadcom flash tool) so far. I'm not sure what will happen if any other settings are used.

Start Serial Console:
NetFn: OEM Request (0x30)
Response Lun: 0x00
Rq Addr: 0x80
Seq: 0x0
Lun: 0x0
Command: 0x06
Data: network byte order ip (4 bytes)
      network byte order udp port (2 bytes)
      null byte (0x00)

      7 bytes of data.

Data comes as UDP packet containing

4x null bytes
   data
6x null bytes

Send Keystrokes:
NetFn: OEM Request (0x30)
Response Lun: 0x00
Rq Addr: 0x80
Seq: 0x0
Lun: 0x0
Command: 0x03
Data: number of bytes inc this one (byte)
      characters...

It looks as though it may not be too difficult to alter the isol part of ipmitool so that it starts a console?

What a weird setup!

TODO: http://www.broadcom.com/drivers/driver-sla.php?driver=570x-diag

IPMI links

• OpenIPMI - kernel driver for the IPMI stuff: http://openipmi.sf.net/ (included in new 2.6 kernels it seems - I just built it in)

• IPMITool - Userland tool for IPMI over openipmi or ipmi over lan.: http://ipmitool.sf.net/

I have installed these, they *DO* work for some commands. If you are using gentoo, you will have /dev/ipmidev/0. You'll want to add 3 lines to /etc/devfsd.conf:

eg. ipmitool lan print 2 is quite interesting. ipmitool arp respond on doesn't work with the firmwares I have (2.04, 1.71)

It works!! (With unreleased firmwares)

ipmitool -I lan -H hostname -U Administrator -a chassis power on

Patch to support some OEM features for ipmitool. ipmitool-1.6.0-tyan.diff

• We are still experimenting with the stuff, as e.g. console redirection still is not working. Also we have the impression that it is wrong how they do it in that latest firmware, using a separate MAC and IP for the SMDC would be maybe much less troublesome than how they do it now (same MAC addr).

I am a bit annoyed with tyan about this, I mean there's no way that this could have been working with any OS. I was explicitly told that the card worked with the S2881 and that's why I went forward with the purchase. Even just being able to reboot the box and see the sensor/event log remotely would be great, but of course without the console it's hard to see if something is going wrong pre-boot. I put my support mail in to tyan when I discovered it wasn't arping (which was a lot easier to discover when using the linux Manager app) -- update - Tyan sent me working files!

I bought me some days ago two different Tyan Dual Opteron Boards. S2882 and S2881 both with SMDC Cards. On the S2882 everything works fine, but only with SMDC Firmware Version 1.69 (I am using openIPMI) But on S2881 I have given up testing. I got two S2881 I Patch Bios to 2.04B Version. SMDC Firmeware 1.74 Patche the Broadcom Bios set the IP with asfcfg and enabled Wake on LAN in NIC Bios. on the one Board everything wirks fine too but on the second I alsway recieve the error Message, that "no LAN connection could have been established"...