B.J. Guillot (Home Page), email:
This page has been accessed times since August 17, 2003
As of 4/12/2008 this web site is being hosted on an Xbox using NanoHTTPD!
Unmodified Xbox Cluster
Paper on Xbox Cluster
- This was submitted to the ICPADS 2004 conference, but was not
accepted for publication.|
The Microsoft Xbox game console has an Intel Celeron 733 MHz processor (16K L1, 128K L2, 133 MHz FSB), 64 MB RAM, 8 or 10 GB of harddisk space, a 10/100 Megabit Ethernet adapter, a DVD/CD drive, four (non-standard) USB ports, and a 130 Watt power supply. New consoles can be bought in in the United States for $180.
Benchmark results: High Performance Linpack (HPL), 540 Megaflops/sec (single node), 1.4 Gigaflops/sec (four-node cluster).
Objective: Build an unmodified Xbox cluster. This means that the nodes in the cluster all are unopened: A mod-chip was never installed. The harddisk and memory have never been upgraded. Software-only exploits are to be used. Hot-swaps should be avoided.
|System||Microsoft Xbox||Sony Playstation 2|
|CPU||Intel Celeron/P3 733 MHz (133 MHz FSB)
16 KB L1 cache, 128 KB L2 cache
Supports MMX and SSE instructions
|"Emotion Engine", 300 MHz MIPS R3000 compatible|
Full MIPS-3 instruction set
with extensions from MIPS-4 and -5
Two Vector Processing Units (VPU)
|Memory||64 MB DDR SDRAM||32 MB RDRAM|
|Network||100 Megabit/sec||Sold* with Sony Linux Kit Not (100 Megabit)|
|Harddisk||8 or 10 GB||Sold* with Sony Linux Kit (40 GB)|
|Removable media ||DVD-ROM/CD-ROM||DVD-ROM/CD-ROM|
|Power supply||96 or 130 Watts||?|
|Special features||Front-panel programmable color LED
(Red, green, orange, off, blinking, etc.)
|Ports||Four USB (non-standard connectors,
requires special cable if you want to use them in Linux)
|Firewire (not accessible from Linux)|
|Extra Costs*||N/A||US$200 Sony Linux Kit (40 GB HDD, Ethernet
US$25 (Sony 8 MB memory card required for Linux)
|7/08/2003||Purchased my first Xbox.|
|7/10/2003||Successfully loaded BusyBox Linux into the Mega-X-Key USB memory card and used the 007 exploit to boot that distro on the Xbox.||7/13/2003||Ed's Debian loaded on Node 1 (using 007 exploit, DVD must remain in the tray).||7/14/2003||A huge mistake was made when installing the Bert and Ernie Reloaded font exploit. The default.xbe file was not signed with the "xbedump -font" prior to rebooting. This killed Node 1 with an Error 21 service screen.||8/02/2003||Node 1 had to be physically opened and a hot-swap procedure was used to fix the mistake. XLinux was burned from its ISO image to a CD-R using Nero on my XP laptop. XLinux boots on a regular PC and can read and write a FATX partition. I used this to put the correct signed file on the C: drive. I was unable to get HDD_Driver to work on Windows XP. Node 1 was repaired and I could boot game DVD's again.||8/03/2003||Succesfully got the font exploit to work--the 007 DVD no longer needed to be in the tray!||8/17/2003||
Purchased second Xbox. Node 2 placed into service.
Removed the discard, daytime, time, smtp, ident daemons.
Configured the NTP time servers, and set the proper timezone. Succesfully
developed a "clone" procedure that allows me to build nodes and have them online in less
than 30 minutes after removing the plastic wrap from new console out of its cardboard box.
Only two files must be manually
modified after the clone procedure (
Removed additional unnecessary daemons (inetd, lpd, apt-proxy). Received e-mail from
Dr. Paris indicating he had a thesis advising slot open and was interested in this project.
lm_sensors was already installed in Ed's Debian. Running "
||8/20/2003||Purchased third Xbox. Node 3 placed into service.||8/23/2003||
Purchased forth Xbox. Node 4 placed into service. The first three Xbox's had been
purchased from Circuit City. This one was purchased from Best Buy. It must have been
from an earlier batch because the Dashboard did not have the "Xbox Live" option
available (i.e., it was Dashboard version 4817). Having read that the exploits do
not always work on the older Dashboards, I went back to the store to purhcase any
Xbox Live-enabled game. The Live-enabled game will automatically update the Dashboard
version once you get to the "New Account" screen. I purchased "Mech Assault" since I
may be able to experiment with it later on with the newer exploits that have been
coming out. Remember that the 007: Agent Under Fire is not a Live-enabled game.
Anyway, I updated the Dashboard to 4920, and then used the same old 007: AUF exploit to
install the font exploit and Ed's Debian just like on Node 2 and 3. Discovered that
Worked on getting ssh configured so that I could rsh commands to the other nodes
from Node 1 without needing a password. Required learning how to use
Web surfing to do research on MPICH (a free implementation of MPI for Linux). Did
not actually install it. Planning to install it with the
Begin the install of MPICH on Nodes 3 and 4. An
||9/1/2003||Installed the ATLAS numerical libraries. First install was a pre-compiled version tailored for the Pentium 3 with 256KB L2 cache. I also installed the GOTO numerical libraries, but again, they were for machines with 256KB L2 cache. A custom version of ATLAS was build. The nice thing about ATLAS is that it is "self-configuring" and tunes itself for the specific machine it gets built on. The tuning process means that the build takes a very long time. HPL (High Performance Linpack) was run using MPICH and I measured a maximum performance using all 4 nodes of 1.4 Gigaflop/sec. This puts my 4-node cluster at a price-performance of 51 US-cents per Megaflop/sec.||9/6/2003||Found and compiled the source code for blink.c, a utility program for Xbox Linux that lets you have full control over the front-panel LED. The LED can be turned off, can be set green, orange, or red, or can be configured to continously switch between any of these states. A script was found that can be run as a cron job to monitor system load every ten minutes and change the LED status to represent load. I have not yet implemented this as a cron job, but may do so in the future. Other possibilities include changing the LED color depending upon measured CPU temperature. Recompiled ATLAS libraries with different compiler flags (using specific Intel compiler flags to optimize for the Pentium 3 and below). Re-running HPL on a single processor to see if I can do any better than 540 Megaflops/sec.||9/14/2003||Reports are starting to come in that Microsoft has released a new version of the Dashboard that plugs the holes the font exploits where using. If you connect to Xbox Live, apparently Microsoft will automatically upgrade your Xbox without your permission.||9/16/2003||
Found the source code to the
||9/17/2003||Installed Linux on Babu's Xbox at school. His machine was an older Xbox that had the original Dashboard. I had to use Mech Assault to upgrade his Dashboard to the Xbox Live-enabled version. We made sure that the Xbox was not connected to the internet when upgrading the Dashboard because we did not want to accidentally pick up the new exploit-unfriendly Dashboard. He is the Advanced OS class with me this semester and will be doing his project on Grid computing using Xbox'es. Once we get his machine configured, we will be able to run some benchmarks using MPICH-G. We will have to find embarrassingly parallel machines for this to work though since the communications link between my four nodes and his node will go over the internet. Both of us have a broadband cable modem connection using Time Warner.||9/17/2003||
Found a Unmodified Xbox Wiki and added the link to the bottom of the page.
It contains very useful information. One of the techniques described
described how to add extra "partitions". Before today, each node had
a 2 GB root filesystem. Now, on Node 1, I added another 2 GB file system
and moved all the home directories to it. Recall /home is exported to all
the other nodes via NFS, so I did not need to do the same procedure on any
of the other nodes. The extra space gives a little more breathing room
for Linux. Both the original and the new "partition" were formatted using
Hardware used: Four Xbox consoles. CyberPower 1500 AWR UPS. Linksys Etherfast 4124 24-port 100 Megabit switch. Mega X-Key USB memory card. (For Internet connectivity: Linksys BEFSR11 EtherFast Cable/DSL Router. Toshiba PCX1000 Cable Modem.)
Software used: EA's 007 Agent Under Fire DVD for Xbox. 007 distro of Ed's Xbox Debian GNU/Linux (version 0.3.1). xbedump. MPICH. HPL. Intel C++ and Fortran compilers for Linux. ATLAS.
Exploits used: 007 Agent Under Fire. Bert and Ernie reloaded.
Ed's Debian Xbox Distro: My cluster uses the 007-ized version of Ed's 0.3.1 distro, which has the 2.4.20 Linux kernel. Ed's 0.4.1 distro is now available, but an 007-ized version is not. The 0.4.1 distro adds features that are not really needed for my cluster (such as VNC server, change of X manager to gdm from xdm, etc.) They did upgrade the Linux kernel from 2.4.20 to 2.4.21, but that isn't that big of a deal. My Redhat 9 workstation is still at 2.4.20 too.
Custom changes to Ed's Debian: Removed automatic boot into X-Windows. Since
the cluster nodes are to be used "headless", it was not necessary to have
X (xdm and xfs) running. Removed 5 out of the 6 tty getty processors that
get started (commented out the lines in
The getty's are not needed because again, the nodes are headless.
Removed various unnessary daemon's (inetd, discard, daytime, time, smtp, ident, apt-proxy, lpd).
/etc/resolv.conf file to select correct DNS server.
link to point to
file to use NTP servers in my time zone.
Issues: Xbox must have an A/V cable plugged into the back of the unit, or it
will not boot. The A/V cable does not need to be connected to any other device,
so this is only a minor annoyance. Major issue: Most of the time Linux reboots,
the "clock loop" familar to many Font exploiters occurs. It does not always occur,
but sometimes it can take 10-30 minutes for the Xbox to get itself out of the loop.
This happens even if the boxes have had continuous power. It must be somehow related
to something either the font exploit or Debian Linux is doing to the clock. Once
the reboot loop stops, the hardware clock is set to 7/4/2003 (Xbox Independence Day).
This is not a showstopper because Ed's Debian is configured to use
ntpdate to automatically set the time from a close-by NTP server.
New techniques emerged in 2003 which enabled the use of Linux without needing a mod-chip. The "007" (Electronic Art's "Agent Under Fire") exploit appeared first and this allowed a user to run an unsigned XBE (Xbox Executable) file by using a buffer-overun bug in the "Load Saved Game" feature. The BusyBox embedded version of Linux was modified to run on the Xbox and people were able to use it to modify files on the Xbox harddrive. A similar buffer overrun bug was later found in Microsoft's very own "Mech Assault" game.
The BusyBox linux distribution is somewhat useful, but you really need something better. Since you had access to the Xbox hard drive, you could then install other versions of Linux, such as Ed's Debian Xbox distribution.
A little later, others found a way to get access to the Xbox hard drive without needing to use either the 007 or Mech Assault exploit. They used a dangerous trick called a "hot-swap". The hot swap exploit requires opening of the Xbox case (voiding the warranty) and disconnecting the Xbox's HD (while powered!) and then connecting the IDE interface to a powered-on PC. The hard disk power connector is never detached. Why did the Xbox and PC need to be running? Because the Xbox harddisk is "locked" with a password. Each Xbox HD has a unique password. Whenever the disk is powered up, you cannot read or write to it until the IDE controller unlocks the disk with the proper password. By doing a hot-swap, the disk never realizes that it was switched computers and stays unlocked.
July 4, 2003: Xbox Independence Day -- The first Font Exploit was released to the world. Now that you had the Linux files copied to the harddisk, you still had the annoying problem of having to use the 007 or Mech Assault exploit to jump-start Linux. The font exploit fixed that problem. You could now boot directly into Linux without needing to have any game DVD's in the tray. The first font exploit was known as the "Bert and Ernie" exploit. A second was called "Bert is Cheating on Ernie". A third was called "Bert and Ernie Reloaded". Several others came after these (various exploits known as "Big Fonts"). Why so many? There still exists a critical flaw with all these font exploits. If the Xbox clock becomes corrupted (which happens way too often, since its battery lasts only a few hours unlike a real PC clock), the Xbox can get stuck in a "reboot loop" whenever it gets powered on. The various exploits have attempted to solve this "reboot loop" issue with varying degrees of success. In my opinion, none of the font exploits really fix the reboot loop issue.
Other exploits followed such as the "Audio exploit". This was a safer way to jump-start Linux without needing a game DVD in the drive, but it required the user to punch a special sequence into the XBox hand controller, and required that a music CD disc be in the DVD tray. This exploit does not suffer from the reboot loop issue.
Xavier University Computer Science Summer Research 2003
5-node Microsoft Xbox cluster using mod-chips (did it over go online?)
Llamma's Xbox Beowulf Cluster
1-node Microsoft Xbox cluster using mod-chips (more nodes planned)
NCSA and University of Illinois Computer Science Dept
70-node Sony Playstation 2 cluster (amazing what you can do with a NSF grant!)
University of Illinois Chemistry Dept
Computational chemistry codes using PS/2 vector instructions
University of Manchester Computing Dept
Implementation of Grid middleware for PS/2