A-Z Index | Phone Book | Careers

InTheloop | 12.16.2002

The Weekly Electronic Newsletter for Berkeley Lab Computubg Sciences Employees

December 16, 2002

NERSC Helps Scientists Get Physics Out of Neutrino Detector Data

(Editor's note: This is an introduction to the full article, which

appears at the end of today's InTheLoop due to the length of the article.)

A U.S. research team gathered data from deep in a mine in Japan to
recently confirm that neutrinos oscillate and have mass. But they had to
tap into computers and high performance data storage systems in Berkeley
to get the science out of the data.

After gathering data for six months starting last January, U.S.
scientists running experiments at KamLAND, an underground neutrino
detector in central Japan, had 800 tapes containing a vast amount of
data. The problem was, there was no clear way to get the data from Japan
to computers in the U.S. for analysis. Thanks to a serendipitous set of
circumstances, computing experts in the NERSC Center were able to
provide the neutrino researchers with the right combination of

Magazine Article Highlights Berkeley Lab's Approach to Cybersecurity

LBLnet and Bro, the system for detecting network intruders in real time,
written by Vern Paxson and used by the Lab to monitor for cybersecurity
attacks, were featured in a Nov. 18 article in Government Computer News
magazine. The article is based partly on an interview with LBLnet's Mike
Bennett, who discusses how the Lab has increased bandwidth to meet
researchers' demand - and tuned Bro to continue providing protection as
network traffic grows by leaps and bounds. Bro is a key element is
keeping the Lab's communications network both open and secure. Read the
article at http://www.gcn.com/21_33/tech-report/20475-1.html.


Planning to Work during the Holiday Shutdown? Info Needed by Dec. 16

The Lab will be closed for the holiday break from Tuesday, December 24,
until Thursday, January 2, 2003. As in the past, the Laboratory needs to
have information regarding who will be working, the location of the
work, and the dates that our employees expect to work during this
period. This information needs to be submitted by close of business on
Monday, Dec. 16 (today).

If you expect to be on site over the break, and you are a NERSC or CRD
employee, please send your information to Zaida McCunney,
ZSMcCunney@lbl.gov. All other employees should send their information to
Susan Green, SLGreen@lbl.gov. Appropriate approvals will be obtained from
department and division management and forwarded to Facilities and


IBM Presents NERSC with Partnership Award

At last week's quarterly meeting of NERSC and IBM staff, Helene
Armitage, IBM Vice President of Technology, presented NERSC with an
etched glass award. The award cites LBNL, NERSC and IBM for
"accelerating Scientific Discovery through Advanced Computing."


Cisco's John Stewart to Speak at Jan. 9 Computer Protection Lecture

John Stewart, Lead Engineer at Cisco Systems and former president of the
World Wide Web Consortium (W3C), will be the featured speaker at the
January 9 Computer Protection Brown Bag in the Bldg. 50 auditorium from
noon to 12:45 p.m. A top-rated speaker, John will talk about the
distributed denial of service threat.

UNIX/Linux Security Hands-on Course to Be Offered Jan. 16

A new course, UNIX/Linux Security Hands-on, will be held on Bldg. 51L
from 9 a.m. to 5 p.m. Thursday, January 16. A description of this
course, which is designed to give attendees practical, hands-on skills
in securing their UNIX and Linux systems, is at
Because this is a hands-on course, space is limited. Visit
https://hris.lbl.gov/ to enroll. Enrollment will be on a first-come,
first-served basis.

Travel News: Effects of United's Bankruptcy, Changes in Boarding Pass Procedures

The following news is from the Lab's Travel Office:

United Airlines Update

Although United Airlines filed bankruptcy last week, most analysts agree
that passengers will not be greatly affected for at least the next few
months. In particular, no major changes are anticipated at San Francisco
International Airport, where United is the largest carrier. The airline
is also expected to continue allowing passengers to accrue and redeem
frequent flier miles. Eventually United will eliminate some routes and
reduce the frequency on others, however these will be planned reductions
and will be announced well in advance.

Transportation Security Administration (TSA) Changing Gate Check-In Procedures

The TSA is moving forward with plans to eliminate gate screening. As a
result, at some passenger security checkpoints you will now be required
to present a boarding pass and photo ID. Airline tickets and ticket
confirmations, such as a travel agent or airline itinerary, will no
longer be accepted at these checkpoints. This procedure is now in place
with the following airlines in the following airports:

  • Detroit: United
  • Los Angeles: American Airlines
  • Long Beach: Jet Blue
  • Minneapolis: United
  • New York Kennedy: Jet Blue
  • LaGuardia: Delta (not Delta shuttle), Northwest
  • Newark: Continental
  • St. Louis: United

Additional information and updates on new locations can be obtained from
the TSA website at http://www.TSA.gov.

To obtain a boarding pass use one of the following options:
1. Airline ticket counter
2. Self-service kiosk
3. Via the Internet (if your airline offers this option)

NERSC Helps Scientists Get Physics Out of Neutrino Detector Data

(Thanks to Lynn Yarris of the Lab's Communications Department for his news release on this research.)

A U.S. research team gathered data from deep in a mine in Japan to
recently confirm that neutrinos oscillate and have mass. But they had to
tap into computers and high performance data storage systems in Berkeley
to get the science out of the data.

After gathering data for six months starting last January, U.S.
scientists running experiments at KamLAND, an underground neutrino
detector in central Japan, had 800 tapes containing a vast amount of
data. The problem was, there was no clear way to get the data from Japan
to computers in the U.S. for analysis. Thanks to a serendipitous set of
circumstances, computing experts in the National Energy Research
Scientific Computing (NERSC) Center Division at Lawrence Berkeley
National Laboratory were able to provide the neutrino researchers with
the right combination of resources.

The experiment in Japan is called KamLAND, which stands for Kamioka
Liquid scintillator Anti-Neutrino Detector. Located in a mine on the
west coast of Japan's main island of Honshu, near the city of Toyama, it
is the largest low-energy anti-neutrino detector ever built. KamLAND
consists of a weather balloon, 13 meters (43 feet) in diameter, filled
with about a kiloton of liquid scintillator, a chemical soup that emits
flashes of light when an incoming anti-neutrino collides with a proton.
These light flashes are detected by a surrounding array of 1,879
photomultiplier light sensors which convert the flashes into electronic
signals that are collected for analysis on computers.

The experiments are generating about 200 gigabytes of data per day. The
amount of data would swamp the slow network connection between the site
and its research host, Tohuku University, so the data are stored on
tapes, then driven by car in a seven-hour trip to the university. There
the data were "read off" of the tapes and stored for analysis by the
Japanese team.

The U.S. team, which includes researchers from the Nuclear Science and
Physics Divisions at Berkeley Lab, UC Berkeley and Stanford, in addition
to 10 other universities, also needed a way to get the data from the
tapes so it could be analyzed in the United States. And while the
existing data were being processed, even more were backing up at the
research site. One of the challenges was that the data were stored on
"LTO" tapes, or Linear Tape Open, a new prototype format developed by a
group of vendors with the goal of providing interoperability between
various mass storage systems.

"By July we realized we had a big problem - we had no good solution for
where to put the data," said Patrick Decowski, a researcher in Berkeley
Lab's Nuclear Science Division who is responsible for integration of the
U.S. analysis software for the project. There was also the issue of
where to analyze the data once they were stored.

Enter the NERSC Center.

As part of a research project, IBM had loaned NERSC an LTO system for
reading tapes and transferring data. NERSC's Mass Storage Group had just
finished their evaluation of the system and were preparing to return it
to IBM when they learned of the problem. An agreement was reached, the
system was kept a few months longer, and Decowski and his crew began
using every minute they could.

"When they first asked us about using LTO, we said we could make it
available so they could read a tape or two," said Harvard Holmes of
NERSC's Mass Storage Group. "But it soon became clear that there was
more than a tape or two. We saw a line of tapes, but we couldn't see
where it ended."

Because of Internet bandwidth limitations, the tapes were FedExed from
Japan to the U.S. beginning in July, 50 tapes per box. In all, more than
800 tapes have been sent here - and returned to Japan - and data are
still being collected. But even after the data were read off by the
system, they had to be stored somewhere where they could be easily
accessed for analysis.

"As part of our research, we had developed software so the LTO library
could interface with our HPSS (High Performance Storage System)," Holmes
said. NERSC's HPSS system uses thousands of tape cartridges to store
data used by users of NERSC computers.

The neutrino group got really good at reading tapes and even wrote
software to automate the process. To avoid interfering with normal NERSC
operations, they used the system nights and weekends. In doing so, they
increased the data transfer in HPSS by up to 80 percent per day. One
weekend they managed to transfer 1.3 terabytes of data in one day. The
only problem was that HPSS was only configured to write one data stream
for their size files, limiting the write rate to about 1 terabyte per
day. The solution? HPSS was quickly reconfigured to write two data
streams for these size files.

Then another issue arose in September - the project was quickly filling
up all the available HPSS tapes and there wasn't money to buy new ones
until the next fiscal year. With some scrambling, new tapes were found
and the transfer continued.

The process was being squeezed on both ends: data continued to
accumulate in Japan and NERSC's loaner LTO system was only available
until the end of December. In the middle was a meeting scheduled for
September. The group wanted to analyze data in time to present results
at the meeting. Others in the field were eagerly anticipating the
results, which they hoped would confirm previous findings from SNO, the
Sudbury Neutrino Observatory in Canada.

The Parallel Distributed Systems Facility, or PDSF, is a 400-processor
cluster computer managed by NERSC for researchers in LBNL's Nuclear
Science and Physics divisions for a number of international experiments.
At the time the KamLAND data began to pour in, one of the largest PDSF
users was idle. With some quick negotiations, the neutrino group was
able to use PDSF to analyze the data, sometimes using the full
400-processor cluster.

"This really shows the value of having a shared resource like PDSF,"
said Shane Canon, a member of NERSC's Computational Systems Group who
manages PDSF. "If the system was dedicated to one project, we couldn't
have accommodated this request. But our flexible schedule environment
allows us to leverage the resource a lot better."

Decowski credited Canon and Iwona Sakrejda of the PDSF team with "doing
their best to make it all run as smoothly as possible. The work by
everyone at NERSC helped us out tremendously to get the physics out of
the data," Decowski said.

Meanwhile, the project continues to pour data into HPSS. Since July, the
group has moved 48 terabytes of data from the tapes from Japan. "This
has been satisfying and enjoyable to help a customer achieve their
objective - and to see the actual scientific results," Holmes said.

The KamLAND results, based on neutrinos generated by nuclear reactors in
Japan and South Korea, indicate that neutrinos have mass and can oscillate or
change from one type to another. These results provide independent
confirmation of earlier studies involving solar neutrinos at the Sudbury
Neutrino Observatory and show that the Standard Model of Particle
Physics, which has successfully explained fundamental physics since the
1970s, is in need of updating. The SNO results were also analyzed using
the PDSF cluster.

For half a century, scientists believed that neutrinos existed, but
believed that they had no mass and came in three types, said Stu Loken,
the General Sciences Deputy for Computing at LBNL. "Now we know that
they have mass and the three types are actually different states of the
same thing," he said.

"The neutrino results are unquestionably the most significant science
story of the 21st century," Loken said. "These results, which give us a
much more complete picture of the universe, wouldn't have been possible
without the support of NERSC."

According to Lee Schroeder, outgoing director of LBNL's Nuclear Science
Division, PDSF has proven to be a valuable workhorse for a number of
Nuclear Science research projects, helping scientists to stretch their

"It's really clear that to be able to carry out world-class science,
both in terms of simulations and analyzing data - computing is essential
to keeping us on the leading edge," Schroeder said. "Without PDSF, we
would not be in a leadership position."

Read more about the KamLAND scientific results at

About Computing Sciences at Berkeley Lab

The Lawrence Berkeley National Laboratory (Berkeley Lab) Computing Sciences organization provides the computing and networking resources and expertise critical to advancing the Department of Energy's research missions: developing new energy sources, improving energy efficiency, developing new materials and increasing our understanding of ourselves, our world and our universe.

ESnet, the Energy Sciences Network, provides the high-bandwidth, reliable connections that link scientists at 40 DOE research sites to each other and to experimental facilities and supercomputing centers around the country. The National Energy Research Scientific Computing Center (NERSC) powers the discoveries of 6,000 scientists at national laboratories and universities, including those at Berkeley Lab's Computational Research Division (CRD). CRD conducts research and development in mathematical modeling and simulation, algorithm design, data storage, management and analysis, computer system architecture and high-performance software implementation. NERSC and ESnet are DOE Office of Science User Facilities.

Lawrence Berkeley National Laboratory addresses the world's most urgent scientific challenges by advancing sustainable energy, protecting human health, creating new materials, and revealing the origin and fate of the universe. Founded in 1931, Berkeley Lab's scientific expertise has been recognized with 13 Nobel prizes. The University of California manages Berkeley Lab for the DOE’s Office of Science.

DOE’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.