Trey Harris

Senior Systems Engineer — UNIX/Linux, Perl, Architecture and Software Integration

Email for full contact information:
trey+resume@lopsa.org

Highlights

Work History

April 2003 - Present

Senior Systems Engineer
Amazon.com
Website Systems Engineering/Infrastructure Automation
Seattle, WA

Senior engineer in retail website team responsible for designing available and frugal solutions based on commodity hardware and open-source software for Amazon.com, Amazon.co.uk, Amazon.co.jp, Amazon.de, and other associate and partner sites. Primary engineer for website media serving platform, comprising ~500 hosts in a complex system and network architecture. Automation programmer writing software running on the entirety of Amazon's Linux fleet. Accomplishments include:

  • Dramatically increased overall availability of media platform, from under 98% (an average of 9.3 customer-impacting events per month) to 99.9% (no customer-impacting events for over 200 days)
  • Engineered and executed live datacenter migration, transparently moving retail operations from a soon-to-be-decommissioned datacenter to a complex of redundant datacenters on the opposite coast
  • Designed and executed company-wide transition of mission-critical API from software load-balancing to hardware load-balancing (on Cisco Content Switching Modules), dramatically improving network efficiency and application performance
  • Implemented edge-caching of SSL secure-image service to reduce bandwidth costs and increase availability
  • Implemented layer-7 routing (on NetScaler load balancing equipment) to improve cache hit rates and reduce hardware costs
  • As rotating primary on-call for all Amazon.com retail and partner websites, performed problem determination and resolution of revenue-impacting issues under extremely high production pressure.
  • Designed, implemented and deployed a several site-wide automation projects eliminating manual operational tasks, to reduce operational load and free resources for other priorities.
  • Taught Perl training classes and assisted in code reviews and software design companywide.
September 2001- November 2002

Associate (Senior UNIX Engineer)
Morgan Stanley
Global UNIX Engineering
New York, NY

Worked in the senior team of ten engineers with final responsibility for the production UNIX infrastructure of a major Wall Street firm, encompassing over 10,000 machines running a heterogeneous mix of operating systems under the most stringent mission-critical conditions. Accomplishments include:

  • Design, implementation and delivery of a network-boot (PXE) infrastructure for the Linux plant which saved the firm over $500,000 and thousands of administrator man-hours in its first year of use.
  • Architecture and preliminary API/runtime implementation of a next-generation software distribution system enabling 72 global datacenters to access production software seamlessly while adhering to stringent policy requirements.
  • Oversight of a pilot program to replace Sun desktops used by developers with commodity Linux desktops, including implementation of an automated build and configuration system.
  • Work on Morgan Stanley's Perl Core Engineering team, maintaining one of the most complicated production Perl installations in the world.
January 2001- August 2001

Senior Systems Engineer
VA Linux Systems, Inc.
New York, NY

Provided technical field support to current and prospective financial-sector and research customers in both a pre-sales and post-sales capacity. Prepared technical presentations for prospective buyers, installed and customized computing clusters on site, and presented the “local engineering face” for a Silicon Valley-based company. Routinely configured and customized multi-million-dollar cluster installations.

October 1999- December 2000

Systems Administrator (Automation Programmer/System Architect/Training Coordinator)
Mail.com, Inc.
New York, NY

Performed general system administration and systems architecture tasks, and worked on training and education issues. Served as a member of the company's systems architecture group. Made substantial contributions to a major redesign of the company's mail storage architecture, including algorithm design for high availability requirements. Developed an in-house introductory Perl class. Taught class to over 50 employees over six weeks, with 50 hours total teaching time.

June 1994-
September 1999

Systems Administrator (Principal System Architect)
Project Isis, Academic Technology and Networks
University of North Carolina at Chapel Hill

Designed and implemented the centralized campus electronic mail system, Isis, and expanded it into the distributed computing infrastructure for the entire campus. Starting in 1994 with around 5,000 users and a few gigabytes of data on two machines, this system became the core of UNC's distributed computing infrastructure, with over 50,000 users, hundreds of workstations (both UNIX and Windows NT), dozens of servers (including three large Sun Enterprise 10000 / SGI Origin 2000-class compute engines) and dozens of terabytes of disk and HSM storage. Technical lead on AFS migration project and on a radical redesign of the IMAP backend, moving over one million mailboxes using custom software written for the task with zero loss of data.

1993-
1994

Info Wizard/Web Wizard
Office of Information Technology
University of North Carolina at Chapel Hill

1993-
1994

Freelance Graphic Designer

1993

Undergraduate System Administrator
Department of Computer Science
University of North Carolina at Chapel Hill

1992-
1993

Teaching Assistant
Department of Computer Science
University of North Carolina at Chapel Hill

Skills and Training

Platforms

General UNIX: some prior experience in Solaris, BSD, AIX, IRIX, HP-UX; strong current experience with Linux and MacOS X

Languages

Fourteen years experience programming system automation tasks in Perl; seven years teaching Perl. Competent in Java, KornShell, Ruby; some experience in PHP3, Python, C, C++, Haskell, Objective-C and Eiffel. Web programming in CGI.pm, Mason, and Ruby on Rails. Contributor/meta-committer to Pugs, the Perl 6 compiler project.

Network Protocols

Extensive knowledge of Internet TCP/IP protocols, including SMTP, IMAPv4rev1 (member of IETF WG), POP, ACAP (member of IETF WG), HTTP, DNS, Kerberos, SNMP. Some experience with Cisco and NetScaler network load-balancers.

Internet Servers

Cyrus Mail Server (IMAP/POP--significant work on internals), MediaWiki (some work on internals), Sendmail, Apache, SSH, CVS, Subversion, MySQL, standard UNIX servers; OpenAFS servers; experience writing TCP/IP clients and servers in Perl and Java, using standard and special-purpose protocols. Experience developing centralized system and performance monitoring tools.

Twenty-four days of tutorial work at USENIX Annual Technical Conferences, System Administration Conferences (LISA), and SANS, on subjects including Perl, UNIX programming, network programming, Java, Sendmail, and Linux and Solaris internals. Five-day Andrew File System (AFS) course from Transarc. Five-day Red Hat kernel internals course. Twelve days of advanced Perl training with Damian Conway. Attendance at LISA, USENIX, and YAPC conferences.

Publications, Presentations and Software Projects

Ongoing

Contributer and meta-committer to Pugs, the Perl 6 compiler project. See http://www.pugscode.org/. Contributions include unit tests, release engineering, test/documentation integration, documentation writing and editing, and Perl 6 language and object model design.

November 2006 (forthcoming)

“Get Ready for Perl 6.” Three-hour tutorial at Phoenix SysAdminDays.

April - July 2006

Internal Perl training course. 20-hour course covering all aspects of Perl was taught to 30 employees. Amazon.com, Seattle.

July 2005

Technical reviewer for Perl Best Practices by Damian Conway, O'Reilly 2005 (acknowledgment, page xxi).

November 2004

Commands::Guarded. Open-source Perl module implementing a deterministic, rectifying variant of Dijkstra's guarded commands in an accessible way to improve system-modifying scripts. See http://search.cpan.org/~trey/Commands-Guarded-0.01/Guarded.pm.

November 2004

“A New Approach to Scripting.” Invited talk, 18th Annual USENIX System Administration Conference (LISA '04), Atlanta. Presented Commands::Guarded.

November 2002

“The case of the 500-mile email.” This widely-forwarded email details a strange troubleshooting incident earlier in my career. Reprinted in different form in Peter van der Linden's Just Java, 6th ed., p. 680. http://www.ibiblio.org/harris/500milemail.html

July 2002

“Fluent Perl.” Article on improving Perl coding style. http://sagewire.sage.org/article.pl?sid=02/07/11/187241

February 2002

“Ask Mr. Know-it-All.” ;login: magazine, February 2002. Humorous article on the use of the term “system administrator.”

March 2001 (scheduled)

“Open-Source Email for the Enterprise.” Three-hour tutorial. Comdex, Chicago.
Note: Comdex organizers cancelled the entire track one week prior to the conference.

December 2000

“Users and Passwords and Scripts, Oh My!” Session chair, refereed papers, 14th Annual USENIX System Administration Conference (LISA 2000), New Orleans.

September - October 2000

Internal Perl training course. 18-hour course was taught to 50 employees in three versions, for beginning, intermediate and experienced programmers. Mail.com, Inc., New York.

November 1999

“Look, Ma, no hands! Coping with Repetitive Strain Injury.” Invited practicum talk, 13th Annual USENIX System Administration Conference (LISA 99), Seattle.

August - December 1998

“Despite Popular Opinion.” Weekly political commentary column, the Daily Tar Heel, Chapel Hill, NC.

December 1998

Panelist, “University Issues Panel”, 12th Annual USENIX System Administration Conference (LISA 98), Boston.

September 1995 - December 1998

Birds-of-a-Feather (BOF) session presenter, “IMAP” (1997 & 1998), “The zillion-line /etc/passwd” (1995 & 1996), 9th - 12th Annual USENIX System Administration Conferences (LISA 95-98) in Monterey, Chicago, San Diego, and Boston.

November 1996

“Selling IMAP to Users.” Talk at the Second International IMAP Conference (under auspices of IETF IMAP Working Group), Seattle.

October 1996

Work-in-Progress (WIP) presenter, “Prop: a scalable system for maintaining and distributing user and group information.” 10th Annual USENIX System Administration Conference (LISA 96), Chicago.

January 1996

“IMAP at the University of North Carolina.” Talk at the First International IMAP Conference (under auspices of IETF IMAP Working Group and University of Washington Computing & Communications), University of Washington, Seattle.

Other Professional Activities

Education

August 1992 - May 1999

Part-time work towards B.S. in Computer Science / B.A. in Linguistics, the University of North Carolina at Chapel Hill

Completed major requirements in Computer Science and Linguistics; remaining was perspective requirements (fine arts, history, etc.). Computer science work includes algorithms, architecture, graphics, and software engineering; linguistics work includes formal syntax and semantics; languages studied include two years each of Russian, Spanish, and Japanese.

Awards

Chancellor's Information Technology Award, the University of North Carolina at Chapel Hill, 1995.
Eagle Scout Award, 1990.