If you've never worked in a sysadmin role or in any other position that would require an on-call rotation, then you may have never seen PagerDuty.
In essence the service provides a great series of integration points with Pingdom or Nagios for handling monitoring. As an infrastructure guy (part time), I can honestly say it's a great tool and I'm grateful to PagerDuty for supporting Jenkins with our own account to help manage project infrastructure.
A couple weekends ago I finished setting up Nagios (read-only username/password:
jenkins) for critical project services which by itself is a good step forward. Combine that with PagerDuty's Nagios integration and a solid on-call rotation, and I'm more confident than I've ever been that Kohsuke or myself could actually take a vacation!
Check them out, and be sure to thank them on Twitter at @PagerDuty for supporting Jenkins!