|
Agenda Item
ASR
Control 12-000563 |
||
|
MEETING DATE: |
04/17/12 |
|
|
legal entity taking action: |
Board of Supervisors |
|
|
board of supervisors
district(s): |
All Districts |
|
|
SUBMITTING
Agency/Department: |
County Executive Office (Approved) |
|
|
Department contact
person(s): |
Mahesh Patel (714) 834-2525 |
|
|
|
KC Roestenberg (714) 834-4409 |
|
Subject:
Orange County Data Center
Emergency Projects
|
ceo Concur |
County
Counsel Review |
Clerk of the Board |
||
|
Concur |
N/A |
Discussion |
||
|
|
|
4/5 Vote |
||
|
|
|
|
||
|
Budgeted: Yes |
Current Year Cost:
$847,040 (not to exceed) |
Annual Cost:
$40,000 (not to exceed) |
||
|
|
|
|
||
|
Staffing Impact: No |
# of Positions:
|
Sole Source:
N/A |
||
|
Current Fiscal Year Revenue: N/A |
||||
|
Funding Source: Information
Technology ISF 289 |
||||
|
|
||||
|
Prior Board Action: N/A
|
||||
RECOMMENDED ACTION(S):
|
1. |
Find that the proposed project is exempt from the provisions of CEQA pursuant to Section 14 CCR 15269(a) of the CEQA Guidelines. |
|
2. |
Declare an emergency condition exists pursuant to Public Contract Code §22050(a)(1)(2). |
|
3. |
Find pursuant to Public Contract Code §22050(a)(1)(2) that the emergency work is necessary to avoid a threat to public health and safety and to maintain ongoing information technology services supporting all essential public services; and that this emergency does not permit a delay resulting from undertaking a competitive solicitation for bids; and that this action is necessary to respond to the emergency. |
|
4. |
Authorize the OC Public Works Director or his designee to negotiate and award an Agreement with Myers, Houghton & Partners, Inc. dba MHP, Inc. in an amount not to exceed $75,000 to provide Architect-Engineer Services for design and construction support for the Emergency Project: Data Center HVAC 2012 Upgrades. |
|
5. |
Authorize the OC Public Works Director or his designee to negotiate and award an Agreement with West Coast Air Conditioning Co., Inc. in an amount not to exceed $677,000 to provide and install new plumbing and electrical infrastructure for the Project as it is designed by Myers, Houghton & Partners, Inc. for the Emergency Project: Data Center HVAC 2012 Upgrades Project. |
|
6. |
Set a review by the Board of Supervisors every 14 days to determine continuing need for emergency action persuant to California Public Contract Code Section 22050, subdivision, (c). |
SUMMARY:
The Orange County Data Center has reached a seasonal threshold where the cooling capacity is no longer adequate to meet the building’s needs. In order to address the anticipated lack of cooling capacity over the summer months, CEO IT requests your Board’s approval to award a contract on an emergency basis to Myers, Houghton & Partners, Inc. dba MHP, Inc. to provide Architect-Engineer Services (A-E) services and construction support for the Emergency Project: Data Center HVAC 2012 Upgrades.
In addition, CEO IT requests your Board’s approval to award a contract on an emergency basis to West Coast Air Conditioning Co., Inc. to provide and install new plumbing and electrical infrastructure for the Emergency Project as designed by Myers, Houghton & Partners, Inc. (MHP) for the Data Center HVAC 2012 Upgrades Project.
BACKGROUND INFORMATION:
In 1991, the County commissioned the County Operations Center (COC) also known as the County Data Center, including approximately 22,200 square feet of dedicated computer rooms to house the County’s central computer operations. The facility has served the County well over the years thus far without any major capital reinvestment in cooling systems.
Modifications were made in 2005 and in 2009 expanding the computer rooms to provide space for the installation of additional servers. Servers have a high heat load. At the time cooling upgrades were made by installing additional computer room air conditioning (CRAC) units. These units are used to directly cool the computer server racks and use chilled water from an in-facility central plant as the source to provide cool air.
With the higher computing capacity of today’s computer servers it is necessary to increase the supportive capacity of collateral systems; electrical and cooling. Computing electrical capacity at the data center (and the subsequent heat) has increased about 23% over the past twelve months; this pace is faster than anticipated for two reasons: demand increased faster than contemplated and the pace of decommissioning older servers was slower than anticipated (newer servers are much more energy efficient). It should be noted that the data center has a seven year capital plan that contemplated $1,500,000 in capital refurbishment to enhance cooling capacity in FY 2012 / 2013 and FY 2013 / 2014. The County collects over $3 million annually in revenue from ACS, LA County, CalOptima, the State of California, etc. Note that most of these contracts have reciprocal six month notice provisions wherein the County can notify the tenants to vacate upon six months’ notice.
There is a long history of focus on cooling capacity at the data center. After submitting a $1.5 million capital request in FY 2007 / 2008 to upgrade the chillers which was not approved, several low cost measures have been implemented to defer chiller replacement. In 2010, the County had a thermal study performed on the computing environment to identify where the “hot spots” were and later inserted under-floor baffling to more optimally direct airflow. At the time, that allowed the County to shut off two of the thirteen CRAC units serving the data center. In 2011, the cooling tower was refurbished and this too resulted in better cooling performance. The proposed FY 2012 / 2013 budget includes $1.0 million for supplementing cooling capacity including pumps and chillers.
The Orange County Data Center has reached a seasonal threshold where the cooling capacity is no longer adequate to meet the building’s needs. During the summer months the added cooling demand, driven by higher outside temperatures, pushes the cooling demand beyond the operational limits of the present chilled water system. On July 15, 2011, a day where the high temperature in Santa Ana was 82 degrees, the data center cooling system went offline due to a minor voltage drop in SCE utility power. When the cooling system failed the computer room temperature raised from an optimal 73 degrees to over 115 degrees in less than 30 minutes. Computer systems, designed to protect themselves, began shutting themselves off at around 100 degrees. Hundreds of critical computer components were taken offline. This impacted almost every critical County application and online system. It took nearly 36 hours to restore all systems. We had two additional occurrences in August and September 2011 but due to increased onsite monitoring, the implementation of additional alarms and improved processes we were able to restart the chillers in time to avert an outage.
OC Public Works Heating Ventilation and Air-Conditioning (HVAC) mechanics have set up the system to run up at higher demand limits which are beyond the chilled water system’s original design specifications. While this helps close the gap in meeting summer cooling demands, it leaves the facility in an unstable operating environment as we are running near overload trip limits. There is no redundant cooling available when operating in this manner. If the collective server and summer cooling demands exceed the chilled water system’s upper threshold limits the system will crash. This would require the shutting down of all servers operating at the Orange County Data Center.
It is believed that owing to the confluence of circumstances, namely that demand has increased and that decommissioning of older equipment has been slower than anticipated, the likelihood of a similar occurrence to that which occurred on July 15, 2011 has increased. Should an outage occur, access to many critical systems would be interrupted; examples of such systems include Probation case management and tracking systems, state-wide systems such as CLETS, CJIS and DOJ, welfare systems, property tax systems, accounting and payroll systems and IT security infrastructure.
CEO IT requests your Board’s approval to award a contract on an emergency basis to Myers, Houghton & Partners, Inc. dba MHP, Inc. to provide Architect-Engineer Services (A-E) services and construction support for the Emergency Project: Data Center HVAC 2012 Upgrades.
CEO IT requests your Board’s approval to award a contract on an emergency basis to West Coast Air Conditioning Co., Inc. to provide and install new plumbing and electrical infrastructure for the Emergency Project as designed by Myers, Houghton & Partners, Inc. (MHP) for the Data Center HVAC 2012 Upgrades Project.
The project will include the minimum design construction documents necessary to meet the County’s immediate cooling requirements for the summer of 2012. The design will provide for the installation of chilled water plumbing lines; relocation of existing CRAC unit(s) and installation of connections for a portable air cooled chiller unit. The project includes minimum electrical upgrades required to support the portable chiller and CRAC unit relocations. The design will be planned to provide the minimum work required to meet 2012 requirements while simultaneously providing systems that can be augmented in future years when a non-emergency project can be implemented to upgrade COC utility systems for HVAC and electrical. The need for augmentation is dependent upon a number of factors including confirming possible renewals and assessing the impact of future demand and the corresponding impact of more efficient computer equipment. The on-going chiller rental is for hot weather periods only.
MHP design documents shall be completed by May 2012 and submitted for permit approval. The contractor, West Coast Air Conditioning Inc., (WCA) will procure materials and equipment and be prepared to begin installation of the plumbing lines in May 2012. Construction work for connection of the portable air cooled chiller unit shall be complete by late June 2012.
MHP currently has a similar contract with the County for the Central Utility Facility (CUF)-Cooling Tower Enclosure Design which includes the upgrade of piping and electrical at the County’s Central Utility Facility. MHP will provide the same structural, mechanical and electrical engineering team for this emergency project. MHP was selected for the CUF-Cooling Tower Enclosure Design project through a County Statement of Qualifications for Providing Architecture-Engineering Services process.
WCA is the current contractor for the CUF- Replace Boiler Feed pumps project. WCA successfully constructed the Cogeneration projects at the CUF and at John Wayne Airport. Their Contractor's License No. #262349 was verified as current and active through the Contractors State License Board. The Contractor is qualified to perform this project based on research of the references previously submitted by the Contractor for the Feedpumps project.
Estimated costs for this upgrade are as follows:
|
One time: |
|
|
Architect-Engineering Services |
$75,000 |
|
Construction Contract |
$677,000 |
|
County Administration, Permits Testing & Inspection |
$85,040 |
|
Total One Time: |
$837,040 |
|
Ongoing: |
|
|
Air cooled chillers 200 tons (per month) |
$10,000 |
Emergency Justification:
The term "Emergency", as used in Section 1102 of the California Public Contract Code, means a sudden, unexpected occurrence that poses a clear and imminent danger, requiring immediate action to prevent or mitigate the loss or impairment of life, health, property, or essential public services.
FINANCIAL IMPACT:
FY 11-12 Budget includes $134,800 (289-4200-P634) for Feasibility Study of Utility Upgrade at Data Center, which will be used for this project and additional budget transfer will be processed within Fund 289 to complete the actual upgrade at the Data Center before summer 2012.
STAFFING IMPACT:
N/A
REVIEWING AGENCIES:
Orange County Public Works (OCPW)