Exercise: Data Centers in Simulation CFD

This hands-on tutorial will guide the user through best practices of setting up data center CFD simulation and reviewing the results.

Learning Objectives

  • Observe how to set up a data center efficiently in Simulation CFD.
  • Provide experience working with multiple data center components in Simulation CFD.
  • Validate simulation inputs and evaluate data center performance.

Assumptions

The simulation will have several simplifying assumptions to reduce overall complexity and expedite completion of this data center tutorial.

  • All exterior walls will be considered adiabatic (i.e., perfectly insulated).
  • Radiation will be assumed to be negligible.
  • Raised floor and ceiling will be considered adiabatic forcing airflow to control temperature distribution.
  • System will be considered as a closed loop with no air being added or removed from the simulation.

This Data Center exercise model is comprised of server racks (1), CRAC units (2), cold aisle supply tiles (3) and hot aisle exhaust tiles (4).

This Data Center exercise model is comprised of server racks (1), CRAC units (2), cold aisle supply tiles (3) and hot aisle exhaust tiles (4).

Simulation Parameters

  • Heat exchanger capacity:
    • 15,000 CFM each
    • Supply temperature of 61F
  • Server racks
    • 800 CFM each
    • 10,000 Watts each

Simulation Process

NOTE: Setup will begin from a support file for which some settings have been predefined to maintain focus on topics specific to data centers. As this an advanced exercise, it is assumed that fundamental skills required to interact with the Simulation CFD interface have already been developed.

1. Download the Exercise-data-center.cfz share file to a local hard drive folder.

2. Load the share file

a. Start the Simulation CFD interface and click Open in the upper left ribbon, then navigate to and open the Exercise-data-center.cfz share file.

3. Define internal fan devices for the server racks:

a. In the Design Study Bar, expand the Groups node, right click on the +X_Servers node and then click on Show only this Group to select the servers which will flow air in the positive X direction and hide all other volumes.

TIP: Groups can be useful in the setup and results review of models with many components; more details about groups can be found here.

b. Assign an Internal Fan device to the selected volumes with the following attributes:

Save to Database

Local

Name

+X 42U Rack

Variation Method

Constant

Value

800 ft3/min

Rotational Speed

0

c. Make certain to set the Flow Direction to 1,0,0 (+X) before clicking on the Apply button.

A common error when assign Internal Fan devices is to forget to set the correct Flow Direction before clicking the Apply button.

A common error when assign Internal Fan devices is to forget to set the correct Flow Direction before clicking the Apply button.

4. Repeat previous step for the servers on the opposite side of the hot aisle:

a. Right click on the Groups:-X_Servers node and then click on Show only this Group to select the servers which will flow air in the negative X direction and hide all other volumes.

b. Define Fan Name as -X 42U Rack

c. Make certain to set the Flow Direction to -1,0,0 (-X) before clicking on the Apply button.

5. Assign Heat Exchanger devices to represent the CRAC units:

a. Right click on the Groups:CRACs node and then click on Show only this Group to select the CRAC volumes and hide all other volumes.

b. Click directly on one of the selected volumes to deselect it (heat exchangers have to be defined one at a time).

c. Click on Edit in the Ribbon and change the material Type to Heat Exchanger.

d. Click on the “...” button next to Inlet Surfaces.

e. Click Select Surface and select the top surface (either surface id 41 or 42 depending on which volume is being assigned) of the selected volume (rotate model if needed).

Surface ids of the top (41, 42) and bottom (9, 12) surfaces of the heat exchangers.  The positive Z axis points up to the ceiling of the data center.

Surface ids of the top (41, 42) and bottom (9, 12) surfaces of the heat exchangers.  The positive Z axis points up to the ceiling of the data center.

f. Click on the “...” button next to Outlet Surfaces.

g. Click Select Surface and select the bottom surface of the selected volume.

h. Bring up the material editor by clicking Edit… in the Materials dialog.

i. Change the Save to database: to Local.

j. For the Name, type in CRAC 1.

k. Set the Flow to Constant at 15000 ft3/min.

l. Select Apply to see the settings update in the properties section.

m. Select the Heat Transfer button from the properties.

n. Select Air Conditioner from the Variation method: dropdown.

o. Make the Set Point Temperature a value of 61 Fahrenheit.

p. Click the Apply button.

q. Click the OK button in the Material Editor dialog.

r. Click the Apply button in the Materials dialog.

6. Repeat step 5 for the other volume representing a CRAC unit:

a. Enter a name of CRAC 2 for this device.

7. Assign surface resistance regions to the floor tiles to represent the pressure drop going through the actual tiles.

a. Right click on the Groups:NoAirCeiling node and then click on Show only this Group which will hide the primary air volumes and the ceiling volume.

b. Left click on the Groups:FloorTiles node to select the top surfaces of the floor tiles.

The 2 selected surfaces above are assigned with surface resistance regions to represent the restriction of flow through the cold aisle tile.

The 2 selected surfaces above are assigned with surface resistance regions to represent the restriction of flow through the cold aisle tile.

c. Define a custom surface resistance material with the following attributes:

Save to Database

Local

Name

40-Open

Variation Method

Free Area Ratio

Value

0.4

d. Click the Apply button and then the OK button.

e. Set the Shell Thickness to 1 inch in the Materials dialog.

f. Click Apply.

8. Repeat step 7 for the ceiling tiles:

a. Left click on the Groups:CeilingTiles node to select only the bottom surface of the volume for the ceiling tiles.

b. Create another custom resistance material with the following attributes:

i. Name: 80 Open

ii. Free Area Ratio: 0.8

9. Assign a heat generation boundary condition to each server to represent the heat load from the server components (16 servers with 10,000 Watts each for a total of 160,000 Watts added to the system).

a. Go into the Boundary Conditions setup task.

b. Left click on the Groups:AllServers node to select all of the servers.

c. Select Edit.

d. Enter 10000 W into the Total Heat Generation quantity.

e. Hit Apply.

NOTE: Initial mesh settings have already been defined. Note that conduction through the ceiling, wall, and the housing volumes for the CRAC units and servers is assumed to be negligible. To save mesh, all of these volumes have been suppressed and will show up as blue in the Mesh Sizing task.

10. Solve the simulation. Settings for a forced convection analysis have already been defined (as detailed here in the AEC Solver Settings page).

a. Solve the analysis for 100 iterations.

11. Verify the model inputs with a few quick sanity checks.

a. When the simulation has completed, review the energy balance in the Summary File:

i. Click on Summary File in the Ribbon.

ii. Scroll down to the Fluid Energy Balance Information section in the Summary File (see Sanity Checks page). The Heat Transfer Due To Sources In Fluid should state 1.6e+005 Watts. If this number is incorrect, then check the volumetric heat generation assignments in the Boundary Conditions section of the setup.

iii. Now scroll down the Summary File a little further to the Data for heat exchanger devices section. The Flow Rate for each heat exchangers should be about 25,000 m^3/hr (15,000 cfm). If it is not, then the heat exchanger device has the wrong inputs.

b. Check the global thermal results.

i. Set the Global Result to Temperature and hide the air volumes to see the server racks. The minimum temperature in the model should be 61F (16.11C), which was the set point for the CRAC units. The hot aisle should be in between the 2 rack aisles. If it is not, then the fan directions were not defined correctly.

In this data center, the highest temperatures in the model are in the hot aisle, as shown above.

In this data center, the highest temperatures in the model are in the hot aisle, as shown above.

12. Check for recirculation in the server racks

a. Zoom in on the server racks and note some areas of potential recirculation which appear at the edges of the end units.

These isolated areas of higher temperatures at the server indicate indicate that there is possible recirculation in the data center.

These isolated areas of higher temperatures at the server indicate indicate that there is possible recirculation in the data center.

Using the options detailed on the Evaluating Data Center Results page, the results can be further interrogated, using Planes and Particle Traces (depicted below), to confirm the extent of recirculation around the servers or in other areas of the data center.


  1. Cold air particle from the CRAC unit enters the floor tiles.
  2. Air particle goes right past the server inlets to the ceiling.
  3. Air particle goes over to the corner of the data center.
  4. Warmer air particle is sucked into upper corner of server.  This was the seed point of the particle trace which was defined by locating a Plane on the server inlet.
  5. Particle exits server and goes through the ceiling tiles. Particle is pulled back into CRAC unit.

Conclusion

The primary focus of this tutorial was to provide practice in the proper setup of a data center simulation. To keep the tutorial manageable, the data center was intentionally limited to 16 servers. However, the same fundamental best practices demonstrated here will directly apply to the largest of data centers, which may contain hundreds of servers. With larger data centers, the higher number of components will increase the probability of user input errors, making model validation a top priority. Finally, as the number of servers increase, so does the mesh size and solver run time.

AttachmentSize
Package icon Dataset - exercise data center.cfz588.97 KB