testing
zeus.utils.testing
Utilities for testing.
_EnergyCounter
Integrates a piecewise constant power draw over a clock.
Power is in mW and time is in seconds, so energy is in mJ.
Source code in zeus/utils/testing.py
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | |
power_mw
property
writable
power_mw
Current power draw. Units: mW.
energy_mj
property
energy_mj
Energy consumed since the counter was created. Units: mJ.
__init__
__init__(power_mw, clock)
Source code in zeus/utils/testing.py
25 26 27 28 29 30 | |
MockGPU
Bases: GPU
A simulated GPU for tests that run without GPU hardware.
The GPU draws power_mw, capped at its current power management limit, and its
energy counter integrates that power over clock. The power draw and the clock
are fixed at construction. The other constructor arguments are stored in attributes
of the same name, which tests can change at any time.
Setters change the simulated state. set_power_management_limit raises
ZeusGPUInvalidArgError for a limit outside power_limit_constraints_mw, and
the clock locking setters raise it when the minimum clock exceeds the maximum.
Attributes:
| Name | Type | Description |
|---|---|---|
persistence_mode |
Whether persistence mode is enabled. |
|
gpu_locked_clocks_mhz |
tuple[int, int] | None
|
Locked GPU clock range, or |
memory_locked_clocks_mhz |
tuple[int, int] | None
|
Locked memory clock range, or |
Source code in zeus/utils/testing.py
62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 | |
supports_nonblocking_setters
property
supports_nonblocking_setters
Return False, since setters take effect immediately.
__init__
__init__(
gpu_index=0,
*,
name="Mock GPU",
power_mw=100000,
memory_power_mw=None,
power_limit_constraints_mw=(100000, 300000),
supported_memory_clocks_mhz=(1000,),
supported_graphics_clocks_mhz=(1500, 1200, 900),
supports_total_energy=True,
temperature_c=40,
clock=time.monotonic,
)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
gpu_index
|
int
|
Index of the GPU. |
0
|
name
|
str
|
Name of the GPU model. |
'Mock GPU'
|
power_mw
|
int
|
Power the GPU draws while its power limit allows it. Units: mW. |
100000
|
memory_power_mw
|
int | None
|
Average power draw of the GPU memory, or |
None
|
power_limit_constraints_mw
|
tuple[int, int]
|
Minimum and maximum power management limits. The maximum is also the default limit. Units: mW. |
(100000, 300000)
|
supported_memory_clocks_mhz
|
Sequence[int]
|
Supported memory clock frequencies. Units: MHz. |
(1000,)
|
supported_graphics_clocks_mhz
|
Sequence[int]
|
Supported graphics clock frequencies. Units: MHz. |
(1500, 1200, 900)
|
supports_total_energy
|
bool
|
Whether the GPU has an energy counter. NVIDIA GPUs older
than Volta do not, and |
True
|
temperature_c
|
int
|
GPU temperature. Units: Celsius. |
40
|
clock
|
Callable[[], float]
|
Returns the current time in seconds. The energy counter integrates power over it. Tests that patch the time seen by the code under test can pass the same function. |
monotonic
|
Source code in zeus/utils/testing.py
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 | |
get_name
get_name()
Return the name of the GPU model.
Source code in zeus/utils/testing.py
133 134 135 | |
get_power_management_limit_constraints
get_power_management_limit_constraints()
Return the minimum and maximum power management limits. Units: mW.
Source code in zeus/utils/testing.py
137 138 139 | |
get_power_management_limit
get_power_management_limit()
Return the current power management limit. Units: mW.
Source code in zeus/utils/testing.py
141 142 143 | |
set_power_management_limit
set_power_management_limit(power_limit_mw, block=True)
Set the GPU's power management limit. Unit: mW.
Source code in zeus/utils/testing.py
145 146 147 148 149 150 151 | |
reset_power_management_limit
reset_power_management_limit(block=True)
Reset the GPU's power management limit to the maximum constraint.
Source code in zeus/utils/testing.py
153 154 155 | |
set_persistence_mode
set_persistence_mode(enabled, block=True)
Set persistence mode.
Source code in zeus/utils/testing.py
157 158 159 | |
get_persistence_mode
get_persistence_mode()
Return whether persistence mode is currently enabled.
Source code in zeus/utils/testing.py
161 162 163 | |
get_supported_memory_clocks
get_supported_memory_clocks()
Return a list of supported memory clock frequencies. Units: MHz.
Source code in zeus/utils/testing.py
165 166 167 | |
set_memory_locked_clocks
set_memory_locked_clocks(
min_clock_mhz, max_clock_mhz, block=True
)
Lock the memory clock to a specified range. Units: MHz.
Source code in zeus/utils/testing.py
169 170 171 | |
reset_memory_locked_clocks
reset_memory_locked_clocks(block=True)
Reset the locked memory clocks to the default.
Source code in zeus/utils/testing.py
173 174 175 | |
get_supported_graphics_clocks
get_supported_graphics_clocks(memory_clock_mhz=None)
Return a list of supported graphics clock frequencies. Units: MHz.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
memory_clock_mhz
|
int | None
|
Ignored. The same graphics clocks are supported at every memory clock. |
None
|
Source code in zeus/utils/testing.py
177 178 179 180 181 182 183 | |
set_gpu_locked_clocks
set_gpu_locked_clocks(
min_clock_mhz, max_clock_mhz, block=True
)
Lock the GPU clock to a specified range. Units: MHz.
Source code in zeus/utils/testing.py
185 186 187 | |
reset_gpu_locked_clocks
reset_gpu_locked_clocks(block=True)
Reset the locked GPU clocks to the default.
Source code in zeus/utils/testing.py
189 190 191 | |
get_average_power_usage
get_average_power_usage()
Return the average power draw of the GPU. Units: mW.
Power is constant between power limit changes, so this equals the instant power draw.
Source code in zeus/utils/testing.py
193 194 195 196 197 198 | |
get_instant_power_usage
get_instant_power_usage()
Return the current power draw of the GPU. Units: mW.
Source code in zeus/utils/testing.py
200 201 202 | |
get_average_memory_power_usage
get_average_memory_power_usage()
Return the average power draw of the GPU's memory. Units: mW.
Source code in zeus/utils/testing.py
204 205 206 207 208 | |
supports_get_total_energy_consumption
supports_get_total_energy_consumption()
Check if the GPU supports retrieving total energy consumption.
Source code in zeus/utils/testing.py
210 211 212 | |
get_total_energy_consumption
get_total_energy_consumption()
Return the energy consumed since the GPU object was created. Units: mJ.
Source code in zeus/utils/testing.py
214 215 216 217 218 | |
get_gpu_temperature
get_gpu_temperature()
Return the current GPU temperature. Units: Celsius.
Source code in zeus/utils/testing.py
220 221 222 | |
MockGPUs
Bases: GPUs
A collection of MockGPU objects that Zeus can use in place of real GPUs.
get_gpus returns zeus.device.gpu._gpus whenever it is set, so patching that
attribute makes every Zeus component in the current process use the simulated GPUs:
def test_training(mocker):
gpus = MockGPUs([MockGPU(0), MockGPU(1)])
mocker.patch("zeus.device.gpu._gpus", gpus)
...
GPU indices are positions in the list. CUDA_VISIBLE_DEVICES is not applied.
Note
Child processes do not see the patch. PowerMonitor polls power in spawned
processes, so it cannot read simulated GPUs. ZeusMonitor starts a
PowerMonitor for GPUs whose supports_total_energy is False.
Source code in zeus/utils/testing.py
225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 | |
gpus
property
gpus
Return the simulated GPUs.
__init__
__init__(gpus, ensure_homogeneous=False)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
gpus
|
Sequence[MockGPU]
|
Simulated GPUs, in index order. |
required |
ensure_homogeneous
|
bool
|
If True, raise |
False
|
Source code in zeus/utils/testing.py
246 247 248 249 250 251 252 253 254 255 256 | |
__del__
__del__()
Do nothing, since there is no management library to shut down.
Source code in zeus/utils/testing.py
263 264 | |
MockCPU
Bases: CPU
A simulated CPU package for tests that run without RAPL or EMI.
The package and its DRAM draw constant power, and their energy counters
integrate that power over clock.
Source code in zeus/utils/testing.py
267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 | |
__init__
__init__(
cpu_index=0,
*,
cpu_power_mw=50000,
dram_power_mw=10000,
clock=time.monotonic,
)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cpu_index
|
int
|
Index of the CPU package. |
0
|
cpu_power_mw
|
int
|
Power draw of the CPU package. Units: mW. |
50000
|
dram_power_mw
|
int | None
|
Power draw of the DRAM, or |
10000
|
clock
|
Callable[[], float]
|
Returns the current time in seconds. The energy counters integrate power over it. |
monotonic
|
Source code in zeus/utils/testing.py
274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 | |
get_total_energy_consumption
get_total_energy_consumption()
Return the energy consumed since the CPU object was created. Units: mJ.
Source code in zeus/utils/testing.py
296 297 298 299 300 301 | |
supports_get_dram_energy_consumption
supports_get_dram_energy_consumption()
Return True if the package has a DRAM zone.
Source code in zeus/utils/testing.py
303 304 305 | |
get_power_limits
get_power_limits()
Return zone power limits that are disabled and have no constraints.
Source code in zeus/utils/testing.py
307 308 309 310 311 312 | |
MockCPUs
Bases: CPUs
A collection of MockCPU objects that Zeus can use in place of real CPUs.
As with MockGPUs, patch zeus.device.cpu._cpus
to make get_cpus return it.
Source code in zeus/utils/testing.py
315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 | |
cpus
property
cpus
Return the simulated CPU packages.
__init__
__init__(cpus)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cpus
|
Sequence[MockCPU]
|
Simulated CPU packages, in index order. |
required |
Source code in zeus/utils/testing.py
322 323 324 325 326 327 328 | |
__del__
__del__()
Do nothing, since there is no monitoring interface to shut down.
Source code in zeus/utils/testing.py
335 336 | |
ReplayZeusMonitor
Bases: ZeusMonitor
A mock ZeusMonitor that replays windows recorded by a real monitor.
This class is for testing only. Based on a CSV log file that records the time
and energy measurements of ZeusMonitor measurement windows, users can drop-in
replace ZeusMonitor with this class to replay the measurement windows and
fast forward training and time/energy measurement.
The methods exposed is identical to or a superset of ZeusMonitor, but behaves
differently. Instead of monitoring the GPU, it replays events from a log file.
The log file generated by ZeusMonitor (log_file) is guaranteed to be compatible
and will replay time and energy measurements just like how the real monitor
experienced them. Note that in the case of concurrent ongoing measurement windows,
the log rows file should record windows in the order of end_window calls.
Attributes:
| Name | Type | Description |
|---|---|---|
gpu_indices |
`list[int]`
|
Indices of all the CUDA devices to monitor. |
Source code in zeus/utils/testing.py
339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 | |
__init__
__init__(
gpu_indices=None,
approx_instant_energy=False,
log_file=None,
ignore_sync_execution=False,
match_window_name=True,
)
The log file should be a CSV file with the following header (e.g. gpu_indices=[0, 2]):
start_time,window_name,elapsed_time,gpu0_energy,gpu2_energy
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
gpu_indices
|
list[int] | None
|
Indices of all the CUDA devices to monitor. This should be consistent
with the indices used in the log file. If |
None
|
approx_instant_energy
|
bool
|
Whether to approximate the instant energy consumption. Not used. |
False
|
log_file
|
str | Path | None
|
Path to the log CSV file to replay events from. |
None
|
ignore_sync_execution
|
bool
|
Whether to ignore |
False
|
match_window_name
|
bool
|
Whether to make sure window names match. (Default: |
True
|
Source code in zeus/utils/testing.py
358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 | |
begin_window
begin_window(key, sync_execution=True, restart=False)
Begin a new window.
This method just pushes the key into a list of ongoing measurement windows, and just makes sure it's unique.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
str
|
Name of the measurement window. |
required |
sync_execution
|
bool
|
Whether to synchronize CUDA before starting the measurement window.
(Default: |
True
|
restart
|
bool
|
If True and the window already exists, cancel the existing window and start a new one. |
False
|
Source code in zeus/utils/testing.py
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 | |
end_window
end_window(key, sync_execution=True, cancel=False)
End an ongoing window.
This method pops the key from a list of ongoing measurement windows and
constructs a Measurement object corresponding to the name of the window
from the log file. If the name of the window does not match the expected
one, a RuntimeError is raised.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
str
|
Name of the measurement window. |
required |
sync_execution
|
bool
|
Whether to synchronize CUDA before ending the measurement window.
(Default: |
True
|
cancel
|
bool
|
Whether to cancel the measurement window. This will not consume a
line from the log file. (Default: |
False
|
Source code in zeus/utils/testing.py
424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 | |
_clock_range
_clock_range(min_clock_mhz, max_clock_mhz)
Return the clock range, or raise if the minimum exceeds the maximum.
Source code in zeus/utils/testing.py
55 56 57 58 59 | |