You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-19 - 13:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Fri Apr 19, 2024 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
206822242166,25sleep20-21swapper/219:07:162
207772237163,24sleep30-21swapper/319:08:303
207652235166,55sleep10-21swapper/119:08:181
207692203165,25sleep00-21swapper/019:08:220
1997121710,2sleep02098799cyclictest21:30:230
20998991101,107cyclictest0-21swapper/200:37:432
209989910298,3cyclictest0-21swapper/219:31:042
209989910197,3cyclictest0-21swapper/222:32:342
59732590,1sleep30-21swapper/319:45:213
92682580,2sleep2311ktimersoftd/222:20:202
191232550,1sleep10-21swapper/120:15:231
2100399412,24cyclictest7932-40dnf22:17:543
2100399411,23cyclictest2785-21ntp_kernel_err22:05:223
2098799411,28cyclictest0-21swapper/000:10:050
2100399391,28cyclictest848-21sssd_nss22:20:153
2099399390,19cyclictest0-21swapper/121:05:211
20987993912,12cyclictest9080-21awk19:55:000
2100399388,22cyclictest20291-21/usr/sbin/munin20:20:193
2100399385,24cyclictest11279-21if_enp2s022:25:153
2100399381,26cyclictest86550irq/126-enp1s0-00:32:263
2100399381,25cyclictest14634-21snmpd23:01:363
2099899382,4cyclictest32276-21kworker/2:222:12:262
2099399381,14cyclictest0-21swapper/119:29:141
2099399381,12cyclictest4980-21awk22:10:201
2100399376,18cyclictest86550irq/126-enp1s0-22:40:163
2100399371,22cyclictest28482-21diskstats23:05:133
2099399371,22cyclictest7104-21diskstats23:30:141
2098799371,15cyclictest0-21swapper/021:08:290
2100399366,21cyclictest22490-21/usr/sbin/munin20:25:153
2100399362,25cyclictest26468-21kworker/u8:122:30:173
2100399361,26cyclictest11149-21/usr/sbin/munin23:40:133
2100399361,22cyclictest26468-21kworker/u8:121:15:273
2099399365,11cyclictest780-21rngd20:51:401
2099399361,11cyclictest18281-21grep22:40:241
2100399354,21cyclictest9693-21diskmemload21:25:143
2100399353,23cyclictest86850irq/129-enp1s0-20:45:123
2100399353,23cyclictest86850irq/129-enp1s0-20:45:123
2100399351,23cyclictest0-21swapper/322:45:143
2100399351,22cyclictest29348-21cut00:20:143
2100399351,20cyclictest4558-21/usr/sbin/munin23:25:143
2100399351,20cyclictest3959-21/usr/sbin/munin22:10:103
2100399350,23cyclictest1218-21ntp_states23:15:263
2099899351,14cyclictest0-21swapper/221:58:142
20993993517,3cyclictest3989-21hddtemp_smartct20:55:161
2099399351,10cyclictest20318-21uniq00:00:161
2098799352,15cyclictest21618-21idleruntime21:35:170
20987993510,12cyclictest23157-21munin-run19:15:020
2100399346,20cyclictest23724-21/usr/sbin/munin22:55:173
2100399344,17cyclictest848-21sssd_nss22:50:293
2100399344,17cyclictest848-21sssd_nss22:50:293
2100399342,28cyclictest25764-40dnf19:16:313
2100399341,29cyclictest86550irq/126-enp1s0-23:30:273
2100399341,27cyclictest0-21swapper/323:49:583
2100399341,26cyclictest26468-21kworker/u8:121:34:043
2100399341,25cyclictest0-21swapper/321:45:183
2100399341,19cyclictest18805-21ntp_states23:55:203
2100399341,19cyclictest18805-21ntp_states23:55:203
2099899341,20cyclictest0-21swapper/220:57:422
2099899341,13cyclictest0-21swapper/200:30:222
2099399341,10cyclictest26834-21cat20:35:011
2098799341,19cyclictest0-21swapper/022:51:010
2098799341,19cyclictest0-21swapper/022:51:010
2098799341,15cyclictest0-21swapper/022:27:330
2100399331,29cyclictest27510-21/usr/sbin/munin21:50:183
2100399331,29cyclictest21186-21/usr/sbin/munin19:10:173
2100399331,25cyclictest0-21swapper/321:10:293
2100399331,24cyclictest0-21swapper/320:30:583
2100399331,22cyclictest26512-21cat00:15:023
2100399331,20cyclictest513-21ntp_states22:00:243
2100399331,17cyclictest30966-21/usr/sbin/munin00:25:243
2099399332,22cyclictest780-21rngd23:51:281
20993993318,3cyclictest21369-21ntp_states20:20:241
2099399331,26cyclictest0-21swapper/119:50:541
2099399331,24cyclictest0-21swapper/100:09:181
2099399331,23cyclictest0-21swapper/123:09:341
2098799331,20cyclictest0-21swapper/023:16:370
2098799331,14cyclictest716-21/usr/sbin/munin00:30:130
2098799331,14cyclictest0-21swapper/022:41:410
2100399326,17cyclictest86550irq/126-enp1s0-20:50:113
2100399321,28cyclictest14634-21snmpd20:14:523
2100399321,27cyclictest9693-21diskmemload23:53:113
2100399321,27cyclictest4578-21/usr/sbin/munin20:55:243
2100399321,22cyclictest22253-21wc00:05:143
2099399325,12cyclictest780-21rngd20:43:061
2099399322,9cyclictest780-21rngd23:40:441
20993993221,3cyclictest0-21swapper/121:10:081
2099399321,18cyclictest15750-21kernelversion22:35:191
2098799321,24cyclictest0-21swapper/022:01:330
2098799321,15cyclictest31506-21kworker/0:222:13:050
2098799321,13cyclictest0-21swapper/023:04:210
2098799321,11cyclictest0-21swapper/023:11:570
2100399318,21cyclictest86850irq/129-enp1s0-19:30:443
2100399313,18cyclictest0-21swapper/321:57:153
2100399312,18cyclictest12290-1kworker/3:1H19:44:553
2100399311,27cyclictest28821-21diskstats19:25:123
2100399311,26cyclictest8408-21/usr/sbin/munin21:05:183
2100399311,26cyclictest7210-21cpuspeed_turbos19:50:113
2100399311,26cyclictest27353-21ntp_states19:20:253
2100399311,23cyclictest2997-21/usr/sbin/munin00:35:263
2100399311,21cyclictest15393-21cut22:35:163
2100399311,20cyclictest16282-21latency_hist21:25:023
2100399311,17cyclictest14634-21snmpd21:37:243
2100399310,23cyclictest86750irq/128-enp1s0-23:35:103
2100399310,21cyclictest0-21swapper/319:35:173
2100399310,17cyclictest0-21swapper/319:55:123
2099899311,11cyclictest0-21swapper/219:25:342
2099399315,24cyclictest86850irq/129-enp1s0-21:52:201
20993993114,3cyclictest1874-21sed19:35:261
2099399310,24cyclictest0-21swapper/119:45:501
2098799311,19cyclictest0-21swapper/023:47:410
2098799311,19cyclictest0-21swapper/022:05:090
2098799311,18cyclictest9693-21diskmemload21:54:290
2098799311,14cyclictest0-21swapper/023:42:450
2098799311,14cyclictest0-21swapper/022:23:490
2098799311,14cyclictest0-21swapper/019:27:170
2098799311,13cyclictest0-21swapper/021:23:410
2098799311,13cyclictest0-21swapper/020:54:450
2098799311,12cyclictest26419-21kworker/0:020:44:510
2098799311,11cyclictest496-21jbd2/dm-0-820:32:210
2100399301,26cyclictest848-21sssd_nss20:05:163
2100399301,26cyclictest5513-21/usr/sbin/munin21:00:163
2100399301,26cyclictest30333-21/usr/sbin/munin23:10:263
2100399301,26cyclictest26574-21/usr/sbin/munin00:15:273
2100399301,26cyclictest23109-21/usr/sbin/munin21:40:193
2100399301,26cyclictest20288-21/usr/sbin/munin00:00:153
2100399301,13cyclictest0-21swapper/320:15:193
2099899301,19cyclictest0-21swapper/223:35:102
20998993011,4cyclictest16916-21kworker/2:023:50:162
2099899301,12cyclictest23091-21sed22:50:302
2099899301,12cyclictest23091-21sed22:50:302
2099399301,23cyclictest0-21swapper/123:03:581
2099399301,12cyclictest0-21swapper/121:39:461
2099399301,11cyclictest0-21swapper/120:03:541
2099399300,2cyclictest86650irq/127-enp1s0-00:35:041
2098799305,12cyclictest7501-21munin-run21:05:000
2098799301,24cyclictest0-21swapper/020:12:470
2098799301,18cyclictest9693-21diskmemload21:17:330
2098799301,17cyclictest0-21swapper/021:55:570
2098799301,12cyclictest0-21swapper/023:59:570
2098799301,12cyclictest0-21swapper/023:59:570
2098799301,12cyclictest0-21swapper/022:48:370
2098799301,11cyclictest7070-21sed19:50:010
2098799301,11cyclictest0-21swapper/022:57:490
2098799301,10cyclictest0-21swapper/020:36:530
2100399292,24cyclictest27237-21diskstats20:35:153
2100399291,25cyclictest11489-21/usr/sbin/munin20:00:183
21003992912,14cyclictest29228-21cut20:40:123
2100399290,26cyclictest2365-21/usr/sbin/munin23:20:133
2099899291,19cyclictest0-21swapper/223:07:182
2099399298,4cyclictest7639-21sort21:05:021
2099399295,12cyclictest780-21rngd20:06:181
2099399291,19cyclictest0-21swapper/120:36:261
2099399291,16cyclictest0-21swapper/122:45:401
2099399291,11cyclictest0-21swapper/119:57:341
2099399290,6cyclictest86850irq/129-enp1s0-23:48:021
2099399290,13cyclictest0-21swapper/120:48:561
2099399290,13cyclictest0-21swapper/120:48:561
2098799291,8cyclictest0-21swapper/023:07:170
2098799291,13cyclictest0-21swapper/023:51:570
2098799291,13cyclictest0-21swapper/020:00:210
2098799291,13cyclictest0-21swapper/019:35:170
2098799291,12cyclictest0-21swapper/023:34:290
2098799291,12cyclictest0-21swapper/019:44:530
2098799291,12cyclictest0-21swapper/000:01:410
2098799291,11cyclictest0-21swapper/019:30:130
2098799291,11cyclictest0-21swapper/000:28:370
2098799291,10cyclictest0-21swapper/021:49:010
2099899284,7cyclictest16916-21kworker/2:000:04:362
2099899283,3cyclictest86750irq/128-enp1s0-19:35:402
2099899283,3cyclictest16916-21kworker/2:022:57:482
2099899282,9cyclictest30-21rcuc/221:03:322
2099899281,9cyclictest0-21swapper/223:55:182
2099899281,9cyclictest0-21swapper/223:55:182
20998992811,2cyclictest16916-21kworker/2:023:40:262
2099899281,12cyclictest0-21swapper/219:53:502
20998992810,3cyclictest86550irq/126-enp1s0-00:12:042
2099399287,4cyclictest0-21swapper/122:55:261
2099399284,17cyclictest0-21swapper/100:15:281
2099399281,17cyclictest0-21swapper/121:22:061
2099399281,13cyclictest0-21swapper/123:10:461
2099399281,13cyclictest0-21swapper/122:19:421
2099399281,12cyclictest0-21swapper/121:55:321
2099399281,10cyclictest0-21swapper/123:24:141
2099399281,10cyclictest0-21swapper/100:12:061
2099399280,22cyclictest0-21swapper/123:28:541
2099399280,11cyclictest0-21swapper/123:59:461
2099399280,11cyclictest0-21swapper/123:59:461
2098799289,16cyclictest857-21munin-run20:50:000
2098799289,16cyclictest857-21munin-run20:50:000
2098799281,12cyclictest0-21swapper/023:29:250
2098799281,12cyclictest0-21swapper/023:22:530
2098799281,12cyclictest0-21swapper/022:36:210
2098799281,12cyclictest0-21swapper/020:06:210
2098799281,10cyclictest0-21swapper/020:16:130
2098799280,11cyclictest31506-21kworker/0:222:15:450
32602270,4sleep230-21rcuc/220:55:022
236192270,4sleep02098799cyclictest20:25:200
20998992711,7cyclictest30-21rcuc/221:15:242
2099899271,13cyclictest0-21swapper/219:15:262
2099899271,11cyclictest0-21swapper/220:37:102
2099399275,10cyclictest0-21swapper/100:34:381
2099399274,8cyclictest2008-40dnf23:18:331
2099399274,10cyclictest780-21rngd19:32:401
2099399271,11cyclictest0-21swapper/100:27:181
2099399270,18cyclictest0-21swapper/121:31:101
2098799271,18cyclictest0-21swapper/021:27:010
2098799271,13cyclictest10063-21kworker/0:122:31:150
2098799271,12cyclictest18141-21kworker/0:221:41:390
2098799271,12cyclictest0-21swapper/019:56:210
2098799271,12cyclictest0-21swapper/000:19:090
2098799271,11cyclictest0-21swapper/000:05:490
2098799271,10cyclictest0-21swapper/019:17:100
2098799271,10cyclictest0-21swapper/000:39:330
2099899261,9cyclictest9693-21diskmemload22:37:502
2099899261,12cyclictest0-21swapper/222:01:342
2099899261,12cyclictest0-21swapper/221:42:462
2099899261,12cyclictest0-21swapper/221:32:542
2099899261,11cyclictest0-21swapper/200:21:342
2099899261,11cyclictest0-21swapper/200:09:102
2099899261,10cyclictest0-21swapper/219:11:572
20998992610,3cyclictest32276-21kworker/2:221:28:542
2099899260,18cyclictest0-21swapper/221:08:382
2099899260,13cyclictest0-21swapper/223:01:502
2099899260,12cyclictest0-21swapper/220:27:582
2099399266,18cyclictest496-21jbd2/dm-0-819:24:301
2099399264,10cyclictest780-21rngd19:40:481
2099399262,2cyclictest780-21rngd19:15:121
2099399261,9cyclictest0-21swapper/100:21:301
2099399261,23cyclictest0-21swapper/119:12:061
20993992611,8cyclictest0-21swapper/120:11:101
2099399261,17cyclictest0-21swapper/122:07:581
2099399261,16cyclictest0-21swapper/122:01:261
2099399261,12cyclictest0-21swapper/121:25:581
2099399261,11cyclictest0-21swapper/121:48:221
2099399260,11cyclictest0-21swapper/121:43:401
2098799261,12cyclictest30951-21kworker/0:220:56:030
2099899251,18cyclictest0-21swapper/200:16:062
2099899251,11cyclictest0-21swapper/223:30:302
2099899251,11cyclictest0-21swapper/222:06:542
2099899251,11cyclictest0-21swapper/220:49:182
2099899251,11cyclictest0-21swapper/220:49:182
2099899251,11cyclictest0-21swapper/219:45:582
2099899251,10cyclictest0-21swapper/223:29:422
2099899250,4cyclictest16916-21kworker/2:023:14:022
2099899250,13cyclictest0-21swapper/220:01:022
2099899250,10cyclictest0-21swapper/221:53:262
2099399251,9cyclictest0-21swapper/121:18:301
2099399251,16cyclictest0-21swapper/122:21:281
2098799252,12cyclictest22267-21munin-run20:25:010
2098799251,9cyclictest0-21swapper/019:20:300
2098799251,14cyclictest0-21swapper/000:20:290
2099899243,3cyclictest16916-21kworker/2:000:28:022
2099899241,12cyclictest0-21swapper/223:49:102
2098799241,10cyclictest0-21swapper/021:11:410
20987992410,2cyclictest0-21swapper/023:35:330
20998992317,3cyclictest3033-21cut23:20:152
2099899231,3cyclictest32276-21kworker/2:222:25:302
2099899230,9cyclictest0-21swapper/222:17:422
2099899230,10cyclictest0-21swapper/222:47:422
2099899230,10cyclictest0-21swapper/220:33:422
2099399231,10cyclictest0-21swapper/122:33:181
2099399230,13cyclictest0-21swapper/122:25:141
2099899223,2cyclictest86650irq/127-enp1s0-21:35:142
20998992217,3cyclictest17865-21turbostat20:15:022
2099899221,14cyclictest0-21swapper/222:43:502
20998992211,3cyclictest0-21swapper/221:47:022
2099899221,10cyclictest0-21swapper/221:13:342
2099899221,10cyclictest0-21swapper/220:19:022
2099399226,3cyclictest0-21swapper/120:26:221
20993992211,5cyclictest0-21swapper/123:38:441
2099399220,11cyclictest0-21swapper/122:51:161
2099399220,11cyclictest0-21swapper/122:51:161
2099899211,13cyclictest0-21swapper/219:21:022
2099899211,10cyclictest0-21swapper/221:24:382
2099899211,10cyclictest0-21swapper/219:42:062
2099899210,11cyclictest0-21swapper/223:15:262
2099899207,2cyclictest32276-21kworker/2:219:55:222
2099899201,12cyclictest0-21swapper/220:21:182
2099899201,11cyclictest0-21swapper/220:07:102
2099899191,12cyclictest0-21swapper/220:40:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional