You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-01 - 13:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Jul 01, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
268632236162,24sleep20-21swapper/219:08:112
267382217174,34sleep00-21swapper/019:06:330
267762204168,24sleep30-21swapper/319:07:033
266642203167,24sleep10-21swapper/119:05:371
271019910798,3cyclictest15796-21diskmemload22:24:352
27101996048,2cyclictest0-21swapper/220:56:392
175952560,5sleep22710199cyclictest23:40:132
84022470,1sleep00-21swapper/000:35:200
320452470,3sleep232047-21missed_timers00:15:212
228752460,1sleep10-21swapper/121:25:111
2710699431,30cyclictest22727-21ntp_states23:50:223
2710699431,21cyclictest15515-21ntp_states22:20:233
2710699421,19cyclictest21644-21gltestperf22:35:173
2710699414,28cyclictest5965-21expr00:30:173
2710699411,30cyclictest5304-21/usr/sbin/munin23:10:223
2710699391,28cyclictest28984-21latency_hist21:40:013
2710699391,28cyclictest28984-21latency_hist21:40:013
2710699391,22cyclictest13583-21fschecks_count19:50:153
2710699381,27cyclictest21324-21runrttasks23:48:503
2710699380,29cyclictest5157-21latency_hist00:30:023
2710699371,28cyclictest31752-21df21:45:143
2710699371,26cyclictest11470-21cut19:45:143
2710699371,22cyclictest9386-21chrt00:38:493
2710699366,23cyclictest27256-21sshd22:48:123
2710699361,27cyclictest18841-21/usr/sbin/munin22:30:203
2710699361,25cyclictest29803-21cat22:55:023
2710699361,24cyclictest1985-21latency_hist19:25:013
2710699361,22cyclictest1930-21hddtemp_smartct21:50:173
2710699361,20cyclictest579-21systemd-journal19:40:013
2710699361,19cyclictest28366-21latency_hist20:25:023
2710699361,12cyclictest87350irq/126-enp1s0-23:30:013
2710699360,27cyclictest0-21swapper/323:42:523
2708899366,14cyclictest7003-21munin-run20:50:020
2710699359,18cyclictest8814-21/usr/sbin/munin19:40:213
2710699356,22cyclictest14839-21munin-run23:35:013
2710699355,19cyclictest30879-21grep19:15:163
2710699355,19cyclictest30879-21grep19:15:153
2710699352,24cyclictest1-21systemd22:40:123
2710699351,31cyclictest16605-21unixbench_singl23:35:263
2710699351,25cyclictest3561-21expr21:55:143
2710699351,24cyclictest26180-21cpuspeed_turbos00:00:123
2710699351,21cyclictest22005-21/usr/sbin/munin20:10:173
2710699350,22cyclictest0-21swapper/319:30:273
2710199351,14cyclictest0-21swapper/220:25:572
2710199351,14cyclictest0-21swapper/220:25:572
2710699343,24cyclictest87350irq/126-enp1s0-23:15:163
2710699341,25cyclictest808-21snmpd20:44:453
2710699341,21cyclictest827-21runrttasks19:11:063
2710699341,21cyclictest827-21runrttasks19:11:063
2710699341,19cyclictest15792-21latency_hist21:10:023
2710699341,19cyclictest0-21swapper/322:05:203
2710199341,15cyclictest0-21swapper/222:54:292
2710699337,17cyclictest0-21swapper/320:05:273
2710699336,19cyclictest2939-21ntpq23:05:203
2710699333,22cyclictest0-21swapper/300:10:193
2710699332,26cyclictest16473-21ntp_states19:55:223
2710699332,23cyclictest23849-21/usr/sbin/munin23:55:123
2710699332,20cyclictest26151-21threads21:30:253
2710699331,23cyclictest3008-21latency_hist00:25:023
2710699331,20cyclictest15796-21diskmemload23:00:013
2710699331,19cyclictest766-21ntpd20:46:163
2710199330,14cyclictest0-21swapper/222:41:092
2710699325,19cyclictest32034-21/usr/sbin/munin23:00:193
2710699321,26cyclictest23873-21smart_sda21:25:223
2710699321,24cyclictest21582-21kworker/u8:120:55:113
2710699321,21cyclictest808-21snmpd00:09:343
2710699321,21cyclictest609-21cpuspeed_turbos20:35:133
2710699321,19cyclictest0-21swapper/321:10:213
2710699321,18cyclictest0-21swapper/320:00:263
2710699321,17cyclictest15796-21diskmemload22:11:433
2710699320,22cyclictest0-21swapper/320:30:133
27101993229,2cyclictest0-21swapper/200:39:332
2710199321,16cyclictest0-21swapper/222:45:172
2710199321,16cyclictest0-21swapper/219:49:102
2710199321,15cyclictest0-21swapper/223:51:062
2710199321,13cyclictest0-21swapper/223:12:132
2708899324,13cyclictest28050-21munin-run00:05:010
2710699316,14cyclictest18469-21ls21:15:123
2710699314,18cyclictest11622-21/usr/sbin/munin21:00:183
2710699311,21cyclictest9039-21fschecks_time23:20:153
2710699310,20cyclictest13916-21kworker/u8:022:25:073
2710199311,15cyclictest0-21swapper/223:19:292
2710699301,26cyclictest7228-21/usr/sbin/munin20:50:233
2710699301,26cyclictest5645-21/usr/sbin/munin22:00:283
2710699301,26cyclictest31094-21/usr/sbin/munin00:15:143
2710699301,26cyclictest28948-21/usr/sbin/munin20:25:143
2710699301,26cyclictest28948-21/usr/sbin/munin20:25:143
2710699301,26cyclictest24198-21/usr/sbin/munin20:15:153
2710699301,21cyclictest30838-21chrt21:40:423
2710699301,19cyclictest0-21swapper/321:24:153
2710699300,17cyclictest87350irq/126-enp1s0-22:20:023
2710199301,14cyclictest0-21swapper/222:15:582
2710199301,13cyclictest0-21swapper/223:37:182
2710199301,13cyclictest0-21swapper/222:13:102
2710199301,13cyclictest0-21swapper/219:35:252
2710199301,13cyclictest0-21swapper/200:08:062
2710199301,12cyclictest0-21swapper/223:05:252
2710199300,13cyclictest0-21swapper/221:28:342
2710199300,10cyclictest0-21swapper/222:38:052
2710699291,25cyclictest2217-21/usr/sbin/munin19:25:203
2710199291,14cyclictest0-21swapper/220:52:212
2710199291,13cyclictest0-21swapper/200:13:382
2710199291,11cyclictest0-21swapper/221:05:392
27095992914,2cyclictest0-21swapper/119:10:161
27095992914,2cyclictest0-21swapper/119:10:151
2708899291,13cyclictest18000-21munin-run21:15:010
27101992812,3cyclictest2608-21munin-run20:40:012
2710199281,16cyclictest0-21swapper/223:45:262
2710199281,13cyclictest0-21swapper/220:12:102
2710199280,12cyclictest0-21swapper/219:41:112
2710199280,12cyclictest0-21swapper/219:20:572
27101992711,3cyclictest5367-21sh20:45:132
2710199271,12cyclictest15796-21diskmemload21:53:022
2710199271,12cyclictest0-21swapper/221:18:252
2710199271,12cyclictest0-21swapper/221:10:172
2710199271,12cyclictest0-21swapper/200:02:182
2710199271,11cyclictest0-21swapper/219:14:222
2710199271,11cyclictest0-21swapper/219:14:222
2710199271,10cyclictest25918-21cut00:00:022
2710199271,10cyclictest15796-21diskmemload23:20:452
2710199270,17cyclictest0-21swapper/223:29:222
2710199270,13cyclictest0-21swapper/221:46:302
2710199270,13cyclictest0-21swapper/221:42:302
2710199270,13cyclictest0-21swapper/200:30:182
2710199270,12cyclictest0-21swapper/219:28:112
2709599273,4cyclictest6747-21fschecks_time23:15:171
2708899275,11cyclictest0-21swapper/022:30:000
2708899271,13cyclictest14219-21munin-run22:20:010
2710199261,15cyclictest0-21swapper/221:39:302
2710199261,15cyclictest0-21swapper/221:39:302
2710199261,13cyclictest0-21swapper/222:57:492
2710199261,12cyclictest0-21swapper/219:15:252
2710199261,12cyclictest0-21swapper/219:15:252
2710199261,11cyclictest0-21swapper/220:05:382
2709599261,4cyclictest0-21swapper/100:13:141
2708899261,13cyclictest28382-21awk20:25:000
2708899260,13cyclictest1665-21awk23:05:020
2710199259,3cyclictest15796-21diskmemload23:33:292
2710199251,9cyclictest0-21swapper/220:23:562
2710199251,8cyclictest0-21swapper/220:18:422
2710199251,12cyclictest0-21swapper/222:26:022
2710199251,11cyclictest0-21swapper/222:31:232
2710199251,11cyclictest0-21swapper/221:33:182
2710199251,11cyclictest0-21swapper/220:30:292
2710199250,10cyclictest0-21swapper/221:02:132
2709599253,11cyclictest87550irq/128-enp1s0-00:35:201
27095992519,3cyclictest17457-21irqstats22:25:181
27095992514,2cyclictest0-21swapper/119:38:191
2709599251,3cyclictest0-21swapper/123:42:121
2709599251,17cyclictest0-21swapper/123:04:251
2709599251,15cyclictest0-21swapper/123:35:301
2709599251,12cyclictest0-21swapper/123:53:441
27088992513,3cyclictest9462-21grep23:20:200
2708899251,11cyclictest12640-21munin-run23:30:020
2708899250,12cyclictest8594-21awk19:40:020
2710199241,8cyclictest0-21swapper/221:22:582
27101992412,3cyclictest0-21swapper/222:05:182
2710199241,11cyclictest0-21swapper/200:28:102
2710199241,10cyclictest0-21swapper/219:57:542
2710199240,9cyclictest0-21swapper/220:42:252
2710199240,10cyclictest0-21swapper/223:00:132
2709599245,7cyclictest0-21swapper/121:32:141
2709599240,22cyclictest0-21swapper/100:04:381
242312240,2sleep10-21swapper/122:40:191
2710199236,3cyclictest0-21swapper/221:56:122
27095992319,3cyclictest0-21swapper/122:10:261
2709599231,6cyclictest0-21swapper/100:09:331
2709599231,3cyclictest9016-21seq20:52:511
2709599231,12cyclictest0-21swapper/121:14:461
2709599230,16cyclictest87550irq/128-enp1s0-21:58:111
27088992316,4cyclictest32473-40dnf20:31:240
2708899231,16cyclictest0-21swapper/021:39:100
2708899231,16cyclictest0-21swapper/021:39:100
2708899231,10cyclictest30437-21tail22:55:160
2710199221,13cyclictest0-21swapper/219:30:132
2710199220,12cyclictest0-21swapper/219:51:122
2710199220,10cyclictest0-21swapper/200:20:502
2709599229,3cyclictest22843-21chrt22:38:021
2709599229,11cyclictest6675-21kworker/u8:019:47:421
2709599228,3cyclictest2198-21missed_timers21:50:211
2709599228,3cyclictest18474-21cat21:15:131
27095992216,3cyclictest7853-21/usr/sbin/munin22:05:231
2709599221,3cyclictest5028-21/usr/sbin/munin20:45:261
2709599221,18cyclictest0-21swapper/121:20:201
2709599221,10cyclictest23736-21taskset20:12:071
27095992210,5cyclictest23-21ksoftirqd/123:26:281
2709599220,5cyclictest0-21swapper/123:49:071
2708899228,8cyclictest0-21swapper/019:25:240
2708899222,10cyclictest1691-21if_enp2s000:20:190
128832220,2sleep112990-21/usr/sbin/munin21:00:241
2710199211,10cyclictest0-21swapper/222:01:382
2709599219,10cyclictest1978-21kworker/u8:220:44:411
2709599211,7cyclictest924-21sshd21:49:121
2709599211,5cyclictest0-21swapper/100:25:281
2709599211,4cyclictest0-21swapper/123:13:091
27095992112,8cyclictest0-21swapper/121:42:481
2709599211,17cyclictest30182-21diskstats22:55:141
2709599210,4cyclictest0-21swapper/123:08:081
2708899218,8cyclictest0-21swapper/019:54:090
2708899217,8cyclictest0-21swapper/021:59:110
2708899216,8cyclictest20897-21timerwakeupswit23:45:260
2708899211,9cyclictest0-21swapper/021:51:240
27088992112,4cyclictest0-21swapper/023:07:120
27088992111,3cyclictest22885-21grep23:50:230
27088992111,2cyclictest7416-21pmu-power23:15:240
2708899211,12cyclictest0-21swapper/021:27:200
2708899211,10cyclictest8435-21kworker/0:119:43:230
2710199200,12cyclictest0-21swapper/220:00:542
2709599209,2cyclictest0-21swapper/119:21:511
2709599208,3cyclictest5734-21fschecks_count00:30:151
2709599208,3cyclictest0-21swapper/100:15:181
2709599205,3cyclictest87550irq/128-enp1s0-22:50:201
2709599204,14cyclictest11389-21latency_hist21:00:011
2709599201,9cyclictest0-21swapper/121:05:301
2709599201,7cyclictest808-21snmpd20:19:401
2709599201,7cyclictest15572-21pmu-power22:20:231
2709599201,6cyclictest9050-21perl23:20:161
2709599201,17cyclictest0-21swapper/120:31:121
27095992011,6cyclictest1978-21kworker/u8:220:38:411
2709599200,4cyclictest0-21swapper/123:59:321
2709599200,4cyclictest0-21swapper/123:30:441
2708899208,2cyclictest21542-21cut21:20:230
2708899206,3cyclictest0-21swapper/020:35:120
2708899204,5cyclictest0-21swapper/021:19:380
2708899203,3cyclictest87550irq/128-enp1s0-22:31:110
2708899201,6cyclictest0-21swapper/021:42:300
2708899201,6cyclictest0-21swapper/000:30:210
2708899201,16cyclictest7628-21munin-run22:05:010
2708899201,11cyclictest27333-21sshd22:49:120
2708899201,10cyclictest0-21swapper/023:10:100
2708899200,10cyclictest0-21swapper/022:20:070
2709599194,6cyclictest0-21swapper/121:36:021
2709599194,6cyclictest0-21swapper/121:36:021
2709599192,4cyclictest5645-21/usr/sbin/munin22:00:161
2709599192,14cyclictest17373-21munin-run20:00:021
27095991916,2cyclictest0-21swapper/122:46:541
27095991912,4cyclictest13544-21df19:50:141
2709599191,17cyclictest0-21swapper/122:32:031
2709599191,17cyclictest0-21swapper/122:15:491
2709599191,17cyclictest0-21swapper/119:44:081
2709599191,15cyclictest3234-21netstat19:25:191
2709599190,3cyclictest0-21swapper/100:21:531
2709599190,10cyclictest0-21swapper/120:05:271
2708899199,3cyclictest23237-21/usr/sbin/munin22:40:120
2708899199,2cyclictest13449-21sshd21:03:120
2708899197,3cyclictest18245-1kworker/u9:200:25:340
2708899194,2cyclictest0-21swapper/019:55:120
2708899192,4cyclictest0-21swapper/020:15:520
2708899191,9cyclictest32235-21awk00:15:230
2708899191,4cyclictest26551-40dnf21:32:110
2708899191,4cyclictest0-21swapper/000:05:280
2708899191,3cyclictest0-21swapper/022:06:390
27088991912,4cyclictest9759-21df20:55:140
2708899191,12cyclictest0-21swapper/021:05:160
2708899191,11cyclictest0-21swapper/023:35:430
2708899191,11cyclictest0-21swapper/022:53:430
2708899191,11cyclictest0-21swapper/020:43:210
27095991812,3cyclictest17780-21cat20:00:121
2709599181,16cyclictest0-21swapper/120:29:421
2709599181,16cyclictest0-21swapper/120:29:421
2709599180,16cyclictest0-21swapper/119:32:121
2709599180,16cyclictest0-21swapper/119:17:361
2709599180,16cyclictest0-21swapper/119:17:351
2708899189,2cyclictest0-21swapper/022:10:160
2708899187,8cyclictest17403-21cpuspeed_turbos23:40:110
2708899187,3cyclictest0-21swapper/020:09:030
2708899186,4cyclictest18245-1kworker/u9:223:55:140
2708899184,2cyclictest28806-21grep19:10:260
2708899184,2cyclictest28806-21grep19:10:260
2708899183,3cyclictest87550irq/128-enp1s0-23:30:130
27088991813,3cyclictest29748-21tr20:25:220
27088991813,3cyclictest29748-21tr20:25:220
2708899181,14cyclictest30371-21sed00:10:210
2708899181,14cyclictest17602-21/usr/sbin/munin20:00:170
27095991712,3cyclictest26395-21/usr/sbin/munin20:20:151
2708899176,4cyclictest21863-21kernelversion22:35:200
2708899171,6cyclictest32258-21kernelversion21:45:180
2708899171,2cyclictest0-21swapper/019:20:130
27088991712,3cyclictest8488-21smart_sda20:50:240
2708899171,14cyclictest31314-21grep19:15:220
2708899171,14cyclictest31314-21grep19:15:220
2708899170,14cyclictest22764-21if_enp2s020:10:170
129492170,2sleep00-21swapper/019:49:470
2708899168,5cyclictest909-21sssd_nss19:35:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional