You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-02 - 09:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 02, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
324182230157,60sleep00-21swapper/019:06:480
326502214162,40sleep10-21swapper/119:09:501
325902204165,26sleep30-21swapper/319:09:043
325172201165,23sleep20-21swapper/219:08:062
2023921440,5sleep03273899cyclictest19:50:260
32750996662,3cyclictest0-21swapper/221:19:592
173382610,3sleep30-21swapper/321:00:123
141182530,8sleep07-21ksoftirqd/020:50:210
6722460,1sleep00-21swapper/022:50:010
6722460,1sleep00-21swapper/022:50:010
32750994441,2cyclictest0-21swapper/220:06:062
21912350,3sleep23275099cyclictest22:50:232
3273899339,2cyclictest24191-21kworker/0:122:55:270
3274699304,17cyclictest0-21swapper/120:02:471
3273899307,3cyclictest11214-21kworker/0:019:11:330
3273899307,3cyclictest11214-21kworker/0:019:11:330
3273899302,13cyclictest24887-21munin-run23:45:000
3273899301,14cyclictest22686-21munin-run23:40:000
3273899292,17cyclictest27187-21kworker/0:320:07:170
3273899291,18cyclictest3718-21kworker/0:120:32:570
3273899290,8cyclictest939-21kworker/0:000:00:450
3275699283,6cyclictest0-21swapper/322:40:233
3273899282,14cyclictest8878-21munin-run21:55:010
3274699278,3cyclictest0-21swapper/121:55:091
3274699277,3cyclictest0-21swapper/123:32:091
3274699273,3cyclictest0-21swapper/100:38:411
3274699271,4cyclictest5518-21irqstats21:45:171
3273899277,2cyclictest24191-21kworker/0:122:51:310
3273899273,11cyclictest0-21swapper/022:10:010
3273899272,12cyclictest5753-21munin-run00:15:010
3275699263,6cyclictest9362-40dnf23:08:243
32746992610,13cyclictest2551-21systemd-journal22:50:391
3273899263,2cyclictest21100-21kworker/0:321:10:190
18022261,3sleep01906-21/usr/sbin/munin21:35:250
18022261,3sleep01906-21/usr/sbin/munin21:35:250
3275699253,8cyclictest18151-21chrt23:28:243
3275699253,2cyclictest8411-21grep21:50:253
3275699252,2cyclictest87350irq/126-enp1s0-22:55:213
3275699251,12cyclictest0-21swapper/321:15:083
3274699259,4cyclictest13787-21/usr/sbin/munin00:30:241
3274699251,9cyclictest0-21swapper/121:21:351
32746992517,2cyclictest5451-21crond19:20:011
32746992513,4cyclictest1900-21systemd-journal20:25:101
32746992513,4cyclictest1900-21systemd-journal20:25:101
3274699251,11cyclictest0-21swapper/121:05:151
3274699251,10cyclictest0-21swapper/121:10:351
3274699250,17cyclictest0-21swapper/121:54:091
3273899252,9cyclictest29496-21kworker/0:100:16:490
3273899251,12cyclictest30855-21munin-run22:45:010
3273899251,12cyclictest12063-21basename19:35:010
3275699243,16cyclictest41-21ksoftirqd/322:20:133
3275699243,11cyclictest0-21swapper/322:26:273
3275699242,4cyclictest26646-21unixbench_singl23:45:243
3275699241,16cyclictest0-21swapper/321:25:153
32756992411,2cyclictest401ktimersoftd/320:10:213
3274699249,3cyclictest0-21swapper/122:56:051
3274699246,3cyclictest0-21swapper/123:06:031
3274699246,3cyclictest0-21swapper/120:51:311
3274699246,3cyclictest0-21swapper/120:18:311
3274699245,5cyclictest0-21swapper/123:21:171
3274699245,3cyclictest0-21swapper/100:10:331
3274699244,2cyclictest0-21swapper/123:52:011
3274699241,9cyclictest0-21swapper/100:05:431
3274699241,10cyclictest0-21swapper/122:33:271
3274699241,10cyclictest0-21swapper/121:35:511
3274699241,10cyclictest0-21swapper/121:35:511
3274699241,10cyclictest0-21swapper/119:13:571
3274699241,10cyclictest0-21swapper/119:13:571
3274699240,10cyclictest0-21swapper/121:40:211
3273899248,2cyclictest1635-21kworker/0:020:40:330
3273899247,3cyclictest13995-21kworker/0:119:58:250
3273899247,2cyclictest1635-21kworker/0:020:45:410
3273899246,3cyclictest21947-21kworker/0:222:27:590
3273899246,2cyclictest21947-21kworker/0:222:30:110
3273899242,2cyclictest87550irq/128-enp1s0-19:25:410
3273899242,2cyclictest24705-21kworker/0:323:50:550
3273899242,10cyclictest13761-21kworker/0:223:21:430
3273899241,9cyclictest14498-21kworker/0:200:38:130
3273899241,7cyclictest23595-21kworker/0:121:28:250
3273899241,12cyclictest32456-21idleruntime-cro21:35:010
3273899240,8cyclictest29569-21kworker/0:220:17:430
129342240,3sleep012935-21gltestperf19:35:130
3275699237,9cyclictest87350irq/126-enp1s0-21:55:243
3275699235,7cyclictest12050irq/123-ahci[0019:55:023
3275699232,3cyclictest0-21swapper/322:04:323
3275699231,7cyclictest0-21swapper/323:17:263
3275699231,6cyclictest0-21swapper/323:00:523
3275099235,7cyclictest26459-21crond22:35:002
3275099231,9cyclictest21459-21diskmemload23:04:262
3275099231,6cyclictest0-21swapper/222:18:262
3275099231,4cyclictest0-21swapper/222:20:192
3274699239,3cyclictest0-21swapper/123:11:331
3274699238,3cyclictest0-21swapper/123:57:331
3274699238,3cyclictest0-21swapper/121:33:491
3274699238,2cyclictest87650irq/129-enp1s0-22:15:261
3274699236,6cyclictest23211-21memory22:25:181
3274699235,3cyclictest0-21swapper/122:42:591
3274699231,8cyclictest16797-21fschecks_time23:25:131
32746992317,3cyclictest16617-21memory22:10:181
3274699231,3cyclictest22690-21perf23:40:001
3274699230,20cyclictest0-21swapper/122:20:141
3273899237,3cyclictest12187-21kworker/0:000:31:330
3273899237,2cyclictest24925-21kworker/0:020:02:470
3273899237,2cyclictest17480-21kworker/0:022:24:110
3273899233,7cyclictest24705-21kworker/0:323:45:130
3273899232,9cyclictest1073-21kworker/0:200:05:430
3273899232,8cyclictest2202-21kworker/0:119:20:310
3273899231,8cyclictest24705-21kworker/0:323:58:250
54452220,1sleep10-21swapper/120:30:241
3275699224,6cyclictest12050irq/123-ahci[0019:30:013
3275699223,8cyclictest401ktimersoftd/300:20:133
32756992217,3cyclictest41-21ksoftirqd/320:08:263
3275699221,12cyclictest0-21swapper/300:08:523
3275699221,11cyclictest808-21snmpd23:21:273
3275699221,11cyclictest0-21swapper/321:45:263
3275699221,11cyclictest0-21swapper/321:30:223
3275699221,10cyclictest808-21snmpd20:25:083
3275699221,10cyclictest808-21snmpd20:25:083
3275699220,9cyclictest0-21swapper/321:13:363
3275099221,6cyclictest0-21swapper/223:50:222
3275099221,6cyclictest0-21swapper/223:09:222
3275099221,5cyclictest0-21swapper/223:55:222
3275099221,4cyclictest0-21swapper/223:44:062
3275099221,3cyclictest0-21swapper/221:40:552
3275099221,11cyclictest0-21swapper/223:28:092
3274699228,3cyclictest0-21swapper/122:47:431
3274699228,3cyclictest0-21swapper/122:47:431
3274699227,9cyclictest0-21swapper/121:28:061
3274699226,3cyclictest0-21swapper/120:21:591
3274699226,3cyclictest0-21swapper/119:20:591
3274699226,3cyclictest0-21swapper/100:18:231
3274699225,2cyclictest0-21swapper/123:00:571
3274699224,3cyclictest0-21swapper/119:35:271
3274699224,3cyclictest0-21swapper/119:32:271
3274699224,3cyclictest0-21swapper/100:04:531
3274699223,6cyclictest27145-40dnf20:06:271
3274699221,8cyclictest0-21swapper/122:37:131
3274699221,18cyclictest24164-21mailstats23:40:221
3273899228,3cyclictest4846-21kworker/0:223:06:030
3273899228,2cyclictest2397-21kworker/0:319:16:110
3273899226,3cyclictest87350irq/126-enp1s0-21:05:150
3273899222,9cyclictest23595-21kworker/0:121:21:350
3273899221,9cyclictest27187-21kworker/0:320:21:590
3273899221,8cyclictest15938-21kworker/0:323:33:390
3273899221,6cyclictest0-21swapper/021:49:540
177352220,3sleep017739-21latency_hist22:15:010
3275699218,5cyclictest12050irq/123-ahci[0020:20:003
3275699218,4cyclictest839-21lldpd22:15:223
3275699216,6cyclictest0-21swapper/322:34:233
3275699215,7cyclictest26688-21/usr/sbin/munin22:35:153
3275699213,10cyclictest401ktimersoftd/300:00:153
3275699212,9cyclictest808-21snmpd20:45:113
3275699211,9cyclictest0-21swapper/322:50:243
3275699211,6cyclictest0-21swapper/323:51:493
3275699211,12cyclictest10074-21/usr/sbin/munin19:30:243
3275699211,11cyclictest808-21snmpd21:35:213
3275699211,11cyclictest808-21snmpd21:35:213
3275699211,11cyclictest0-21swapper/323:35:593
3275699211,11cyclictest0-21swapper/300:14:223
3275699210,9cyclictest0-21swapper/300:28:443
3274699218,2cyclictest0-21swapper/120:39:291
3274699217,3cyclictest0-21swapper/122:06:131
3274699215,3cyclictest0-21swapper/120:43:211
32746992112,3cyclictest17060-21tune2fs19:45:141
3274699211,18cyclictest0-21swapper/123:15:051
32746992111,3cyclictest29765-21cat20:15:011
3274699211,13cyclictest0-21swapper/100:20:021
3273899217,2cyclictest10846-21kworker/0:122:00:070
3273899217,2cyclictest0-21swapper/020:25:110
3273899217,2cyclictest0-21swapper/020:25:110
3273899214,2cyclictest87450irq/127-enp1s0-19:42:330
3273899211,8cyclictest15938-21kworker/0:323:28:070
3273899211,12cyclictest9729-21/usr/sbin/munin23:10:150
64702200,4sleep011-21rcuc/020:35:130
3275699209,3cyclictest87650irq/129-enp1s0-20:40:173
3275699209,3cyclictest808-21snmpd19:24:563
3275699203,6cyclictest87350irq/126-enp1s0-19:15:263
3275699202,9cyclictest87350irq/126-enp1s0-20:20:123
3275699202,9cyclictest14469-21/usr/sbin/munin19:40:243
3275699202,10cyclictest16669-21/usr/sbin/munin19:45:143
3275699201,9cyclictest18884-21fschecks_count23:30:153
3275699201,9cyclictest0-21swapper/322:10:113
3275699201,8cyclictest0-21swapper/320:00:413
3275699201,5cyclictest0-21swapper/321:24:223
3275699201,4cyclictest0-21swapper/321:44:593
3275699201,4cyclictest0-21swapper/300:15:463
3275699201,3cyclictest0-21swapper/323:40:293
3275699201,11cyclictest0-21swapper/300:34:373
3275099209,2cyclictest30-21rcuc/220:25:182
3275099209,2cyclictest30-21rcuc/220:25:182
3275099206,3cyclictest4525-21ntpq19:15:192
3275099201,4cyclictest0-21swapper/223:38:142
32750992011,3cyclictest87550irq/128-enp1s0-22:00:022
3275099200,6cyclictest0-21swapper/219:55:172
3274699207,3cyclictest0-21swapper/119:43:031
3274699206,3cyclictest0-21swapper/123:45:551
32746992015,3cyclictest11215-21grep20:45:141
3274699201,3cyclictest0-21swapper/120:56:191
3274699201,16cyclictest9842-21latency_hist19:30:011
3273899207,3cyclictest29496-21kworker/0:100:20:030
3273899206,3cyclictest21100-21kworker/0:321:15:470
3273899204,4cyclictest5880-21perl23:00:140
3273899202,8cyclictest2088-21kworker/0:221:41:070
3273899201,6cyclictest0-21swapper/022:16:590
3275699196,4cyclictest0-21swapper/322:45:423
3275699196,4cyclictest0-21swapper/322:45:423
3275699195,2cyclictest21469-21basename21:10:013
3275699193,5cyclictest405-1kworker/u9:023:55:303
3275699191,9cyclictest0-21swapper/320:59:273
3275699191,7cyclictest0-21swapper/319:55:133
3275699191,3cyclictest0-21swapper/322:09:023
3275699191,10cyclictest0-21swapper/300:36:063
3275699190,5cyclictest39-21rcuc/323:10:173
3275099196,4cyclictest7142-21grep20:35:182
3275099195,4cyclictest32027-21grep21:30:262
3275099193,4cyclictest0-21swapper/221:11:522
3275099192,6cyclictest12610-21df19:35:102
3275099192,4cyclictest0-21swapper/223:16:442
3275099192,14cyclictest13175-21head00:30:162
3275099191,7cyclictest0-21swapper/221:39:152
3275099191,7cyclictest0-21swapper/221:39:152
3275099191,6cyclictest0-21swapper/221:21:202
3275099191,5cyclictest0-21swapper/222:47:542
3275099191,5cyclictest0-21swapper/222:47:542
3275099191,5cyclictest0-21swapper/221:50:592
32750991914,3cyclictest30862-21perf22:45:012
32750991912,4cyclictest8957-21tune2fs20:40:142
3275099191,15cyclictest11586-21chrt23:13:392
3274699191,2cyclictest0-21swapper/119:57:011
3274699191,16cyclictest0-21swapper/119:50:261
3273899196,3cyclictest12599-21kworker/0:120:59:330
3273899194,3cyclictest88950irq/132-enp2s0-20:10:550
3273899191,6cyclictest0-21swapper/022:35:100
3273899191,6cyclictest0-21swapper/000:25:320
183962191,2sleep20-21swapper/219:47:042
54802181,2sleep20-21swapper/200:11:432
3275099189,2cyclictest87550irq/128-enp1s0-00:15:332
3275099185,4cyclictest16640-21sed20:55:262
3275099184,4cyclictest4770-21egrep21:45:122
3275099183,3cyclictest20547-21ntp_states21:05:202
3275099183,3cyclictest0-21swapper/222:35:242
3275099181,8cyclictest10748-21egrep20:45:102
3275099181,8cyclictest10147-21munin-run00:25:012
3275099181,7cyclictest20961-21cat19:55:012
3275099181,7cyclictest18479-21kworker/u8:221:25:222
3275099181,6cyclictest4006-21latency22:55:182
3275099181,6cyclictest0-21swapper/222:13:312
3275099181,6cyclictest0-21swapper/200:25:202
3275099181,5cyclictest0-21swapper/223:30:312
3275099181,4cyclictest0-21swapper/221:55:262
3275099181,10cyclictest0-21swapper/223:49:052
3274699184,3cyclictest11644-21df_inode22:00:131
3274699181,6cyclictest0-21swapper/100:26:471
3274699181,16cyclictest0-21swapper/121:17:511
3274699181,16cyclictest0-21swapper/121:00:111
3273899185,2cyclictest9293-21kworker/0:123:15:110
3273899181,8cyclictest0-21swapper/021:02:270
3273899181,6cyclictest0-21swapper/021:56:430
173562180,3sleep011-21rcuc/019:45:160
146382180,2sleep30-21swapper/320:52:223
3275699177,7cyclictest6471-21expr20:35:133
3275699176,3cyclictest87350irq/126-enp1s0-20:30:133
3275699171,3cyclictest0-21swapper/319:36:463
3275099172,5cyclictest15947-21grep19:40:242
3275099171,7cyclictest2156-21sed19:10:262
3275099171,7cyclictest2156-21sed19:10:262
3275099171,6cyclictest827-21runrttasks20:00:122
3275099171,6cyclictest1579-21grep00:05:112
3275099171,5cyclictest31495-21kworker/u8:200:00:122
3275099171,4cyclictest14662-21proc_pri22:05:252
3275099171,4cyclictest0-21swapper/223:20:592
3275099171,4cyclictest0-21swapper/222:25:132
3275099171,4cyclictest0-21swapper/200:35:392
32750991711,4cyclictest0-21swapper/219:25:132
32750991711,3cyclictest32054-21/usr/sbin/munin20:20:202
317802170,3sleep20-21swapper/220:19:342
25172171,2sleep30-21swapper/319:14:343
25172171,2sleep30-21swapper/319:14:343
3275099167,3cyclictest0-21swapper/220:32:192
3275099165,3cyclictest12044-21crond19:35:012
3275099161,6cyclictest27662-21/usr/sbin/munin20:10:142
32750991612,3cyclictest0-21swapper/220:53:532
32750991612,3cyclictest0-21swapper/219:22:382
32750991611,3cyclictest17277-21/usr/sbin/munin21:00:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional