You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-21 - 18:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sun Jun 21, 2026 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
310512239167,60sleep20-21swapper/207:07:512
309292238161,25sleep00-21swapper/007:06:150
309352204168,23sleep10-21swapper/107:06:201
309892202166,24sleep30-21swapper/307:07:043
2948721500,19sleep23131199cyclictest10:45:012
31311999389,3cyclictest0-21swapper/209:31:372
31311996966,2cyclictest0-21swapper/207:16:232
282712570,4sleep339-21rcuc/309:25:243
115602530,3sleep211564-21cpuspeed_turbos08:50:122
31311994339,3cyclictest0-21swapper/210:53:202
3130599391,27cyclictest0-21swapper/108:11:101
3130599391,26cyclictest0-21swapper/108:54:221
3130599371,25cyclictest0-21swapper/107:18:141
3131699361,22cyclictest0-21swapper/312:38:313
31311993611,22cyclictest18658-21/usr/sbin/munin10:20:172
3129799361,4cyclictest0-21swapper/009:11:160
3131199353,4cyclictest31560-40dnf10:49:262
3131199351,9cyclictest27164-21df09:25:122
31311993511,3cyclictest4234-21/usr/sbin/munin07:20:152
3130599350,26cyclictest0-21swapper/112:28:501
3130599350,26cyclictest0-21swapper/112:28:501
3129799351,2cyclictest0-21swapper/009:33:100
3129799351,11cyclictest12737-21sshd07:39:240
3129799351,10cyclictest2655-21/usr/sbin/munin08:30:260
3129799350,11cyclictest20645-21kworker/0:007:56:080
3131199341,5cyclictest1691-21/usr/sbin/munin10:55:252
31311993411,3cyclictest25252-21/usr/sbin/munin10:35:202
31311993411,3cyclictest15838-21/usr/sbin/munin09:00:272
31311993410,4cyclictest13504-21timerwakeupswit10:05:292
31311993410,4cyclictest13504-21timerwakeupswit10:05:292
3130599341,23cyclictest0-21swapper/108:28:341
3129799341,10cyclictest6743-21kworker/0:208:41:050
3131199331,4cyclictest15518-21awk10:10:252
31311993311,3cyclictest10489-21/usr/sbin/munin11:15:232
31311993311,3cyclictest10489-21/usr/sbin/munin11:15:232
3129799331,11cyclictest22653-21kworker/0:310:35:480
3129799331,10cyclictest808-21snmpd12:07:550
3131699321,12cyclictest5144-21/usr/sbin/munin07:20:183
3131199329,3cyclictest7043-21/usr/sbin/munin08:40:132
3131199328,3cyclictest3267-21/usr/sbin/munin09:45:232
3131199325,17cyclictest0-21swapper/211:50:292
31311993211,3cyclictest19618-21/usr/sbin/munin07:55:192
3129799328,3cyclictest19906-21kworker/0:209:24:450
3129799321,9cyclictest6743-21kworker/0:208:51:210
3129799321,6cyclictest18429-21sh10:20:010
3131199319,7cyclictest28191-21awk08:15:012
3131199319,3cyclictest4850-21/usr/sbin/munin08:35:132
3131199319,3cyclictest30612-21/usr/sbin/munin08:20:122
3131199318,3cyclictest1161-21ntpq12:05:222
3131199317,5cyclictest909-21sssd_nss08:25:212
3131199317,17cyclictest0-21swapper/209:11:112
3131199316,4cyclictest0-21swapper/208:30:182
3131199311,9cyclictest0-21swapper/209:36:012
3131199311,6cyclictest24075-21fschecks_count11:45:142
31311993115,13cyclictest16324-21ls07:45:212
31311993115,13cyclictest16324-21ls07:45:212
31311993111,2cyclictest12685-21/usr/sbin/munin11:20:122
31311993110,3cyclictest827-21runrttasks10:15:052
3130599311,13cyclictest0-21swapper/111:50:301
3130599311,12cyclictest0-21swapper/109:41:031
3129799312,8cyclictest22653-21kworker/0:310:41:250
3129799311,5cyclictest23579-21sshd08:02:240
3129799310,10cyclictest29380-21kworker/0:010:52:010
3131699301,14cyclictest0-21swapper/309:39:343
3131199308,3cyclictest4508-21/usr/sbin/munin12:15:192
3131199303,20cyclictest0-21swapper/212:31:182
3131199301,2cyclictest0-21swapper/209:44:242
3131199301,2cyclictest0-21swapper/208:05:172
3131199301,10cyclictest0-21swapper/212:22:032
3131199301,10cyclictest0-21swapper/209:57:282
31311993010,3cyclictest5724-21expr09:50:142
31311993010,3cyclictest14524-21sort08:55:172
31311993010,3cyclictest14524-21sort08:55:172
3130599301,9cyclictest0-21swapper/107:13:481
3130599301,17cyclictest0-21swapper/109:24:491
3130599301,12cyclictest0-21swapper/110:29:261
3130599301,11cyclictest0-21swapper/107:47:181
3130599301,11cyclictest0-21swapper/107:47:181
3130599300,23cyclictest0-21swapper/110:40:001
3130599300,18cyclictest0-21swapper/109:14:511
3129799306,3cyclictest32253-21kworker/0:212:10:560
3129799302,10cyclictest29380-21kworker/0:011:02:370
3129799301,7cyclictest11839-21latency_hist10:05:010
3129799301,5cyclictest20022-21diskmemload12:00:070
3129799301,19cyclictest12982-21kworker/0:012:34:160
3129799301,10cyclictest21165-21kworker/0:211:48:480
3131199299,3cyclictest827-21runrttasks11:37:142
3131199299,3cyclictest14119-21uname12:35:172
3131199297,3cyclictest9242-21/usr/sbin/munin08:45:162
3131199296,4cyclictest3562-21/usr/sbin/munin12:10:232
3131199296,4cyclictest10391-21/usr/sbin/munin10:00:162
3131199295,3cyclictest87350irq/126-enp1s0-07:25:212
3131199294,5cyclictest0-21swapper/209:05:162
3131199291,6cyclictest25022-21latency_hist10:35:012
3131199291,2cyclictest0-21swapper/211:30:112
31311992910,2cyclictest827-21runrttasks07:36:342
3130599291,9cyclictest0-21swapper/110:55:181
3130599291,13cyclictest0-21swapper/107:56:381
3130599291,11cyclictest0-21swapper/108:34:381
3130599291,10cyclictest0-21swapper/108:16:261
3129799296,3cyclictest19906-21kworker/0:209:16:130
3129799291,9cyclictest30329-21kworker/0:308:27:530
3129799291,10cyclictest5671-21kworker/0:211:10:130
3129799291,10cyclictest12162-21kworker/0:011:22:000
3131699287,8cyclictest0-21swapper/307:31:343
3131699281,12cyclictest0-21swapper/311:19:463
3131699281,12cyclictest0-21swapper/311:19:463
3131199289,3cyclictest6428-21df11:05:142
3131199287,3cyclictest30382-21latency_hist08:20:012
3131199281,6cyclictest0-21swapper/209:15:112
3131199281,5cyclictest14881-21/usr/sbin/munin11:25:152
3131199281,3cyclictest12050irq/123-ahci[0007:45:022
3131199281,2cyclictest87350irq/126-enp1s0-12:30:012
3131199281,2cyclictest87350irq/126-enp1s0-12:30:012
3130599289,13cyclictest0-21swapper/111:23:461
3130599281,9cyclictest0-21swapper/109:26:471
3130599281,3cyclictest0-21swapper/109:56:061
3130599281,12cyclictest0-21swapper/109:48:431
3130599281,11cyclictest808-21snmpd12:02:541
3130599281,11cyclictest0-21swapper/112:05:301
3130599281,11cyclictest0-21swapper/111:18:371
3130599281,11cyclictest0-21swapper/111:18:371
3130599281,11cyclictest0-21swapper/111:02:381
3130599281,11cyclictest0-21swapper/109:08:351
3130599281,10cyclictest0-21swapper/111:05:271
3130599281,10cyclictest0-21swapper/108:59:111
3130599281,10cyclictest0-21swapper/108:59:101
3130599280,9cyclictest0-21swapper/109:31:031
3129799286,3cyclictest21165-21kworker/0:211:50:080
3129799284,8cyclictest0-21swapper/011:08:210
3129799281,8cyclictest5031-21kworker/0:009:56:200
31297992811,3cyclictest30329-21kworker/0:308:19:400
3129799281,11cyclictest698-21kworker/0:109:48:120
3131699271,9cyclictest0-21swapper/312:20:193
31316992713,7cyclictest0-21swapper/312:30:153
31316992712,2cyclictest18818-21timerandwakeup07:50:273
3131699271,15cyclictest0-21swapper/310:50:183
3131699270,2cyclictest18342-21/usr/sbin/munin11:30:223
3131199278,3cyclictest9577-21missed_timers07:30:202
3131199278,3cyclictest827-21runrttasks07:54:442
3131199278,3cyclictest4290-21fschecks_count11:00:152
3131199277,3cyclictest22135-21wc10:25:222
3131199273,7cyclictest21580-21tr11:40:102
3131199271,9cyclictest87350irq/126-enp1s0-11:11:372
3131199271,5cyclictest0-21swapper/209:23:322
3131199271,4cyclictest0-21swapper/208:02:512
3131199271,3cyclictest87550irq/128-enp1s0-11:55:222
3130599271,9cyclictest0-21swapper/112:32:301
3130599271,9cyclictest0-21swapper/107:34:271
3130599271,6cyclictest0-21swapper/111:30:431
3130599271,2cyclictest0-21swapper/107:44:231
3130599271,12cyclictest11126-21smart_sda10:00:241
3130599271,12cyclictest0-21swapper/110:18:201
3130599271,12cyclictest0-21swapper/107:20:031
3130599271,11cyclictest0-21swapper/111:35:041
3130599271,11cyclictest0-21swapper/107:53:101
3130599270,16cyclictest0-21swapper/110:14:121
3129799277,2cyclictest3830-21kworker/0:207:23:290
31297992711,2cyclictest21165-21kworker/0:211:55:280
3131699261,9cyclictest3613-21diskstats09:45:143
3131699261,14cyclictest0-21swapper/311:36:543
3131699261,12cyclictest0-21swapper/307:29:463
3130599261,9cyclictest0-21swapper/112:16:061
3130599261,9cyclictest0-21swapper/111:40:141
3130599261,9cyclictest0-21swapper/109:01:151
3130599261,20cyclictest0-21swapper/111:14:471
3130599261,17cyclictest0-21swapper/111:55:421
3130599261,13cyclictest0-21swapper/108:45:241
3130599261,12cyclictest0-21swapper/108:37:061
3130599261,11cyclictest0-21swapper/112:36:141
3130599261,10cyclictest0-21swapper/110:40:501
3130599261,10cyclictest0-21swapper/110:21:201
3130599261,10cyclictest0-21swapper/108:03:061
3130599261,10cyclictest0-21swapper/107:26:191
3130599260,9cyclictest0-21swapper/108:22:461
3130599260,11cyclictest0-21swapper/108:44:521
3129799266,4cyclictest30329-21kworker/0:308:20:040
3129799261,9cyclictest25742-21kworker/0:208:07:320
3129799261,9cyclictest22653-21kworker/0:310:28:280
3129799261,12cyclictest12162-21kworker/0:011:33:160
3129799261,10cyclictest12162-21kworker/0:011:38:480
3129799260,24cyclictest13194-21kworker/0:009:03:210
3131699251,9cyclictest0-21swapper/307:41:303
3131699251,17cyclictest0-21swapper/308:24:113
3131699251,15cyclictest0-21swapper/308:29:113
3131199258,3cyclictest827-21runrttasks07:12:172
3131199251,9cyclictest0-21swapper/212:00:332
3130599258,3cyclictest0-21swapper/110:51:421
3130599251,9cyclictest0-21swapper/108:06:221
3130599251,10cyclictest0-21swapper/107:37:591
3130599250,11cyclictest0-21swapper/112:13:321
3129799259,3cyclictest9584-21kworker/0:110:06:160
3129799259,3cyclictest9584-21kworker/0:110:06:160
3129799259,2cyclictest19906-21kworker/0:209:29:360
3129799256,3cyclictest6112-21kworker/0:007:29:200
3129799252,9cyclictest29380-21kworker/0:010:49:450
195832251,2sleep319685-21ntp_kernel_err10:20:203
3131699249,3cyclictest3466-21irqstats08:30:193
3131699246,4cyclictest0-21swapper/310:16:553
3131699241,11cyclictest0-21swapper/309:01:263
31316992410,3cyclictest0-21swapper/307:15:143
3130599241,14cyclictest30709-21ntp_states10:45:221
3130599241,14cyclictest0-21swapper/112:20:141
3130599241,13cyclictest0-21swapper/109:17:311
3130599241,11cyclictest0-21swapper/111:48:221
3130599241,11cyclictest0-21swapper/110:05:021
3130599241,11cyclictest0-21swapper/110:05:021
3129799249,3cyclictest8330-21kworker/0:107:32:080
3129799248,2cyclictest698-21kworker/0:109:41:160
3129799248,2cyclictest21165-21kworker/0:211:44:160
3129799247,4cyclictest3429-21kworker/0:110:57:170
3129799242,11cyclictest31223-21/usr/sbin/munin09:35:200
3129799242,10cyclictest5671-21kworker/0:211:16:290
3129799242,10cyclictest5671-21kworker/0:211:16:280
3129799241,5cyclictest0-21swapper/010:11:550
156782240,2sleep00-21swapper/009:00:020
156782240,2sleep00-21swapper/009:00:020
3131699231,15cyclictest0-21swapper/310:06:513
3131699231,15cyclictest0-21swapper/310:06:503
3130599231,9cyclictest0-21swapper/110:30:181
3130599231,12cyclictest0-21swapper/111:27:351
3130599230,12cyclictest0-21swapper/109:51:271
3130599230,10cyclictest0-21swapper/109:37:471
3129799238,3cyclictest14283-21kworker/0:010:30:480
3129799238,2cyclictest7770-21kworker/0:112:25:200
3129799238,2cyclictest7770-21kworker/0:112:25:200
3129799237,3cyclictest7770-21kworker/0:112:23:320
3129799232,11cyclictest745-21kworker/0:007:13:560
3131699228,8cyclictest0-21swapper/312:08:273
3131699228,3cyclictest14881-21/usr/sbin/munin11:25:233
3131699228,1cyclictest0-21swapper/310:49:503
3131699227,4cyclictest0-21swapper/312:27:353
3131699227,4cyclictest0-21swapper/312:27:353
3131699221,13cyclictest0-21swapper/312:15:153
3131699221,11cyclictest0-21swapper/310:11:553
3131699221,11cyclictest0-21swapper/309:41:263
3129799228,3cyclictest336-21kworker/0:008:36:050
3129799226,4cyclictest5031-21kworker/0:009:51:000
3129799226,3cyclictest25742-21kworker/0:208:11:080
3129799221,10cyclictest4116-21kworker/0:012:16:200
3129799221,10cyclictest13023-21/usr/sbin/munin07:40:140
3131699219,3cyclictest0-21swapper/310:41:033
3131699216,6cyclictest0-21swapper/311:06:163
3131699211,4cyclictest0-21swapper/310:58:123
31316992112,7cyclictest26547-21systemd-journal09:24:273
3131699211,11cyclictest0-21swapper/310:27:353
3131699211,10cyclictest0-21swapper/307:13:153
3129799217,6cyclictest10914-21kworker/0:108:48:130
3129799217,3cyclictest13297-21kworker/0:107:17:290
3129799217,2cyclictest14459-21kworker/0:111:26:520
3129799216,2cyclictest10474-21kworker/0:307:47:280
3129799216,2cyclictest10474-21kworker/0:307:47:280
3129799215,2cyclictest10474-21kworker/0:307:53:000
3131699209,3cyclictest18044-21/usr/sbin/munin09:05:163
3131699208,4cyclictest0-21swapper/310:00:273
3131699205,9cyclictest16974-40dnf07:47:253
3131699205,9cyclictest16974-40dnf07:47:253
3131699205,3cyclictest0-21swapper/311:43:383
3131699203,6cyclictest0-21swapper/312:03:423
3131699202,10cyclictest29029-21/usr/sbin/munin09:30:163
3131699201,7cyclictest0-21swapper/309:10:223
3131699201,3cyclictest20430-21irqstats07:55:173
3131699201,13cyclictest0-21swapper/308:45:303
31316992010,3cyclictest0-21swapper/308:41:383
3131699200,13cyclictest0-21swapper/308:17:313
3129799200,11cyclictest7770-21kworker/0:112:36:130
3131699195,3cyclictest0-21swapper/311:47:063
3131699193,3cyclictest0-21swapper/311:57:303
3131699191,7cyclictest25022-21sh10:35:023
3131699191,6cyclictest8564-1kworker/u9:109:15:303
31316991913,4cyclictest24023-21/usr/sbin/munin08:05:143
3131699191,11cyclictest8287-21/usr/sbin/munin11:10:213
3131699191,10cyclictest0-21swapper/312:10:313
3131699191,10cyclictest0-21swapper/310:37:353
3129799191,7cyclictest0-21swapper/010:20:100
3129799190,11cyclictest13194-21kworker/0:009:09:090
3131699184,7cyclictest87550irq/128-enp1s0-08:02:253
3131699184,3cyclictest87350irq/126-enp1s0-11:01:423
3131699182,13cyclictest12789-21crond07:40:003
3131699181,8cyclictest11442-21/usr/sbin/munin08:50:163
3131699181,10cyclictest25865-21/usr/sbin/munin11:50:173
3131699180,9cyclictest26587-21kworker/u8:109:55:103
3131699180,12cyclictest0-21swapper/308:12:253
3131699175,3cyclictest0-21swapper/308:35:263
3131699172,8cyclictest87350irq/126-enp1s0-09:50:173
3131699171,13cyclictest20022-21diskmemload11:23:423
3131699171,13cyclictest13639-21/usr/sbin/munin08:55:163
3131699171,13cyclictest13639-21/usr/sbin/munin08:55:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional