You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-11 - 18:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Thu Dec 11, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
63402237163,25sleep30-21swapper/307:06:593
64652236162,24sleep20-21swapper/207:08:352
65622234161,24sleep00-21swapper/007:09:500
65632233167,54sleep10-21swapper/107:09:501
2984621710,4sleep3668199cyclictest10:25:263
753621550,2sleep1667099cyclictest10:50:121
667799106103,2cyclictest0-21swapper/210:18:352
6677998783,3cyclictest0-21swapper/211:05:462
6677998178,2cyclictest0-21swapper/208:11:272
6677998178,2cyclictest0-21swapper/208:11:272
306682640,1sleep10-21swapper/110:30:011
9602550,4sleep011-21rcuc/010:35:140
36812500,1sleep30-21swapper/311:55:143
326682500,1sleep20-21swapper/209:20:142
668199421,29cyclictest30183-21/usr/sbin/munin09:15:263
668199408,17cyclictest25782-21/usr/sbin/munin09:05:233
668199401,23cyclictest15031-21wc11:05:243
668199391,25cyclictest86550irq/126-enp1s0-11:00:133
668199383,7cyclictest26751-40dnf07:50:333
666499378,13cyclictest8836-21munin-run07:15:010
668199363,28cyclictest9018-40dnf10:52:343
668199361,7cyclictest19177-21/usr/sbin/munin08:50:163
668199361,5cyclictest2220-21/usr/sbin/munin09:25:253
668199361,4cyclictest14041-21grep09:50:183
668199360,23cyclictest0-21swapper/308:35:243
668199354,20cyclictest0-21swapper/309:13:353
668199351,6cyclictest6376-21tune2fs08:20:153
6681993516,16cyclictest22324-21find12:35:273
668199351,5cyclictest31169-21taskset11:44:313
668199351,25cyclictest0-21swapper/308:41:243
668199351,24cyclictest0-21swapper/312:03:143
6681993510,22cyclictest16832-21grep12:25:153
6677993532,2cyclictest0-21swapper/208:17:172
668199341,6cyclictest31065-21cstates10:30:143
668199340,8cyclictest9543-21kworker/u8:212:13:323
668199336,3cyclictest86550irq/126-enp1s0-11:10:273
668199333,20cyclictest780-21rngd09:02:523
668199331,22cyclictest26417-21/usr/sbin/munin10:20:263
668199331,21cyclictest11236-21kworker/u8:011:15:323
668199331,13cyclictest0-21swapper/310:04:173
6681993310,20cyclictest21665-21processes11:20:233
6677993330,2cyclictest0-21swapper/208:57:092
6677993324,2cyclictest15864-21kworker/2:209:36:182
668199324,2cyclictest0-21swapper/310:15:143
668199323,3cyclictest19228-21df12:30:143
668199323,3cyclictest19228-21df12:30:143
668199322,4cyclictest27399-21diskstats11:35:193
668199321,5cyclictest0-21swapper/309:46:513
668199321,3cyclictest0-21swapper/311:25:463
668199321,3cyclictest0-21swapper/311:25:463
6681993210,19cyclictest21378-21/usr/sbin/munin08:55:223
668199319,19cyclictest7017-21idleruntime-cro10:50:023
668199319,19cyclictest28029-21hwlatdetect07:55:193
668199312,4cyclictest11649-21smart_sda08:30:243
668199311,4cyclictest1591-21/usr/sbin/munin08:10:173
668199311,4cyclictest1591-21/usr/sbin/munin08:10:173
668199311,3cyclictest9451-21/usr/sbin/munin10:55:183
668199311,3cyclictest86550irq/126-enp1s0-12:15:153
668199311,27cyclictest11233-21grep07:15:273
668199311,26cyclictest3788-21/usr/sbin/munin08:15:193
6681993111,17cyclictest3323-21grep10:40:163
6681993110,18cyclictest14634-21snmpd12:08:483
6681993110,18cyclictest14634-21snmpd07:22:533
6681993110,17cyclictest15091-21gltestperf12:20:163
666499312,16cyclictest18666-21munin-run12:30:000
286612310,1sleep10-21swapper/111:35:321
668199308,18cyclictest25823-21missed_timers11:30:213
668199307,20cyclictest8721-21fschecks_time08:25:173
668199302,6cyclictest32718-21df09:20:143
668199301,5cyclictest17131-21if_err_enp1s007:30:183
668199301,4cyclictest978-21/usr/sbin/munin10:35:163
668199301,4cyclictest30218-21hddtemp_smartct08:00:173
668199301,26cyclictest16898-21unixbench-2d09:55:263
668199301,21cyclictest14634-21snmpd07:47:583
6681993011,16cyclictest9169-21diskstats09:40:153
6681993010,17cyclictest21061-21fgrep10:05:253
6677993020,2cyclictest15864-21kworker/2:210:23:382
6677993019,3cyclictest15864-21kworker/2:208:01:542
668199291,4cyclictest7696-21uname07:10:173
668199291,4cyclictest21873-21/usr/sbin/munin07:40:233
668199291,4cyclictest19464-21memory07:35:203
668199291,4cyclictest17885-21/usr/sbin/munin08:45:203
668199291,3cyclictest32667-21/usr/sbin/munin11:45:263
668199291,3cyclictest23559-21/usr/sbin/munin10:10:273
668199291,3cyclictest14634-21snmpd08:08:003
668199291,3cyclictest0-21swapper/311:50:173
667799290,12cyclictest0-21swapper/211:15:182
6664992911,15cyclictest19613-21perf10:05:000
668199282,3cyclictest15070-21memory07:25:193
668199281,3cyclictest6969-21/usr/sbin/munin09:35:153
668199281,3cyclictest4768-21diskstats09:30:163
667799281,11cyclictest0-21swapper/211:55:262
666499289,16cyclictest29956-21munin-run09:15:000
666499281,14cyclictest23356-21idleruntime-cro09:00:000
667799271,10cyclictest0-21swapper/211:51:082
667799261,12cyclictest0-21swapper/209:43:262
667099263,5cyclictest86750irq/128-enp1s0-12:20:001
667099263,15cyclictest0-21swapper/110:34:531
667099261,17cyclictest7713-21open_inodes09:35:221
6670992610,3cyclictest9797-21/usr/sbin/munin09:40:191
666499260,12cyclictest18044-21awk11:15:000
667799258,3cyclictest15864-21kworker/2:212:00:142
667799252,14cyclictest15864-21kworker/2:211:30:222
667099257,10cyclictest0-21swapper/109:14:541
667099255,16cyclictest26751-40dnf07:50:331
667099251,17cyclictest0-21swapper/111:11:271
667099251,12cyclictest0-21swapper/109:00:111
666499252,10cyclictest11319-21kworker/0:011:03:550
6677992415,2cyclictest15864-21kworker/2:209:58:362
6677992414,2cyclictest15864-21kworker/2:210:34:582
6677992413,2cyclictest15864-21kworker/2:208:50:202
667099246,10cyclictest0-21swapper/111:25:311
667099246,10cyclictest0-21swapper/111:25:311
667099241,3cyclictest8200-21cat07:10:251
6670992411,6cyclictest16110-21systemd-journal07:29:591
667099240,18cyclictest0-21swapper/110:20:471
666499249,12cyclictest13004-21awk09:50:010
666499241,9cyclictest15139-21kworker/0:209:59:030
666499241,6cyclictest0-21swapper/012:02:200
666499241,12cyclictest29248-21/usr/sbin/munin11:40:130
667799239,3cyclictest15864-21kworker/2:207:40:222
667799239,2cyclictest15864-21kworker/2:212:22:482
667799238,3cyclictest15864-21kworker/2:211:39:082
667799233,10cyclictest15864-21kworker/2:208:27:282
667799231,9cyclictest15864-21kworker/2:210:36:342
6677992312,2cyclictest15864-21kworker/2:208:21:182
6677992310,3cyclictest15864-21kworker/2:208:44:562
667099239,3cyclictest0-21swapper/108:42:091
667099238,8cyclictest0-21swapper/109:47:481
667099233,2cyclictest0-21swapper/110:15:241
667099231,9cyclictest0-21swapper/109:25:131
6670992312,3cyclictest4652-21proc_pri11:55:231
667099231,16cyclictest0-21swapper/111:32:431
667099231,15cyclictest86750irq/128-enp1s0-12:12:591
666499238,3cyclictest134112cat09:50:120
666499230,12cyclictest26814-21idleruntime-cro11:35:010
667799225,8cyclictest13361-21kworker/2:107:25:242
667799225,4cyclictest15864-21kworker/2:210:46:582
667799221,8cyclictest0-21swapper/210:40:522
6677992215,2cyclictest15864-21kworker/2:209:33:002
6677992214,4cyclictest836-21runrttasks11:45:192
667099222,12cyclictest86750irq/128-enp1s0-12:35:231
667099221,3cyclictest6331-21diskstats08:20:131
667099221,19cyclictest838-21taskset11:47:541
667099221,19cyclictest0-21swapper/110:03:161
667099221,15cyclictest0-21swapper/110:40:101
667099221,15cyclictest0-21swapper/109:54:451
667099221,10cyclictest0-21swapper/111:19:291
6670992210,11cyclictest11236-21kworker/u8:011:10:001
667099220,5cyclictest0-21swapper/110:39:291
667099220,2cyclictest0-21swapper/107:32:021
667099220,16cyclictest0-21swapper/111:03:191
667099220,16cyclictest0-21swapper/110:06:221
667099220,16cyclictest0-21swapper/107:20:161
667099220,15cyclictest86750irq/128-enp1s0-09:23:391
38942220,2sleep10-21swapper/108:15:131
175632220,3sleep2667799cyclictest08:45:172
667799216,2cyclictest15864-21kworker/2:209:30:022
6677992116,3cyclictest23180-21cut11:25:172
6677992116,3cyclictest23180-21cut11:25:172
667099219,3cyclictest11625-21sendmail_mailqu08:30:231
667099214,10cyclictest0-21swapper/112:09:511
667099212,16cyclictest18666-21munin-run12:30:001
667099211,2cyclictest10822-21ntpq07:15:231
667099211,18cyclictest0-21swapper/108:10:061
667099211,18cyclictest0-21swapper/108:10:061
667099211,15cyclictest0-21swapper/109:30:261
667099211,15cyclictest0-21swapper/108:55:031
667099211,15cyclictest0-21swapper/108:53:071
667099211,14cyclictest0-21swapper/107:40:021
667099210,10cyclictest0-21swapper/107:45:051
666499219,3cyclictest25141-21/usr/sbin/munin07:50:120
6664992115,3cyclictest28838-21grep07:55:270
6664992110,8cyclictest26060-21sshd10:18:290
5372211,3sleep1603-21sort08:05:221
228132211,16sleep022897-21latency10:10:180
667799208,3cyclictest15864-21kworker/2:211:03:142
667799207,3cyclictest15864-21kworker/2:210:55:142
667799206,4cyclictest15864-21kworker/2:209:05:062
667799206,2cyclictest15864-21kworker/2:212:18:162
667799206,2cyclictest13361-21kworker/2:107:10:442
667799205,3cyclictest15864-21kworker/2:208:05:202
667799205,2cyclictest32-21ksoftirqd/209:03:102
667799204,2cyclictest15864-21kworker/2:212:35:202
667799202,11cyclictest1361-21kworker/2:008:30:102
667799201,7cyclictest302052egrep09:15:132
667799201,11cyclictest15864-21kworker/2:209:10:142
667799201,10cyclictest23083-21ntp_states10:10:212
667099208,3cyclictest0-21swapper/112:31:201
667099208,3cyclictest0-21swapper/112:31:201
667099208,3cyclictest0-21swapper/109:16:031
6670992016,3cyclictest0-21swapper/110:55:021
667099201,3cyclictest0-21swapper/108:00:121
6670992013,2cyclictest0-21swapper/110:49:431
6670992013,2cyclictest0-21swapper/108:28:021
667099201,18cyclictest0-21swapper/107:55:201
667099201,18cyclictest0-21swapper/107:36:581
667099201,14cyclictest0-21swapper/109:56:391
6670992011,3cyclictest86750irq/128-enp1s0-12:01:401
667099201,11cyclictest12621-21cpu08:35:131
667099201,11cyclictest0-21swapper/112:20:161
667099200,4cyclictest0-21swapper/111:24:301
667099200,2cyclictest86750irq/128-enp1s0-11:41:061
667099200,15cyclictest0-21swapper/110:10:381
667099200,15cyclictest0-21swapper/109:05:271
666499206,5cyclictest0-21swapper/009:20:150
666499203,14cyclictest9875-21turbostat12:10:000
6664992014,4cyclictest7-21ksoftirqd/011:50:190
666499201,11cyclictest5985-21/usr/sbin/munin08:20:150
666499201,11cyclictest4419-21/usr/sbin/munin09:30:190
666499201,11cyclictest30183-21/usr/sbin/munin09:15:160
666499201,11cyclictest10386-21/usr/sbin/munin08:30:170
666499201,10cyclictest4614-21sshd10:42:290
666499201,10cyclictest0-21swapper/011:55:270
666499200,5cyclictest86550irq/126-enp1s0-09:35:160
667799196,4cyclictest13361-21kworker/2:107:21:502
667799196,3cyclictest32-21ksoftirqd/207:50:002
667799195,6cyclictest14677-21unixbench-2d09:50:252
667799195,3cyclictest15864-21kworker/2:210:54:322
667799194,3cyclictest86650irq/127-enp1s0-07:31:582
667799193,8cyclictest15864-21kworker/2:210:04:162
667799193,3cyclictest15864-21kworker/2:212:14:082
667799191,9cyclictest15864-21kworker/2:211:42:402
667799191,10cyclictest20510-21/usr/sbin/munin10:05:182
6677991910,2cyclictest15864-21kworker/2:209:45:222
667099191,10cyclictest17011-21cpu08:45:131
666499199,8cyclictest7-21ksoftirqd/007:35:160
666499199,3cyclictest18955-21awk08:50:010
666499193,2cyclictest12050irq/123-ahci[0010:25:010
666499191,5cyclictest11-21rcuc/012:10:150
666499191,5cyclictest0-21swapper/010:51:300
666499191,4cyclictest19473-21kworker/u8:010:31:320
666499191,11cyclictest0-21swapper/007:15:140
666499191,10cyclictest23570-21/usr/sbin/munin09:00:160
666499191,10cyclictest22777-21sshd12:36:300
666499191,10cyclictest0-21swapper/012:30:260
666499191,10cyclictest0-21swapper/012:30:260
666499191,10cyclictest0-21swapper/011:45:400
33152191,2sleep00-21swapper/008:12:000
33152191,2sleep00-21swapper/008:12:000
223572191,2sleep00-21swapper/011:24:200
667799188,2cyclictest86850irq/129-enp1s0-12:32:402
667799188,2cyclictest86850irq/129-enp1s0-12:32:402
667799186,3cyclictest15864-21kworker/2:211:20:382
667799185,3cyclictest86550irq/126-enp1s0-12:28:102
667799184,2cyclictest32-21ksoftirqd/207:55:482
667799183,3cyclictest86850irq/129-enp1s0-07:53:282
667799181,8cyclictest29471-21expr10:25:212
667799181,7cyclictest0-21swapper/212:07:432
667099181,3cyclictest839-21kworker/u8:111:50:061
666499187,3cyclictest16467-21sshd08:41:280
666499186,6cyclictest12575-21kworker/0:008:50:430
666499182,9cyclictest9040-21kworker/0:110:58:350
666499181,9cyclictest31746-21/usr/sbin/munin08:05:190
666499181,7cyclictest0-21swapper/011:38:400
666499181,6cyclictest0-21swapper/011:25:230
666499181,6cyclictest0-21swapper/011:25:230
6664991812,3cyclictest2241-21egrep09:25:120
666499181,15cyclictest0-21swapper/012:20:090
666499181,14cyclictest13927-21grep07:25:010
666499181,10cyclictest21507-21sshd10:06:280
66612180,2sleep00-21swapper/010:45:340
55722181,2sleep00-21swapper/008:17:460
104942181,2sleep00-21swapper/009:41:190
97602171,2sleep20-21swapper/207:15:142
667799174,2cyclictest15864-21kworker/2:211:11:522
667799172,4cyclictest15864-21kworker/2:207:35:382
6677991711,3cyclictest12985-21cut08:35:172
666499173,8cyclictest0-21swapper/012:15:110
666499173,6cyclictest7-21ksoftirqd/011:05:160
666499171,9cyclictest5275-21kworker/0:208:26:150
666499171,3cyclictest0-21swapper/010:27:460
666499171,3cyclictest0-21swapper/007:30:180
666499171,14cyclictest22726-21awk07:45:010
6664991711,3cyclictest14645-21cat08:40:010
666499171,10cyclictest0-21swapper/007:25:210
193422170,2sleep00-21swapper/011:15:330
666499161,2cyclictest0-21swapper/008:02:320
6664991611,2cyclictest836-21runrttasks09:06:160
6664991610,3cyclictest24010-21sh07:45:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional