You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-19 - 06:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sun Jul 19, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
311392237162,25sleep00-21swapper/019:08:540
311752232162,57sleep10-21swapper/119:09:221
309592203167,24sleep20-21swapper/219:06:332
309482203169,22sleep30-21swapper/319:06:263
313209910399,2cyclictest20574-21/usr/sbin/munin19:55:192
19172550,7sleep123-21ksoftirqd/121:40:141
72322520,1sleep20-21swapper/200:20:162
192552480,2sleep20-21swapper/222:20:182
3132099451,16cyclictest0-21swapper/200:05:282
3132599402,4cyclictest16293-21munin-run22:15:013
31310994010,14cyclictest855-21munin-run21:40:020
31310994010,14cyclictest855-21munin-run21:40:010
3132599391,6cyclictest31916-21sed21:35:163
3132599391,6cyclictest31916-21sed21:35:163
3132599371,7cyclictest29925-21taskset23:58:073
3132099371,10cyclictest0-21swapper/223:45:212
31310993711,12cyclictest3714-21munin-run23:00:010
3132599362,20cyclictest15132-40dnf00:37:333
3132599361,4cyclictest7485-21latency_hist21:55:023
3131099368,13cyclictest18496-21perf22:20:010
3132599353,4cyclictest27439-21perl21:25:163
31325993512,20cyclictest17126-21/usr/sbin/munin23:30:243
3132099351,20cyclictest0-21swapper/220:29:162
3132099351,20cyclictest0-21swapper/220:29:162
3131099352,22cyclictest14481-21kworker/0:123:32:310
3132599342,28cyclictest11269-21basename20:50:013
31325993411,20cyclictest7095-21/usr/sbin/munin20:40:243
31325993411,20cyclictest25918-21/usr/sbin/munin23:50:223
31320993421,3cyclictest0-21swapper/223:19:392
3132099341,20cyclictest0-21swapper/220:09:112
3132599332,4cyclictest13796-21ntp_states23:20:223
3132599331,4cyclictest16509-21/usr/sbin/munin22:15:173
3132599331,29cyclictest808-21snmpd22:26:073
31325993312,18cyclictest5590-210anacron23:01:013
3132599331,15cyclictest0-21swapper/321:31:233
31325993311,19cyclictest1742-21/usr/sbin/munin22:55:213
31325993310,20cyclictest4583-21/usr/sbin/munin00:15:173
31325993310,20cyclictest22490-21/usr/sbin/munin21:15:203
31325993310,19cyclictest9095-21cat21:55:253
3132599323,2cyclictest0-21swapper/319:15:183
3132599323,2cyclictest0-21swapper/319:15:173
3132599321,21cyclictest0-21swapper/319:41:033
31325993212,17cyclictest12118-21/usr/sbin/munin22:05:133
3132599321,14cyclictest0-21swapper/322:45:083
3132599321,14cyclictest0-21swapper/322:45:083
3132599321,13cyclictest0-21swapper/321:07:483
3132599321,10cyclictest808-21snmpd22:31:083
3132599321,10cyclictest20394-21sshd22:21:303
3132599320,7cyclictest9588-21/usr/sbin/munin23:10:233
31320993219,3cyclictest0-21swapper/200:29:552
31320993219,3cyclictest0-21swapper/200:15:562
3131099322,23cyclictest791-21kworker/0:019:35:440
3131099322,23cyclictest32161-21kworker/0:200:18:570
3132599319,19cyclictest4899-21/usr/sbin/munin20:35:263
3132599319,19cyclictest17230-21crond19:50:013
3132599311,8cyclictest11348-21irqstats23:15:193
3132599311,15cyclictest0-21swapper/322:00:073
3132599310,14cyclictest0-21swapper/322:44:523
3132599310,14cyclictest0-21swapper/321:46:193
3132099311,20cyclictest0-21swapper/223:09:232
3132099311,20cyclictest0-21swapper/220:50:392
3131099316,12cyclictest0-21swapper/021:15:000
31310993111,17cyclictest9069-21munin-run20:45:010
3132599309,3cyclictest9683-21idleruntime00:25:173
3132599309,18cyclictest8444-21basename19:30:013
3132599309,18cyclictest32672-21if_err_enp1s022:50:183
3132599308,4cyclictest31971-21/usr/sbin/munin20:20:253
3132599308,3cyclictest30434-21dump-pmu-power20:20:013
3132599305,4cyclictest909-21sssd_nss21:00:173
3132599302,9cyclictest10643-21latency_hist19:35:013
3132599302,3cyclictest87350irq/126-enp1s0-00:24:313
3132599301,4cyclictest808-21snmpd19:12:143
3132599301,4cyclictest3260-21/usr/sbin/munin00:10:183
3132599301,12cyclictest0-21swapper/323:38:553
31325993011,16cyclictest25300-21/usr/sbin/munin22:35:233
31325993010,17cyclictest687-21expr20:25:123
31325993010,17cyclictest687-21expr20:25:123
31325993010,17cyclictest13203-21wc00:35:033
3132599300,11cyclictest0-21swapper/323:25:033
3132099308,3cyclictest12620-21sshd19:37:172
3132099301,20cyclictest0-21swapper/222:07:072
3132099301,20cyclictest0-21swapper/220:24:352
3132099301,19cyclictest0-21swapper/221:27:552
31320993011,3cyclictest0-21swapper/219:49:392
3131099304,12cyclictest25706-21perf23:50:010
3131099301,9cyclictest520-21kworker/0:121:42:070
125642300,5sleep03131099cyclictest22:05:140
3132599299,2cyclictest25320-21gltestperf21:20:153
3132599299,18cyclictest4878-21sed19:20:153
3132599299,17cyclictest11719-21expr20:50:143
3132599298,4cyclictest0-21swapper/323:05:213
3132599298,3cyclictest1762-21head21:40:133
3132599293,4cyclictest21200-21memory21:10:213
3132599293,3cyclictest11211-21/usr/sbin/munin19:35:123
3132599292,5cyclictest26616-21diskstats20:10:163
3132599292,5cyclictest0-21swapper/300:00:273
31325992912,14cyclictest19009-21grep19:50:263
31320992912,3cyclictest22328-21fschecks_count20:00:152
3132099291,10cyclictest0-21swapper/200:33:352
3131099292,21cyclictest29290-21kworker/0:322:48:190
3131099292,21cyclictest29290-21kworker/0:322:48:190
3131099291,21cyclictest10739-21kworker/0:100:31:370
3132599288,2cyclictest827-21runrttasks00:05:563
3132599282,5cyclictest25327-21smart_sda20:05:243
3132599281,9cyclictest22508-21strings23:40:203
3132599281,6cyclictest0-21swapper/321:00:023
3132599281,2cyclictest0-21swapper/323:45:183
3132599280,2cyclictest0-21swapper/320:00:123
3132099286,5cyclictest0-21swapper/222:34:002
31320992818,3cyclictest0-21swapper/221:21:232
3132099281,12cyclictest0-21swapper/221:12:152
3132099281,12cyclictest0-21swapper/220:17:272
3132099281,12cyclictest0-21swapper/219:28:112
3132099281,12cyclictest0-21swapper/219:11:512
3132099281,11cyclictest0-21swapper/223:38:072
3132099281,11cyclictest0-21swapper/222:38:552
3132099281,11cyclictest0-21swapper/221:37:192
3132099281,11cyclictest0-21swapper/221:37:192
3132099281,11cyclictest0-21swapper/221:31:512
3132099281,11cyclictest0-21swapper/220:42:032
3132099281,11cyclictest0-21swapper/219:19:402
3132099281,11cyclictest0-21swapper/219:19:392
3132099281,10cyclictest0-21swapper/223:21:272
3131099283,12cyclictest23491-21munin-run23:45:010
31310992811,4cyclictest0-21swapper/000:40:020
3131099280,21cyclictest28177-21kworker/0:020:27:030
3131099280,21cyclictest28177-21kworker/0:020:27:030
42842270,4sleep011-21rcuc/019:20:120
3132599278,3cyclictest20537-21latency19:55:183
3132099279,3cyclictest0-21swapper/222:26:042
3132099278,3cyclictest0-21swapper/221:49:432
3132099271,13cyclictest0-21swapper/223:27:152
3132099271,12cyclictest0-21swapper/200:11:442
3132099271,11cyclictest0-21swapper/223:12:592
3132099271,11cyclictest0-21swapper/222:16:002
3132099271,11cyclictest0-21swapper/222:13:272
3132099271,11cyclictest0-21swapper/221:50:472
3132099271,11cyclictest0-21swapper/200:36:472
3132099271,10cyclictest0-21swapper/223:59:472
3132099271,10cyclictest0-21swapper/222:55:392
3132099271,10cyclictest0-21swapper/220:58:272
3131099272,9cyclictest1380-21kworker/0:223:15:460
3131099271,9cyclictest4363-21kworker/0:320:36:290
3131099271,21cyclictest4363-21kworker/0:320:32:210
3131099271,16cyclictest25393-21kworker/0:323:53:550
3131099271,10cyclictest25929-21kworker/0:320:09:050
3132099269,3cyclictest4291-21dnf23:00:152
3132099269,3cyclictest0-21swapper/220:30:392
3132099268,3cyclictest0-21swapper/221:02:072
31320992612,8cyclictest0-21swapper/221:06:072
31320992611,3cyclictest0-21swapper/222:41:592
3132099261,12cyclictest0-21swapper/221:42:112
3132099261,12cyclictest0-21swapper/219:32:352
3131099261,10cyclictest12611-21kworker/0:122:10:170
3131099260,21cyclictest27750-21kworker/0:023:58:250
3131099260,12cyclictest5309-21perf21:50:010
311112260,4sleep232-21ksoftirqd/200:00:182
3132599252,4cyclictest802-21sssd_be20:30:233
3132099258,3cyclictest0-21swapper/219:52:152
3132099258,3cyclictest0-21swapper/219:41:312
3132099251,9cyclictest0-21swapper/222:50:352
3132099251,9cyclictest0-21swapper/219:22:032
31320992511,3cyclictest0-21swapper/222:03:592
3132099251,11cyclictest0-21swapper/221:57:552
3132099251,10cyclictest0-21swapper/223:30:472
3132099251,10cyclictest0-21swapper/222:48:472
3132099251,10cyclictest0-21swapper/222:48:472
3132099251,10cyclictest0-21swapper/220:35:352
3131599254,4cyclictest773-21NetworkManager22:38:081
3131599252,6cyclictest23-21ksoftirqd/119:40:221
3131099252,8cyclictest2059-21kworker/0:000:10:210
3131099252,16cyclictest20524-21kworker/0:222:28:030
3131099252,10cyclictest8091-21kworker/0:219:31:080
3131099251,11cyclictest520-21kworker/0:121:58:290
3131099251,10cyclictest791-21kworker/0:019:47:360
3131099251,10cyclictest15609-21kworker/0:021:22:330
3131099250,9cyclictest14481-21kworker/0:123:37:330
3132099248,3cyclictest0-21swapper/223:51:592
3132099241,12cyclictest0-21swapper/223:42:232
31320992410,3cyclictest0-21swapper/220:47:552
31320992410,3cyclictest0-21swapper/220:12:192
3131599242,15cyclictest87650irq/129-enp1s0-23:29:101
3131599242,11cyclictest29474-21munin-run22:45:011
3131599241,17cyclictest0-21swapper/122:46:411
3131599241,17cyclictest0-21swapper/122:46:411
3131099249,3cyclictest791-21kworker/0:019:58:090
3131099247,15cyclictest25086-21awk22:35:010
3131099242,9cyclictest28511-21kworker/0:221:27:030
3131099242,9cyclictest15609-21kworker/0:021:09:390
3131099242,9cyclictest10106-21kworker/0:323:13:260
3131099242,8cyclictest791-21kworker/0:019:53:410
3131099242,8cyclictest29290-21kworker/0:322:52:270
3131099241,9cyclictest22177-21kworker/0:321:17:370
3131099241,8cyclictest3615-21kworker/0:023:02:030
3131099241,8cyclictest32547-21kworker/0:120:24:550
3131099241,8cyclictest11051-21kworker/0:120:58:530
3131099241,15cyclictest25929-21kworker/0:320:11:210
3131099241,11cyclictest8504-21kworker/0:000:28:330
3132099239,3cyclictest0-21swapper/221:18:592
3131599238,3cyclictest0-21swapper/119:10:361
31315992315,3cyclictest0-21swapper/123:17:141
3131599231,19cyclictest12452-21df22:05:121
3131599231,16cyclictest0-21swapper/122:30:081
3131599231,16cyclictest0-21swapper/121:26:271
3131599231,16cyclictest0-21swapper/121:15:021
3131099236,2cyclictest20524-21kworker/0:222:38:530
3131099235,15cyclictest29484-21awk22:45:010
3131099231,8cyclictest791-21kworker/0:019:25:420
3131099231,8cyclictest32161-21kworker/0:200:09:490
3131099231,8cyclictest1380-21kworker/0:223:09:000
3131099231,8cyclictest12611-21kworker/0:122:23:250
3131099230,14cyclictest32161-21kworker/0:200:03:270
3131599222,18cyclictest0-21swapper/121:00:251
3131599222,15cyclictest0-21swapper/121:37:111
3131599222,15cyclictest0-21swapper/121:37:111
3131599222,15cyclictest0-21swapper/119:58:281
3131599221,9cyclictest0-21swapper/123:01:261
31315992216,4cyclictest0-21swapper/100:35:211
31315992214,3cyclictest0-21swapper/122:50:211
31315992214,3cyclictest0-21swapper/121:55:501
31315992214,3cyclictest0-21swapper/121:23:591
31315992214,3cyclictest0-21swapper/120:15:011
3131599221,3cyclictest7692-21memory00:20:201
3131599221,16cyclictest0-21swapper/121:10:471
3131599221,16cyclictest0-21swapper/119:26:471
3131599221,15cyclictest0-21swapper/121:08:051
3131099222,8cyclictest30744-21kworker/0:321:32:300
3131099222,8cyclictest11051-21kworker/0:121:02:090
3131099222,11cyclictest791-21kworker/0:019:44:420
3131099221,9cyclictest520-21kworker/0:121:53:350
3131099221,9cyclictest21652-21kworker/0:120:02:590
3131099221,7cyclictest28177-21kworker/0:020:19:030
3131099221,10cyclictest791-21kworker/0:019:16:550
3131099221,10cyclictest791-21kworker/0:019:16:550
3131599212,10cyclictest3050-21diskstats20:30:141
31315992117,3cyclictest0-21swapper/122:18:481
3131599211,3cyclictest910-21systemd-logind00:04:301
3131599211,17cyclictest0-21swapper/121:33:411
3131599211,15cyclictest0-21swapper/120:39:351
3131599211,15cyclictest0-21swapper/120:09:231
3131599211,15cyclictest0-21swapper/120:02:541
3131599211,15cyclictest0-21swapper/119:37:481
3131599211,14cyclictest0-21swapper/121:45:191
3131599211,13cyclictest3621-21kworker/u8:123:13:471
3131599211,13cyclictest0-21swapper/123:07:291
3131599211,10cyclictest2203-21grep19:15:131
3131599211,10cyclictest2203-21grep19:15:131
3131599210,16cyclictest0-21swapper/122:03:301
3131099211,9cyclictest10106-21kworker/0:323:22:010
3131099211,14cyclictest8815-21kworker/0:020:50:550
3131099211,11cyclictest1380-21kworker/0:223:27:490
3131099210,9cyclictest8815-21kworker/0:020:48:190
3131599202,3cyclictest0-21swapper/120:19:241
31315992016,3cyclictest0-21swapper/123:46:531
31315992016,3cyclictest0-21swapper/122:56:011
3131599201,4cyclictest3458-21ntp_states00:10:221
31315992013,2cyclictest0-21swapper/120:50:021
31315992013,2cyclictest0-21swapper/119:52:161
31315992012,3cyclictest0-21swapper/120:22:491
3131599201,17cyclictest0-21swapper/123:38:041
3131599201,14cyclictest0-21swapper/123:20:281
3131599201,14cyclictest0-21swapper/122:25:241
3131599201,14cyclictest0-21swapper/122:11:531
3131599201,14cyclictest0-21swapper/119:45:411
3131599201,14cyclictest0-21swapper/119:30:481
3131599201,12cyclictest87650irq/129-enp1s0-21:52:261
3131599201,11cyclictest29379-21smart_sda23:55:241
3131599201,11cyclictest23125-21sshd23:40:311
3131599201,11cyclictest11259-21latency_hist20:50:011
3131099202,10cyclictest9651-21kworker/0:022:02:550
31310992014,4cyclictest32158-21cut19:10:160
3131599199,2cyclictest20692-21awk22:25:001
3131599191,3cyclictest18430-21switchtime23:30:271
3131599191,17cyclictest0-21swapper/120:42:391
3131599191,11cyclictest12321-21pmu-power00:30:221
3131599191,10cyclictest26254-21df23:50:161
3131599191,10cyclictest1797-21sed20:25:241
3131599191,10cyclictest1797-21sed20:25:241
3131599190,14cyclictest0-21swapper/119:21:361
3131599190,13cyclictest0-21swapper/100:05:041
3131099197,2cyclictest32161-21kworker/0:200:20:410
31315991813,3cyclictest6149-21unixbench_singl00:15:261
3131599181,16cyclictest0-21swapper/100:26:071
31315991811,2cyclictest0-21swapper/120:56:291
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional