You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-30 - 04:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat May 30, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
288712239163,23sleep20-21swapper/219:05:572
291142202165,24sleep10-21swapper/119:09:061
290112202164,25sleep00-21swapper/019:07:440
272222202165,24sleep30-21swapper/319:05:053
1843121310,4sleep02926999cyclictest21:10:120
295852460,1sleep00-21swapper/000:00:240
2926999351,20cyclictest9792-21kworker/0:123:21:470
2928699321,11cyclictest0-21swapper/320:42:273
2927599321,22cyclictest3917-21munin-run23:05:011
2926999322,16cyclictest19419-21kworker/0:120:04:350
247162320,5sleep22928299cyclictest22:35:262
2928699311,12cyclictest0-21swapper/323:13:233
2928699301,19cyclictest0-21swapper/320:13:123
2928699301,19cyclictest0-21swapper/320:13:113
2928699301,10cyclictest0-21swapper/320:36:433
2926999301,11cyclictest30846-21kworker/0:021:44:550
2926999292,10cyclictest23113-21kworker/0:223:54:510
2926999291,10cyclictest31813-21kworker/0:123:00:320
195612290,3sleep02926999cyclictest20:00:010
2928699281,13cyclictest0-21swapper/319:35:193
2928699281,13cyclictest0-21swapper/300:01:273
2928699281,12cyclictest0-21swapper/300:29:243
2927599288,10cyclictest22353-21seq21:19:231
2926999282,10cyclictest19917-21kworker/0:321:33:590
2928699279,3cyclictest0-21swapper/323:33:033
29286992711,3cyclictest0-21swapper/323:07:033
2928699271,12cyclictest0-21swapper/300:38:193
2928699271,12cyclictest0-21swapper/300:15:593
2928699271,11cyclictest0-21swapper/321:21:473
2928699270,18cyclictest0-21swapper/320:46:353
2927599278,7cyclictest0-21swapper/121:20:131
2927599270,11cyclictest25932-21/usr/sbin/munin21:25:221
2926999272,9cyclictest30846-21kworker/0:021:53:590
2926999271,9cyclictest28022-21kworker/0:120:30:270
29269992713,2cyclictest14616-21kworker/0:023:42:590
2926999271,10cyclictest6078-21kworker/0:023:14:470
2926999271,10cyclictest22862-21kworker/0:322:33:190
2928699266,3cyclictest0-21swapper/322:21:033
2928699261,12cyclictest0-21swapper/322:52:153
2928699261,12cyclictest0-21swapper/320:08:313
2928699261,11cyclictest0-21swapper/321:58:313
2928699261,11cyclictest0-21swapper/319:54:083
2928699260,12cyclictest0-21swapper/321:27:593
2928699260,11cyclictest0-21swapper/320:55:153
29275992622,3cyclictest0-21swapper/120:17:071
2927599261,15cyclictest0-21swapper/123:25:011
2927599260,20cyclictest0-21swapper/123:50:351
2926999267,4cyclictest24520-21kworker/0:021:26:390
2926999267,2cyclictest32423-21kworker/0:300:20:550
2926999262,10cyclictest19917-21kworker/0:321:19:430
29269992612,2cyclictest3085-21kworker/0:122:06:070
2926999261,11cyclictest14616-21kworker/0:023:36:390
2926999261,11cyclictest14616-21kworker/0:023:36:390
2926999261,11cyclictest10800-21kworker/0:000:31:510
2926999261,10cyclictest8213-21kworker/0:222:02:110
2926999261,10cyclictest16275-21kworker/0:222:20:270
29269992610,3cyclictest24742-21kworker/0:119:10:230
8332250,4sleep230-21rcuc/220:30:132
2928699258,3cyclictest0-21swapper/323:38:473
2928699258,3cyclictest0-21swapper/323:38:473
2928699257,2cyclictest0-21swapper/322:31:163
2928699256,10cyclictest0-21swapper/323:00:203
2928699251,11cyclictest0-21swapper/322:46:273
29286992510,3cyclictest0-21swapper/323:57:113
2928699250,9cyclictest0-21swapper/320:25:133
2928699250,10cyclictest0-21swapper/300:09:153
2928299258,4cyclictest0-21swapper/222:15:152
29282992513,9cyclictest10291-21systemd-journal23:17:262
2928299250,14cyclictest0-21swapper/223:20:172
29275992522,2cyclictest0-21swapper/122:28:451
29275992522,2cyclictest0-21swapper/120:02:531
29275992521,3cyclictest0-21swapper/120:43:401
2927599251,7cyclictest0-21swapper/123:49:061
2927599251,5cyclictest0-21swapper/122:03:011
2927599251,17cyclictest0-21swapper/123:55:191
2927599251,16cyclictest0-21swapper/100:02:321
2927599251,12cyclictest6480-21irqstats21:55:161
2927599251,12cyclictest0-21swapper/123:42:491
2926999256,2cyclictest14616-21kworker/0:023:29:270
2926999251,9cyclictest29554-21kworker/0:022:56:430
2926999251,9cyclictest1726-21kworker/0:223:05:070
2926999251,13cyclictest11662-21kworker/0:119:42:350
2926999251,10cyclictest32466-21kworker/0:020:35:470
2926999251,10cyclictest28022-21kworker/0:120:25:270
2926999251,10cyclictest23113-21kworker/0:223:48:470
2926999251,10cyclictest19917-21kworker/0:321:22:110
29269992510,3cyclictest16832-21kworker/0:223:34:070
2928699249,3cyclictest0-21swapper/321:43:313
2928699241,17cyclictest0-21swapper/323:22:453
2928699241,16cyclictest0-21swapper/321:39:273
2928699241,14cyclictest0-21swapper/322:00:153
2928699241,10cyclictest0-21swapper/321:30:313
29286992410,3cyclictest0-21swapper/319:55:043
2928699240,10cyclictest0-21swapper/321:51:473
2928299241,16cyclictest0-21swapper/219:16:372
29275992421,2cyclictest0-21swapper/123:06:141
29275992421,2cyclictest0-21swapper/120:36:581
2927599242,11cyclictest7150-21cstates00:25:141
29275992412,2cyclictest18018-21diskmemload21:49:561
2927599241,19cyclictest0-21swapper/121:54:361
2927599241,12cyclictest19284-21ntp_kernel_pll_21:10:191
2927599241,11cyclictest6672-21df23:10:121
2927599241,11cyclictest5117-21df_inode00:20:141
2927599241,11cyclictest18018-21diskmemload22:50:121
2927599241,11cyclictest14144-21rt-features23:25:231
2927599241,11cyclictest1-21systemd23:00:011
2926999247,4cyclictest19419-21kworker/0:120:07:470
2926999247,2cyclictest25123-21kworker/0:222:51:390
2926999246,4cyclictest14899-21kworker/0:219:50:310
2926999242,9cyclictest32364-21kworker/0:000:08:270
2926999242,11cyclictest26780-21kworker/0:121:36:590
2926999240,10cyclictest11753-21kworker/0:022:16:030
2928699239,3cyclictest0-21swapper/320:22:393
2928699238,3cyclictest0-21swapper/319:19:073
2928699236,9cyclictest0-21swapper/322:58:033
2928699231,9cyclictest0-21swapper/319:40:233
2928699231,12cyclictest0-21swapper/319:23:153
2928699231,10cyclictest25238-21switchtime23:50:233
2928699231,10cyclictest0-21swapper/321:45:273
2928699230,10cyclictest0-21swapper/321:15:553
2928699230,10cyclictest0-21swapper/320:33:433
2927599232,10cyclictest4580-21cstates19:25:111
29275992319,3cyclictest0-21swapper/120:25:091
29275992317,3cyclictest9748-21sed19:35:191
29275992312,7cyclictest0-21swapper/100:10:401
2927599231,20cyclictest0-21swapper/119:10:141
2927599231,16cyclictest0-21swapper/122:05:551
2927599231,16cyclictest0-21swapper/121:35:431
29275992311,3cyclictest0-21swapper/123:38:221
29275992311,3cyclictest0-21swapper/123:38:221
2927599231,11cyclictest944-21diskstats20:30:141
2927599231,10cyclictest1959-21cat19:19:561
2927599230,21cyclictest0-21swapper/121:40:161
2927599230,21cyclictest0-21swapper/120:45:121
2926999237,2cyclictest3085-21kworker/0:121:56:110
2926999236,4cyclictest6696-21kworker/0:120:52:110
2926999234,2cyclictest11753-21kworker/0:022:27:030
2926999231,10cyclictest30846-21kworker/0:021:45:350
29269992310,3cyclictest14616-21kworker/0:023:55:230
2926999230,10cyclictest32423-21kworker/0:300:26:510
2928699228,3cyclictest0-21swapper/321:07:313
2928699221,14cyclictest0-21swapper/319:27:433
2928699221,11cyclictest0-21swapper/323:44:073
2928699221,10cyclictest0-21swapper/300:14:153
29286992210,3cyclictest18018-21diskmemload22:05:173
2928699220,9cyclictest0-21swapper/322:12:153
2928299226,4cyclictest8937-21latency_hist00:30:012
2928299225,9cyclictest30792-21cpu20:25:112
2928299225,9cyclictest14661-21sed19:45:242
2927599222,17cyclictest10471-21ntp_states20:50:201
2927599222,11cyclictest0-21swapper/119:46:081
29275992218,3cyclictest0-21swapper/122:45:121
29275992218,3cyclictest0-21swapper/120:22:571
29275992214,3cyclictest0-21swapper/123:31:361
2927599221,3cyclictest0-21swapper/122:13:551
2927599221,19cyclictest0-21swapper/122:33:171
2927599221,19cyclictest0-21swapper/122:17:001
2927599221,17cyclictest12382-21netstat00:35:191
29275992211,3cyclictest0-21swapper/100:16:001
2927599221,10cyclictest24538-21forks20:10:141
2927599221,10cyclictest24538-21forks20:10:141
2927599221,10cyclictest17633-21cpu19:55:121
2926999227,8cyclictest30238-21kworker/0:220:22:310
2926999226,4cyclictest10479-21kworker/0:020:55:440
2926999226,2cyclictest13240-21kworker/0:221:00:230
2926999225,4cyclictest24742-21kworker/0:119:25:270
2926999222,9cyclictest14005-21kworker/0:122:13:350
29269992211,2cyclictest14899-21kworker/0:219:49:030
2926999221,12cyclictest11366-21/usr/sbin/munin00:35:140
2926999221,10cyclictest25123-21kworker/0:222:48:350
29269992210,2cyclictest32423-21kworker/0:300:17:150
2928699219,3cyclictest0-21swapper/300:22:123
2928699217,3cyclictest0-21swapper/320:51:233
2928699212,9cyclictest26923-21fschecks_time20:15:153
2928699211,6cyclictest0-21swapper/322:25:123
29286992114,4cyclictest15138-21hddtemp_smartct22:15:153
2928699211,13cyclictest0-21swapper/322:35:033
2928699211,11cyclictest0-21swapper/319:10:553
2928699210,14cyclictest0-21swapper/321:11:193
2928699210,13cyclictest0-21swapper/323:25:033
2928299217,4cyclictest0-21swapper/221:59:062
2928299215,4cyclictest909-21sssd_nss00:10:232
2928299213,6cyclictest30-21rcuc/221:45:252
2928299212,15cyclictest1-21systemd19:52:222
2928299211,7cyclictest17235-21tune2fs22:20:152
2928299211,3cyclictest18992-21kworker/u8:221:14:562
2928299211,11cyclictest5150-21fschecks_count00:20:152
2927599219,3cyclictest0-21swapper/119:51:581
2927599212,17cyclictest0-21swapper/120:56:551
2927599211,18cyclictest0-21swapper/119:30:231
2927599211,18cyclictest0-21swapper/100:30:191
2927599211,11cyclictest0-21swapper/121:30:531
2927599211,11cyclictest0-21swapper/119:22:331
2927599211,10cyclictest0-21swapper/122:22:091
2927599211,10cyclictest0-21swapper/121:00:151
2927599211,10cyclictest0-21swapper/100:06:571
2927599210,20cyclictest0-21swapper/122:40:231
2927599210,19cyclictest0-21swapper/122:35:131
2927599210,10cyclictest0-21swapper/120:05:201
2926999218,2cyclictest9792-21kworker/0:123:19:510
2926999215,8cyclictest23910-21kworker/0:020:16:230
2926999211,11cyclictest18508-21kworker/0:122:40:350
2926999211,10cyclictest4494-21kworker/0:320:44:070
2926999211,10cyclictest15487-21kworker/0:121:06:590
237322210,3sleep09-21rcu_preempt22:35:160
2928699205,5cyclictest0-21swapper/300:33:433
2928699201,10cyclictest0-21swapper/320:04:233
2928299205,4cyclictest3303-21if_enp1s000:15:162
2928299205,4cyclictest12547-21unixbench_singl19:40:242
2928299205,3cyclictest87350irq/126-enp1s0-21:50:122
2928299205,3cyclictest31246-21ls22:50:242
2928299204,3cyclictest17469-21sed21:05:252
2928299203,6cyclictest32-21ksoftirqd/223:05:182
2928299201,7cyclictest9980-21irqstats00:30:172
2928299201,7cyclictest27205-21ls21:30:102
2928299201,7cyclictest26803-21latency_hist21:30:012
2928299201,6cyclictest21499-21awk23:45:012
2928299201,6cyclictest0-21swapper/223:47:552
2928299201,6cyclictest0-21swapper/221:16:212
2928299201,10cyclictest8348-21sshd19:32:222
2927599208,4cyclictest0-21swapper/123:18:431
2927599200,18cyclictest0-21swapper/119:41:271
2926999208,3cyclictest32423-21kworker/0:300:10:250
2926999208,2cyclictest24742-21kworker/0:119:15:430
2926999204,3cyclictest8539-21kworker/0:219:38:110
2926999202,9cyclictest6288-21kworker/0:019:32:270
2926999202,4cyclictest21665-21kworker/0:220:11:150
2926999202,4cyclictest21665-21kworker/0:220:11:150
2926999201,9cyclictest6696-21kworker/0:120:48:430
2928699191,8cyclictest0-21swapper/319:30:153
2928699191,5cyclictest0-21swapper/323:18:353
2928699191,12cyclictest0-21swapper/323:46:393
2928699191,12cyclictest0-21swapper/322:43:233
2928299197,3cyclictest87650irq/129-enp1s0-20:55:252
2928299195,3cyclictest87350irq/126-enp1s0-20:40:122
2928299194,8cyclictest30-21rcuc/221:20:242
2928299194,3cyclictest22012-21memory22:30:172
2928299193,8cyclictest87350irq/126-enp1s0-22:55:362
2928299192,5cyclictest13532-21smart_sda22:10:232
2928299191,7cyclictest0-21swapper/200:00:302
2928299191,6cyclictest32334-21kworker/2:000:35:252
2928299191,6cyclictest28571-21/usr/sbin/munin20:20:132
2928299191,6cyclictest25443-21sshd20:10:232
2928299191,6cyclictest25443-21sshd20:10:232
2928299191,6cyclictest20914-21cat22:30:012
2928299191,6cyclictest18648-21ntp_states19:55:212
2928299191,5cyclictest30049-21irqstats21:35:192
2928299191,4cyclictest0-21swapper/223:58:082
29282991913,3cyclictest22317-21df20:05:142
2928299191,2cyclictest0-21swapper/223:10:182
2927599190,17cyclictest0-21swapper/121:05:191
2926999192,10cyclictest3984-21kworker/0:219:23:070
2928699187,3cyclictest14946-21runrttasks21:00:213
2928299186,2cyclictest87550irq/128-enp1s0-23:25:222
2928299185,3cyclictest32-21ksoftirqd/222:05:172
2928299184,3cyclictest2733-21interrupts23:00:162
2928299183,3cyclictest87350irq/126-enp1s0-19:10:152
2928299183,3cyclictest31532-21grep00:05:212
2928299183,3cyclictest0-21swapper/222:45:162
2928299182,5cyclictest808-21snmpd21:00:072
2928299182,4cyclictest3115-21memory19:20:182
2928299181,9cyclictest7868-21/usr/sbin/munin22:00:162
2928299181,6cyclictest31763-21cstates21:40:132
2928299181,6cyclictest0-21swapper/223:55:012
2928299181,6cyclictest0-21swapper/223:30:532
2928299181,5cyclictest2814-21/usr/sbin/munin20:35:232
2928299181,5cyclictest19411-21cat23:40:012
2928299181,5cyclictest19411-21cat23:40:012
2928299181,4cyclictest9773-21uniq20:50:142
29282991812,3cyclictest20426-21gltestperf20:00:132
2928299180,6cyclictest9177-21fschecks_count19:35:142
273492181,2sleep20-21swapper/222:43:412
2928699171,9cyclictest0-21swapper/319:49:433
2928299171,5cyclictest0-21swapper/220:15:212
29282991712,3cyclictest8175-21sed20:45:192
29282991611,3cyclictest4738-21df_inode19:25:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional