You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-28 - 03:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue Jul 28, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
254682238163,25sleep20-21swapper/219:05:362
256962235162,25sleep00-21swapper/019:08:320
255962203167,24sleep30-21swapper/319:07:153
256922202166,24sleep10-21swapper/119:08:291
2891621520,5sleep02589799cyclictest00:08:170
3239521490,2sleep22590999cyclictest00:15:182
2382321470,3sleep12590299cyclictest20:15:181
25909991071,2cyclictest0-21swapper/220:52:102
325782590,4sleep02589799cyclictest21:50:120
66302580,3sleep26634-21unixbench_singl23:15:262
207662540,1sleep20-21swapper/220:10:012
207662540,1sleep20-21swapper/220:10:012
194532530,5sleep22590999cyclictest22:33:002
194532530,5sleep22590999cyclictest22:33:002
206372520,4sleep339-21rcuc/322:35:183
263682450,1sleep00-21swapper/020:20:240
2589799369,12cyclictest24028-21awk22:45:020
2589799315,13cyclictest30611-21munin-run23:00:010
2589799313,5cyclictest7-21ksoftirqd/023:50:140
25897993112,16cyclictest9593-21munin-run19:45:000
75802300,5sleep32591499cyclictest23:20:123
2589799303,12cyclictest13996-21awk19:55:000
2590299291,9cyclictest0-21swapper/122:50:521
2590999281,10cyclictest0-21swapper/220:19:532
25902992711,5cyclictest22750-40dnf20:14:491
25897992710,14cyclictest32208-21idleruntime-cro21:50:010
2589799270,12cyclictest31272-21idleruntime-cro00:15:010
33372260,2sleep20-21swapper/223:10:162
25909992623,2cyclictest0-21swapper/222:39:322
2590299266,3cyclictest18245-1kworker/u9:200:31:391
2590299261,9cyclictest0-21swapper/100:39:391
2590299261,13cyclictest0-21swapper/123:40:121
2590299261,13cyclictest0-21swapper/123:40:121
2589799267,4cyclictest27530-21kworker/0:222:08:490
2589799261,5cyclictest0-21swapper/023:30:050
2591499251,10cyclictest0-21swapper/323:00:173
2590999255,5cyclictest27477-21cpuspeed_turbos20:25:132
2590299257,7cyclictest0-21swapper/122:07:071
2590299251,8cyclictest21047-21kworker/u8:021:24:561
2590299251,7cyclictest0-21swapper/123:17:151
2590299251,11cyclictest0-21swapper/123:31:541
2589799251,6cyclictest0-21swapper/021:40:490
2591499241,14cyclictest0-21swapper/323:06:323
2591499241,10cyclictest0-21swapper/322:15:263
2590299241,12cyclictest0-21swapper/123:26:401
2590299241,12cyclictest0-21swapper/100:17:041
2590299241,12cyclictest0-21swapper/100:03:461
2590299241,10cyclictest0-21swapper/100:27:421
2590299240,7cyclictest0-21swapper/121:44:291
2590299240,2cyclictest87350irq/126-enp1s0-23:04:281
2590299240,16cyclictest0-21swapper/122:15:481
2589799247,14cyclictest5506-21idleruntime-cro00:30:000
2589799241,6cyclictest28689-21kworker/u8:200:35:580
2589799241,6cyclictest0-21swapper/000:24:290
25897992416,6cyclictest7-21ksoftirqd/020:55:090
2589799240,11cyclictest7403-21basename19:40:000
2591499231,15cyclictest0-21swapper/321:50:273
2591499231,14cyclictest0-21swapper/300:24:213
2590299236,6cyclictest0-21swapper/121:10:371
2590299233,3cyclictest0-21swapper/121:27:561
2590299232,9cyclictest0-21swapper/122:20:191
2590299231,6cyclictest0-21swapper/122:33:541
2590299231,6cyclictest0-21swapper/122:33:541
2590299231,5cyclictest0-21swapper/123:35:221
2590299231,16cyclictest0-21swapper/122:11:591
2590299231,16cyclictest0-21swapper/121:53:191
2590299231,15cyclictest14602-21diskmemload22:43:331
2590299231,11cyclictest684-21cpuspeed_turbos23:05:131
2590299230,16cyclictest18245-1kworker/u9:219:41:431
2589799233,10cyclictest11-21rcuc/020:20:020
2589799231,6cyclictest0-21swapper/023:09:460
2589799231,6cyclictest0-21swapper/021:37:480
2589799231,12cyclictest0-21swapper/020:35:020
2589799230,6cyclictest0-21swapper/021:27:550
25914992216,3cyclictest32121-21gltestperf00:15:153
2591499221,4cyclictest0-21swapper/321:45:103
2591499221,15cyclictest0-21swapper/321:19:073
2591499221,15cyclictest0-21swapper/321:19:073
2591499221,10cyclictest13494-21unixbench-2d19:50:263
2591499220,9cyclictest0-21swapper/320:40:143
2590999222,9cyclictest808-21snmpd21:50:132
2590999221,4cyclictest0-21swapper/221:37:002
2590299225,2cyclictest87550irq/128-enp1s0-19:30:201
25902992214,2cyclictest0-21swapper/120:50:421
2590299221,18cyclictest27630-21expr20:25:141
2590299221,12cyclictest28220-21munin-node20:45:301
2590299221,10cyclictest0-21swapper/123:50:401
25902992210,3cyclictest0-21swapper/120:20:521
2590299220,6cyclictest0-21swapper/122:48:111
2590299220,6cyclictest0-21swapper/121:37:051
2590299220,16cyclictest0-21swapper/121:30:141
2590299220,16cyclictest0-21swapper/120:05:181
2590299220,16cyclictest0-21swapper/120:05:181
2589799225,5cyclictest0-21swapper/000:01:550
2589799223,12cyclictest14602-21diskmemload23:16:250
2589799221,6cyclictest0-21swapper/023:27:500
2589799221,6cyclictest0-21swapper/023:01:200
2589799221,6cyclictest0-21swapper/022:04:200
2589799221,5cyclictest0-21swapper/022:49:520
2589799221,5cyclictest0-21swapper/021:11:470
2589799221,5cyclictest0-21swapper/000:31:200
2589799221,11cyclictest3619-21perf20:45:010
2589799220,6cyclictest0-21swapper/022:22:100
152742221,3sleep00-21swapper/019:55:230
2591499217,8cyclictest0-21swapper/321:24:463
2591499213,3cyclictest0-21swapper/323:25:333
2591499211,7cyclictest0-21swapper/322:52:473
2591499211,6cyclictest0-21swapper/322:05:133
2591499211,5cyclictest0-21swapper/323:55:073
2591499211,5cyclictest0-21swapper/322:42:193
2591499211,5cyclictest0-21swapper/321:31:493
2591499211,14cyclictest0-21swapper/323:30:593
25914992111,3cyclictest2258-21/usr/sbin/munin21:55:233
2591499211,11cyclictest20810-21/usr/sbin/munin20:10:173
2590999217,3cyclictest0-21swapper/222:48:532
2590999215,4cyclictest6434-21idleruntime00:30:172
2590999214,3cyclictest15732-21sshd23:38:482
2590999211,8cyclictest27188-21perf20:25:012
2590999211,6cyclictest2697-21ntp_states20:40:202
2590999211,6cyclictest13381-21taskset23:31:502
25909992115,3cyclictest22429-21crond23:55:022
2590999211,10cyclictest0-21swapper/223:20:202
2590299219,3cyclictest4233-21munin-run22:00:011
2590299219,3cyclictest0-21swapper/123:11:241
2590299211,9cyclictest0-21swapper/123:24:151
2590299211,15cyclictest0-21swapper/120:31:501
2590299211,14cyclictest0-21swapper/122:39:431
2590299211,11cyclictest28150-21ntp_states00:05:231
2590299210,6cyclictest0-21swapper/100:21:191
2590299210,5cyclictest0-21swapper/122:55:511
2590299210,15cyclictest0-21swapper/123:55:141
2590299210,15cyclictest0-21swapper/121:18:031
2590299210,15cyclictest0-21swapper/121:18:031
2590299210,15cyclictest0-21swapper/120:40:071
2589799212,1cyclictest0-21swapper/000:16:480
2589799212,11cyclictest9814-21/usr/sbin/munin19:45:260
2589799211,7cyclictest0-21swapper/021:21:000
2589799211,6cyclictest0-21swapper/022:10:120
2589799211,5cyclictest0-21swapper/023:58:370
2589799211,5cyclictest0-21swapper/022:32:270
2589799211,5cyclictest0-21swapper/022:32:270
2589799211,4cyclictest0-21swapper/022:53:250
2589799211,4cyclictest0-21swapper/022:39:330
2589799211,11cyclictest12634-21/usr/sbin/munin21:05:230
2589799210,6cyclictest0-21swapper/022:18:210
117722211,2sleep311852-21latency_hist19:50:023
73762200,2sleep20-21swapper/219:40:002
294622200,3sleep029466-21fschecks_count19:15:370
2591499207,7cyclictest0-21swapper/319:20:253
2591499206,2cyclictest0-21swapper/323:45:133
2591499205,8cyclictest0-21swapper/322:26:283
2591499205,4cyclictest9198-21df_inode22:10:143
2591499205,2cyclictest0-21swapper/320:48:483
2591499204,9cyclictest0-21swapper/321:25:293
2591499202,3cyclictest0-21swapper/323:36:203
2591499201,6cyclictest30806-21NetworkManager19:18:083
2591499201,6cyclictest0-21swapper/323:50:173
2591499201,5cyclictest0-21swapper/323:43:293
2591499201,5cyclictest0-21swapper/323:43:293
2591499201,5cyclictest0-21swapper/322:57:583
2591499201,5cyclictest0-21swapper/300:10:163
2591499201,16cyclictest23921-21memory20:15:183
2591499201,13cyclictest0-21swapper/323:15:323
2591499201,13cyclictest0-21swapper/300:36:293
2591499201,12cyclictest0-21swapper/300:32:253
2591499200,10cyclictest3483-21grep23:10:173
2590999205,3cyclictest87350irq/126-enp1s0-22:15:182
2590999201,7cyclictest0-21swapper/223:03:312
2590999201,7cyclictest0-21swapper/200:39:212
2590999201,6cyclictest0-21swapper/221:44:282
2590999201,10cyclictest0-21swapper/221:29:382
2590999200,5cyclictest827-21runrttasks00:08:382
2590299209,3cyclictest8568-21df20:55:141
2590299203,2cyclictest28689-21kworker/u8:200:13:051
2590299201,3cyclictest0-21swapper/121:45:591
25902992013,3cyclictest0-21swapper/119:39:051
25902992012,3cyclictest0-21swapper/119:50:301
2590299201,14cyclictest0-21swapper/123:47:281
2590299201,14cyclictest0-21swapper/122:01:201
2590299201,14cyclictest0-21swapper/121:05:531
2590299201,14cyclictest0-21swapper/120:00:441
2590299201,14cyclictest0-21swapper/119:48:441
2590299201,14cyclictest0-21swapper/119:25:181
2590299201,14cyclictest0-21swapper/119:17:441
2590299201,11cyclictest26912-21irqstats19:10:191
2590299201,10cyclictest15808-21/usr/sbin/munin22:25:151
2589799201,4cyclictest0-21swapper/021:32:580
2589799201,11cyclictest16064-21/usr/sbin/munin23:40:160
2589799201,11cyclictest16064-21/usr/sbin/munin23:40:160
194652201,3sleep30-21swapper/322:33:103
194652201,3sleep30-21swapper/322:33:103
280512191,3sleep00-21swapper/019:14:550
2591499195,3cyclictest0-21swapper/320:25:133
2591499193,5cyclictest0-21swapper/321:44:053
2591499193,13cyclictest2533-21unixbench_multi19:25:263
2591499191,7cyclictest0-21swapper/300:00:133
2591499191,6cyclictest766-21ntpd19:41:053
2591499191,5cyclictest0-21swapper/300:09:243
25914991912,6cyclictest9468-21kworker/u8:119:55:093
25914991912,4cyclictest28068-21latency_hist19:15:013
2591499191,15cyclictest4314-21seq19:30:223
2591499191,15cyclictest14311-21ntpq22:20:203
2591499191,12cyclictest0-21swapper/322:03:313
2590999195,3cyclictest87650irq/129-enp1s0-19:45:142
2590999195,3cyclictest87650irq/129-enp1s0-00:10:142
2590999195,3cyclictest16808-21timerandwakeup22:25:262
2590999193,3cyclictest87350irq/126-enp1s0-22:55:142
2590999193,3cyclictest12300-21sshd21:03:492
2590999191,6cyclictest766-21ntpd19:58:592
2590999191,5cyclictest17137-21ntp_states23:40:232
2590999191,5cyclictest17137-21ntp_states23:40:222
25909991914,3cyclictest18797-21grep23:45:172
2590299198,3cyclictest32264-21open_files19:20:211
2590299191,16cyclictest0-21swapper/120:35:091
2590299190,13cyclictest0-21swapper/121:00:131
2589799191,5cyclictest0-21swapper/023:13:250
2589799191,11cyclictest0-21swapper/020:48:080
2589799191,10cyclictest808-21snmpd21:55:140
2589799191,10cyclictest808-21snmpd21:00:040
2591499182,13cyclictest6029-21/usr/sbin/munin20:50:173
25914991815,2cyclictest0-21swapper/300:25:163
2591499181,3cyclictest0-21swapper/321:39:093
2590999183,3cyclictest87350irq/126-enp1s0-22:20:122
2590999182,4cyclictest7004-21df_inode22:05:142
2590999181,6cyclictest0-21swapper/221:13:262
2590999181,5cyclictest31309-21sshd20:31:482
2590999181,5cyclictest11412-21sshd23:29:482
2590999181,5cyclictest0-21swapper/223:57:402
2590999181,5cyclictest0-21swapper/200:29:352
2590999181,4cyclictest5192-21turbostat19:35:022
2590999181,4cyclictest4859-21/usr/sbin/munin20:45:212
2590999181,4cyclictest3296-21ntp_kernel_pll_21:55:222
2590999181,4cyclictest12634-21/usr/sbin/munin21:05:192
2590999181,3cyclictest24856-21/usr/sbin/munin00:00:132
25909991812,3cyclictest31898-21cat19:20:162
2590999181,11cyclictest0-21swapper/221:17:382
2590999181,11cyclictest0-21swapper/221:17:382
2590299187,2cyclictest14231-21egrep19:55:131
2589799189,2cyclictest18245-1kworker/u9:223:21:220
2589799183,9cyclictest808-21snmpd20:14:500
2589799183,7cyclictest0-21swapper/023:36:180
2589799181,9cyclictest808-21snmpd20:29:490
2589799181,9cyclictest0-21swapper/022:29:580
2589799181,8cyclictest3010-21awk19:30:010
2589799181,14cyclictest17014-21head20:00:170
2589799181,11cyclictest0-21swapper/021:16:480
2589799181,11cyclictest0-21swapper/021:16:480
148292181,3sleep30-21swapper/321:10:113
2591499173,12cyclictest401ktimersoftd/320:00:153
2591499171,4cyclictest25302-21ntp_states22:45:213
25914991712,3cyclictest586-21sendmail_mailst20:35:243
2591499171,14cyclictest0-21swapper/321:07:473
25914991711,3cyclictest8578-21diskstats20:55:153
2591499171,13cyclictest19604-21grep20:05:203
2591499171,13cyclictest19604-21grep20:05:193
2590999173,4cyclictest0-21swapper/223:05:262
2590999172,3cyclictest87350irq/126-enp1s0-21:24:412
2590999172,3cyclictest1385-21latency_hist20:40:012
2590999171,8cyclictest0-21swapper/219:52:532
2590999171,6cyclictest0-21swapper/221:49:422
2590999171,6cyclictest0-21swapper/200:21:572
2590999171,5cyclictest0-21swapper/222:53:222
2590999171,5cyclictest0-21swapper/222:10:482
2590999171,4cyclictest22230-21wc22:40:132
2590999171,15cyclictest0-21swapper/219:12:102
25909991711,3cyclictest4950-21cut22:00:152
25909991711,3cyclictest24097-21uniq21:30:142
25909991711,3cyclictest10195-21munin-plugin-st21:00:012
2589799171,8cyclictest808-21snmpd19:24:370
2589799171,6cyclictest0-21swapper/020:51:430
2589799171,3cyclictest0-21swapper/020:39:110
2591499161,6cyclictest0-21swapper/320:22:433
25914991613,2cyclictest0-21swapper/321:04:543
25914991611,3cyclictest5421-21/usr/sbin/munin19:35:133
2590999162,8cyclictest16505-21cpuspeed_turbos20:00:132
2590999161,4cyclictest20810-21/usr/sbin/munin20:10:152
25909991611,3cyclictest30514-21unixbench_multi19:15:482
25909991611,3cyclictest2548-21grep19:25:262
2589799162,7cyclictest87550irq/128-enp1s0-23:45:270
2589799161,6cyclictest0-21swapper/020:05:140
2589799161,6cyclictest0-21swapper/020:05:130
25897991610,3cyclictest4081-21cat19:30:190
2591499158,4cyclictest30256-21sed20:30:163
25909991510,3cyclictest9148-21unixbench_multi19:40:272
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional