You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-21 - 12:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Tue Jul 21, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
116892234161,23sleep10-21swapper/119:07:561
118052233162,58sleep20-21swapper/219:09:262
118412204166,25sleep30-21swapper/319:09:563
116582201164,25sleep00-21swapper/019:07:310
11943998985,3cyclictest0-21swapper/220:02:322
253292560,3sleep20-21swapper/219:35:262
84262540,2sleep30-21swapper/321:25:203
86262430,1sleep30-21swapper/322:40:163
11933993810,13cyclictest11028-21idleruntime-cro20:20:020
1193399366,14cyclictest28326-21munin-run00:40:020
1193399366,14cyclictest28326-21munin-run00:40:020
1195099351,12cyclictest24588-21ntp_states23:15:223
1193399356,13cyclictest27692-21munin-run23:25:000
1195099341,21cyclictest0-21swapper/321:15:093
1195099341,20cyclictest0-21swapper/321:05:403
1195099331,26cyclictest0-21swapper/323:38:093
1195099330,15cyclictest0-21swapper/300:35:093
1195099330,15cyclictest0-21swapper/300:35:083
11950993216,3cyclictest0-21swapper/320:38:243
11950993216,10cyclictest0-21swapper/320:00:523
1195099321,19cyclictest0-21swapper/321:10:043
1195099321,13cyclictest0-21swapper/322:54:443
1195099321,13cyclictest0-21swapper/322:09:133
11950993117,8cyclictest0-21swapper/300:18:373
11950993116,8cyclictest0-21swapper/322:39:163
11950993115,8cyclictest0-21swapper/323:05:483
1195099311,21cyclictest0-21swapper/323:55:373
1195099310,18cyclictest0-21swapper/320:33:083
1195099310,14cyclictest0-21swapper/323:42:453
11950993012,3cyclictest0-21swapper/300:31:213
1195099301,20cyclictest0-21swapper/322:27:083
1195099301,20cyclictest0-21swapper/320:49:483
1195099301,13cyclictest0-21swapper/321:58:293
1195099300,14cyclictest0-21swapper/321:50:173
1195099300,13cyclictest0-21swapper/321:21:213
1195099300,11cyclictest0-21swapper/321:39:293
1195099291,13cyclictest495-21jbd2/dm-0-823:10:123
1195099291,13cyclictest0-21swapper/322:18:283
1195099291,11cyclictest0-21swapper/323:30:573
1195099291,10cyclictest0-21swapper/300:25:013
1195099290,27cyclictest0-21swapper/300:28:493
1195099290,13cyclictest0-21swapper/319:34:493
1195099290,12cyclictest0-21swapper/323:24:483
133102280,3sleep19-21rcu_preempt20:25:021
1195099281,24cyclictest17958-21awk23:00:223
1195099281,13cyclictest0-21swapper/300:07:493
1195099281,12cyclictest547-21diskmemload22:47:403
1195099281,11cyclictest0-21swapper/319:52:213
1195099280,19cyclictest0-21swapper/323:46:573
1193399289,16cyclictest24844-21basename22:05:010
197652270,10sleep230-21rcuc/219:25:152
11950992715,3cyclictest0-21swapper/320:52:283
11950992715,3cyclictest0-21swapper/320:28:363
11950992715,3cyclictest0-21swapper/320:07:083
11950992714,3cyclictest0-21swapper/320:17:003
1195099271,18cyclictest0-21swapper/322:04:453
1195099271,12cyclictest0-21swapper/322:20:563
1195099271,12cyclictest0-21swapper/320:22:323
1195099271,12cyclictest0-21swapper/320:13:083
1195099271,11cyclictest0-21swapper/319:58:093
1195099271,11cyclictest0-21swapper/319:58:083
1195099270,12cyclictest0-21swapper/321:42:213
1195099270,12cyclictest0-21swapper/300:13:573
11939992710,8cyclictest30588-21systemd-journal21:02:201
1195099267,9cyclictest0-21swapper/319:42:053
1195099260,12cyclictest0-21swapper/322:30:243
1195099260,12cyclictest0-21swapper/322:12:493
1193399267,16cyclictest16693-21munin-run23:00:010
1193399261,13cyclictest28603-21munin-run21:00:000
1195099259,9cyclictest0-21swapper/323:25:413
1195099257,8cyclictest0-21swapper/322:58:083
11950992515,3cyclictest0-21swapper/300:03:333
1195099251,13cyclictest0-21swapper/319:13:483
1195099251,12cyclictest0-21swapper/321:30:133
1195099251,12cyclictest0-21swapper/320:44:043
1193999256,9cyclictest0-21swapper/123:59:181
1193999251,12cyclictest22149-21grep21:55:261
1193999251,12cyclictest11017-21cat00:00:111
1193999250,17cyclictest0-21swapper/122:58:251
1193399254,4cyclictest30191-21awk19:50:010
1195099249,3cyclictest0-21swapper/319:28:413
1195099241,13cyclictest0-21swapper/320:56:523
1195099240,8cyclictest0-21swapper/323:50:253
1194399241,10cyclictest2218-21munin-run20:00:002
1194399241,10cyclictest2218-21munin-run20:00:002
1194399241,10cyclictest0-21swapper/221:26:292
1193999243,11cyclictest18160-21gltestperf00:15:151
1193999242,10cyclictest13284-21cat22:50:181
1193999241,11cyclictest10725-21ntp_states21:30:211
1195099237,3cyclictest28209-21/usr/sbin/munin19:45:253
1195099231,15cyclictest0-21swapper/321:45:413
1195099231,13cyclictest0-21swapper/319:19:203
1193999232,13cyclictest87650irq/129-enp1s0-21:21:321
1193999231,15cyclictest0-21swapper/120:48:161
1193999231,12cyclictest12855-21strings21:35:201
1193399237,3cyclictest0-21swapper/021:10:000
242422220,5sleep11193999cyclictest23:15:181
140332221,17sleep014150-21head00:05:180
1195099229,3cyclictest0-21swapper/321:01:003
1195099229,3cyclictest0-21swapper/319:20:053
1194399226,8cyclictest87650irq/129-enp1s0-20:50:172
1194399226,4cyclictest0-21swapper/200:10:262
1194399223,5cyclictest22393-40dnf21:57:162
1194399221,6cyclictest0-21swapper/223:32:462
1194399221,6cyclictest0-21swapper/222:32:562
1194399221,6cyclictest0-21swapper/221:44:582
1194399221,6cyclictest0-21swapper/200:09:042
1194399221,5cyclictest0-21swapper/221:24:542
1193999222,9cyclictest0-21swapper/122:40:441
1193999222,18cyclictest0-21swapper/122:34:331
1193999222,18cyclictest0-21swapper/119:25:501
11939992217,4cyclictest0-21swapper/121:11:071
1193999221,3cyclictest15098-21taskset00:09:371
1193999221,19cyclictest0-21swapper/120:58:321
1193999221,19cyclictest0-21swapper/100:22:241
1193999221,14cyclictest0-21swapper/122:03:581
1193999221,11cyclictest17231-21acpi19:20:121
1193999221,11cyclictest14103-21latency_hist19:15:011
1193999220,9cyclictest0-21swapper/123:24:401
1193999220,2cyclictest0-21swapper/100:12:231
1193399224,3cyclictest23521-21/usr/sbin/munin23:15:140
1193399221,9cyclictest11284-21df_inode00:00:120
1193399221,18cyclictest31263-21head23:30:220
11933992210,3cyclictest19209-21basename19:25:010
1194399213,6cyclictest0-21swapper/223:24:522
1194399211,6cyclictest18245-1kworker/u9:223:03:092
1194399211,6cyclictest0-21swapper/221:31:062
1194399210,3cyclictest3393-21kworker/u8:222:35:122
1193999218,3cyclictest19516-21cat21:50:221
1193999215,4cyclictest0-21swapper/121:27:071
1193999212,18cyclictest0-21swapper/122:17:341
1193999212,17cyclictest0-21swapper/100:31:391
1193999211,4cyclictest0-21swapper/123:15:011
1193999211,18cyclictest0-21swapper/121:43:041
1193999211,10cyclictest2235-21sleep20:00:011
1193999211,10cyclictest2235-21sleep20:00:001
1193999210,8cyclictest0-21swapper/123:29:161
1193399219,3cyclictest1793-21memory21:10:180
1193399213,11cyclictest0-21swapper/022:22:190
1193399213,10cyclictest4651-21/usr/sbin/munin20:05:180
1193399211,4cyclictest16956-21users00:10:270
1193399211,11cyclictest18117-21fschecks_time00:15:160
1193399211,10cyclictest808-21snmpd19:50:070
11933992110,3cyclictest87550irq/128-enp1s0-19:57:170
11933992110,3cyclictest87550irq/128-enp1s0-19:57:170
1193399210,3cyclictest0-21swapper/023:40:470
1195099201,11cyclictest0-21swapper/319:35:253
1194399208,3cyclictest30193-21awk19:50:002
1194399204,8cyclictest87350irq/126-enp1s0-00:30:172
1194399203,10cyclictest0-21swapper/223:08:222
1194399201,5cyclictest0-21swapper/222:53:142
1194399201,5cyclictest0-21swapper/200:19:132
1194399201,3cyclictest0-21swapper/222:40:092
1194399200,5cyclictest27404-21kworker/u8:223:36:552
1193999205,4cyclictest26611-21grep22:05:251
1193999203,7cyclictest547-21diskmemload23:45:191
1193999202,3cyclictest0-21swapper/122:37:151
1193999202,3cyclictest0-21swapper/121:05:111
1193999202,16cyclictest0-21swapper/120:44:371
1193999201,8cyclictest0-21swapper/123:09:281
1193999201,18cyclictest0-21swapper/123:44:461
1193999201,18cyclictest0-21swapper/122:10:431
1193999201,18cyclictest0-21swapper/100:38:061
1193999201,18cyclictest0-21swapper/100:38:061
1193999201,17cyclictest0-21swapper/121:17:031
1193999201,17cyclictest0-21swapper/120:15:161
1193999201,13cyclictest0-21swapper/100:27:341
1193999201,11cyclictest28273-21cpuspeed_turbos19:45:121
1193999201,10cyclictest25415-21needreboot20:50:211
1193999201,10cyclictest1272-21chrt22:25:021
1193999200,4cyclictest0-21swapper/123:30:041
1193999200,18cyclictest0-21swapper/123:53:551
1193999200,18cyclictest0-21swapper/120:11:321
1193999200,14cyclictest0-21swapper/122:46:321
1193399209,4cyclictest25073-21memory00:30:200
1193399208,2cyclictest3393-21kworker/u8:222:38:200
1193399204,3cyclictest0-21swapper/022:30:190
1193399201,5cyclictest0-21swapper/021:32:570
11933992013,4cyclictest4440-21grep23:45:130
11933992012,4cyclictest22005-21dump-pmu-power20:45:000
1193399201,13cyclictest0-21swapper/023:09:240
1193399201,13cyclictest0-21swapper/022:18:440
1193399201,12cyclictest0-21swapper/021:56:000
1193399200,5cyclictest0-21swapper/023:38:410
1193399200,3cyclictest0-21swapper/021:51:130
1193399200,13cyclictest0-21swapper/022:40:100
68762191,2sleep00-21swapper/021:20:310
136852190,4sleep00-21swapper/020:25:120
1194399194,8cyclictest547-21diskmemload21:48:112
1194399192,9cyclictest8738-21/usr/sbin/munin23:55:142
1194399192,9cyclictest0-21swapper/200:37:392
1194399192,9cyclictest0-21swapper/200:37:392
1194399192,8cyclictest2108250kworker/u8:023:25:352
1194399191,9cyclictest13708-21grep19:10:252
1194399191,8cyclictest0-21swapper/200:02:162
1194399191,7cyclictest27629-21fschecks_count22:10:152
1194399191,7cyclictest25182-21grep22:05:112
11943991914,2cyclictest87650irq/129-enp1s0-22:01:292
1194399191,3cyclictest0-21swapper/221:50:322
1194399191,15cyclictest3253-21cstates21:15:132
11943991910,3cyclictest6309-21latency_hist23:50:022
1193999192,16cyclictest0-21swapper/123:02:201
11939991915,3cyclictest0-21swapper/122:25:141
11939991915,3cyclictest0-21swapper/120:32:401
11939991915,3cyclictest0-21swapper/119:32:141
1193999191,4cyclictest0-21swapper/121:46:291
1193999191,17cyclictest0-21swapper/120:28:011
1193999191,16cyclictest0-21swapper/123:35:051
1193999191,16cyclictest0-21swapper/119:41:541
1193999191,10cyclictest32417-21cut19:55:011
1193999191,10cyclictest24984-21ps19:35:221
1193999190,16cyclictest17845-21acpi20:35:141
1193399196,2cyclictest7-21ksoftirqd/021:25:160
1193399195,7cyclictest30388-21sed21:00:250
1193399195,7cyclictest13555-21timerwakeupswit19:10:250
1193399194,3cyclictest0-21swapper/022:27:200
1193399193,12cyclictest26011-21/usr/sbin/munin19:40:140
11933991914,3cyclictest87550irq/128-enp1s0-21:35:270
1193399191,3cyclictest0-21swapper/022:50:190
1193399191,3cyclictest0-21swapper/021:16:220
1193399191,13cyclictest0-21swapper/000:22:290
1193399191,12cyclictest0-21swapper/022:05:090
1193399191,12cyclictest0-21swapper/021:42:420
1193399191,10cyclictest22160-21cut23:10:180
239582181,2sleep00-21swapper/020:47:000
1194399185,4cyclictest18245-1kworker/u9:221:13:302
1194399183,4cyclictest12856-21needreboot21:35:212
1194399183,11cyclictest87350irq/126-enp1s0-22:21:392
1194399182,10cyclictest0-21swapper/200:26:472
1194399181,7cyclictest0-21swapper/223:13:062
1194399181,7cyclictest0-21swapper/222:47:412
1194399181,6cyclictest0-21swapper/223:16:082
1194399181,6cyclictest0-21swapper/220:12:462
1194399181,5cyclictest15031-21/usr/sbin/munin19:15:272
1194399181,4cyclictest0-21swapper/200:22:292
11943991812,2cyclictest87650irq/129-enp1s0-22:17:432
11943991811,4cyclictest16469-40dnf22:57:232
1194399181,14cyclictest0-21swapper/219:25:022
1194399180,6cyclictest0-21swapper/219:40:232
1193999181,3cyclictest6160-21unixbench_multi20:05:281
1193999181,16cyclictest0-21swapper/120:02:291
1193999181,10cyclictest15957-21missed_timers19:15:211
1193399184,6cyclictest23816-21/usr/sbin/munin19:35:150
1193399184,4cyclictest7110-21cat20:10:130
1193399181,5cyclictest909-21sssd_nss21:46:150
1193399181,5cyclictest0-21swapper/023:26:150
1193399181,5cyclictest0-21swapper/023:00:310
1193399181,5cyclictest0-21swapper/022:47:040
11933991812,3cyclictest827-21runrttasks20:50:210
11933991812,3cyclictest22484-21sed19:30:180
1193399181,12cyclictest0-21swapper/000:27:320
11933991810,5cyclictest16927-21sshd19:19:160
1193399180,5cyclictest0-21swapper/023:55:150
1193399180,11cyclictest0-21swapper/023:50:150
234082171,2sleep20-21swapper/219:32:502
1194399175,3cyclictest26646-21cpu20:55:142
1194399175,3cyclictest12050irq/123-ahci[0019:55:002
1194399172,6cyclictest4651-21/usr/sbin/munin20:05:212
1194399171,6cyclictest808-21snmpd20:40:152
1194399171,6cyclictest0-21swapper/223:50:092
1194399171,6cyclictest0-21swapper/222:28:572
11943991711,3cyclictest10635-21users20:15:272
11943991710,4cyclictest909-21sssd_nss20:35:242
1193399172,8cyclictest808-21snmpd22:10:340
1193399172,12cyclictest11239-21/usr/sbin/munin20:20:250
1193399171,9cyclictest808-21snmpd20:35:140
1193399171,8cyclictest15642-21/usr/sbin/munin20:30:120
1193399170,15cyclictest21411-21perf19:30:010
41622160,2sleep00-21swapper/020:01:500
1194399161,6cyclictest16624-21sed20:30:182
1194399161,6cyclictest13444-21/usr/sbin/munin20:25:262
1194399161,6cyclictest0-21swapper/223:40:162
1194399161,6cyclictest0-21swapper/220:20:192
11943991611,3cyclictest28828-21/usr/sbin/munin21:00:172
1194399160,14cyclictest0-21swapper/220:45:122
1194399152,5cyclictest909-21sssd_nss21:05:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional