You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-10 - 07:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun May 10, 2026 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1332021700,1sleep30-21swapper/300:22:053
25059216399,34sleep30-21swapper/319:09:493
248882140104,25sleep10-21swapper/119:07:361
24907213996,34sleep00-21swapper/019:07:500
247832138103,24sleep20-21swapper/219:06:162
165432870,4chrt0-21swapper/322:00:243
32362790,2sleep10-21swapper/100:15:131
289202660,2sleep10-21swapper/121:21:371
117892630,2sleep20-21swapper/219:50:242
278622600,2sleep30-21swapper/323:45:133
173282600,3sleep00-21swapper/022:49:480
313492570,1sleep20-21swapper/221:24:022
108442570,1sleep10-21swapper/119:50:151
287802540,1sleep30-21swapper/300:10:093
15512540,2sleep20-21swapper/219:30:102
185432500,9sleep334-21ksoftirqd/322:25:393
4412490,2sleep00-21swapper/020:40:230
175722480,1chrt0-21swapper/000:00:550
126322480,1sleep10-21swapper/119:55:141
169382420,1sleep00-21swapper/023:36:350
57362410,2chrt0-21swapper/222:16:232
302902410,2chrt0-21swapper/122:59:171
52942380,3sleep30-21swapper/322:15:573
41902350,1sleep10-21swapper/122:15:221
43902330,3sleep228-21ksoftirqd/219:35:162
25347993316,4cyclictest28-21ksoftirqd/221:10:222
79262310,11sleep334-21ksoftirqd/300:18:283
25347993119,2cyclictest141rcu_preempt00:08:412
25345993123,6cyclictest2639-21munin-run22:15:000
25347993016,4cyclictest28-21ksoftirqd/222:21:532
25347992912,5cyclictest28-21ksoftirqd/221:58:502
125542290,2sleep10-21swapper/122:45:241
25347992819,6cyclictest10370-21ssh23:31:462
25347992811,4cyclictest28-21ksoftirqd/223:15:482
2534799279,10cyclictest28-21ksoftirqd/200:32:022
25347992713,4cyclictest28-21ksoftirqd/223:26:282
2534799271,2cyclictest0-21swapper/219:11:322
25347992712,5cyclictest28-21ksoftirqd/221:25:172
25346992720,5cyclictest22-21ksoftirqd/123:50:001
25347992616,2cyclictest0-21swapper/200:12:012
25347992614,5cyclictest28-21ksoftirqd/200:00:502
25347992611,6cyclictest17300-1kworker/u9:023:20:342
25347992610,4cyclictest28-21ksoftirqd/221:30:042
25347992610,4cyclictest28-21ksoftirqd/200:17:122
25347992610,2cyclictest141rcu_preempt23:41:562
2534699261,1cyclictest0-21swapper/119:11:291
2534899251,23cyclictest0-21swapper/319:11:483
2534799258,6cyclictest28-21ksoftirqd/222:12:372
25347992514,4cyclictest28-21ksoftirqd/222:36:522
25347992511,2cyclictest141rcu_preempt22:42:192
25347992510,4cyclictest28-21ksoftirqd/222:50:302
25347992510,4cyclictest28-21ksoftirqd/222:32:492
25347992510,4cyclictest28-21ksoftirqd/221:36:032
25347992510,4cyclictest28-21ksoftirqd/200:22:412
2534699252,16cyclictest28751-21taskset22:10:121
25346992514,7cyclictest22748-21cut22:05:181
2534599258,3cyclictest141rcu_preempt23:10:240
25345992512,9cyclictest560-21dbus-daemon00:24:590
2534599251,1cyclictest0-21swapper/019:11:230
25348992414,7cyclictest278-21systemd-journal23:35:003
25348992413,7cyclictest9820-21apt-get22:20:123
25348992410,8cyclictest7070-21ssh21:30:133
2534799248,3cyclictest141rcu_preempt21:45:582
2534799244,11cyclictest28-21ksoftirqd/222:45:202
25347992418,4cyclictest28-21ksoftirqd/223:59:592
25347992417,5cyclictest28-21ksoftirqd/221:00:012
25347992412,4cyclictest28-21ksoftirqd/222:01:472
25347992412,4cyclictest28-21ksoftirqd/221:19:362
25347992411,2cyclictest141rcu_preempt00:26:132
25346992411,7cyclictest12034-21diskmemload21:25:101
247082240,2sleep00-21swapper/021:19:100
138342240,2sleep30-21swapper/322:46:193
262652230,2sleep0131rcuc/023:44:100
2534899237,12cyclictest13491-1kworker/u9:123:35:113
25348992315,6cyclictest30938-21ssh21:23:393
2534799238,8cyclictest28-21ksoftirqd/222:28:382
2534799238,4cyclictest28-21ksoftirqd/223:46:532
2534799234,5cyclictest28-21ksoftirqd/221:41:532
2534799233,5cyclictest28-21ksoftirqd/221:51:352
2534799233,2cyclictest141rcu_preempt00:39:542
25347992317,4cyclictest28-21ksoftirqd/223:40:002
2534699239,9cyclictest648-21ntpd23:35:211
25346992317,4cyclictest4041-21sendmail22:40:011
25346992313,5cyclictest12034-21diskmemload23:05:251
25346992313,5cyclictest12034-21diskmemload23:05:241
25345992313,7cyclictest27883-21diskstats20:30:150
2534899227,3cyclictest278122sh22:33:083
25348992214,6cyclictest6183-21diskstats20:55:153
25348992211,8cyclictest21077-21diskstats21:40:173
25348992210,9cyclictest560-21dbus-daemon21:10:003
2534799227,3cyclictest141rcu_preempt20:35:202
2534799225,9cyclictest28-21ksoftirqd/223:04:412
2534799225,3cyclictest28-21ksoftirqd/219:25:152
2534799222,11cyclictest28-21ksoftirqd/222:55:182
2534799221,11cyclictest28-21ksoftirqd/223:14:162
2534799221,11cyclictest28-21ksoftirqd/222:06:172
25347992210,2cyclictest141rcu_preempt23:07:192
25347992210,2cyclictest141rcu_preempt23:07:182
2534699221,17cyclictest31457-21ssh23:00:101
2534599227,3cyclictest141rcu_preempt22:58:060
2534599224,4cyclictest12034-21diskmemload23:01:570
2534599222,14cyclictest20453-21packagekitd21:40:040
2534599221,7cyclictest17433-21memory22:25:170
2534599221,4cyclictest0-21swapper/023:51:010
2534599221,4cyclictest0-21swapper/023:51:000
25345992213,7cyclictest23143-21ntpq00:05:210
25345992212,7cyclictest4630-21timerwakeupswit22:15:240
107002220,2chrt0-21swapper/123:32:071
2534899212,13cyclictest22131-21sh00:05:133
25348992112,6cyclictest278-21systemd-journal00:00:003
25348992112,6cyclictest24449-21cut22:55:013
25348992111,7cyclictest2562-21awk20:45:223
2534899211,10cyclictest12034-21diskmemload00:31:303
25348992110,8cyclictest25409-21apt-get20:25:123
2534799219,4cyclictest28-21ksoftirqd/220:40:242
2534799219,2cyclictest141rcu_preempt20:05:122
2534799213,3cyclictest141rcu_preempt23:51:432
2534799213,3cyclictest141rcu_preempt23:51:432
25347992116,3cyclictest28-21ksoftirqd/219:20:012
25347992114,5cyclictest12019-21cron21:10:012
2534699211,5cyclictest12640-21ssh23:57:471
25346992115,4cyclictest22530-21ssh22:29:381
25346992113,6cyclictest22-21ksoftirqd/123:25:011
25346992113,5cyclictest12498-21/usr/sbin/munin23:10:111
25346992110,7cyclictest5004-21tail22:40:141
2534599219,8cyclictest18914-21apt-get20:10:120
2534599214,3cyclictest12034-21diskmemload23:48:360
2534599214,3cyclictest12034-21diskmemload21:33:310
2534599212,14cyclictest12452-21apt-get21:10:130
2534599211,7cyclictest8171-21missed_timers23:30:200
2534599211,6cyclictest9924-21ls21:55:240
25345992113,6cyclictest2458-21ntp_kernel_pll_20:45:210
25345992113,6cyclictest20237-21seq00:27:160
25345992111,7cyclictest10211-21cat21:05:130
2534599211,16cyclictest12034-21diskmemload22:30:070
25345992110,7cyclictest52750irq/139-enp3s0-23:05:130
25345992110,7cyclictest52750irq/139-enp3s0-23:05:120
242512210,2sleep10-21swapper/121:18:441
2534899202,13cyclictest6064-21apt-get19:40:133
2534899201,6cyclictest12034-21diskmemload21:37:543
25348992013,5cyclictest4316-21cut19:35:153
25348992013,5cyclictest25768-21sed21:20:003
25348992013,5cyclictest21389-21diskstats20:15:153
25348992012,6cyclictest8878-21seq21:55:143
25348992012,6cyclictest665-21diskstats21:25:123
25348992012,5cyclictest31870-21awk00:35:263
25348992011,7cyclictest560-21dbus-daemon19:30:003
25348992011,7cyclictest29348-21seq19:19:163
2534899201,16cyclictest12034-21diskmemload00:04:563
25348992011,5cyclictest53050irq/142-enp3s0-21:45:123
2534799208,5cyclictest28-21ksoftirqd/220:04:312
25347992014,5cyclictest28-21ksoftirqd/221:05:002
25347992014,4cyclictest28-21ksoftirqd/219:44:592
25347992014,4cyclictest24506-21timerandwakeup20:20:232
25347992013,5cyclictest28-21ksoftirqd/220:50:012
25347992011,6cyclictest13707-21cat19:55:242
2534699209,8cyclictest278-21systemd-journal00:29:591
2534699205,4cyclictest66050lldpd23:17:511
2534699203,10cyclictest597-21ssh23:25:121
25346992015,4cyclictest22-21ksoftirqd/122:04:591
25346992012,6cyclictest30801-21tr19:20:211
25346992012,6cyclictest25623-21seq22:30:581
25346992012,6cyclictest11407-21cat21:05:251
2534599209,8cyclictest28782-21rm00:10:090
2534599203,3cyclictest141rcu_preempt22:40:230
2534599203,3cyclictest12034-21diskmemload00:19:140
25345992012,6cyclictest9154-21pmu-power19:45:210
25345992012,6cyclictest24015-21seq23:18:260
25345992012,6cyclictest2024-21vmstat21:25:230
25345992011,6cyclictest23660-21cat00:30:120
2534899199,7cyclictest30539-21grep19:20:193
2534899197,10cyclictest12034-21diskmemload22:55:153
2534899192,12cyclictest31847-21apt-get20:40:133
2534899192,12cyclictest23263-21apt-get20:20:123
2534899191,9cyclictest141rcu_preempt23:00:153
2534899191,9cyclictest0-21swapper/322:36:173
25348991912,5cyclictest34-21ksoftirqd/323:15:063
25348991912,5cyclictest3062-21vmstat19:30:253
25348991910,6cyclictest53050irq/142-enp3s0-23:10:233
2534799198,2cyclictest141rcu_preempt20:18:262
2534799197,4cyclictest28-21ksoftirqd/220:13:402
25347991914,3cyclictest271rcuc/219:20:122
2534799191,3cyclictest141rcu_preempt20:34:132
25347991911,6cyclictest27216-21seq20:29:112
2534699199,7cyclictest20689-21apt-get21:40:131
2534699193,9cyclictest1842-21ssh23:50:121
2534699193,9cyclictest1842-21ssh23:50:111
2534699192,12cyclictest15358-21apt-get00:00:131
2534699192,11cyclictest30163-21apt-get00:35:111
25346991912,5cyclictest29684-21awk20:35:111
25346991911,5cyclictest27759-21cat19:15:151
25346991911,4cyclictest13418-21needreboot21:10:201
2534699191,12cyclictest636-21runrttasks00:30:111
25346991910,7cyclictest6228-21cat19:40:121
2534599191,4cyclictest2469-21irqstats21:50:160
25345991914,3cyclictest6366-21grep20:55:170
25345991914,3cyclictest5776-21sh23:28:570
25345991914,3cyclictest17107-21sh20:05:130
25345991913,4cyclictest21634-21sh22:52:230
25345991913,4cyclictest17283-21sh22:00:440
25345991912,5cyclictest29969-21cat20:35:120
25345991912,5cyclictest28277-21date19:15:190
25345991912,5cyclictest11571-21vmstat22:20:270
25345991911,6cyclictest31773-21cat19:25:010
25345991911,6cyclictest27662-21ls21:45:130
25345991911,6cyclictest26837-21cat21:20:190
25345991911,6cyclictest25599-21cat20:25:130
25345991911,6cyclictest22954-21seq20:19:430
25345991911,6cyclictest22454-21grep22:05:150
25345991911,6cyclictest14221-21perl21:35:140
25345991910,7cyclictest10057-21cat21:05:010
25345991910,6cyclictest16730-21cat20:05:010
24482190,4sleep32447-21hddtemp_smartct23:50:153
24482190,4sleep32447-21hddtemp_smartct23:50:143
2534899189,7cyclictest12334-21cat19:55:013
2534899188,8cyclictest1700-21/usr/sbin/munin21:50:143
2534899183,4cyclictest34-21ksoftirqd/319:59:593
2534899181,8cyclictest5284-21ssh23:28:263
2534899181,8cyclictest0-21swapper/321:13:353
25348991813,3cyclictest53050irq/142-enp3s0-23:21:463
25348991811,5cyclictest8459-21expr19:45:133
25348991811,5cyclictest6863-21ntpq23:05:223
25348991811,5cyclictest6863-21ntpq23:05:223
25348991811,5cyclictest5671-21date20:55:013
25348991811,5cyclictest21564-21sed23:40:163
25348991811,5cyclictest17395-21cat20:05:153
25348991811,5cyclictest16945-21expr00:25:153
25348991810,6cyclictest8493-21seq22:42:423
25348991810,6cyclictest30881-21cat20:35:223
25348991810,6cyclictest29148-21seq22:10:163
25348991810,6cyclictest20421-21who20:10:263
25348991810,5cyclictest9116-21mailstats21:00:233
2534799186,9cyclictest28-21ksoftirqd/220:50:122
25347991810,6cyclictest8793-21basename19:45:172
2534699189,7cyclictest5154-21grep19:35:241
2534699189,6cyclictest6595-21ls20:55:191
2534699189,6cyclictest19178-21wc22:50:191
2534699188,7cyclictest1570-21apt-get20:45:131
2534699186,3cyclictest141rcu_preempt20:05:001
2534699182,11cyclictest640-21snmpd21:00:241
25346991811,5cyclictest23509-21expr20:20:121
25346991810,6cyclictest2667-21expr21:50:171
25346991810,6cyclictest22919-21seq23:40:491
25346991810,6cyclictest17111-21diskstats20:05:141
25346991810,6cyclictest12114-21seq21:57:201
25345991813,3cyclictest6039-21/usr/sbin/munin19:40:140
25345991811,5cyclictest24184-21seq20:20:210
25345991810,6cyclictest5469-21seq19:36:020
25345991810,6cyclictest4515-21cat20:50:200
25345991810,6cyclictest3450-21basename19:35:010
25345991810,6cyclictest31091-21ls00:35:190
25345991810,6cyclictest30764-21basename22:35:150
25345991810,6cyclictest27546-21cat23:20:260
25345991810,6cyclictest10038-21cat23:55:230
25348991711,4cyclictest25028-21ssh22:06:433
25348991710,4cyclictest28207-21uname20:30:183
2534699179,6cyclictest22375-21cat20:15:251
2534699176,9cyclictest0-21swapper/122:22:021
2534699172,11cyclictest14049-21df21:35:131
2534699172,10cyclictest29184-21df00:10:131
25346991712,3cyclictest1937-21fschecks_count19:30:141
2534699171,14cyclictest8269-21ssh21:30:221
2534699171,14cyclictest10282-21df_inode00:20:151
25346991711,3cyclictest27622-21cpuspeed_turbos20:30:121
2534699171,12cyclictest28799-21chrt21:45:251
25346991710,5cyclictest7939-21cron19:45:011
25346991710,5cyclictest32588-21sed20:40:191
25346991710,5cyclictest32159-21df19:25:151
25346991710,5cyclictest24626-21seq00:06:131
2534599172,10cyclictest12476-21apt-get19:55:120
25345991712,3cyclictest10739-21fschecks_time19:50:160
25345991710,5cyclictest1028-21seq19:26:430
2534899169,5cyclictest16132-21sed20:00:243
2534699168,6cyclictest25604-21ls20:25:141
2534699168,6cyclictest19293-21grep20:10:131
2534699157,6cyclictest4692-21grep20:50:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional