You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-26 - 04:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Fri Jun 26, 2026 00:44:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
181371721140,2sleep10-21swapper/123:25:201
175079821040,4sleep00-21swapper/020:30:160
17827532940,3sleep20-21swapper/222:00:132
171998427946,23sleep00-21swapper/019:05:110
172131626747,14sleep30-21swapper/319:08:113
172117826643,14sleep10-21swapper/119:06:251
172125626543,15sleep20-21swapper/219:07:272
18358862510,2sleep20-21swapper/200:30:132
17568542400,3sleep23056-21PK-Backend20:45:222
139099370,3rtkit-daemon0-21swapper/223:21:092
139099370,3rtkit-daemon0-21swapper/221:25:282
139099370,3rtkit-daemon0-21swapper/200:05:102
18098742350,13sleep342-21ksoftirqd/323:15:173
139099350,3rtkit-daemon0-21swapper/223:28:092
139099350,3rtkit-daemon0-21swapper/221:50:142
139099350,3rtkit-daemon0-21swapper/221:16:582
139099340,3rtkit-daemon0-21swapper/223:32:332
139099340,3rtkit-daemon0-21swapper/223:18:542
139099340,3rtkit-daemon0-21swapper/222:53:492
139099330,3rtkit-daemon0-21swapper/223:04:202
17456222300,3sleep03056-21PK-Backend20:15:160
17260142290,12sleep126-21ksoftirqd/119:20:121
17455242250,2sleep30-21swapper/320:15:153
1721601992512,12cyclictest1767934-21unin-run21:20:013
1721594992517,6cyclictest436-20systemd-journal21:25:001
17296222240,3sleep30-21swapper/319:30:133
172159499241,21cyclictest1838914-21/usr/sbin/munin00:40:011
18259092230,2sleep20-21swapper/200:00:272
172160199233,19cyclictest0-21swapper/321:26:323
1721601992321,1cyclictest0-21swapper/321:21:203
1721598992312,8cyclictest436-20systemd-journal19:30:002
1721594992314,7cyclictest1776822-21apt-get21:44:591
17442292220,4chrt3056-21PK-Backend20:10:190
1721601992220,1cyclictest3056-21PK-Backend19:49:323
1721601992220,1cyclictest3056-21PK-Backend00:13:493
1721601992220,1cyclictest0-21swapper/322:25:343
1721601992220,1cyclictest0-21swapper/322:22:513
1721601992220,1cyclictest0-21swapper/321:13:353
1721601992220,1cyclictest0-21swapper/320:25:023
1721601992220,1cyclictest0-21swapper/319:14:243
172160199220,21cyclictest3056-21PK-Backend23:23:553
172160199220,20cyclictest3056-21PK-Backend19:50:083
172160199220,20cyclictest3056-21PK-Backend00:31:543
1721598992220,1cyclictest0-21swapper/223:38:522
1721598992220,1cyclictest0-21swapper/200:26:322
172159499226,14cyclictest1819557-21sh23:45:001
1721594992220,1cyclictest0-21swapper/122:14:141
1721594992219,2cyclictest3056-21PK-Backend20:10:111
1721594992215,5cyclictest1781976-21sshd21:56:131
1721594992213,7cyclictest1824777-21/usr/sbin/munin00:00:001
172159199229,5cyclictest1768019-21latency_hist21:20:020
172159199220,21cyclictest0-21swapper/020:20:060
139099220,14rtkit-daemon0-21swapper/219:39:322
18215532210,9sleep013-21ksoftirqd/023:50:090
17872212210,4chrt0-21swapper/022:10:210
17818252210,4sleep00-21swapper/021:55:210
1721601992119,1cyclictest3056-21PK-Backend23:49:503
1721601992119,1cyclictest3056-21PK-Backend23:07:383
1721601992119,1cyclictest3056-21PK-Backend22:11:343
1721601992113,6cyclictest1760719-21/usr/sbin/munin21:00:013
1721601992113,6cyclictest1747790-21threads20:20:203
172160199210,8cyclictest1789558-21unin-run22:20:013
172160199210,20cyclictest3056-21PK-Backend23:34:153
172160199210,20cyclictest3056-21PK-Backend21:58:233
172160199210,20cyclictest0-21swapper/323:10:353
172160199210,20cyclictest0-21swapper/321:35:043
172160199210,20cyclictest0-21swapper/321:07:403
172160199210,20cyclictest0-21swapper/320:35:223
172160199210,19cyclictest3056-21PK-Backend23:56:113
172160199210,19cyclictest3056-21PK-Backend21:42:073
1721598992119,1cyclictest0-21swapper/223:50:552
1721598992119,1cyclictest0-21swapper/223:46:022
1721598992119,1cyclictest0-21swapper/222:58:032
1721598992119,1cyclictest0-21swapper/222:49:582
1721598992119,1cyclictest0-21swapper/221:37:322
1721598992119,1cyclictest0-21swapper/221:32:262
1721598992119,1cyclictest0-21swapper/221:22:112
1721598992119,1cyclictest0-21swapper/220:53:182
1721598992119,1cyclictest0-21swapper/220:05:532
1721598992119,1cyclictest0-21swapper/219:23:552
1721598992119,1cyclictest0-21swapper/200:11:572
172159899210,20cyclictest0-21swapper/219:40:322
1721594992119,1cyclictest3056-21PK-Backend21:03:301
1721594992119,1cyclictest3056-21PK-Backend20:31:531
1721594992119,1cyclictest3056-21PK-Backend19:55:571
1721594992119,1cyclictest3056-21PK-Backend19:12:111
1721594992119,1cyclictest0-21swapper/122:40:231
1721594992119,1cyclictest0-21swapper/122:35:511
1721594992119,1cyclictest0-21swapper/122:33:061
1721594992119,1cyclictest0-21swapper/122:07:321
1721594992119,1cyclictest0-21swapper/119:48:481
1721594992119,1cyclictest0-21swapper/100:20:201
1721594992119,1cyclictest0-21swapper/100:09:471
1721594992118,2cyclictest3056-21PK-Backend21:31:281
1721594992118,2cyclictest1894999-21snapd23:48:431
1721594992115,4cyclictest1733647-21grep19:40:171
172159499210,20cyclictest3056-21PK-Backend23:30:041
172159499210,20cyclictest3056-21PK-Backend21:15:401
172159499210,20cyclictest3056-21PK-Backend21:13:311
172159499210,20cyclictest3056-21PK-Backend00:03:231
172159499210,20cyclictest0-21swapper/122:15:081
172159499210,20cyclictest0-21swapper/120:27:201
172159199218,11cyclictest1817231-21python323:35:200
1721591992119,1cyclictest0-21swapper/023:14:110
1721591992119,1cyclictest0-21swapper/019:57:290
1721591992114,6cyclictest1-21systemd00:00:000
1721591992111,2cyclictest141rcu_preempt20:40:000
172159199210,20cyclictest0-21swapper/023:27:500
18171332200,3sleep10-21swapper/123:35:201
1721601992018,1cyclictest3056-21PK-Backend23:39:083
1721601992018,1cyclictest3056-21PK-Backend21:45:193
1721601992018,1cyclictest0-21swapper/322:46:303
1721601992013,5cyclictest1742802-21sshd20:06:123
1721601992012,6cyclictest1728438-21cat19:25:213
172160199200,1cyclictest3056-21PK-Backend23:53:013
172160199200,1cyclictest3056-21PK-Backend22:42:173
172160199200,1cyclictest0-21swapper/300:04:023
172160199200,19cyclictest3056-21PK-Backend23:42:533
172160199200,19cyclictest3056-21PK-Backend22:58:123
172160199200,19cyclictest3056-21PK-Backend22:00:323
172160199200,19cyclictest0-21swapper/322:50:423
172160199200,19cyclictest0-21swapper/320:11:093
172160199200,19cyclictest0-21swapper/319:21:423
172160199200,18cyclictest1761351-21diskstats21:00:133
172159899206,12cyclictest1791291-21unin-run22:25:002
172159899202,12cyclictest1739259-21users19:55:212
1721598992018,1cyclictest0-21swapper/223:55:152
1721598992018,1cyclictest0-21swapper/222:43:222
1721598992018,1cyclictest0-21swapper/222:37:212
1721598992018,1cyclictest0-21swapper/221:12:172
1721598992018,1cyclictest0-21swapper/220:16:002
1721598992018,1cyclictest0-21swapper/219:46:462
1721598992013,5cyclictest1792153-21cat22:25:192
1721598992013,4cyclictest1724700-21strings19:15:172
172159899200,1cyclictest0-21swapper/221:46:442
172159899200,1cyclictest0-21swapper/220:26:552
172159899200,19cyclictest0-21swapper/221:07:372
172159899200,19cyclictest0-21swapper/220:33:182
172159899200,19cyclictest0-21swapper/219:52:142
172159899200,19cyclictest0-21swapper/219:34:172
172159899200,19cyclictest0-21swapper/200:39:082
1721594992018,1cyclictest3056-21PK-Backend23:14:571
1721594992018,1cyclictest3056-21PK-Backend20:23:211
1721594992018,1cyclictest0-21swapper/123:23:151
1721594992018,1cyclictest0-21swapper/123:06:061
1721594992018,1cyclictest0-21swapper/122:01:221
1721594992018,1cyclictest0-21swapper/121:38:521
1721594992018,1cyclictest0-21swapper/119:26:431
1721594992018,1cyclictest0-21swapper/100:19:121
1721594992017,2cyclictest3056-21PK-Backend20:59:241
1721594992017,2cyclictest3056-21PK-Backend19:37:381
1721594992015,3cyclictest1823017-21cron23:55:001
172159499200,19cyclictest3056-21PK-Backend21:25:331
172159499200,19cyclictest3056-21PK-Backend20:42:281
172159499200,19cyclictest0-21swapper/120:45:451
172159499200,19cyclictest0-21swapper/100:31:491
172159499200,19cyclictest0-21swapper/100:25:481
172159499200,19cyclictest0-21swapper/100:10:551
1721591992018,1cyclictest3056-21PK-Backend20:05:590
1721591992018,1cyclictest3056-21PK-Backend19:54:380
1721591992018,1cyclictest0-21swapper/021:04:130
1721591992018,1cyclictest0-21swapper/019:28:150
1721591992015,3cyclictest1831722-21turbostat00:20:010
1721591992013,6cyclictest1804625-21cat23:00:150
172159199200,6cyclictest1802180-21awk22:55:000
172159199200,19cyclictest0-21swapper/023:42:240
172159199200,19cyclictest0-21swapper/023:08:170
172159199200,19cyclictest0-21swapper/020:53:180
172159199200,19cyclictest0-21swapper/019:15:450
172159199200,19cyclictest0-21swapper/000:33:410
172159199200,19cyclictest0-21swapper/000:28:130
172159199200,19cyclictest0-21swapper/000:22:110
172159199200,18cyclictest3056-21PK-Backend22:45:190
172160199192,13cyclictest1738174-21apt19:55:113
1721601991917,1cyclictest3056-21PK-Backend00:19:173
1721601991917,1cyclictest0-21swapper/323:01:433
1721601991914,3cyclictest1750320-21apt20:30:113
1721601991912,5cyclictest42-21ksoftirqd/320:00:213
172160199190,18cyclictest3056-21PK-Backend22:36:493
172160199190,18cyclictest1827646-21python300:05:223
172160199190,17cyclictest1754033-21basename20:40:093
172159899198,8cyclictest883-21dbus-daemon23:15:002
172159899192,13cyclictest1793524-21apt22:30:132
1721598991917,1cyclictest0-21swapper/223:08:372
1721598991917,1cyclictest0-21swapper/222:05:302
1721598991917,1cyclictest0-21swapper/220:11:482
1721598991917,1cyclictest0-21swapper/219:13:272
1721598991917,1cyclictest0-21swapper/200:15:222
1721598991911,6cyclictest1741303-21sshd20:04:132
172159899190,1cyclictest0-21swapper/222:19:292
1721594991917,1cyclictest3056-21PK-Backend22:23:401
1721594991917,1cyclictest3056-21PK-Backend21:06:011
1721594991917,1cyclictest0-21swapper/121:52:441
1721594991917,1cyclictest0-21swapper/120:15:311
1721594991913,5cyclictest1803744-21seq22:58:041
1721594991912,5cyclictest1-21systemd20:05:001
172159499190,1cyclictest3056-21PK-Backend19:53:401
172159499190,1cyclictest0-21swapper/122:50:371
172159499190,1cyclictest0-21swapper/121:48:251
172159499190,1cyclictest0-21swapper/120:50:321
172159499190,18cyclictest3056-21PK-Backend19:19:481
172159199196,3cyclictest0-21swapper/021:45:160
1721591991917,1cyclictest0-21swapper/021:43:480
1721591991917,1cyclictest0-21swapper/021:06:430
1721591991916,2cyclictest3056-21PK-Backend22:55:320
172159199190,1cyclictest0-21swapper/022:35:380
172159199190,1cyclictest0-21swapper/022:29:160
172159199190,1cyclictest0-21swapper/021:50:230
172159199190,1cyclictest0-21swapper/021:35:090
172159199190,1cyclictest0-21swapper/000:38:340
172159199190,18cyclictest3056-21PK-Backend20:40:270
172159199190,18cyclictest3056-21PK-Backend00:06:440
172159199190,18cyclictest0-21swapper/023:17:100
172159199190,18cyclictest0-21swapper/022:01:110
172159199190,18cyclictest0-21swapper/021:31:170
172159199190,18cyclictest0-21swapper/020:47:030
172159199190,18cyclictest0-21swapper/020:28:590
172159199190,18cyclictest0-21swapper/019:33:070
172159199190,18cyclictest0-21swapper/019:24:300
172159199190,18cyclictest0-21swapper/000:12:110
172160199182,14cyclictest1758658-21taskset20:51:083
1721601991813,3cyclictest1832657-21idleruntime00:20:173
1721601991813,3cyclictest1731461-21irqstats19:35:163
1721601991813,3cyclictest1724033-21apt19:15:103
172160199181,14cyclictest1813765-21python323:25:213
172160199180,6cyclictest1793354-21apt-config22:30:003
172159899186,3cyclictest141rcu_preempt21:55:132
172159899182,12cyclictest1832207-21apt-get00:20:042
1721598991816,1cyclictest0-21swapper/223:40:342
1721598991816,1cyclictest0-21swapper/220:56:572
1721598991816,1cyclictest0-21swapper/220:36:362
1721598991815,2cyclictest3056-21PK-Backend22:10:032
1721598991811,5cyclictest1775808-21awk21:40:142
1721598991811,4cyclictest1762579-21latency_hist21:05:012
1721594991816,1cyclictest3056-21PK-Backend23:16:261
1721594991816,1cyclictest3056-21PK-Backend22:26:231
1721594991810,7cyclictest1730714-21cron19:35:001
1721591991816,1cyclictest3056-21PK-Backend21:10:280
1721591991816,1cyclictest0-21swapper/023:46:440
1721591991816,1cyclictest0-21swapper/023:20:060
1721591991816,1cyclictest0-21swapper/022:23:280
1721591991816,1cyclictest0-21swapper/022:16:470
1721591991816,1cyclictest0-21swapper/020:00:520
1721591991814,2cyclictest1816107-21sh23:34:590
1721591991812,4cyclictest1769639-21ls21:25:010
1721591991811,5cyclictest1794238-21ntpq22:30:190
1721601991714,2cyclictest3056-21PK-Backend20:45:173
172159899170,7cyclictest1747211-21sshd20:20:112
1721594991713,2cyclictest26-21ksoftirqd/122:50:001
172159199179,6cyclictest886-21NetworkManager19:10:180
1721591991713,3cyclictest1733424-21if_enp2s019:40:150
1721591991712,3cyclictest1769907-21cpu21:25:110
1721591991711,5cyclictest1784905-21expr22:05:140
1721591991711,4cyclictest1797157-21grep22:40:120
1721591991710,5cyclictest1825175-21wc00:00:170
1721601991614,1cyclictest3056-21PK-Backend22:09:453
1721601991614,1cyclictest3056-21PK-Backend21:54:593
1721601991614,1cyclictest3056-21PK-Backend00:27:353
1721601991614,1cyclictest0-21swapper/321:31:123
172159899163,8cyclictest1754045-21apt20:40:092
1721591991614,1cyclictest0-21swapper/020:58:320
1721591991610,4cyclictest1735682-21timerandwakeup19:45:200
1721601991513,1cyclictest3056-21PK-Backend00:36:473
172159499152,9cyclictest1804181-21apt23:00:101
1721594991512,2cyclictest1753511-21sh20:39:591
1721591991512,2cyclictest3056-21PK-Backend19:35:240
1721594991410,2cyclictest26-21ksoftirqd/120:05:121
172160199128,3cyclictest3056-21PK-Backend19:42:313
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional