You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-13 - 05:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed May 13, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
198158921160,3sleep00-21swapper/019:20:220
206010821140,3sleep30-21swapper/323:05:163
201692821120,3sleep00-21swapper/021:00:220
20269132990,3sleep00-21swapper/021:30:140
139099980,51rtkit-daemon0-21swapper/319:06:073
197495327947,22sleep20-21swapper/219:05:122
197629527638,31sleep00-21swapper/019:08:190
197626927141,24sleep10-21swapper/119:08:001
20055292590,2sleep00-21swapper/020:30:110
20780562480,3sleep00-21swapper/023:55:180
19968942480,3sleep30-21swapper/320:05:163
20708282430,4sleep13056-21PK-Backend23:35:181
1976571993016,12cyclictest883-21dbus-daemon22:45:000
1976571993010,17cyclictest436-20systemd-journal22:55:000
197657199298,19cyclictest2068526-21cat23:30:120
197657199291,25cyclictest2007840-21python320:35:190
197657199291,25cyclictest2007840-21python320:35:190
197657199289,17cyclictest2054134-21latency_hist22:50:010
197657199285,21cyclictest1-21systemd00:19:270
1976571992811,15cyclictest2057923-21tr23:00:140
20145632270,3sleep10-21swapper/120:55:141
197657199279,16cyclictest1987505-21grep19:40:010
197657199279,16cyclictest1987505-21grep19:40:000
197657199277,6cyclictest1-21systemd23:10:000
197657199277,5cyclictest2011048-21ls20:45:130
197657199276,7cyclictest2001878-21sort20:20:120
1976571992717,7cyclictest436-20systemd-journal23:50:000
1976571992716,9cyclictest2076153-21latency23:50:170
1976571992710,15cyclictest2084021-21latency_hist00:15:010
20654012260,3chrt0-21swapper/323:20:163
197657199269,15cyclictest2045377-21perf22:29:560
197657199269,15cyclictest1095-21runrttasks20:07:130
197657199267,17cyclictest1999344-21timerwakeupswit20:10:230
197657199261,4cyclictest0-21swapper/000:35:120
20376752250,4chrt2037673-21grep22:00:192
20110812250,4sleep30-21swapper/320:45:143
1976584992514,8cyclictest1997777-21apt-get20:10:002
197657199259,14cyclictest2156-21gnome-shell19:13:100
197657199259,14cyclictest2071682-21latency_hist23:40:000
197657199259,14cyclictest2043635-21latency_hist22:20:010
197657199259,14cyclictest1095-21runrttasks20:41:420
197657199256,3cyclictest2035254-21cpu21:55:120
197657199255,5cyclictest2033552-21cpuspeed21:50:120
197657199252,7cyclictest2061229-21apt23:10:140
197657199251,9cyclictest2091417-21idleruntime-cro00:35:000
1976571992515,9cyclictest2038414-21sh22:05:000
1976571992515,6cyclictest1095-21runrttasks23:15:220
1976571992513,11cyclictest2049074-21/usr/sbin/munin22:35:210
197657199250,22cyclictest2018280-21grep21:05:160
197658699241,3cyclictest2082934-21df_inode00:10:163
197657199245,3cyclictest2003704-21cpuspeed_turbos20:25:130
197657199241,7cyclictest436-20systemd-journal23:28:380
197657199241,7cyclictest2073609-21dump-pmu-power23:45:000
1976571992417,5cyclictest2015634-21perf21:00:000
197657199240,7cyclictest1994709-21tr20:00:110
1976579992321,1cyclictest0-21swapper/100:35:341
197657199237,5cyclictest2001366-21seq20:18:210
197657199237,2cyclictest20329812chrt21:47:060
197657199236,3cyclictest2056752-21grep22:55:190
197657199236,2cyclictest0-21swapper/000:25:050
197657199236,16cyclictest0-21swapper/019:54:580
197657199236,16cyclictest0-21swapper/000:03:080
197657199235,2cyclictest0-21swapper/021:29:060
197657199231,8cyclictest2040794-21df_inode22:10:130
197657199231,7cyclictest2066436-21idleruntime-cro23:25:010
197657199231,7cyclictest2021079-21sort21:15:120
197657199231,7cyclictest2013299-21python320:50:210
197657199231,5cyclictest2086646-21cpuspeed_turbos00:20:130
197657199230,21cyclictest1978686-21df19:15:020
197658499222,14cyclictest1095-21runrttasks21:40:102
1976584992220,1cyclictest0-21swapper/221:08:262
1976584992215,5cyclictest2043618-21cron22:20:002
1976579992220,1cyclictest3056-21PK-Backend22:57:461
1976579992220,1cyclictest0-21swapper/123:09:561
1976579992218,2cyclictest141rcu_preempt22:50:161
1976579992217,4cyclictest26-21ksoftirqd/100:19:381
1976579992213,7cyclictest2016270-21df_inode21:00:151
197657999220,20cyclictest3056-21PK-Backend21:27:151
197657199222,5cyclictest1985658-21munin-run19:35:000
1976571992220,1cyclictest0-21swapper/021:41:300
197657199221,7cyclictest2024212-21chrt21:24:040
197657199221,3cyclictest2040161-21/usr/sbin/munin22:10:010
20901412210,3sleep33056-21PK-Backend00:30:133
20547862210,2sleep33056-21PK-Backend22:50:173
1976586992118,2cyclictest3056-21PK-Backend23:10:213
1976586992118,2cyclictest3056-21PK-Backend20:22:243
1976586992115,4cyclictest2156-21gnome-shell22:24:093
1976586992113,6cyclictest1989301-21date19:45:003
197658699210,20cyclictest0-21swapper/321:25:163
197658499212,17cyclictest2070801-21grep23:35:182
1976584992119,1cyclictest0-21swapper/223:04:332
1976584992119,1cyclictest0-21swapper/222:25:322
1976584992119,1cyclictest0-21swapper/222:22:422
1976584992119,1cyclictest0-21swapper/221:17:532
1976584992119,1cyclictest0-21swapper/221:04:142
1976584992119,1cyclictest0-21swapper/220:36:362
1976584992119,1cyclictest0-21swapper/220:36:362
1976584992119,1cyclictest0-21swapper/220:25:272
1976584992119,1cyclictest0-21swapper/219:49:042
1976584992119,1cyclictest0-21swapper/219:44:462
1976584992119,1cyclictest0-21swapper/200:26:112
1976584992119,1cyclictest0-21swapper/200:01:092
1976584992113,6cyclictest1978628-21python319:15:022
197658499210,19cyclictest0-21swapper/220:02:292
1976579992119,1cyclictest3056-21PK-Backend23:48:431
1976579992119,1cyclictest3056-21PK-Backend23:00:371
1976579992119,1cyclictest3056-21PK-Backend22:11:501
1976579992119,1cyclictest3056-21PK-Backend21:07:351
1976579992119,1cyclictest3056-21PK-Backend19:21:021
1976579992119,1cyclictest0-21swapper/123:41:471
1976579992119,1cyclictest0-21swapper/121:57:231
1976579992119,1cyclictest0-21swapper/100:33:311
1976579992118,2cyclictest3056-21PK-Backend19:51:381
1976579992118,2cyclictest3056-21PK-Backend19:45:151
197657999210,20cyclictest0-21swapper/123:54:511
197657999210,1cyclictest141rcu_preempt20:00:001
197657999210,19cyclictest3056-21PK-Backend22:25:081
197657999210,19cyclictest3056-21PK-Backend22:15:141
197657999210,19cyclictest0-21swapper/122:43:331
197657199217,3cyclictest2019889-21grep21:10:210
197657199217,13cyclictest0-21swapper/019:44:200
197657199215,2cyclictest1993765-21wc19:55:210
197657199210,7cyclictest2081853-21sed00:05:230
197657199210,7cyclictest2043905-21grep22:20:120
197657199210,7cyclictest2029013-21awk21:35:180
197657199210,7cyclictest1982906-21if_err_enp2s019:25:160
197658699203,15cyclictest1981987-21sshd19:22:383
1976586992012,6cyclictest2064425-21apt-get23:20:003
197658699200,8cyclictest1985056-21taskset19:30:213
197658699200,19cyclictest3056-21PK-Backend00:15:303
197658699200,19cyclictest0-21swapper/322:29:183
197658699200,19cyclictest0-21swapper/319:38:373
197658699200,19cyclictest0-21swapper/319:38:363
197658699200,18cyclictest2079835-21/usr/sbin/munin00:00:193
1976584992018,1cyclictest0-21swapper/223:46:372
1976584992018,1cyclictest0-21swapper/223:16:222
1976584992018,1cyclictest0-21swapper/222:37:422
1976584992018,1cyclictest0-21swapper/221:11:032
1976584992018,1cyclictest0-21swapper/220:49:002
1976584992018,1cyclictest0-21swapper/219:51:202
1976584992018,1cyclictest0-21swapper/200:30:172
1976584992014,4cyclictest2156-21gnome-shell19:30:092
197658499200,1cyclictest0-21swapper/220:51:512
197658499200,19cyclictest0-21swapper/222:32:012
197658499200,19cyclictest0-21swapper/222:12:482
197658499200,19cyclictest0-21swapper/220:24:262
1976579992018,1cyclictest3056-21PK-Backend21:19:241
1976579992018,1cyclictest3056-21PK-Backend21:11:471
1976579992018,1cyclictest0-21swapper/123:14:501
1976579992018,1cyclictest0-21swapper/120:29:291
1976579992017,2cyclictest3056-21PK-Backend22:37:041
1976579992017,2cyclictest3056-21PK-Backend19:38:531
1976579992017,2cyclictest3056-21PK-Backend19:38:531
1976579992013,5cyclictest2008087-21sshd20:35:381
1976579992013,5cyclictest2008087-21sshd20:35:381
197657999200,19cyclictest3056-21PK-Backend23:32:481
197657999200,19cyclictest3056-21PK-Backend22:01:361
197657999200,19cyclictest3056-21PK-Backend21:22:011
197657999200,19cyclictest3056-21PK-Backend20:47:201
197657999200,19cyclictest0-21swapper/100:23:311
197657199207,3cyclictest2048335-21timerandwakeup22:30:200
197657199200,6cyclictest1990234-21gunzip19:45:190
139099200,14rtkit-daemon0-21swapper/222:43:202
139099200,14rtkit-daemon0-21swapper/220:42:482
20698942190,10sleep342-21ksoftirqd/323:35:003
20675512190,3sleep3411rcuc/323:25:213
197658699197,10cyclictest1977625-21python319:10:203
197658699192,15cyclictest2050891-21cpuspeed22:40:133
1976586991913,4cyclictest1983166-21ntp_kernel_pll_19:25:193
1976586991912,6cyclictest2050206-21sshd22:35:383
1976586991912,5cyclictest2019154-21sed21:10:013
197658699191,16cyclictest2081566-21ntp_states00:05:193
1976586991911,6cyclictest2074544-21grep23:45:173
1976586991911,6cyclictest2026956-21fw_packets21:30:163
197658699190,18cyclictest3056-21PK-Backend23:37:593
197658699190,18cyclictest0-21swapper/322:15:593
197658699190,16cyclictest1993216-21/usr/sbin/munin19:55:123
197658499194,9cyclictest2085686-21ld-2.23.so00:19:152
197658499192,5cyclictest2062089-21python323:10:232
197658499192,12cyclictest2059494-21apt23:05:112
1976584991917,1cyclictest3056-21PK-Backend20:31:572
1976584991917,1cyclictest0-21swapper/222:58:502
1976584991917,1cyclictest0-21swapper/222:53:102
1976584991917,1cyclictest0-21swapper/221:58:082
1976584991917,1cyclictest0-21swapper/221:48:422
1976584991917,1cyclictest0-21swapper/220:59:142
1976584991917,1cyclictest0-21swapper/219:28:292
1976584991917,1cyclictest0-21swapper/219:23:092
1976584991914,3cyclictest978-21snmpd21:39:382
1976584991913,4cyclictest2156-21gnome-shell00:13:092
1976584991911,6cyclictest2027636-21awk21:30:232
197658499190,1cyclictest0-21swapper/223:50:432
197658499190,1cyclictest0-21swapper/223:42:312
197658499190,1cyclictest0-21swapper/222:07:342
197658499190,1cyclictest0-21swapper/220:11:482
197658499190,18cyclictest0-21swapper/222:48:172
197658499190,18cyclictest0-21swapper/221:20:312
1976579991917,1cyclictest3056-21PK-Backend21:43:581
1976579991917,1cyclictest3056-21PK-Backend19:17:371
1976579991917,1cyclictest0-21swapper/123:26:531
1976579991917,1cyclictest0-21swapper/122:21:301
1976579991917,1cyclictest0-21swapper/120:41:391
1976579991917,1cyclictest0-21swapper/120:11:391
197657999190,1cyclictest0-21swapper/120:31:181
197657999190,1cyclictest0-21swapper/120:03:331
197657999190,18cyclictest0-21swapper/120:21:251
20308772180,3sleep30-21swapper/321:40:193
1976586991814,3cyclictest2156-21gnome-shell21:45:093
1976586991813,3cyclictest2012919-21kernelversion20:50:173
1976586991813,3cyclictest2006105-21python320:30:193
197658699180,1cyclictest0-21swapper/319:19:163
197658699180,16cyclictest2047364-21apt22:30:103
1976584991816,1cyclictest0-21swapper/223:33:322
1976584991816,1cyclictest0-21swapper/220:18:112
1976584991816,1cyclictest0-21swapper/200:21:042
1976584991811,5cyclictest1993855-21cat19:55:212
1976584991811,5cyclictest1977680-21ls19:10:212
1976584991810,6cyclictest2025689-21python321:25:222
1976579991816,1cyclictest3056-21PK-Backend19:34:341
1976579991816,1cyclictest0-21swapper/123:17:541
1976579991816,1cyclictest0-21swapper/100:26:351
1976579991813,3cyclictest1997876-21apt-key20:10:011
1976579991812,5cyclictest2053606-21ls22:45:221
1976579991811,6cyclictest2034525-21sed21:50:211
1976579991811,6cyclictest1-21systemd00:00:001
197657999181,13cyclictest2000018-21/usr/sbin/munin20:15:231
139099180,3rtkit-daemon3056-21PK-Backend20:15:363
1976586991715,1cyclictest3056-21PK-Backend00:20:443
1976586991713,3cyclictest2019374-21apt21:10:143
1976586991712,3cyclictest2023866-21timerwakeupswit21:20:203
1976586991712,3cyclictest2015638-21turbostat21:00:003
1976586991710,5cyclictest2038855-21grep22:05:123
1976586991710,5cyclictest2029355-21cat21:35:213
197658699170,3cyclictest2056772-21ntp_states22:55:203
197658699170,15cyclictest2058130-21df_inode23:00:153
197658699170,15cyclictest1989387-21apt19:45:103
197658499173,9cyclictest2080830-21apt00:05:122
197658499172,5cyclictest34-21ksoftirqd/200:35:222
1976584991715,1cyclictest0-21swapper/221:51:202
1976584991715,1cyclictest0-21swapper/219:38:232
1976584991715,1cyclictest0-21swapper/219:38:232
197657999171,8cyclictest26-21ksoftirqd/121:35:001
1976579991715,1cyclictest0-21swapper/123:24:501
20760142160,2sleep33056-21PK-Backend23:50:143
1976586991614,1cyclictest3056-21PK-Backend21:55:173
1976586991614,1cyclictest3056-21PK-Backend00:25:443
1976586991612,3cyclictest2017060-21chrt21:01:183
1976586991610,4cyclictest2021947-21gzip21:15:213
197658699160,15cyclictest3056-21PK-Backend23:59:013
197658699160,15cyclictest3056-21PK-Backend20:27:443
197658499168,7cyclictest0-21swapper/200:00:012
1976584991614,1cyclictest0-21swapper/223:20:282
1976579991614,1cyclictest3056-21PK-Backend22:31:371
1976579991614,1cyclictest0-21swapper/122:08:251
1976579991614,1cyclictest0-21swapper/121:49:461
197658699158,5cyclictest436-20systemd-journal22:15:003
197658699158,5cyclictest2072558-21basename23:40:173
1976586991510,4cyclictest42-21ksoftirqd/319:50:113
197658699150,1cyclictest3056-21PK-Backend22:47:023
197658699150,14cyclictest0-21swapper/322:02:343
197657999158,5cyclictest918-21rs:main1
1976579991513,1cyclictest3056-21PK-Backend19:29:131
1976579991513,1cyclictest3056-21PK-Backend00:05:191
1976586991412,1cyclictest3056-21PK-Backend20:11:493
197658699140,1cyclictest3056-21PK-Backend21:52:203
197658699140,1cyclictest3056-21PK-Backend20:38:393
197658699140,1cyclictest3056-21PK-Backend20:38:393
197658699140,1cyclictest3056-21PK-Backend00:35:453
197658499149,3cyclictest2066817-21tr23:25:132
1976579991410,3cyclictest3056-21PK-Backend21:37:251
1976586991311,1cyclictest3056-21PK-Backend20:02:293
197658699130,1cyclictest3056-21PK-Backend20:44:143
197657999138,4cyclictest3056-21PK-Backend19:43:121
197657999138,4cyclictest3056-21PK-Backend00:01:361
197657999131,1cyclictest141rcu_preempt19:15:001
197657999124,2cyclictest141rcu_preempt20:50:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional