You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-30 - 14:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Thu Apr 30, 2026 00:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
309707321120,4sleep20-21swapper/222:30:192
309198121080,3sleep20-21swapper/222:10:142
30893352990,4sleep03056-21PK-Backend22:00:140
139099930,57rtkit-daemon0-21swapper/319:06:173
302893926545,14sleep00-21swapper/019:06:550
302821726543,14sleep10-21swapper/119:05:201
302911326444,14sleep20-21swapper/219:09:112
30555422470,3sleep00-21swapper/020:20:210
30863252410,3sleep1251rcuc/121:50:011
30900382340,3sleep10-21swapper/122:00:231
139099330,2rtkit-daemon0-21swapper/221:04:172
30636612320,3sleep13056-21PK-Backend20:45:121
31014852310,4sleep00-21swapper/022:50:140
3029317993014,11cyclictest883-21dbus-daemon22:35:263
3029317992716,6cyclictest3118248-21apt00:00:113
302931599278,17cyclictest3124603-21date00:25:012
302931599269,15cyclictest1095-21runrttasks20:42:352
302931599268,16cyclictest3108624-21expr23:20:122
3029315992611,13cyclictest1009-21lldpd00:01:112
302931599259,14cyclictest436-20systemd-journal23:05:002
302931599258,15cyclictest3047835-21cstates20:00:132
302931599257,16cyclictest3031747-21cpu19:15:122
302931599255,6cyclictest3033761-21expr19:20:112
302931599255,5cyclictest3070580-21cat21:05:142
302931599250,3cyclictest436-20systemd-journal21:57:052
3029311992521,3cyclictest26-21ksoftirqd/122:55:281
3029311992517,4cyclictest26-21ksoftirqd/122:50:271
302931599248,14cyclictest3089871-21switchtime22:00:232
302931599247,15cyclictest978-21snmpd20:14:212
302931599240,6cyclictest3127467-21awk00:40:002
302931199247,4cyclictest26-21ksoftirqd/121:40:011
3029311992412,9cyclictest883-21dbus-daemon22:45:261
3029310992419,3cyclictest3057814-21apt-get20:30:050
30442932230,2sleep10-21swapper/119:50:111
3029317992321,1cyclictest3056-21PK-Backend21:23:043
3029317992320,2cyclictest3056-21PK-Backend21:27:383
302931799230,21cyclictest3056-21PK-Backend23:46:073
302931599237,14cyclictest978-21snmpd21:54:302
302931599237,14cyclictest978-21snmpd21:54:302
302931599237,14cyclictest3116919-21cat23:50:212
302931599237,14cyclictest3050077-21ntp_kernel_pll_20:05:182
302931599236,15cyclictest3067630-21missed_timers20:55:172
302931599232,5cyclictest3118204-21cut00:00:012
302931599231,9cyclictest436-20systemd-journal23:30:052
3029315992310,3cyclictest30442942expr19:50:112
3029311992318,3cyclictest26-21ksoftirqd/100:00:001
3029311992315,5cyclictest1-21systemd20:30:241
3029311992313,4cyclictest26-21ksoftirqd/123:16:051
3029310992321,1cyclictest0-21swapper/023:53:040
3029310992315,6cyclictest3111380-21cut23:30:170
31084262220,4sleep03056-21PK-Backend23:20:020
3029317992220,1cyclictest0-21swapper/320:10:033
3029317992212,7cyclictest3090436-21cron22:05:003
302931799220,20cyclictest3056-21PK-Backend20:15:243
302931599229,12cyclictest3055516-21switchtime20:20:212
302931599229,11cyclictest0-21swapper/223:39:092
302931599225,15cyclictest2156-21gnome-shell19:12:092
302931599224,7cyclictest3105997-21cpuspeed_turbos23:10:122
3029315992220,1cyclictest0-21swapper/223:19:502
3029315992217,4cyclictest0-21swapper/220:29:092
3029315992217,4cyclictest0-21swapper/200:09:182
302931599221,19cyclictest3046766-21grep19:55:202
3029315992210,11cyclictest3065255-21/usr/sbin/munin20:50:182
302931599220,21cyclictest0-21swapper/222:40:162
302931199223,2cyclictest141rcu_preempt22:10:001
3029311992220,1cyclictest0-21swapper/123:46:051
3029311992220,1cyclictest0-21swapper/123:12:531
3029311992220,1cyclictest0-21swapper/120:25:231
3029311992220,1cyclictest0-21swapper/119:11:081
3029311992220,1cyclictest0-21swapper/100:15:391
3029311992216,4cyclictest26-21ksoftirqd/119:35:221
3029311992211,8cyclictest883-21dbus-daemon20:20:011
3029310992220,1cyclictest0-21swapper/021:39:470
3029310992220,1cyclictest0-21swapper/020:54:590
3029310992220,1cyclictest0-21swapper/019:33:540
3029310992220,1cyclictest0-21swapper/019:20:510
3029310992216,4cyclictest3076662-21sed21:20:220
3029310992214,6cyclictest436-20systemd-journal00:00:000
3029310992213,7cyclictest436-20systemd-journal21:15:000
139099220,4rtkit-daemon1389-21rtkit-daemon20:18:462
302931799217,13cyclictest1-21systemd21:05:003
302931799212,14cyclictest3033478-21apt19:20:113
3029317992119,1cyclictest3056-21PK-Backend22:24:223
3029317992119,1cyclictest3056-21PK-Backend19:39:143
3029317992119,1cyclictest3056-21PK-Backend00:37:173
3029317992119,1cyclictest3056-21PK-Backend00:13:593
3029317992119,1cyclictest0-21swapper/322:55:173
3029317992119,1cyclictest0-21swapper/321:10:133
3029317992119,1cyclictest0-21swapper/320:56:203
3029317992119,1cyclictest0-21swapper/319:14:413
3029317992118,1cyclictest3056-21PK-Backend20:35:253
3029317992113,6cyclictest3066576-21sshd20:53:043
3029317992111,7cyclictest436-20systemd-journal20:50:003
3029317992111,7cyclictest3038411-21munin-plugin-st19:35:003
302931799210,20cyclictest3056-21PK-Backend19:40:023
302931799210,19cyclictest3056-21PK-Backend23:10:383
302931799210,19cyclictest3056-21PK-Backend21:51:573
302931799210,19cyclictest3056-21PK-Backend21:51:573
302931799210,19cyclictest3056-21PK-Backend00:22:173
302931599216,13cyclictest2156-21gnome-shell23:05:092
302931599215,5cyclictest3125730-21entropy00:30:142
302931599215,2cyclictest30907122chrt22:05:412
302931599214,4cyclictest3081560-21netstat21:35:182
302931599214,15cyclictest0-21swapper/220:45:122
3029315992119,1cyclictest0-21swapper/222:47:052
3029315992119,1cyclictest0-21swapper/219:42:182
302931599211,8cyclictest3042733-21memory19:45:162
3029315992113,6cyclictest3036665-21cron19:30:002
302931599210,9cyclictest3058704-21df_inode20:30:132
302931599210,20cyclictest0-21swapper/223:29:102
302931599210,20cyclictest0-21swapper/200:10:192
302931599210,19cyclictest3077159-21awk21:25:002
302931199214,9cyclictest26-21ksoftirqd/123:55:001
3029311992119,1cyclictest3056-21PK-Backend20:05:301
3029311992119,1cyclictest3056-21PK-Backend20:03:531
3029311992119,1cyclictest0-21swapper/123:03:401
3029311992119,1cyclictest0-21swapper/122:42:381
3029311992119,1cyclictest0-21swapper/122:19:261
3029311992119,1cyclictest0-21swapper/121:52:361
3029311992119,1cyclictest0-21swapper/121:52:361
3029311992119,1cyclictest0-21swapper/121:23:361
3029311992119,1cyclictest0-21swapper/120:50:441
3029311992119,1cyclictest0-21swapper/120:37:531
3029311992119,1cyclictest0-21swapper/119:31:011
3029311992119,1cyclictest0-21swapper/100:37:231
3029311992119,1cyclictest0-21swapper/100:25:131
3029311992119,1cyclictest0-21swapper/100:11:471
3029311992118,2cyclictest3056-21PK-Backend00:00:391
3029311992115,4cyclictest26-21ksoftirqd/120:15:001
302931199210,20cyclictest0-21swapper/123:39:371
302931199210,19cyclictest3056-21PK-Backend21:08:081
3029310992119,1cyclictest3056-21PK-Backend19:45:180
3029310992119,1cyclictest0-21swapper/023:37:290
3029310992119,1cyclictest0-21swapper/023:26:200
3029310992119,1cyclictest0-21swapper/022:49:580
3029310992119,1cyclictest0-21swapper/022:35:440
3029310992119,1cyclictest0-21swapper/022:24:560
3029310992119,1cyclictest0-21swapper/022:12:460
3029310992119,1cyclictest0-21swapper/021:30:280
3029310992119,1cyclictest0-21swapper/020:14:440
3029310992119,1cyclictest0-21swapper/000:15:210
3029310992119,1cyclictest0-21swapper/000:02:570
3029310992118,2cyclictest0-21swapper/000:27:450
302931099210,20cyclictest0-21swapper/023:15:130
302931099210,20cyclictest0-21swapper/022:42:200
302931099210,20cyclictest0-21swapper/022:16:450
302931099210,20cyclictest0-21swapper/021:27:170
302931099210,20cyclictest0-21swapper/020:42:080
302931099210,20cyclictest0-21swapper/019:14:280
302931099210,20cyclictest0-21swapper/000:23:530
302931099210,1cyclictest0-21swapper/020:09:090
302931099210,19cyclictest3056-21PK-Backend20:55:470
139099210,17rtkit-daemon1788643-21eadedCompositor22:57:522
3029317992018,1cyclictest3056-21PK-Backend23:43:303
3029317992018,1cyclictest3056-21PK-Backend23:26:343
3029317992018,1cyclictest3056-21PK-Backend22:11:373
3029317992018,1cyclictest3056-21PK-Backend22:08:263
3029317992018,1cyclictest3056-21PK-Backend21:42:513
3029317992018,1cyclictest3056-21PK-Backend19:52:473
3029317992018,1cyclictest3056-21PK-Backend19:49:353
3029317992018,1cyclictest3056-21PK-Backend00:33:393
3029317992017,2cyclictest0-21swapper/321:07:503
3029317992012,6cyclictest3032740-21sed19:15:213
302931799200,19cyclictest0-21swapper/323:20:183
302931799200,19cyclictest0-21swapper/319:56:453
302931799200,19cyclictest0-21swapper/300:29:473
302931599206,12cyclictest978-21snmpd23:45:452
302931599206,12cyclictest0-21swapper/223:40:092
302931599205,2cyclictest30840242chrt21:44:362
302931599203,3cyclictest0-21swapper/219:33:172
302931599202,5cyclictest3074823-21grep21:15:202
3029315992018,1cyclictest0-21swapper/222:20:012
302931599201,4cyclictest2361-21gdbus22:35:262
302931599200,6cyclictest3124859-21taskset00:27:172
302931599200,6cyclictest3093715-21sleep22:20:002
302931599200,6cyclictest3039717-21grep19:35:212
302931599200,5cyclictest3072175-21sshd21:10:052
302931599200,1cyclictest0-21swapper/220:37:402
3029311992018,1cyclictest3056-21PK-Backend21:55:471
3029311992018,1cyclictest0-21swapper/123:29:291
3029311992018,1cyclictest0-21swapper/121:40:531
3029311992018,1cyclictest0-21swapper/121:15:521
3029311992018,1cyclictest0-21swapper/121:10:331
3029311992018,1cyclictest0-21swapper/119:56:091
302931199200,1cyclictest0-21swapper/121:34:311
302931199200,19cyclictest0-21swapper/122:30:141
302931199200,19cyclictest0-21swapper/121:26:001
302931199200,19cyclictest0-21swapper/100:07:551
3029310992018,1cyclictest0-21swapper/023:40:200
3029310992018,1cyclictest0-21swapper/023:04:100
3029310992018,1cyclictest0-21swapper/022:06:240
3029310992018,1cyclictest0-21swapper/021:52:180
3029310992018,1cyclictest0-21swapper/021:52:180
3029310992018,1cyclictest0-21swapper/020:36:130
3029310992018,1cyclictest0-21swapper/019:39:290
3029310992018,1cyclictest0-21swapper/019:16:240
3029310992012,6cyclictest3073937-21apt21:15:090
302931099200,1cyclictest0-21swapper/000:32:250
302931099200,19cyclictest0-21swapper/023:14:100
302931099200,19cyclictest0-21swapper/022:59:100
302931099200,19cyclictest0-21swapper/021:04:530
302931099200,19cyclictest0-21swapper/020:18:440
30777992190,11sleep234-21ksoftirqd/221:25:152
3029317991917,1cyclictest3056-21PK-Backend23:51:363
3029317991917,1cyclictest3056-21PK-Backend22:43:073
3029317991916,2cyclictest3056-21PK-Backend23:06:403
302931799190,1cyclictest3056-21PK-Backend00:09:193
302931799190,18cyclictest3056-21PK-Backend21:47:243
302931799190,18cyclictest3056-21PK-Backend20:31:403
302931799190,18cyclictest0-21swapper/320:07:403
302931799190,18cyclictest0-21swapper/320:04:293
302931799190,17cyclictest3056-21PK-Backend23:57:363
302931799190,17cyclictest3056-21PK-Backend23:32:493
302931599193,3cyclictest0-21swapper/200:15:372
3029315991917,1cyclictest0-21swapper/222:54:422
302931599190,7cyclictest3086064-21apt-get21:50:002
302931599190,3cyclictest3079841-21/usr/sbin/munin21:30:172
302931599190,1cyclictest3056-21PK-Backend22:25:012
3029311991917,1cyclictest0-21swapper/123:07:531
3029311991917,1cyclictest0-21swapper/119:29:261
3029311991915,3cyclictest26-21ksoftirqd/123:35:011
3029311991911,6cyclictest3041896-21/usr/sbin/munin19:45:021
302931199190,1cyclictest3056-21PK-Backend19:17:501
302931199190,1cyclictest0-21swapper/123:24:501
302931199190,1cyclictest0-21swapper/122:39:141
302931199190,1cyclictest0-21swapper/100:21:211
3029310991917,1cyclictest0-21swapper/021:45:220
3029310991917,1cyclictest0-21swapper/020:00:380
3029310991917,1cyclictest0-21swapper/000:11:290
3029310991917,1cyclictest0-21swapper/000:07:370
302931099190,1cyclictest0-21swapper/023:47:360
302931099190,18cyclictest3056-21PK-Backend19:57:270
30798692180,4chrt3056-21PK-Backend21:30:183
3029317991816,1cyclictest3056-21PK-Backend20:23:093
3029317991816,1cyclictest3056-21PK-Backend00:16:493
302931799181,15cyclictest3088722-21munin-run22:00:003
302931799180,16cyclictest3057624-21chrt20:27:543
3029311991816,1cyclictest0-21swapper/123:42:271
3029311991816,1cyclictest0-21swapper/100:34:321
3029311991815,2cyclictest3056-21PK-Backend21:04:231
302931199180,1cyclictest3056-21PK-Backend20:41:401
302931099188,8cyclictest13-21ksoftirqd/019:55:000
3029310991817,1cyclictest0-21swapper/021:41:230
3029310991816,1cyclictest0-21swapper/022:29:560
3029310991812,4cyclictest3041923-21wc19:45:010
3029310991810,6cyclictest3064620-21users20:45:220
302931099180,17cyclictest0-21swapper/021:06:280
3029317991715,1cyclictest3056-21PK-Backend21:18:313
3029317991712,4cyclictest3107996-21systemd-run23:20:003
3029317991711,5cyclictest3062679-21smartctl20:40:213
302931199177,4cyclictest26-21ksoftirqd/122:10:501
302931199172,10cyclictest3054570-21unattended-upgr20:20:061
3029311991715,1cyclictest3056-21PK-Backend20:55:171
3029311991715,1cyclictest0-21swapper/122:24:271
3029310991711,4cyclictest3057828-21basename20:30:000
3029317991614,1cyclictest3056-21PK-Backend23:39:043
3029317991614,1cyclictest3056-21PK-Backend19:27:453
302931799160,14cyclictest3100897-21grep22:50:003
3029311991612,3cyclictest3034421-21timerandwakeup19:20:201
302931199161,13cyclictest3096046-21cron22:30:001
302931199161,11cyclictest3042157-21apt19:45:111
3029310991614,1cyclictest0-21swapper/023:09:580
3029310991610,5cyclictest3096971-21needreboot22:30:180
3029310991610,4cyclictest2156-21gnome-shell00:36:090
3029317991510,4cyclictest3056-21PK-Backend23:04:333
302931799150,3cyclictest3097705-21sshd22:34:053
302931099158,6cyclictest3035428-21cat19:25:130
3029310991513,1cyclictest0-21swapper/021:55:290
3029317991412,1cyclictest3056-21PK-Backend22:15:363
3029317991412,1cyclictest3056-21PK-Backend21:38:183
3029317991411,2cyclictest3101455-21df_inode22:50:133
302931799118,2cyclictest3056-21PK-Backend22:25:513
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional