You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 06:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri May 15, 2026 00:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
313935421290,3sleep20-21swapper/220:20:012
313154721280,2sleep00-21swapper/019:55:210
311698721110,4sleep33056-21PK-Backend19:15:193
314187721070,4sleep33056-21PK-Backend20:25:153
321156121000,2sleep33056-21PK-Backend23:45:003
31748702970,3sleep10-21swapper/122:00:121
139099860,73rtkit-daemon0-21swapper/319:09:303
31333682770,2sleep33056-21PK-Backend20:00:213
311345626941,20sleep20-21swapper/219:07:032
311362326444,14sleep00-21swapper/019:09:100
311339326243,13sleep10-21swapper/119:06:141
31641202600,3sleep00-21swapper/021:30:130
31198942600,3sleep20-21swapper/219:25:112
31179312450,2sleep13056-21PK-Backend19:20:011
31185242390,2sleep30-21swapper/319:20:153
3113804993211,6cyclictest3136792-21sed20:10:210
32292502310,4sleep13056-21PK-Backend00:35:151
31382452300,8sleep1261ksoftirqd/120:15:141
311381399309,19cyclictest3145394-21ls20:35:183
139099300,4rtkit-daemon1389-21rtkit-daemon00:14:383
32228732290,3chrt0-21swapper/200:15:192
31296512290,3sleep23056-21PK-Backend19:50:212
311381399291,8cyclictest3137793-21taskset20:15:033
3113813992910,17cyclictest3160619-21/usr/sbin/munin21:20:173
3113804992911,5cyclictest3197266-21seq23:04:350
311381399288,18cyclictest3201602-21expr23:15:133
311381399288,17cyclictest3181826-21cpuspeed_turbos22:20:113
311381399287,5cyclictest3167676-21cat21:40:113
311381399281,9cyclictest3229079-21apt00:35:143
3113813992810,16cyclictest3157151-21/usr/sbin/munin21:10:213
311380499289,6cyclictest3154185-21awk21:00:160
311380499289,5cyclictest3117277-21sed19:15:210
139099280,4rtkit-daemon1389-21rtkit-daemon23:32:212
311381399279,16cyclictest3121183-21latency_hist19:30:013
311381399277,6cyclictest3137333-21munin-run20:15:013
311381399272,8cyclictest3123186-21apt19:35:103
311381399272,21cyclictest3183441-21apt22:25:093
311381399271,4cyclictest0-21swapper/320:20:103
3113804992712,6cyclictest3188376-21cron22:40:000
311381399268,16cyclictest3185378-21grep22:30:123
311381399267,5cyclictest3170053-21sh21:45:203
311381399267,5cyclictest3170053-21sh21:45:203
311381399265,5cyclictest3121455-21dpkg19:30:103
311381399260,2cyclictest31878962chrt22:35:233
311380499269,6cyclictest3192982-21python322:50:210
311380499269,4cyclictest3222426-21expr00:15:130
311380499269,4cyclictest3141470-21grep20:25:110
311380499268,4cyclictest3198569-21timerandwakeup23:05:220
311381399258,14cyclictest3178688-21/usr/sbin/munin22:10:163
311381399257,2cyclictest3225332-21cron00:25:003
311381399256,6cyclictest3163657-21sshd21:27:533
311381399255,5cyclictest3199115-21latency_hist23:10:013
311381399255,18cyclictest3056-21PK-Backend21:05:003
311381399251,7cyclictest3143043-21apt20:30:103
311381399251,3cyclictest0-21swapper/323:10:143
311381399250,23cyclictest3166023-21cpuspeed_turbos21:35:143
311381399250,1cyclictest3056-21PK-Backend19:40:553
311380499258,5cyclictest3216753-21sort00:00:010
311380499258,5cyclictest3208175-21date23:35:010
3113804992511,2cyclictest3213434-21/usr/sbin/munin23:50:210
311380499250,19cyclictest0-21swapper/023:37:340
311381399249,14cyclictest3056-21PK-Backend23:03:503
311381399246,3cyclictest3056-21PK-Backend22:47:543
311381399243,5cyclictest3180060-21ls22:15:123
311381399243,5cyclictest3148473-21expr20:45:123
311381399242,6cyclictest3174505-21dpkg22:00:103
311381399241,7cyclictest436-20systemd-journal00:35:013
311381399241,7cyclictest3207264-21ntp_states23:30:193
311381399241,7cyclictest3147373-21/usr/sbin/munin20:40:213
311381399240,8cyclictest3226151-21/usr/sbin/munin00:25:163
311381399240,8cyclictest3177228-21/usr/sbin/munin22:05:183
311381399240,22cyclictest3056-21PK-Backend00:00:153
311380499248,4cyclictest3156386-21timerandwakeup21:05:210
3113804992411,2cyclictest3149946-21/usr/sbin/munin20:50:200
311380499240,22cyclictest3134479-21diskstats20:05:150
311381399238,14cyclictest3056-21PK-Backend21:54:053
311381399230,8cyclictest3223140-21/usr/sbin/munin00:15:223
311381399230,8cyclictest3164387-21expr21:30:133
311381399230,7cyclictest3129662-21sort19:50:213
311380899230,21cyclictest3217416-21irqstats00:00:151
311380499235,3cyclictest0-21swapper/023:25:130
311380499233,6cyclictest3194116-21apt22:55:120
311380499231,8cyclictest3209995-21apt23:40:120
3113804992318,4cyclictest0-21swapper/020:33:550
3113804992316,5cyclictest883-21dbus-daemon20:25:000
31742862220,4sleep10-21swapper/122:00:001
311381399227,3cyclictest3056-21PK-Backend21:06:143
311381399220,7cyclictest3193194-21users22:50:233
311381399220,6cyclictest3174331-21cut22:00:003
311381399220,20cyclictest3056-21PK-Backend23:59:093
311381399220,20cyclictest3056-21PK-Backend19:45:433
3113811992220,1cyclictest0-21swapper/220:38:082
311381199220,20cyclictest31841412chrt22:25:162
3113808992220,1cyclictest0-21swapper/119:40:451
3113808992220,1cyclictest0-21swapper/100:12:321
3113808992219,2cyclictest3056-21PK-Backend00:17:391
3113808992213,6cyclictest3193578-21apt-get22:55:001
311380899220,21cyclictest3056-21PK-Backend21:32:251
311380899220,21cyclictest0-21swapper/119:36:121
311380899220,20cyclictest3056-21PK-Backend22:21:531
311380899220,20cyclictest0-21swapper/123:35:141
3113804992220,1cyclictest3056-21PK-Backend21:20:050
3113804992220,1cyclictest0-21swapper/021:25:250
311380499221,6cyclictest3220406-21apt00:10:120
3113804992213,7cyclictest436-20systemd-journal19:15:000
311380499221,10cyclictest3158040-21python321:10:220
311380499220,21cyclictest0-21swapper/022:13:310
139099220,18rtkit-daemon1788643-21eadedCompositor23:43:042
139099220,15rtkit-daemon3056-21PK-Backend23:21:193
311381399217,2cyclictest3056-21PK-Backend20:59:113
311381399215,3cyclictest3056-21PK-Backend23:53:263
311381399210,6cyclictest3160154-21seq21:17:253
311381399210,6cyclictest3151456-21munin-plugin-st20:55:003
311381399210,1cyclictest3056-21PK-Backend23:25:203
311381399210,1cyclictest3056-21PK-Backend19:59:593
311381399210,19cyclictest3056-21PK-Backend23:38:293
311381399210,19cyclictest3056-21PK-Backend22:57:053
3113811992119,1cyclictest3056-21PK-Backend23:46:342
3113811992119,1cyclictest0-21swapper/223:04:092
3113811992119,1cyclictest0-21swapper/222:16:032
3113811992119,1cyclictest0-21swapper/222:11:572
3113811992119,1cyclictest0-21swapper/221:24:532
3113811992119,1cyclictest0-21swapper/220:55:192
3113811992119,1cyclictest0-21swapper/220:30:522
3113811992119,1cyclictest0-21swapper/220:24:102
3113811992119,1cyclictest0-21swapper/200:00:542
3113811992118,2cyclictest3056-21PK-Backend21:10:192
311381199210,20cyclictest0-21swapper/223:56:482
311381199210,20cyclictest0-21swapper/223:28:092
311381199210,20cyclictest0-21swapper/223:08:152
311381199210,20cyclictest0-21swapper/222:46:452
311381199210,20cyclictest0-21swapper/222:38:342
311381199210,20cyclictest0-21swapper/222:21:102
311381199210,20cyclictest0-21swapper/222:04:472
311381199210,20cyclictest0-21swapper/221:35:272
311381199210,20cyclictest0-21swapper/220:12:132
311381199210,20cyclictest0-21swapper/200:09:392
311380899217,12cyclictest3128407-21unin-run19:50:001
311380899212,13cyclictest978-21snmpd21:05:111
3113808992119,1cyclictest3056-21PK-Backend21:27:381
3113808992119,1cyclictest3056-21PK-Backend20:39:531
3113808992119,1cyclictest0-21swapper/123:57:111
3113808992119,1cyclictest0-21swapper/123:48:451
3113808992119,1cyclictest0-21swapper/122:55:391
3113808992118,2cyclictest3056-21PK-Backend22:05:581
3113808992118,2cyclictest0-21swapper/121:04:471
3113808992113,6cyclictest3227963-21ls00:30:181
3113808992110,8cyclictest883-21dbus-daemon21:55:001
311380899210,2cyclictest3219418-21ntp_states00:05:221
311380899210,20cyclictest0-21swapper/119:55:181
311380899210,20cyclictest0-21swapper/119:50:181
311380899210,19cyclictest3056-21PK-Backend20:40:401
311380899210,19cyclictest3056-21PK-Backend00:20:431
311380499215,3cyclictest0-21swapper/020:55:160
3113804992119,1cyclictest3056-21PK-Backend21:17:070
3113804992119,1cyclictest3056-21PK-Backend20:40:370
3113804992119,1cyclictest0-21swapper/023:14:350
3113804992119,1cyclictest0-21swapper/022:32:580
3113804992119,1cyclictest0-21swapper/022:25:480
3113804992119,1cyclictest0-21swapper/022:18:380
3113804992119,1cyclictest0-21swapper/022:03:310
3113804992119,1cyclictest0-21swapper/021:40:400
3113804992119,1cyclictest0-21swapper/020:17:120
3113804992119,1cyclictest0-21swapper/019:25:480
3113804992119,1cyclictest0-21swapper/000:29:250
311380499210,20cyclictest0-21swapper/022:21:430
311380499210,20cyclictest0-21swapper/021:51:280
311380499210,20cyclictest0-21swapper/020:39:030
311380499210,20cyclictest0-21swapper/020:03:260
311380499210,20cyclictest0-21swapper/000:23:160
3113813992018,1cyclictest3056-21PK-Backend22:41:443
3113813992017,2cyclictest3056-21PK-Backend20:06:303
311381399200,6cyclictest3213191-21apt23:50:003
311381399200,4cyclictest3115505-21turbostat19:15:003
311381399200,19cyclictest3056-21PK-Backend00:05:193
3113811992018,1cyclictest0-21swapper/220:08:342
3113811992018,1cyclictest0-21swapper/219:22:122
311381199200,1cyclictest0-21swapper/222:52:542
311381199200,19cyclictest0-21swapper/223:15:382
311381199200,19cyclictest0-21swapper/222:31:242
311381199200,19cyclictest0-21swapper/221:52:442
311381199200,19cyclictest0-21swapper/221:27:162
311381199200,19cyclictest0-21swapper/220:53:292
311381199200,19cyclictest0-21swapper/220:28:422
311381199200,19cyclictest0-21swapper/220:04:562
311381199200,19cyclictest0-21swapper/219:30:492
311381199200,19cyclictest0-21swapper/219:11:572
3113808992018,1cyclictest3056-21PK-Backend21:46:581
3113808992018,1cyclictest3056-21PK-Backend21:46:581
3113808992018,1cyclictest3056-21PK-Backend21:40:161
3113808992018,1cyclictest3056-21PK-Backend20:12:011
3113808992018,1cyclictest3056-21PK-Backend19:23:141
3113808992018,1cyclictest0-21swapper/120:01:211
3113808992018,1cyclictest0-21swapper/119:13:001
3113808992017,2cyclictest3056-21PK-Backend19:33:561
3113808992012,6cyclictest3159654-21ntp_states21:15:221
311380899200,1cyclictest0-21swapper/123:15:191
311380899200,1cyclictest0-21swapper/120:25:331
311380899200,19cyclictest0-21swapper/123:28:521
311380899200,19cyclictest0-21swapper/122:44:101
311380899200,19cyclictest0-21swapper/122:38:021
311380499202,6cyclictest13-21ksoftirqd/019:22:080
3113804992018,1cyclictest0-21swapper/023:21:590
3113804992018,1cyclictest0-21swapper/022:05:340
3113804992018,1cyclictest0-21swapper/020:49:220
3113804992012,6cyclictest3212668-21seq23:45:210
311380499200,19cyclictest0-21swapper/022:41:090
311380499200,19cyclictest0-21swapper/021:45:330
311380499200,19cyclictest0-21swapper/021:45:330
311380499200,19cyclictest0-21swapper/021:38:370
311380499200,19cyclictest0-21swapper/019:54:340
311380499200,19cyclictest0-21swapper/019:38:590
311380499200,19cyclictest0-21swapper/019:30:070
139099200,13rtkit-daemon0-21swapper/219:16:112
3113811991917,1cyclictest0-21swapper/222:08:532
3113811991917,1cyclictest0-21swapper/221:06:272
3113811991917,1cyclictest0-21swapper/200:30:072
3113811991913,4cyclictest2156-21gnome-shell23:14:092
311381199190,1cyclictest0-21swapper/222:42:402
311381199190,1cyclictest0-21swapper/221:19:332
311381199190,1cyclictest0-21swapper/219:38:402
311381199190,18cyclictest0-21swapper/219:46:582
3113808991917,1cyclictest3056-21PK-Backend21:39:141
3113808991917,1cyclictest3056-21PK-Backend21:22:181
3113808991917,1cyclictest0-21swapper/123:20:401
3113808991917,1cyclictest0-21swapper/123:10:131
3113808991917,1cyclictest0-21swapper/120:48:381
311380899190,1cyclictest3056-21PK-Backend22:32:551
311380899190,1cyclictest3056-21PK-Backend22:19:501
311380899190,1cyclictest3056-21PK-Backend20:21:551
311380899190,18cyclictest0-21swapper/120:06:341
311380899190,16cyclictest436-20systemd-journal23:05:001
3113804991917,1cyclictest0-21swapper/021:55:330
3113804991912,5cyclictest3126864-21apt-config19:45:010
311380499190,1cyclictest0-21swapper/019:40:150
311380499190,18cyclictest3190645-21expr22:45:140
311380499190,18cyclictest0-21swapper/000:32:290
311381199189,7cyclictest3229939-21python300:35:232
3113811991816,1cyclictest0-21swapper/219:41:102
3113811991816,1cyclictest0-21swapper/200:26:022
3113811991813,4cyclictest3167715-21grep21:40:112
3113811991812,5cyclictest3147385-21awk20:40:202
3113811991812,4cyclictest1095-21runrttasks21:45:132
3113811991812,4cyclictest1095-21runrttasks21:45:132
3113808991816,1cyclictest3056-21PK-Backend22:25:591
3113808991811,5cyclictest3120366-21tr19:25:161
3113808991810,6cyclictest3150709-21seq20:50:181
311380899180,1cyclictest0-21swapper/122:48:161
3113804991813,3cyclictest3217585-21grep00:00:160
311381199179,6cyclictest3224036-21expr00:20:122
311381199179,6cyclictest3148973-21ntpq20:45:182
3113811991715,1cyclictest0-21swapper/221:55:482
3113811991715,1cyclictest0-21swapper/221:03:372
3113811991714,2cyclictest3056-21PK-Backend00:12:432
311380899179,7cyclictest3211453-21perf23:45:001
3113808991715,1cyclictest3056-21PK-Backend22:12:541
3113808991711,4cyclictest2156-21gnome-shell21:11:091
3113808991710,5cyclictest3206499-21seq23:30:101
3113804991712,3cyclictest3219419-21sh00:05:220
3113804991710,5cyclictest3201610-21expr23:15:130
311381199163,12cyclictest1095-21runrttasks21:32:212
3113811991614,1cyclictest0-21swapper/223:20:592
3113811991611,3cyclictest3195618-21grep23:00:002
3113811991610,4cyclictest3209286-21sed23:35:232
3113808991614,1cyclictest0-21swapper/120:56:301
3113808991614,1cyclictest0-21swapper/100:25:501
3113808991612,3cyclictest3199047-21sh23:10:001
3113808991611,3cyclictest3143530-21kernelversion20:30:161
3113811991513,1cyclictest0-21swapper/223:50:402
3113811991513,1cyclictest0-21swapper/219:55:432
311380899156,7cyclictest3213919-21cat23:50:161
311380499149,3cyclictest1095-21runrttasks00:39:480
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional