You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-04 - 21:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Aug 04, 2026 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
139099960,52rtkit-daemon0-21swapper/207:08:432
315819427740,30sleep00-21swapper/007:05:350
315822526849,13sleep10-21swapper/107:06:001
315706426342,14sleep30-21swapper/307:05:113
32514992430,3sleep30-21swapper/311:30:123
315867699287,19cyclictest436-20systemd-journal09:40:013
139099280,19rtkit-daemon0-21swapper/211:40:152
32184442270,3sleep20-21swapper/209:55:232
315867699278,17cyclictest3258476-21tr11:50:133
315867699277,18cyclictest436-20systemd-journal10:44:013
315867699277,18cyclictest3240374-21date11:00:013
315867699272,22cyclictest3246176-21apt-get11:15:033
315867699270,7cyclictest3262284-21awk12:00:163
139099270,18rtkit-daemon3184713-21/usr/sbin/munin08:20:173
139099270,18rtkit-daemon0-21swapper/211:55:222
315867699269,15cyclictest436-20systemd-journal11:24:553
315867699269,15cyclictest436-20systemd-journal11:24:553
315867699269,15cyclictest3188716-21cat08:30:223
315867699268,16cyclictest3264425-21switchtime12:05:223
315867699268,15cyclictest3172022-21cpuspeed_turbos07:45:143
315867699266,18cyclictest978-21snmpd09:15:193
3158676992613,11cyclictest3190188-21ntp_states08:35:203
315867699260,24cyclictest3208266-21munin-run09:30:003
315867699260,24cyclictest3208266-21munin-run09:30:003
315867199260,9cyclictest32701572switchtime12:20:220
315867699256,17cyclictest3244806-21ls11:10:163
315867699256,17cyclictest3163868-21sed07:20:223
315867699252,20cyclictest0-21swapper/307:40:143
315867699251,22cyclictest3274651-21cpuspeed_turbos12:35:143
315867699251,22cyclictest3226762-21needreboot10:20:183
315867699250,3cyclictest3237124-21df10:50:133
315867699250,23cyclictest3056-21PK-Backend10:16:073
315867699250,22cyclictest31739372sleep307:50:133
139099250,17rtkit-daemon0-21swapper/211:20:212
139099250,17rtkit-daemon0-21swapper/211:20:212
315867699248,14cyclictest3222847-21ls10:10:133
315867699247,15cyclictest436-20systemd-journal07:40:013
315867699247,15cyclictest3229975-21idleruntime10:30:153
315867699247,15cyclictest3182696-21cstates08:15:133
315867699245,17cyclictest3216456-21ntpq09:50:203
315867699245,17cyclictest3181255-21kernelversion08:10:183
315867699243,19cyclictest3240805-21expr11:00:133
315867699242,19cyclictest3268736-21apt-get12:20:033
315867699240,23cyclictest3056-21PK-Backend07:58:383
315867699240,23cyclictest0-21swapper/308:07:383
315867699240,22cyclictest3212071-21readlink09:40:013
315867599240,22cyclictest3244217-21apt11:10:102
32093062230,4sleep13056-21PK-Backend09:30:201
315867699238,14cyclictest3164564-21acpi07:25:123
315867699238,13cyclictest3220639-21latency_hist10:05:013
315867699237,14cyclictest42-21ksoftirqd/309:45:123
315867699237,14cyclictest3231761-21/usr/sbin/munin10:35:143
315867699237,14cyclictest0-21swapper/307:15:023
315867699236,15cyclictest3208481-21/usr/sbin/munin09:30:213
315867699234,17cyclictest3268919-21latency_hist12:20:003
315867699233,17cyclictest3250419-21ntp_states11:25:213
315867699232,3cyclictest0-21swapper/312:25:133
315867699231,20cyclictest3206330-21sshd09:23:013
315867699230,3cyclictest3273608-21sensors_temp12:30:223
315867699230,3cyclictest3265511-21seq12:10:073
3158675992317,4cyclictest957-21gdbus07:37:282
3158671992316,6cyclictest3180211-21cron08:10:000
315867699228,12cyclictest3159390-21latency07:10:173
315867699226,14cyclictest3166310-21grep07:30:123
315867699226,14cyclictest0-21swapper/311:45:173
315867699226,14cyclictest0-21swapper/310:26:523
315867699226,14cyclictest0-21swapper/310:26:523
315867699224,16cyclictest3261291-21sshd11:57:013
315867699223,17cyclictest0-21swapper/311:35:053
315867699222,18cyclictest0-21swapper/310:45:113
315867699222,17cyclictest0-21swapper/309:55:133
3158676992211,9cyclictest3242414-21tr11:05:143
315867699220,5cyclictest3220635-20cron10:05:003
315867699220,20cyclictest3255259-21diskstats11:40:163
315867699220,20cyclictest3056-21PK-Backend08:57:283
3158675992214,6cyclictest3177396-21df_inode08:00:142
3158675992213,7cyclictest3270009-21ntp_states12:20:202
3158674992220,1cyclictest0-21swapper/109:50:321
3158674992219,2cyclictest3056-21PK-Backend11:28:131
3158674992213,7cyclictest3233494-21/usr/sbin/munin10:40:151
3158674992212,8cyclictest3204774-21cron09:19:591
315867499220,20cyclictest3242075-21wc11:05:011
315867499220,20cyclictest3056-21PK-Backend11:20:221
315867499220,20cyclictest3056-21PK-Backend11:20:221
315867499220,20cyclictest3056-21PK-Backend08:32:111
3158671992219,2cyclictest3056-21PK-Backend09:22:250
315867699216,14cyclictest3191388-21cstates08:40:143
315867699212,18cyclictest3198841-21ntp_offset09:00:193
3158676992116,4cyclictest0-21swapper/309:14:443
3158675992119,1cyclictest0-21swapper/207:21:492
3158675992118,2cyclictest0-21swapper/212:15:122
3158675992112,7cyclictest436-20systemd-journal12:10:002
315867599210,20cyclictest0-21swapper/210:24:132
315867599210,20cyclictest0-21swapper/207:56:302
3158674992119,1cyclictest0-21swapper/110:48:321
3158674992119,1cyclictest0-21swapper/109:58:231
3158674992119,1cyclictest0-21swapper/109:05:571
3158674992119,1cyclictest0-21swapper/108:53:271
3158674992119,1cyclictest0-21swapper/108:46:171
3158674992119,1cyclictest0-21swapper/108:41:511
3158674992119,1cyclictest0-21swapper/108:06:081
3158674992119,1cyclictest0-21swapper/107:34:181
3158674992118,2cyclictest3056-21PK-Backend11:53:551
315867499210,20cyclictest0-21swapper/107:11:131
315867499210,19cyclictest3056-21PK-Backend10:25:331
315867499210,19cyclictest3056-21PK-Backend10:25:331
315867499210,19cyclictest3056-21PK-Backend08:11:501
3158671992119,1cyclictest0-21swapper/012:15:370
3158671992119,1cyclictest0-21swapper/011:33:050
3158671992119,1cyclictest0-21swapper/011:21:150
3158671992119,1cyclictest0-21swapper/011:21:150
3158671992119,1cyclictest0-21swapper/011:08:240
3158671992119,1cyclictest0-21swapper/010:32:350
3158671992119,1cyclictest0-21swapper/009:26:040
3158671992119,1cyclictest0-21swapper/009:26:040
3158671992119,1cyclictest0-21swapper/008:46:430
3158671992119,1cyclictest0-21swapper/008:37:380
3158671992119,1cyclictest0-21swapper/008:32:570
3158671992119,1cyclictest0-21swapper/008:29:390
3158671992118,2cyclictest3056-21PK-Backend10:09:160
3158671992118,2cyclictest0-21swapper/010:17:210
3158671992118,1cyclictest0-21swapper/008:00:410
3158671992114,5cyclictest883-21dbus-daemon10:55:000
3158671992114,5cyclictest13-21ksoftirqd/011:15:010
315867199210,9cyclictest32039362missed_timers09:15:180
315867199210,20cyclictest0-21swapper/011:00:200
315867199210,20cyclictest0-21swapper/009:52:130
315867199210,20cyclictest0-21swapper/008:19:250
139099210,15rtkit-daemon1211-21kerneloops10:17:492
315867699206,12cyclictest32010982chrt09:07:363
315867699204,14cyclictest0-21swapper/308:50:223
315867699202,15cyclictest3177893-21python308:00:223
3158675992018,1cyclictest0-21swapper/211:39:572
3158675992018,1cyclictest0-21swapper/209:51:212
3158675992018,1cyclictest0-21swapper/209:42:292
3158675992018,1cyclictest0-21swapper/209:21:002
3158675992018,1cyclictest0-21swapper/208:52:282
3158675992018,1cyclictest0-21swapper/208:12:462
3158675992017,2cyclictest0-21swapper/210:45:222
3158675992013,6cyclictest3239484-21awk10:55:192
3158675992013,5cyclictest436-20systemd-journal08:40:002
315867599200,19cyclictest0-21swapper/211:31:532
315867599200,19cyclictest0-21swapper/211:18:282
315867599200,19cyclictest0-21swapper/210:51:382
315867599200,19cyclictest0-21swapper/210:09:472
3158674992018,1cyclictest3056-21PK-Backend10:32:161
3158674992018,1cyclictest3056-21PK-Backend07:42:091
3158674992018,1cyclictest3056-21PK-Backend07:21:341
3158674992018,1cyclictest0-21swapper/110:21:211
3158674992018,1cyclictest0-21swapper/108:55:431
3158674992017,2cyclictest3198510-21smartctl09:00:161
3158674992017,2cyclictest3056-21PK-Backend10:57:581
3158674992017,2cyclictest3056-21PK-Backend09:46:191
3158674992012,6cyclictest3223003-21df_inode10:10:131
3158674992012,6cyclictest3186211-21cat08:25:151
315867499200,19cyclictest3056-21PK-Backend08:17:441
315867499200,19cyclictest0-21swapper/108:24:271
315867499200,18cyclictest3056-21PK-Backend07:50:131
3158671992018,1cyclictest0-21swapper/011:17:300
3158671992018,1cyclictest0-21swapper/010:12:210
3158671992018,1cyclictest0-21swapper/009:38:210
3158671992018,1cyclictest0-21swapper/007:38:030
3158671992013,5cyclictest3195005-21df08:50:140
3158671992012,6cyclictest3167758-21apt-get07:35:000
315867199200,19cyclictest0-21swapper/012:37:400
315867199200,19cyclictest0-21swapper/012:32:130
315867199200,19cyclictest0-21swapper/012:25:440
315867199200,19cyclictest0-21swapper/012:00:360
315867199200,19cyclictest0-21swapper/011:40:220
315867199200,19cyclictest0-21swapper/011:39:340
315867199200,19cyclictest0-21swapper/010:55:130
315867199200,19cyclictest0-21swapper/010:37:150
315867199200,19cyclictest0-21swapper/009:40:510
315867199200,19cyclictest0-21swapper/008:40:210
315867199200,19cyclictest0-21swapper/008:23:520
315867199200,19cyclictest0-21swapper/007:29:100
139099200,14rtkit-daemon0-21swapper/207:50:362
3158675991917,1cyclictest0-21swapper/212:31:492
3158675991917,1cyclictest0-21swapper/212:28:582
3158675991917,1cyclictest0-21swapper/211:28:552
3158675991917,1cyclictest0-21swapper/209:37:292
3158675991917,1cyclictest0-21swapper/209:16:132
3158675991917,1cyclictest0-21swapper/208:55:182
3158675991917,1cyclictest0-21swapper/208:20:162
3158675991917,1cyclictest0-21swapper/207:47:462
3158675991914,3cyclictest1095-21runrttasks11:45:112
3158675991912,5cyclictest3187426-21cron08:30:002
315867599190,1cyclictest0-21swapper/211:53:222
315867599190,1cyclictest0-21swapper/211:08:342
315867599190,1cyclictest0-21swapper/210:26:092
315867599190,1cyclictest0-21swapper/210:26:092
315867599190,1cyclictest0-21swapper/208:49:022
315867599190,1cyclictest0-21swapper/207:18:122
315867599190,18cyclictest3056-21PK-Backend09:10:182
315867599190,18cyclictest0-21swapper/212:37:162
315867599190,18cyclictest0-21swapper/210:12:172
315867599190,18cyclictest0-21swapper/209:31:552
315867499199,9cyclictest3056-21PK-Backend07:40:001
3158674991917,1cyclictest3056-21PK-Backend09:42:071
3158674991917,1cyclictest0-21swapper/111:16:031
3158674991917,1cyclictest0-21swapper/110:52:511
3158674991917,1cyclictest0-21swapper/110:37:301
3158674991916,2cyclictest3056-21PK-Backend12:13:491
3158674991915,2cyclictest3224371-21apt10:15:141
315867499190,1cyclictest3056-21PK-Backend12:37:081
315867499190,1cyclictest3056-21PK-Backend12:24:511
315867499190,1cyclictest3056-21PK-Backend11:59:291
315867499190,18cyclictest0-21swapper/111:45:441
315867199199,9cyclictest3199599-21unin-run09:05:000
3158671991917,1cyclictest0-21swapper/011:56:240
3158671991917,1cyclictest0-21swapper/007:47:550
3158671991917,1cyclictest0-21swapper/007:22:410
3158671991916,2cyclictest1095-21runrttasks08:59:560
3158671991911,6cyclictest3219556-21cat10:00:170
315867199190,1cyclictest0-21swapper/011:50:220
315867199190,1cyclictest0-21swapper/010:46:340
315867199190,1cyclictest0-21swapper/010:29:180
315867199190,1cyclictest0-21swapper/010:29:170
315867199190,17cyclictest3218905-21apt10:00:010
3158676991815,2cyclictest3056-21PK-Backend08:25:013
3158675991816,1cyclictest0-21swapper/212:04:242
3158675991816,1cyclictest0-21swapper/210:36:372
3158675991816,1cyclictest0-21swapper/209:46:552
3158675991816,1cyclictest0-21swapper/208:33:482
3158675991816,1cyclictest0-21swapper/207:28:592
3158675991812,5cyclictest34-21ksoftirqd/208:40:122
315867599180,17cyclictest3183230-21ntp_states08:15:212
3158674991816,1cyclictest3056-21PK-Backend09:36:461
3158674991816,1cyclictest0-21swapper/111:32:321
3158674991816,1cyclictest0-21swapper/110:02:011
3158674991815,2cyclictest3056-21PK-Backend12:27:411
3158674991815,2cyclictest3056-21PK-Backend08:38:331
3158674991814,3cyclictest26-21ksoftirqd/111:10:011
3158674991811,5cyclictest3164976-21ls07:25:181
3158674991810,6cyclictest3245674-21cron11:15:001
315867499180,1cyclictest0-21swapper/109:25:321
315867499180,1cyclictest0-21swapper/109:25:311
315867199189,7cyclictest3233957-21grep10:40:190
3158671991816,1cyclictest0-21swapper/011:27:580
315867199181,16cyclictest3160337-21grep07:14:590
315867699170,2cyclictest31940432sleep308:45:593
315867599178,7cyclictest3241431-21/usr/sbin/munin11:00:222
315867599174,11cyclictest3266988-21cron12:15:002
3158675991715,1cyclictest0-21swapper/210:30:012
3158675991715,1cyclictest0-21swapper/208:09:422
3158675991715,1cyclictest0-21swapper/207:40:142
3158675991712,3cyclictest3207263-21kernelversion09:25:182
3158675991712,3cyclictest3207263-21kernelversion09:25:182
315867499179,6cyclictest3273478-21awk12:30:201
315867499175,11cyclictest3205710-21needreboot09:20:171
3158674991715,1cyclictest3056-21PK-Backend12:15:241
3158674991715,1cyclictest3056-21PK-Backend07:45:061
3158674991715,1cyclictest3056-21PK-Backend07:18:501
3158674991711,4cyclictest3222039-21sshd10:06:011
315867199178,7cyclictest883-21dbus-daemon12:10:000
3158671991715,1cyclictest0-21swapper/008:12:430
3158671991715,1cyclictest0-21swapper/007:50:190
3158671991715,1cyclictest0-21swapper/007:15:170
3158671991710,5cyclictest3265494-21sshd12:10:010
3158675991614,1cyclictest0-21swapper/210:01:562
3158675991610,4cyclictest3234769-21perf10:45:002
3158674991613,2cyclictest3056-21PK-Backend11:38:131
3158674991611,4cyclictest3176813-21cut08:00:021
315867499161,11cyclictest3177030-21apt08:00:121
315867199169,5cyclictest3175998-21ntp_kernel_err07:55:190
3158671991614,1cyclictest0-21swapper/007:42:350
3158671991612,3cyclictest3208642-21cpuspeed_turbos09:30:120
3158671991611,3cyclictest3199816-21acpi09:05:140
139099160,3rtkit-daemon0-21swapper/207:33:532
3158675991511,3cyclictest3199815-21/usr/sbin/munin09:05:172
3158675991510,4cyclictest28650irq/123-ahci[0009:05:002
315867499157,6cyclictest3201992-21cat09:10:171
3158671991513,1cyclictest0-21swapper/011:46:570
3158671991513,1cyclictest0-21swapper/009:49:430
3158671991510,4cyclictest3226366-21expr10:20:120
3158671991510,3cyclictest1095-21runrttasks09:11:120
3158674991412,1cyclictest3056-21PK-Backend11:41:451
315867599138,4cyclictest3159113-21ls07:10:132
315867499139,3cyclictest3056-21PK-Backend12:03:301
315867499137,4cyclictest0-21swapper/112:05:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional