You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-24 - 00:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Jun 23, 2026 12:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30471122700,21sleep3421ksoftirqd/307:10:003
30325722200,8sleep20-21swapper/207:05:132
35829021270,3sleep33056-21PK-Backend09:40:013
33190921140,4sleep13056-21PK-Backend08:25:011
30321728669,10sleep10-21swapper/107:05:131
30450727551,18sleep00-21swapper/007:08:050
3200312570,1chrt319645-21/usr/sbin/munin07:50:132
3922532500,3sleep10-21swapper/111:15:151
3603842370,1sleep10-21swapper/109:45:121
3939932290,3sleep0161rcuc/011:20:140
3518842260,4sleep33056-21PK-Backend09:20:163
139099250,17rtkit-daemon3056-21PK-Backend10:50:153
139099240,4rtkit-daemon0-21swapper/307:10:133
304799992314,7cyclictest419244-21cron12:35:013
30479999230,15cyclictest141rcu_preempt11:18:533
304795992314,7cyclictest343806-21users08:55:202
304790992321,1cyclictest0-21swapper/011:00:510
139099230,16rtkit-daemon3056-21PK-Backend07:32:053
304799992220,1cyclictest3056-21PK-Backend11:45:153
304799992218,3cyclictest311838-21chrt07:26:123
304799992216,4cyclictest42-21ksoftirqd/311:35:013
304799992216,4cyclictest375397-21latency_hist10:30:003
304795992220,1cyclictest0-21swapper/211:35:432
304795992220,1cyclictest0-21swapper/211:20:502
304795992220,1cyclictest0-21swapper/210:49:062
304790992220,1cyclictest0-21swapper/009:29:390
304790992220,1cyclictest0-21swapper/007:31:290
304790992219,2cyclictest0-21swapper/010:25:220
304790992213,7cyclictest379488-21df10:40:110
30479099220,21cyclictest306136-21chrt07:11:010
30479099220,1cyclictest0-21swapper/009:42:360
139099220,14rtkit-daemon3056-21PK-Backend07:40:133
139099220,14rtkit-daemon0-21swapper/309:07:173
304799992119,1cyclictest0-21swapper/312:22:253
304799992119,1cyclictest0-21swapper/310:08:423
304799992118,2cyclictest3056-21PK-Backend11:41:023
304799992118,2cyclictest3056-21PK-Backend11:06:213
304799992115,4cyclictest42-21ksoftirqd/312:20:013
304799992113,6cyclictest328975-21df08:15:133
304799992112,8cyclictest330320-21/usr/sbin/munin08:20:133
30479999210,20cyclictest3056-21PK-Backend10:38:373
30479999210,20cyclictest0-21swapper/310:20:253
30479999210,1cyclictest3056-21PK-Backend07:58:503
30479999210,1cyclictest3056-21PK-Backend07:58:503
30479999210,19cyclictest3056-21PK-Backend07:50:123
304795992119,1cyclictest0-21swapper/212:00:302
304795992119,1cyclictest0-21swapper/211:26:452
304795992119,1cyclictest0-21swapper/211:09:002
304795992119,1cyclictest0-21swapper/211:00:152
304795992119,1cyclictest0-21swapper/210:42:102
304795992119,1cyclictest0-21swapper/210:26:012
304795992119,1cyclictest0-21swapper/209:47:492
304795992119,1cyclictest0-21swapper/209:37:282
304795992119,1cyclictest0-21swapper/209:20:452
304795992119,1cyclictest0-21swapper/209:16:192
304795992119,1cyclictest0-21swapper/209:06:122
304795992119,1cyclictest0-21swapper/207:56:512
304795992119,1cyclictest0-21swapper/207:56:502
304795992114,5cyclictest365217-21cut10:00:012
304795992113,6cyclictest332940-21python308:25:222
30479599210,20cyclictest0-21swapper/212:38:442
30479599210,20cyclictest0-21swapper/208:35:162
30479599210,20cyclictest0-21swapper/208:14:342
30479599210,20cyclictest0-21swapper/207:44:122
30479599210,20cyclictest0-21swapper/207:25:272
30479599210,20cyclictest0-21swapper/207:18:242
30479599210,19cyclictest0-21swapper/210:53:332
304793992119,1cyclictest3056-21PK-Backend08:33:221
30479099218,4cyclictest13-21ksoftirqd/009:52:380
30479099218,4cyclictest13-21ksoftirqd/008:16:590
30479099218,4cyclictest13-21ksoftirqd/008:12:540
30479099217,5cyclictest13-21ksoftirqd/008:38:560
30479099217,5cyclictest13-21ksoftirqd/008:24:200
304790992119,1cyclictest0-21swapper/011:10:440
304790992119,1cyclictest0-21swapper/010:54:560
304790992119,1cyclictest0-21swapper/010:35:430
304790992119,1cyclictest0-21swapper/010:08:520
304790992119,1cyclictest0-21swapper/009:49:190
304790992119,1cyclictest0-21swapper/009:37:500
304790992119,1cyclictest0-21swapper/008:49:030
304790992119,1cyclictest0-21swapper/008:41:320
304790992119,1cyclictest0-21swapper/008:25:370
304790992118,2cyclictest0-21swapper/012:38:040
304790992113,6cyclictest324595-21cron08:05:000
304790992112,7cyclictest393261-21perf11:20:000
30479099210,20cyclictest0-21swapper/009:06:130
139099210,15rtkit-daemon3056-21PK-Backend11:10:173
139099210,14rtkit-daemon3056-21PK-Backend12:05:073
139099210,13rtkit-daemon0-21swapper/308:56:083
4007732200,4sleep00-21swapper/011:40:120
3604432200,4chrt3056-21PK-Backend09:45:143
30479999204,3cyclictest141rcu_preempt11:25:293
304799992018,1cyclictest3056-21PK-Backend10:47:153
304799992018,1cyclictest3056-21PK-Backend08:00:463
304799992018,1cyclictest0-21swapper/310:17:553
304799992018,1cyclictest0-21swapper/309:10:423
304799992018,1cyclictest0-21swapper/309:04:413
304799992018,1cyclictest0-21swapper/308:27:233
304799992017,2cyclictest3056-21PK-Backend07:36:473
304799992015,3cyclictest361928-21grep09:50:123
30479999201,4cyclictest398716-21apt11:35:123
304799992014,4cyclictest416907-21python312:25:213
30479999201,3cyclictest141rcu_preempt07:20:313
304799992013,5cyclictest42-21ksoftirqd/308:50:103
304799992013,5cyclictest326589-21cat08:10:013
30479999200,1cyclictest3056-21PK-Backend10:55:203
30479999200,1cyclictest3056-21PK-Backend09:28:543
30479999200,1cyclictest0-21swapper/310:12:203
30479999200,19cyclictest3056-21PK-Backend10:31:543
30479999200,19cyclictest3056-21PK-Backend09:19:283
30479999200,19cyclictest0-21swapper/308:49:263
30479999200,19cyclictest0-21swapper/307:15:583
304795992018,1cyclictest0-21swapper/211:30:572
304795992018,1cyclictest0-21swapper/210:37:102
304795992018,1cyclictest0-21swapper/210:11:492
304795992018,1cyclictest0-21swapper/209:41:062
304795992018,1cyclictest0-21swapper/209:30:592
304795992018,1cyclictest0-21swapper/209:00:582
304795992018,1cyclictest0-21swapper/208:30:362
304795992018,1cyclictest0-21swapper/208:19:002
30479599200,1cyclictest0-21swapper/208:44:012
30479599200,19cyclictest0-21swapper/212:16:402
30479599200,19cyclictest0-21swapper/211:44:432
30479599200,19cyclictest0-21swapper/210:32:102
30479599200,19cyclictest0-21swapper/210:23:312
30479599200,19cyclictest0-21swapper/210:17:572
30479599200,19cyclictest0-21swapper/210:03:102
30479599200,19cyclictest0-21swapper/209:57:362
30479599200,19cyclictest0-21swapper/208:50:102
30479599200,19cyclictest0-21swapper/208:47:402
304793992018,1cyclictest3056-21PK-Backend10:12:511
304793992013,6cyclictest389521-21cron11:10:001
304793992012,6cyclictest436-20systemd-journal10:50:001
30479099207,4cyclictest13-21ksoftirqd/011:31:470
30479099207,4cyclictest13-21ksoftirqd/010:14:120
304790992018,1cyclictest3056-21PK-Backend07:40:350
304790992018,1cyclictest0-21swapper/011:35:320
304790992018,1cyclictest0-21swapper/009:58:320
304790992018,1cyclictest0-21swapper/009:58:320
304790992018,1cyclictest0-21swapper/008:59:030
304790992018,1cyclictest0-21swapper/007:50:560
304790992017,2cyclictest0-21swapper/007:25:210
304790992014,5cyclictest345920-21cron09:05:000
304790992013,5cyclictest356385-21date09:35:010
30479099200,1cyclictest0-21swapper/012:24:080
30479099200,19cyclictest3056-21PK-Backend10:01:360
30479099200,19cyclictest0-21swapper/010:47:250
30479099200,19cyclictest0-21swapper/009:18:160
30479099200,19cyclictest0-21swapper/009:14:110
30479099200,19cyclictest0-21swapper/008:34:430
139099200,13rtkit-daemon0-21swapper/207:39:462
4041312190,2sleep20-21swapper/211:50:122
30479999194,2cyclictest141rcu_preempt10:00:203
304799991917,1cyclictest3056-21PK-Backend10:41:073
304799991917,1cyclictest0-21swapper/309:56:253
304799991917,1cyclictest0-21swapper/309:56:253
304799991917,1cyclictest0-21swapper/309:43:003
30479999191,3cyclictest141rcu_preempt12:36:083
30479999191,3cyclictest141rcu_preempt12:01:193
30479999191,3cyclictest141rcu_preempt07:49:093
30479999190,1cyclictest0-21swapper/308:37:273
30479999190,17cyclictest327266-21chrt08:10:153
304795991917,1cyclictest0-21swapper/209:54:312
304795991917,1cyclictest0-21swapper/208:05:082
304795991912,5cyclictest407096-21cron12:00:002
304795991912,5cyclictest306334-21sshd07:13:362
304795991910,7cyclictest436-20systemd-journal12:25:002
304793991917,1cyclictest0-21swapper/109:37:091
304793991917,1cyclictest0-21swapper/109:24:041
304793991912,5cyclictest402464-21expr11:45:131
304793991912,5cyclictest396507-21sshd11:27:381
30479399190,1cyclictest3056-21PK-Backend09:55:001
30479399190,1cyclictest0-21swapper/112:04:521
30479399190,1cyclictest0-21swapper/110:26:571
30479099197,4cyclictest13-21ksoftirqd/012:34:500
30479099196,4cyclictest13-21ksoftirqd/012:12:570
304790991917,1cyclictest0-21swapper/009:23:570
304790991914,4cyclictest13-21ksoftirqd/007:45:120
304790991913,4cyclictest883-21dbus-daemon08:00:000
304790991913,4cyclictest883-21dbus-daemon08:00:000
304790991913,4cyclictest371918-21cron10:20:000
304790991912,6cyclictest396122-21awk11:25:200
30479099190,1cyclictest0-21swapper/011:54:230
30479099190,1cyclictest0-21swapper/008:05:570
30479099190,18cyclictest0-21swapper/011:49:510
30479099190,18cyclictest0-21swapper/008:52:070
139099190,13rtkit-daemon0-21swapper/207:47:512
30479999181,3cyclictest141rcu_preempt11:52:013
30479999181,3cyclictest141rcu_preempt11:02:333
304799991812,4cyclictest411587-21ntp_states12:10:193
304799991810,6cyclictest42-21ksoftirqd/311:24:193
304795991816,1cyclictest0-21swapper/211:14:552
304795991816,1cyclictest0-21swapper/210:05:062
304795991816,1cyclictest0-21swapper/208:20:572
304795991816,1cyclictest0-21swapper/208:01:162
304795991811,6cyclictest411763-21sed12:10:222
304793991816,1cyclictest3056-21PK-Backend12:20:201
304793991816,1cyclictest3056-21PK-Backend08:58:431
304793991816,1cyclictest3056-21PK-Backend08:13:411
304793991816,1cyclictest0-21swapper/110:35:161
304793991814,3cyclictest313874-21cron07:35:001
304793991813,3cyclictest354601-21latency_hist09:30:011
304793991812,5cyclictest410388-21seq12:07:511
304793991812,4cyclictest2156-21gnome-shell07:55:091
304793991812,4cyclictest2156-21gnome-shell07:55:091
304793991811,6cyclictest394473-21sed11:20:201
304793991811,6cyclictest317507-21cat07:45:011
304793991811,5cyclictest390196-21grep11:10:141
30479399181,15cyclictest382688-21cpuspeed_turbos10:50:131
304793991811,5cyclictest355582-21ntpq09:30:181
30479099188,2cyclictest141rcu_preempt12:09:190
30479099185,4cyclictest13-21ksoftirqd/012:26:320
30479099185,4cyclictest13-21ksoftirqd/012:17:500
30479099185,4cyclictest13-21ksoftirqd/010:20:330
304790991816,1cyclictest0-21swapper/007:39:130
304790991812,4cyclictest388698-21grep11:05:150
304790991811,5cyclictest306766-21missed_timers07:15:020
30479099181,13cyclictest308956-21apt07:20:110
30479099180,17cyclictest0-21swapper/012:01:060
30479999179,6cyclictest337463-21grep08:40:123
30479999173,2cyclictest141rcu_preempt11:55:093
304799991715,1cyclictest0-21swapper/308:30:413
30479999171,3cyclictest141rcu_preempt09:32:143
30479599179,6cyclictest347936-21apt09:10:122
304795991715,1cyclictest0-21swapper/211:47:132
304795991713,2cyclictest354602-21dump-pmu-power09:30:002
30479399179,6cyclictest384955-21df_inode10:55:131
30479399179,6cyclictest337006-21cron08:40:001
304793991715,1cyclictest3056-21PK-Backend12:26:281
304793991715,1cyclictest3056-21PK-Backend07:14:411
304793991715,1cyclictest0-21swapper/110:01:021
304793991715,1cyclictest0-21swapper/107:45:361
304793991712,3cyclictest2156-21gnome-shell11:59:091
304793991710,5cyclictest436-20systemd-journal12:39:361
304793991710,5cyclictest436-20systemd-journal11:44:371
304793991710,5cyclictest379116-21basename10:40:081
30479399170,16cyclictest0-21swapper/107:51:101
304790991715,1cyclictest0-21swapper/010:32:380
304790991712,3cyclictest385197-21missed_timers10:55:160
304795991614,1cyclictest0-21swapper/207:32:582
30479599161,12cyclictest416042-21apt12:25:112
304795991610,5cyclictest418178-21awk12:30:152
30479399169,6cyclictest404540-21cat11:50:171
30479399169,6cyclictest325399-21ls08:05:161
30479399169,5cyclictest314141-21find07:35:101
30479399163,11cyclictest375723-21cpuspeed10:30:121
304793991612,3cyclictest323938-21wc08:00:201
304793991612,2cyclictest386795-21interrupts11:00:161
304793991611,3cyclictest311564-21switchtime07:25:211
304793991610,5cyclictest309147-21expr07:20:111
30479399160,15cyclictest3056-21PK-Backend09:55:341
30479399160,15cyclictest3056-21PK-Backend09:55:341
30479399160,15cyclictest3056-21PK-Backend08:49:511
30479099160,14cyclictest436-20systemd-journal11:56:380
30479599159,5cyclictest918-21rs:main2
30479399159,4cyclictest26-21ksoftirqd/109:40:191
30479399158,6cyclictest373673-21/usr/sbin/munin10:25:001
30479399158,5cyclictest883-21dbus-daemon09:20:011
304793991513,1cyclictest3056-21PK-Backend08:51:331
304793991513,1cyclictest3056-21PK-Backend07:16:491
304793991513,1cyclictest0-21swapper/112:10:051
30479399150,14cyclictest3056-21PK-Backend12:34:061
30479399150,14cyclictest3056-21PK-Backend09:00:251
30479599147,5cyclictest436-20systemd-journal12:10:002
30479599147,5cyclictest309372-21cat07:20:142
304795991411,2cyclictest384563-21/usr/sbin/munin10:55:132
30479399149,4cyclictest3056-21PK-Backend09:08:391
304793991412,1cyclictest3056-21PK-Backend08:15:571
30479399140,13cyclictest3056-21PK-Backend10:05:211
30479399139,3cyclictest3056-21PK-Backend09:10:041
304793991311,1cyclictest3056-21PK-Backend12:16:141
304793991311,1cyclictest3056-21PK-Backend11:36:061
304793991311,1cyclictest3056-21PK-Backend11:30:181
304793991210,1cyclictest3056-21PK-Backend08:25:241
30479399117,3cyclictest3717032sleep110:17:211
30479399117,2cyclictest337761-21uname08:40:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional