You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-30 - 20:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Jul 30, 2026 12:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31785622300,8sleep23056-21PK-Backend07:05:252
41799121210,3sleep20-21swapper/211:15:152
3876642980,3sleep13056-21PK-Backend09:50:131
3335702960,4sleep13056-21PK-Backend07:50:111
139099840,69rtkit-daemon0-21swapper/307:05:253
31837427152,13sleep00-21swapper/007:09:130
3384172610,4sleep13056-21PK-Backend08:05:111
31843526141,13sleep10-21swapper/107:09:581
4046542490,4sleep23056-21PK-Backend10:35:222
4336722480,3sleep13056-21PK-Backend12:00:131
3647392460,4sleep03056-21PK-Backend08:47:560
3717982370,1sleep3371800-21expr09:05:123
31860499329,10cyclictest403416-21dump-pmu-power10:35:013
318602993227,3cyclictest34-21ksoftirqd/208:46:362
31860499318,9cyclictest404057-21df_inode10:35:133
31860499318,13cyclictest0-21swapper/308:15:303
31860499307,14cyclictest0-21swapper/307:25:033
31860499307,10cyclictest331857-21entropy07:45:143
31860499305,14cyclictest326649-21cat07:30:163
318604993010,18cyclictest351349-21wc08:40:173
31860499298,4cyclictest3383892basename08:05:103
31860499297,8cyclictest0-21swapper/309:35:123
318604992916,9cyclictest957-21gdbus08:46:373
31860499289,8cyclictest339860-21apt-get08:10:033
31860499288,18cyclictest446094-21sshd12:35:113
318604992812,14cyclictest401633-21latency_hist10:30:023
318604992811,15cyclictest396952-21ntpq10:15:173
318604992810,16cyclictest441193-21/usr/sbin/munin12:20:193
318604992710,15cyclictest347095-21systemd-run08:30:003
31860299276,2cyclictest141rcu_preempt10:35:002
139099270,19rtkit-daemon378149-21dpkg09:23:403
31860499262,20cyclictest423024-21apt11:30:133
318597992620,4cyclictest26-21ksoftirqd/108:46:341
31860499255,8cyclictest0-21swapper/312:25:173
31860499254,8cyclictest0-21swapper/312:10:123
31860499254,20cyclictest0-21swapper/311:39:193
31860499254,13cyclictest0-21swapper/309:15:003
318604992516,6cyclictest430823-21taskset11:50:223
318604992515,8cyclictest390516-21awk10:00:013
318604992513,11cyclictest413017-21strings11:00:173
318604992512,11cyclictest440287-21latency_hist12:20:013
318604992510,13cyclictest384568-21ntp_states09:40:183
31860499249,13cyclictest385959-21smartctl09:45:173
31860499249,13cyclictest337413-21ntp_states08:00:193
318604992421,2cyclictest0-21swapper/311:15:383
318604992414,8cyclictest329482-21grep07:40:003
31860499240,22cyclictest428622-21smartctl11:45:123
31860499240,22cyclictest428622-21smartctl11:45:123
318604992319,3cyclictest0-21swapper/307:14:413
318604992315,5cyclictest415942-21cpuspeed_turbos11:10:123
318604992315,5cyclictest415550-21idleruntime-cro11:10:003
318604992313,8cyclictest426862-21chrt11:40:143
318604992313,8cyclictest395687-21sh10:15:003
318604992312,8cyclictest436-20systemd-journal10:50:253
318604992311,10cyclictest344037-21cut08:20:123
31860299231,9cyclictest4336732ls12:00:132
318602992318,4cyclictest34-21ksoftirqd/208:10:012
31859499232,20cyclictest0-21swapper/008:17:030
318594992321,1cyclictest0-21swapper/011:20:010
31859499230,4cyclictest416820-21timerwakeupswit11:10:210
31860499229,11cyclictest407364-21cstates10:45:123
318604992214,6cyclictest398362-21taskset10:20:143
318604992214,6cyclictest349215-21grep08:35:123
318604992213,7cyclictest444217-21grep12:30:123
318604992213,7cyclictest434087-21needreboot12:00:193
318604992213,7cyclictest419881-21missed_timers11:20:153
318604992213,7cyclictest381161-21chrt09:30:233
318604992213,7cyclictest367847-21basename08:54:593
318604992212,8cyclictest436-20systemd-journal10:00:113
318604992210,10cyclictest335649-21tr07:55:183
318602992220,1cyclictest0-21swapper/211:37:002
318602992220,1cyclictest0-21swapper/209:52:022
318602992220,1cyclictest0-21swapper/208:20:032
318602992220,1cyclictest0-21swapper/207:27:172
318602992217,4cyclictest34-21ksoftirqd/212:25:002
318602992214,6cyclictest322714-21date07:20:002
318594992220,1cyclictest0-21swapper/012:09:560
318594992220,1cyclictest0-21swapper/011:49:550
318594992220,1cyclictest0-21swapper/011:49:550
318594992220,1cyclictest0-21swapper/009:23:220
318594992220,1cyclictest0-21swapper/007:43:030
318594992220,1cyclictest0-21swapper/007:26:410
318594992219,2cyclictest3056-21PK-Backend09:26:530
31859499220,21cyclictest1317-21systemd09:19:490
139099220,14rtkit-daemon0-21swapper/211:13:432
139099220,14rtkit-daemon0-21swapper/210:57:122
3824012210,4sleep13056-21PK-Backend09:35:131
3413092210,3sleep20-21swapper/208:10:222
31860499217,12cyclictest410589-21/usr/sbin/munin10:55:173
318604992119,1cyclictest0-21swapper/309:19:463
318604992114,5cyclictest368798-21netstat08:55:203
318604992113,6cyclictest435945-21wc12:05:203
318604992113,6cyclictest405578-21cpuspeed_turbos10:40:103
318604992113,6cyclictest330751-21timerwakeupswit07:40:213
318604992113,6cyclictest321009-21/usr/sbin/munin07:15:103
318604992113,5cyclictest433148-21turbostat12:00:003
318604992112,7cyclictest370441-21grep09:00:163
318604992111,7cyclictest0-21swapper/310:05:113
31860499210,20cyclictest0-21swapper/307:24:083
318602992119,1cyclictest0-21swapper/212:16:552
318602992119,1cyclictest0-21swapper/211:56:402
318602992119,1cyclictest0-21swapper/211:40:452
318602992119,1cyclictest0-21swapper/211:22:472
318602992119,1cyclictest0-21swapper/211:06:262
318602992119,1cyclictest0-21swapper/211:02:392
318602992119,1cyclictest0-21swapper/210:52:122
318602992119,1cyclictest0-21swapper/210:45:292
318602992119,1cyclictest0-21swapper/209:38:182
318602992119,1cyclictest0-21swapper/209:23:232
318602992119,1cyclictest0-21swapper/208:00:022
318602992119,1cyclictest0-21swapper/207:54:412
318602992119,1cyclictest0-21swapper/207:39:212
31860299210,2cyclictest0-21swapper/209:10:192
31860299210,20cyclictest3056-21PK-Backend08:37:002
31860299210,20cyclictest0-21swapper/210:08:462
31860299210,20cyclictest0-21swapper/209:42:372
31860299210,20cyclictest0-21swapper/207:46:372
31860299210,20cyclictest0-21swapper/207:14:062
318597992119,1cyclictest3056-21PK-Backend11:54:371
318597992119,1cyclictest0-21swapper/108:41:311
31859799210,4cyclictest141rcu_preempt08:55:001
31859799210,1cyclictest3056-21PK-Backend10:46:231
31859799210,1cyclictest141rcu_preempt11:25:011
31859799210,19cyclictest3056-21PK-Backend10:30:541
31859799210,19cyclictest3056-21PK-Backend10:04:311
31859799210,19cyclictest3056-21PK-Backend08:00:211
318594992119,1cyclictest0-21swapper/012:15:030
318594992119,1cyclictest0-21swapper/011:56:250
318594992119,1cyclictest0-21swapper/009:52:210
318594992119,1cyclictest0-21swapper/007:54:060
318594992119,1cyclictest0-21swapper/007:36:340
318594992118,2cyclictest3056-21PK-Backend10:21:280
31859499210,20cyclictest0-21swapper/010:31:070
31859499210,20cyclictest0-21swapper/010:17:420
31859499210,20cyclictest0-21swapper/009:42:200
31859499210,20cyclictest0-21swapper/009:11:380
31859499210,20cyclictest0-21swapper/009:01:170
31859499210,20cyclictest0-21swapper/008:35:010
139099210,14rtkit-daemon0-21swapper/309:29:403
139099210,14rtkit-daemon0-21swapper/212:37:392
139099210,13rtkit-daemon0-21swapper/209:46:442
3825672200,3sleep0382571-21latency09:35:160
31860499207,12cyclictest0-21swapper/311:30:003
318604992018,1cyclictest0-21swapper/307:52:533
318604992013,4cyclictest387266-21/usr/sbin/munin09:50:183
318604992011,7cyclictest348040-21ntp_kernel_err08:30:203
318602992018,1cyclictest0-21swapper/212:07:302
318602992018,1cyclictest0-21swapper/210:26:432
318602992018,1cyclictest0-21swapper/210:15:282
318602992018,1cyclictest0-21swapper/207:18:252
31860299200,1cyclictest0-21swapper/212:13:572
31860299200,1cyclictest0-21swapper/210:11:442
31860299200,19cyclictest0-21swapper/212:27:022
31860299200,19cyclictest0-21swapper/210:22:102
31860299200,19cyclictest0-21swapper/209:03:292
31860299200,19cyclictest0-21swapper/208:16:532
318597992018,1cyclictest0-21swapper/112:31:201
31859799200,1cyclictest3056-21PK-Backend11:29:011
31859799200,19cyclictest3056-21PK-Backend12:28:021
31859799200,19cyclictest3056-21PK-Backend11:48:361
31859799200,19cyclictest3056-21PK-Backend11:48:351
31859799200,19cyclictest3056-21PK-Backend10:26:011
31859799200,19cyclictest3056-21PK-Backend10:08:371
31859799200,19cyclictest3056-21PK-Backend09:59:041
31859799200,19cyclictest3056-21PK-Backend09:00:371
31859799200,18cyclictest3056-21PK-Backend09:31:261
31859799200,18cyclictest3056-21PK-Backend07:27:291
318594992018,1cyclictest0-21swapper/012:21:530
318594992018,1cyclictest0-21swapper/011:44:140
318594992018,1cyclictest0-21swapper/011:19:130
318594992018,1cyclictest0-21swapper/010:47:300
318594992018,1cyclictest0-21swapper/010:12:350
318594992018,1cyclictest0-21swapper/009:08:540
318594992018,1cyclictest0-21swapper/008:54:000
318594992018,1cyclictest0-21swapper/007:31:130
318594992018,1cyclictest0-21swapper/007:16:270
318594992012,6cyclictest406088-21strings10:40:150
318594992011,7cyclictest347340-21apt08:30:140
31859499200,1cyclictest3056-21PK-Backend11:05:140
31859499200,1cyclictest0-21swapper/008:10:070
31859499200,19cyclictest0-21swapper/012:02:050
31859499200,19cyclictest0-21swapper/011:29:280
31859499200,19cyclictest0-21swapper/010:56:230
31859499200,19cyclictest0-21swapper/010:27:230
31859499200,19cyclictest0-21swapper/010:09:380
31859499200,19cyclictest0-21swapper/009:56:410
31859499200,19cyclictest0-21swapper/009:33:070
31859499200,19cyclictest0-21swapper/008:08:310
31859499200,19cyclictest0-21swapper/007:10:450
318602991917,1cyclictest0-21swapper/208:54:032
318602991917,1cyclictest0-21swapper/207:40:082
318602991910,7cyclictest405787-21df_inode10:40:122
31860299190,1cyclictest0-21swapper/211:49:252
31860299190,1cyclictest0-21swapper/211:49:242
31860299190,1cyclictest0-21swapper/209:56:222
31860299190,1cyclictest0-21swapper/208:58:222
31860299190,1cyclictest0-21swapper/208:40:322
31860299190,1cyclictest0-21swapper/208:33:282
31860299190,1cyclictest0-21swapper/207:55:292
31860299190,18cyclictest0-21swapper/211:50:112
31860299190,18cyclictest0-21swapper/209:07:362
31859799199,9cyclictest3056-21PK-Backend10:44:591
31859799199,9cyclictest3056-21PK-Backend10:15:001
318597991915,2cyclictest341883-21idleruntime-cro08:15:001
318597991914,4cyclictest26-21ksoftirqd/109:23:501
31859799190,1cyclictest3056-21PK-Backend11:57:201
31859799190,1cyclictest3056-21PK-Backend11:41:451
31859799190,1cyclictest3056-21PK-Backend11:18:261
31859799190,1cyclictest3056-21PK-Backend07:55:271
31859799190,1cyclictest3056-21PK-Backend07:47:041
31859799190,1cyclictest3056-21PK-Backend07:33:101
31859799190,1cyclictest0-21swapper/109:48:101
31859799190,1cyclictest0-21swapper/107:35:201
31859799190,18cyclictest3056-21PK-Backend12:37:421
31859799190,18cyclictest3056-21PK-Backend11:31:111
31859799190,18cyclictest3056-21PK-Backend09:40:451
31859799190,18cyclictest3056-21PK-Backend09:13:011
31859799190,18cyclictest3056-21PK-Backend07:14:591
31859799190,18cyclictest0-21swapper/109:26:461
318594991917,1cyclictest0-21swapper/012:10:440
318594991917,1cyclictest0-21swapper/011:33:130
318594991917,1cyclictest0-21swapper/011:00:080
318594991917,1cyclictest0-21swapper/010:53:250
318594991917,1cyclictest0-21swapper/010:37:500
318594991917,1cyclictest0-21swapper/007:58:380
318594991916,2cyclictest3056-21PK-Backend08:21:020
31859499190,1cyclictest0-21swapper/012:34:300
31859499190,1cyclictest0-21swapper/008:58:210
31859499190,1cyclictest0-21swapper/007:22:560
31859499190,18cyclictest0-21swapper/007:48:100
139099190,13rtkit-daemon0-21swapper/207:33:082
318602991815,2cyclictest0-21swapper/211:30:172
318602991815,2cyclictest0-21swapper/209:25:332
318602991813,3cyclictest375231-21/usr/sbin/munin09:15:122
318602991811,6cyclictest346335-21ntp_states08:25:162
318597991816,1cyclictest3056-21PK-Backend07:22:371
318597991816,1cyclictest0-21swapper/108:36:031
31859799180,16cyclictest3056-21PK-Backend12:21:401
31859799180,16cyclictest3056-21PK-Backend09:16:201
318594991810,6cyclictest352267-21munin-run08:44:590
31859499180,1cyclictest3056-21PK-Backend09:48:010
31859499180,1cyclictest0-21swapper/012:37:140
139099180,10rtkit-daemon34-21ksoftirqd/210:02:562
318602991710,6cyclictest421003-21find11:25:102
31859799179,6cyclictest410346-21cron10:55:001
318597991715,1cyclictest3056-21PK-Backend11:06:441
318597991715,1cyclictest3056-21PK-Backend11:01:501
318597991714,2cyclictest3056-21PK-Backend08:23:341
318597991710,5cyclictest399430-21sh10:25:011
31859799170,16cyclictest3056-21PK-Backend08:33:191
318594991715,1cyclictest0-21swapper/011:50:430
318594991715,1cyclictest0-21swapper/011:39:070
318594991715,1cyclictest0-21swapper/008:03:590
318594991710,5cyclictest436-20systemd-journal12:25:120
3206812160,4chrt3056-21PK-Backend07:15:031
318602991612,3cyclictest34-21ksoftirqd/212:34:412
31859799162,6cyclictest3056-21PK-Backend09:05:111
318597991614,1cyclictest3056-21PK-Backend12:10:251
318597991614,1cyclictest3056-21PK-Backend10:58:191
318597991614,1cyclictest3056-21PK-Backend07:40:141
31859799160,5cyclictest141rcu_preempt11:39:591
139099160,3rtkit-daemon0-21swapper/209:33:172
31859799154,3cyclictest3424391cstates08:15:131
31859799152,5cyclictest141rcu_preempt12:10:001
318597991513,1cyclictest3056-21PK-Backend08:27:181
318597991512,2cyclictest3056-21PK-Backend11:14:211
318597991511,3cyclictest403745-21cpuspeed10:35:121
31859799150,6cyclictest3056-21PK-Backend08:56:411
318594991511,3cyclictest345968-21sshd08:25:120
318594991510,4cyclictest392235-21basename10:05:000
318597991412,1cyclictest3056-21PK-Backend12:15:531
31859799130,6cyclictest3056-21PK-Backend10:15:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional