You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-01 - 13:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Sep 01, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
190126721400,3sleep00-21swapper/022:50:210
182646321270,3sleep33056-21PK-Backend19:20:203
188990321260,3sleep23056-21PK-Backend22:20:192
181991627446,17sleep00-21swapper/019:05:120
182085627349,15sleep30-21swapper/319:05:283
182050126647,13sleep10-21swapper/119:05:221
182120926344,13sleep20-21swapper/219:08:042
18881532530,4sleep33056-21PK-Backend22:15:203
18850302470,3sleep20-21swapper/222:05:242
19046652460,4sleep10-21swapper/123:00:161
139099390,4rtkit-daemon882-21avahi-daemon22:51:403
139099380,5rtkit-daemon1788643-21eadedCompositor20:48:103
139099360,2rtkit-daemon3056-21PK-Backend23:23:583
139099360,2rtkit-daemon3056-21PK-Backend22:01:083
139099350,2rtkit-daemon3056-21PK-Backend22:39:463
139099340,2rtkit-daemon3056-21PK-Backend21:22:553
139099340,2rtkit-daemon3056-21PK-Backend20:36:113
1821512992914,12cyclictest1901808-21cron22:54:592
139099270,4rtkit-daemon1389-21rtkit-daemon22:19:002
1821503992611,12cyclictest1909045-21unin-run23:15:000
139099250,4rtkit-daemon1389-21rtkit-daemon20:07:302
182150399241,6cyclictest1883703-21munin-run22:04:590
1821503992416,6cyclictest1888899-21perf22:20:010
139099240,3rtkit-daemon1389-21rtkit-daemon21:26:232
18845522230,3sleep10-21swapper/122:05:191
182150899232,5cyclictest26-21ksoftirqd/119:15:001
139099230,3rtkit-daemon1389-21rtkit-daemon22:02:142
1821514992220,1cyclictest3056-21PK-Backend22:13:343
1821514992214,6cyclictest1871298-21cron21:30:013
1821514992213,7cyclictest1935115-21fw_packets00:30:163
182151499220,20cyclictest3056-21PK-Backend23:58:253
1821512992220,1cyclictest0-21swapper/223:53:152
1821512992220,1cyclictest0-21swapper/221:40:532
1821512992220,1cyclictest0-21swapper/219:38:042
1821512992220,1cyclictest0-21swapper/219:30:412
182151299220,21cyclictest34-21ksoftirqd/221:54:452
1821508992220,1cyclictest0-21swapper/119:29:541
1821508992216,4cyclictest1865775-21runrttasks21:11:011
182150899221,19cyclictest0-21swapper/123:35:391
182150899220,20cyclictest0-21swapper/120:32:061
1821503992220,1cyclictest0-21swapper/022:26:280
1821503992220,1cyclictest0-21swapper/020:58:400
1821503992220,1cyclictest0-21swapper/019:12:200
1821503992220,1cyclictest0-21swapper/000:05:500
1821503992215,5cyclictest13-21ksoftirqd/019:56:220
1821503992213,7cyclictest883-21dbus-daemon19:55:000
19375122210,2sleep20-21swapper/200:35:202
19370972210,2chrt0-21swapper/100:35:161
1821514992118,2cyclictest3056-21PK-Backend23:48:253
1821514992118,2cyclictest3056-21PK-Backend23:48:243
1821514992112,7cyclictest436-20systemd-journal19:45:003
182151499210,20cyclictest3056-21PK-Backend20:10:173
182151499210,20cyclictest3056-21PK-Backend00:35:423
182151499210,20cyclictest0-21swapper/323:03:373
182151499210,19cyclictest3056-21PK-Backend23:16:213
182151499210,19cyclictest3056-21PK-Backend20:08:283
1821512992119,1cyclictest0-21swapper/223:55:522
1821512992119,1cyclictest0-21swapper/223:47:122
1821512992119,1cyclictest0-21swapper/223:47:122
1821512992119,1cyclictest0-21swapper/222:57:252
1821512992119,1cyclictest0-21swapper/222:48:182
1821512992119,1cyclictest0-21swapper/222:48:182
1821512992119,1cyclictest0-21swapper/222:12:362
1821512992119,1cyclictest0-21swapper/221:55:332
1821512992119,1cyclictest0-21swapper/221:37:222
1821512992119,1cyclictest0-21swapper/221:10:592
1821512992119,1cyclictest0-21swapper/220:51:242
1821512992119,1cyclictest0-21swapper/220:43:272
1821512992119,1cyclictest0-21swapper/220:36:452
1821512992119,1cyclictest0-21swapper/220:01:092
1821512992119,1cyclictest0-21swapper/219:56:432
1821512992119,1cyclictest0-21swapper/219:51:282
1821512992119,1cyclictest0-21swapper/200:17:072
1821512992119,1cyclictest0-21swapper/200:02:412
1821512992114,4cyclictest1848237-21dump-pmu-power20:25:002
1821512992113,6cyclictest1867455-21vmstat21:15:232
182151299210,20cyclictest0-21swapper/223:24:142
182151299210,20cyclictest0-21swapper/222:26:362
182151299210,20cyclictest0-21swapper/221:06:532
182151299210,20cyclictest0-21swapper/220:49:282
1821508992119,1cyclictest0-21swapper/121:55:071
1821508992119,1cyclictest0-21swapper/119:57:521
1821508992114,5cyclictest1895605-21ntp_kernel_err22:35:191
182150899210,20cyclictest3056-21PK-Backend20:29:561
182150899210,20cyclictest0-21swapper/122:31:371
182150899210,20cyclictest0-21swapper/122:16:171
182150899210,20cyclictest0-21swapper/122:00:011
182150899210,20cyclictest0-21swapper/120:24:161
182150899210,20cyclictest0-21swapper/119:18:461
1821503992119,1cyclictest0-21swapper/023:15:080
1821503992119,1cyclictest0-21swapper/022:58:380
1821503992119,1cyclictest0-21swapper/022:43:370
1821503992119,1cyclictest0-21swapper/022:11:060
1821503992119,1cyclictest0-21swapper/021:51:190
1821503992119,1cyclictest0-21swapper/021:28:140
1821503992119,1cyclictest0-21swapper/020:49:140
1821503992119,1cyclictest0-21swapper/020:13:100
1821503992119,1cyclictest0-21swapper/020:02:090
1821503992119,1cyclictest0-21swapper/019:22:010
1821503992119,1cyclictest0-21swapper/000:34:430
1821503992119,1cyclictest0-21swapper/000:21:520
1821503992113,6cyclictest1869252-21sshd21:21:080
1821503992112,7cyclictest1853715-21cron20:40:000
182150399210,20cyclictest3056-21PK-Backend23:45:360
182150399210,20cyclictest3056-21PK-Backend23:45:360
182150399210,20cyclictest0-21swapper/022:36:070
182150399210,20cyclictest0-21swapper/022:20:330
182150399210,20cyclictest0-21swapper/020:51:440
182150399210,20cyclictest0-21swapper/000:02:250
139099210,4rtkit-daemon1389-21rtkit-daemon22:36:582
182151499206,13cyclictest1867840-21sh21:20:003
1821514992018,1cyclictest3056-21PK-Backend21:06:013
1821514992018,1cyclictest3056-21PK-Backend00:08:253
1821514992018,1cyclictest0-21swapper/323:25:063
1821514992018,1cyclictest0-21swapper/321:39:203
182151499201,7cyclictest1916698-21irqstats23:35:163
1821514992017,2cyclictest3056-21PK-Backend22:30:183
1821514992017,2cyclictest3056-21PK-Backend20:01:263
182151499201,5cyclictest1877036-21apt21:45:113
182151499201,5cyclictest1877036-21apt21:45:113
1821514992012,6cyclictest1879158-21awk21:50:153
182151499200,1cyclictest3056-21PK-Backend22:41:183
182151499200,1cyclictest3056-21PK-Backend20:58:183
182151499200,1cyclictest3056-21PK-Backend19:17:043
182151499200,19cyclictest3056-21PK-Backend22:25:113
182151499200,19cyclictest3056-21PK-Backend21:00:143
182151499200,19cyclictest3056-21PK-Backend20:43:373
182151499200,19cyclictest3056-21PK-Backend19:51:123
182151499200,19cyclictest0-21swapper/319:25:173
182151299208,8cyclictest1932454-21apt-get00:25:012
1821512992018,1cyclictest0-21swapper/223:35:372
1821512992018,1cyclictest0-21swapper/220:25:082
1821512992018,1cyclictest0-21swapper/220:16:432
1821512992014,3cyclictest1915956-21idleruntime-cro23:35:012
1821512992012,6cyclictest1850866-21df20:30:122
182151299200,19cyclictest0-21swapper/222:32:312
182151299200,19cyclictest0-21swapper/221:04:092
182151299200,19cyclictest0-21swapper/220:58:212
182151299200,19cyclictest0-21swapper/220:10:152
182151299200,19cyclictest0-21swapper/219:49:392
1821508992017,2cyclictest3056-21PK-Backend23:46:141
1821508992017,2cyclictest3056-21PK-Backend23:46:131
1821508992013,6cyclictest1930209-21expr00:15:201
1821508992013,6cyclictest1897616-21ntpq22:40:181
182150899200,1cyclictest0-21swapper/121:17:021
182150899200,19cyclictest3056-21PK-Backend22:56:181
182150899200,19cyclictest0-21swapper/123:05:591
182150899200,19cyclictest0-21swapper/122:26:301
182150899200,19cyclictest0-21swapper/122:23:331
182150899200,19cyclictest0-21swapper/121:50:131
182150899200,19cyclictest0-21swapper/120:09:221
1821503992018,1cyclictest3056-21PK-Backend20:24:540
1821503992018,1cyclictest0-21swapper/023:53:130
1821503992018,1cyclictest0-21swapper/021:10:290
1821503992018,1cyclictest0-21swapper/020:17:170
1821503992018,1cyclictest0-21swapper/000:10:030
1821503992013,5cyclictest1871544-21sshd21:30:060
182150399200,1cyclictest0-21swapper/023:58:130
182150399200,19cyclictest3056-21PK-Backend21:49:100
182150399200,19cyclictest3056-21PK-Backend21:49:100
182150399200,19cyclictest0-21swapper/022:06:460
182150399200,19cyclictest0-21swapper/021:57:340
182150399200,19cyclictest0-21swapper/020:44:330
182150399200,19cyclictest0-21swapper/019:28:570
182150399200,19cyclictest0-21swapper/000:38:090
1821514991917,1cyclictest3056-21PK-Backend23:52:373
1821514991917,1cyclictest3056-21PK-Backend22:07:183
1821514991917,1cyclictest3056-21PK-Backend00:21:033
1821514991917,1cyclictest3056-21PK-Backend00:18:263
1821514991917,1cyclictest3056-21PK-Backend00:13:253
1821514991917,1cyclictest3056-21PK-Backend00:02:373
1821514991917,1cyclictest0-21swapper/323:10:463
1821514991917,1cyclictest0-21swapper/319:30:033
1821514991917,1cyclictest0-21swapper/319:10:013
1821514991911,6cyclictest1848238-21apt-get20:25:013
182151499190,1cyclictest3056-21PK-Backend23:33:373
182151499190,1cyclictest3056-21PK-Backend20:19:163
182151499190,1cyclictest0-21swapper/323:41:353
182151499190,1cyclictest0-21swapper/300:27:043
182151499190,18cyclictest3056-21PK-Backend20:32:023
182151499190,18cyclictest0-21swapper/319:55:253
182151499190,17cyclictest1849515-21ntp_states20:25:203
182151299192,12cyclictest1917952-21apt23:40:112
1821512991917,1cyclictest0-21swapper/223:01:582
1821512991917,1cyclictest0-21swapper/221:45:452
1821512991917,1cyclictest0-21swapper/221:45:452
1821512991917,1cyclictest0-21swapper/200:28:372
182151299190,18cyclictest0-21swapper/222:40:492
182150899192,13cyclictest1868110-21apt21:20:121
1821508991917,1cyclictest3056-21PK-Backend21:42:371
1821508991917,1cyclictest0-21swapper/123:40:261
1821508991917,1cyclictest0-21swapper/123:30:521
1821508991917,1cyclictest0-21swapper/122:50:231
1821508991917,1cyclictest0-21swapper/122:46:391
1821508991917,1cyclictest0-21swapper/122:46:391
1821508991917,1cyclictest0-21swapper/121:27:221
1821508991917,1cyclictest0-21swapper/120:10:241
1821508991917,1cyclictest0-21swapper/119:40:291
1821508991917,1cyclictest0-21swapper/100:33:321
1821508991913,5cyclictest1860433-21sed20:55:211
1821508991911,6cyclictest1834679-21sort19:45:131
182150899190,1cyclictest3056-21PK-Backend00:27:301
182150899190,1cyclictest0-21swapper/121:01:061
182150899190,1cyclictest0-21swapper/120:18:011
182150899190,18cyclictest3056-21PK-Backend20:49:111
182150899190,18cyclictest0-21swapper/120:51:411
182150899190,18cyclictest0-21swapper/120:42:211
182150399194,13cyclictest1867838-21sh21:20:010
1821503991917,1cyclictest0-21swapper/022:32:210
1821503991917,1cyclictest0-21swapper/021:02:310
1821503991917,1cyclictest0-21swapper/020:08:230
1821503991917,1cyclictest0-21swapper/000:15:040
1821503991911,6cyclictest1850242-21perf20:30:010
182150399190,1cyclictest0-21swapper/023:02:230
182150399190,1cyclictest0-21swapper/022:47:230
182150399190,1cyclictest0-21swapper/022:47:230
182150399190,1cyclictest0-21swapper/021:42:530
182150399190,1cyclictest0-21swapper/020:32:450
182150399190,18cyclictest0-21swapper/023:42:110
182150399190,18cyclictest0-21swapper/000:28:410
1821514991813,4cyclictest1881996-21systemctl22:00:003
1821514991812,4cyclictest1889691-21expr22:20:183
1821514991811,5cyclictest1831117-21apt19:35:123
182151499180,16cyclictest1906561-21ntp_states23:05:193
182151299188,2cyclictest141rcu_preempt19:15:212
182151299183,13cyclictest1927679-21cut00:10:082
182151299182,12cyclictest1828249-21python319:25:232
1821512991816,1cyclictest3056-21PK-Backend00:05:172
1821512991816,1cyclictest0-21swapper/223:06:312
1821512991816,1cyclictest0-21swapper/219:21:542
1821512991816,1cyclictest0-21swapper/219:13:512
1821512991816,1cyclictest0-21swapper/200:30:262
1821508991816,1cyclictest0-21swapper/100:24:531
1821508991812,5cyclictest1873066-21systemd-run21:35:001
1821508991811,6cyclictest1921533-21wc23:50:111
182150399188,9cyclictest1832579-21perf19:40:000
1821503991816,1cyclictest0-21swapper/021:05:490
1821503991811,5cyclictest1823675-21cat19:15:030
182150399180,1cyclictest0-21swapper/023:39:470
182150399180,1cyclictest0-21swapper/021:35:160
1821514991712,4cyclictest1876343-21sshd21:43:073
1821514991710,5cyclictest1902739-21grep22:55:183
182151299172,11cyclictest1832849-21apt19:40:122
1821512991715,1cyclictest0-21swapper/221:33:502
182150899179,6cyclictest436-20systemd-journal23:20:011
182150899179,6cyclictest1926222-21sed00:00:221
182150899179,6cyclictest1874009-21ntp_kernel_err21:35:171
182150899174,2cyclictest141rcu_preempt00:10:081
182150899173,12cyclictest1886429-21ntpq22:10:201
1821508991715,1cyclictest0-21swapper/123:25:171
1821508991711,5cyclictest1836843-21sed19:50:171
1821508991711,4cyclictest1911259-21expr23:20:111
1821508991710,5cyclictest1878078-21sort21:45:221
1821508991710,5cyclictest1878078-21sort21:45:221
182150399178,7cyclictest1911410-21fw_packets23:20:140
1821503991715,1cyclictest0-21swapper/019:43:290
1821503991711,4cyclictest1834569-21basename19:45:110
1821514991614,1cyclictest3056-21PK-Backend22:48:023
1821514991614,1cyclictest3056-21PK-Backend22:48:013
1821514991614,1cyclictest3056-21PK-Backend21:30:223
1821514991614,1cyclictest3056-21PK-Backend21:13:583
182151299169,5cyclictest1869574-21munin-run21:25:002
1821512991612,3cyclictest1910745-21idleruntime-cro23:20:002
182150899168,6cyclictest1923144-21awk23:55:111
1821508991614,1cyclictest0-21swapper/120:35:041
1821508991614,1cyclictest0-21swapper/100:09:391
1821508991612,3cyclictest1825863-21cstates19:20:121
182150899161,13cyclictest1907750-21wc23:10:121
1821514991511,3cyclictest3056-21PK-Backend19:47:033
182151299159,4cyclictest1908200-21needreboot23:10:182
182150899158,5cyclictest1863895-21sed21:05:221
182150899158,5cyclictest1832319-21sshd19:36:531
1821508991513,1cyclictest0-21swapper/120:04:551
1821508991511,3cyclictest1829094-21/usr/sbin/munin19:30:171
182150399159,5cyclictest1913467-21ntpq23:25:190
1821503991510,3cyclictest1095-21runrttasks23:30:190
1821503991510,3cyclictest1095-21runrttasks23:06:330
182150399149,3cyclictest1829233-21cpuspeed_turbos19:30:120
1821514991311,1cyclictest3056-21PK-Backend20:52:093
182151299131,8cyclictest1913330-21/usr/sbin/munin23:25:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional