You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-08 - 14:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Jun 08, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
415291421260,3sleep00-21swapper/021:35:120
415147921230,3sleep20-21swapper/221:30:122
40132830,1sleep04012-21cut23:40:180
139099740,62rtkit-daemon0-21swapper/219:05:202
139099740,62rtkit-daemon0-21swapper/219:05:192
410125126740,21sleep10-21swapper/119:09:211
410125126740,21sleep10-21swapper/119:09:211
409982926744,14sleep00-21swapper/019:05:100
409982926744,14sleep00-21swapper/019:05:100
410109826444,13sleep30-21swapper/319:07:223
410109826444,13sleep30-21swapper/319:07:223
41294702440,4sleep33056-21PK-Backend20:28:383
41115532400,4sleep13056-21PK-Backend19:35:191
41244222280,4chrt3056-21PK-Backend20:15:011
41056082260,4sleep2331rcuc/219:20:122
410147599263,4cyclictest42-21ksoftirqd/321:55:003
4101475992518,5cyclictest4119182-21cron20:00:003
41236442240,3sleep34123637-21ntp_kernel_pll_20:10:183
410147599242,7cyclictest22341-21python300:30:213
410147599242,5cyclictest0-21swapper/322:45:123
147322240,4sleep13056-21PK-Backend00:10:191
41394172230,4sleep013-21ksoftirqd/020:55:140
4101475992321,1cyclictest0-21swapper/300:38:203
4101475992320,2cyclictest411rcuc/321:21:493
4101475992320,2cyclictest3056-21PK-Backend23:33:503
4101475992319,3cyclictest0-21swapper/300:00:133
4101475992310,3cyclictest42-21ksoftirqd/321:04:303
410147599230,6cyclictest42-21ksoftirqd/322:10:143
410147599230,6cyclictest3056-21PK-Backend21:48:493
410147599230,21cyclictest4103351-21irqstats19:15:023
4101465992312,7cyclictest4147446-21perf21:20:010
41921552220,3sleep20-21swapper/223:25:212
41921552220,3sleep20-21swapper/223:25:212
41465232220,3chrt0-21swapper/321:15:183
4101475992220,1cyclictest3056-21PK-Backend20:37:073
4101475992220,1cyclictest0-21swapper/322:17:333
4101475992220,1cyclictest0-21swapper/319:21:243
4101475992219,2cyclictest411rcuc/323:28:043
4101475992219,2cyclictest411rcuc/323:28:043
4101475992219,2cyclictest411rcuc/322:31:263
4101475992219,2cyclictest3056-21PK-Backend21:34:193
4101475992219,2cyclictest3056-21PK-Backend20:34:453
4101475992212,3cyclictest42-21ksoftirqd/321:36:493
4101475992211,3cyclictest42-21ksoftirqd/323:45:223
410147599220,8cyclictest42-21ksoftirqd/321:45:003
410147599220,7cyclictest42-21ksoftirqd/323:20:073
410147599220,7cyclictest42-21ksoftirqd/323:00:113
410147599220,7cyclictest3056-21PK-Backend23:19:113
410147599220,6cyclictest42-21ksoftirqd/323:35:463
410147599220,6cyclictest3056-21PK-Backend23:13:463
410147599220,21cyclictest0-21swapper/320:46:143
4101472992220,1cyclictest0-21swapper/222:07:142
4101472992220,1cyclictest0-21swapper/221:48:012
4101472992220,1cyclictest0-21swapper/221:21:172
4101472992220,1cyclictest0-21swapper/220:13:232
4101472992219,2cyclictest3056-21PK-Backend23:40:212
410147299220,3cyclictest10878-21memory00:00:232
410147299220,21cyclictest0-21swapper/221:35:512
4101469992220,1cyclictest3056-21PK-Backend23:28:501
4101469992220,1cyclictest3056-21PK-Backend23:28:491
410146999220,20cyclictest3056-21PK-Backend20:07:201
410146599220,3cyclictest1723-21df_inode23:35:130
139099220,16rtkit-daemon3056-21PK-Backend21:57:383
139099220,14rtkit-daemon0-21swapper/220:27:262
410147599211,9cyclictest42-21ksoftirqd/319:10:153
4101475992119,1cyclictest0-21swapper/322:52:003
4101475992119,1cyclictest0-21swapper/322:24:493
4101475992118,2cyclictest411rcuc/323:54:153
4101475992118,2cyclictest411rcuc/320:20:263
4101475992118,2cyclictest411rcuc/320:05:063
4101475992118,2cyclictest411rcuc/319:26:373
4101475992118,2cyclictest411rcuc/300:26:583
4101475992117,3cyclictest411rcuc/321:11:073
4101475992117,3cyclictest411rcuc/319:53:353
4101475992117,3cyclictest411rcuc/319:42:093
4101475992117,3cyclictest411rcuc/319:33:503
4101475992117,3cyclictest411rcuc/300:17:533
410147599210,7cyclictest42-21ksoftirqd/323:40:533
410147599210,7cyclictest42-21ksoftirqd/322:59:503
410147599210,7cyclictest42-21ksoftirqd/322:06:163
410147599210,7cyclictest42-21ksoftirqd/321:26:113
410147599210,6cyclictest3056-21PK-Backend23:07:273
410147599210,6cyclictest3056-21PK-Backend22:01:003
410147599210,20cyclictest0-21swapper/322:40:383
410147599210,20cyclictest0-21swapper/320:53:583
410147599210,20cyclictest0-21swapper/300:10:553
4101472992119,1cyclictest0-21swapper/223:04:262
4101472992119,1cyclictest0-21swapper/222:51:492
4101472992119,1cyclictest0-21swapper/222:42:142
4101472992119,1cyclictest0-21swapper/222:38:232
4101472992119,1cyclictest0-21swapper/222:02:482
4101472992119,1cyclictest0-21swapper/221:15:422
4101472992119,1cyclictest0-21swapper/219:55:122
4101472992113,6cyclictest17892-21expr00:20:132
410147299210,20cyclictest0-21swapper/223:33:192
410147299210,20cyclictest0-21swapper/223:17:022
410147299210,20cyclictest0-21swapper/223:13:102
410147299210,20cyclictest0-21swapper/221:55:242
410147299210,20cyclictest0-21swapper/219:54:112
410147299210,20cyclictest0-21swapper/219:46:542
410147299210,1cyclictest0-21swapper/222:25:262
4101469992119,1cyclictest3056-21PK-Backend23:58:171
4101469992119,1cyclictest3056-21PK-Backend19:18:451
4101469992119,1cyclictest0-21swapper/123:36:141
4101469992119,1cyclictest0-21swapper/122:01:441
4101469992119,1cyclictest0-21swapper/121:47:441
4101469992119,1cyclictest0-21swapper/121:22:491
4101469992118,2cyclictest3056-21PK-Backend19:23:331
4101469992117,2cyclictest3056-21PK-Backend22:40:301
410146999210,20cyclictest0-21swapper/123:02:141
410146999210,20cyclictest0-21swapper/122:20:151
410146999210,20cyclictest0-21swapper/121:33:111
410146999210,20cyclictest0-21swapper/121:25:141
410146999210,19cyclictest3056-21PK-Backend20:46:541
410146999210,19cyclictest3056-21PK-Backend19:30:561
4101465992118,2cyclictest4122929-21apt20:10:110
410146599210,20cyclictest0-21swapper/021:54:140
41860972200,5sleep10-21swapper/123:10:121
410147599202,16cyclictest0-21swapper/323:57:223
4101475992018,1cyclictest0-21swapper/320:04:153
4101475992017,2cyclictest411rcuc/320:59:123
4101475992017,2cyclictest411rcuc/320:43:363
4101475992017,2cyclictest411rcuc/319:49:023
4101475992017,2cyclictest411rcuc/300:21:543
4101475992017,2cyclictest411rcuc/300:09:153
4101475992016,3cyclictest411rcuc/322:36:193
4101475992016,3cyclictest411rcuc/322:25:053
4101475992016,3cyclictest411rcuc/319:37:113
410147599200,7cyclictest4124635-21/usr/sbin/munin20:15:133
410147299205,3cyclictest41410402expr21:00:132
4101472992018,1cyclictest0-21swapper/223:56:032
4101472992018,1cyclictest0-21swapper/223:45:562
4101472992018,1cyclictest0-21swapper/223:20:552
4101472992018,1cyclictest0-21swapper/223:06:292
4101472992018,1cyclictest0-21swapper/222:46:062
4101472992018,1cyclictest0-21swapper/222:12:412
4101472992018,1cyclictest0-21swapper/220:54:142
4101472992018,1cyclictest0-21swapper/220:43:192
4101472992018,1cyclictest0-21swapper/219:38:502
4101472992018,1cyclictest0-21swapper/219:28:222
4101472992017,2cyclictest0-21swapper/200:25:032
410147299200,19cyclictest0-21swapper/223:50:292
410147299200,19cyclictest0-21swapper/221:10:082
410147299200,19cyclictest0-21swapper/220:55:012
410147299200,19cyclictest0-21swapper/220:18:582
410147299200,19cyclictest0-21swapper/219:40:392
410147299200,19cyclictest0-21swapper/200:35:232
410147299200,19cyclictest0-21swapper/200:34:092
4101469992018,1cyclictest3056-21PK-Backend22:35:511
4101469992018,1cyclictest3056-21PK-Backend20:02:471
4101469992018,1cyclictest3056-21PK-Backend00:26:501
4101469992018,1cyclictest3056-21PK-Backend00:02:371
4101469992018,1cyclictest0-21swapper/123:24:091
4101469992018,1cyclictest0-21swapper/122:05:081
4101469992018,1cyclictest0-21swapper/120:42:211
4101469992018,1cyclictest0-21swapper/120:29:371
4101469992017,2cyclictest3056-21PK-Backend22:30:231
4101469992017,2cyclictest3056-21PK-Backend00:36:571
410146999200,19cyclictest3056-21PK-Backend21:18:041
410146999200,19cyclictest3056-21PK-Backend19:54:421
410146999200,19cyclictest0-21swapper/123:16:401
410146999200,19cyclictest0-21swapper/123:06:051
410146999200,18cyclictest3056-21PK-Backend22:45:101
410146599206,5cyclictest141rcu_preempt23:00:010
410146599205,3cyclictest141rcu_preempt19:55:000
4101465992018,1cyclictest0-21swapper/023:24:180
4101465992018,1cyclictest0-21swapper/019:59:080
4101465992013,6cyclictest1-21systemd00:00:000
4101465992012,6cyclictest4157563-21sshd21:46:390
4101465992010,7cyclictest436-20systemd-journal20:25:010
410146599200,19cyclictest0-21swapper/023:06:330
410146599200,19cyclictest0-21swapper/022:30:230
410146599200,19cyclictest0-21swapper/022:21:590
410146599200,19cyclictest0-21swapper/020:38:570
410146599200,19cyclictest0-21swapper/019:19:140
410147599190,10cyclictest42-21ksoftirqd/321:05:133
4101472991917,1cyclictest0-21swapper/220:01:412
4101472991917,1cyclictest0-21swapper/219:16:122
4101472991917,1cyclictest0-21swapper/200:09:002
4101472991913,5cyclictest17539-21sed00:20:002
4101472991912,5cyclictest4127100-21ntp_kernel_err20:20:192
410147299190,1cyclictest0-21swapper/222:33:442
410147299190,1cyclictest0-21swapper/220:31:222
410147299190,1cyclictest0-21swapper/220:08:382
410147299190,18cyclictest0-21swapper/222:20:462
410147299190,17cyclictest4110333-21sshd19:32:372
4101469991917,1cyclictest3056-21PK-Backend22:57:191
4101469991917,1cyclictest3056-21PK-Backend19:55:441
4101469991917,1cyclictest3056-21PK-Backend19:48:261
4101469991917,1cyclictest0-21swapper/123:40:471
4101469991916,2cyclictest3056-21PK-Backend19:41:241
4101469991916,1cyclictest3056-21PK-Backend21:10:201
410146999190,1cyclictest3056-21PK-Backend22:15:501
410146999190,1cyclictest3056-21PK-Backend22:14:011
410146999190,1cyclictest3056-21PK-Backend21:00:471
410146999190,1cyclictest0-21swapper/123:34:591
410146999190,1cyclictest0-21swapper/120:37:011
410146999190,18cyclictest3056-21PK-Backend20:59:111
410146599199,7cyclictest436-20systemd-journal23:34:590
410146599192,13cyclictest4182629-21apt23:00:110
410146599192,13cyclictest4161835-21apt22:00:110
4101465991917,1cyclictest0-21swapper/022:47:130
4101465991917,1cyclictest0-21swapper/021:31:490
4101465991915,3cyclictest4101644-21apt19:10:110
4101465991913,4cyclictest41350382sleep020:45:000
4101465991911,6cyclictest4149054-21sshd21:23:380
410146599190,6cyclictest141rcu_preempt21:30:000
410146599190,18cyclictest0-21swapper/021:58:130
410146599190,18cyclictest0-21swapper/000:28:320
4101472991816,1cyclictest0-21swapper/222:58:312
4101472991816,1cyclictest0-21swapper/220:45:422
4101472991811,5cyclictest4151022-21date21:30:002
4101469991816,1cyclictest3056-21PK-Backend22:26:441
4101469991816,1cyclictest0-21swapper/123:46:201
4101469991812,5cyclictest4160552-21if_err_enp2s021:55:171
4101469991812,5cyclictest19214-21cron00:24:591
410146599182,11cyclictest4187823-21apt23:15:110
4101465991816,1cyclictest3056-21PK-Backend00:38:330
4101465991816,1cyclictest0-21swapper/022:35:240
4101465991816,1cyclictest0-21swapper/022:25:230
4101465991816,1cyclictest0-21swapper/020:51:070
4101465991814,3cyclictest1095-21runrttasks20:34:310
4101465991812,5cyclictest4141715-21sed21:00:200
4101465991812,5cyclictest16353-21ls00:15:140
4101465991811,5cyclictest4110902-21cat19:35:120
4101465991810,6cyclictest883-21dbus-daemon19:35:000
4101465991810,6cyclictest4129188-21cat20:25:240
410146599180,1cyclictest0-21swapper/021:43:190
4101472991715,1cyclictest0-21swapper/222:16:192
4101472991715,1cyclictest0-21swapper/221:05:222
4101472991715,1cyclictest0-21swapper/219:13:492
4101469991715,1cyclictest3056-21PK-Backend23:52:561
4101469991715,1cyclictest3056-21PK-Backend19:12:381
4101469991714,2cyclictest3056-21PK-Backend21:05:331
4101469991713,3cyclictest4154369-21sh21:40:001
410146599172,11cyclictest4186095-21apt23:10:130
4101465991715,1cyclictest0-21swapper/020:48:580
4101465991715,1cyclictest0-21swapper/019:25:090
4101465991715,1cyclictest0-21swapper/000:13:320
4101465991715,1cyclictest0-21swapper/000:05:140
4101465991712,4cyclictest4113985-21cron19:45:000
410146599171,14cyclictest4164639-21users22:05:240
4101465991710,5cyclictest5740-21grep23:45:170
4101465991710,5cyclictest465-21systemd-udevd22:52:350
410146599170,16cyclictest0-21swapper/023:50:470
410146599170,16cyclictest0-21swapper/023:26:070
410146599170,16cyclictest0-21swapper/023:26:060
4101472991614,1cyclictest0-21swapper/223:35:492
4101472991614,1cyclictest0-21swapper/220:37:582
4101472991610,5cyclictest14569-21sed00:10:162
410146999168,6cyclictest4126126-21cron20:20:011
4101469991614,1cyclictest3056-21PK-Backend22:52:391
4101469991614,1cyclictest3056-21PK-Backend00:31:231
4101469991614,1cyclictest3056-21PK-Backend00:10:001
4101469991614,1cyclictest0-21swapper/121:53:311
4101469991614,1cyclictest0-21swapper/120:34:371
4101469991611,3cyclictest4155090-21if_enp2s021:40:171
410146599169,5cyclictest4120263-21ps20:00:220
4101465991614,1cyclictest0-21swapper/020:05:520
4101465991614,1cyclictest0-21swapper/019:46:590
4101465991614,1cyclictest0-21swapper/019:22:110
4101465991611,4cyclictest4144945-21awk21:10:180
4101465991611,3cyclictest4167084-21cpuspeed22:15:120
4101472991510,4cyclictest34-21ksoftirqd/221:55:002
4101469991513,1cyclictest3056-21PK-Backend20:53:491
410146999150,6cyclictest3056-21PK-Backend19:27:141
410146599159,5cyclictest10479-21tr00:00:140
410146599159,4cyclictest4142665-21tr21:05:120
4101465991511,3cyclictest4124635-21/usr/sbin/munin20:15:210
4101465991511,3cyclictest21452-21/usr/sbin/munin00:30:210
4101465991510,4cyclictest18190-21awk00:20:160
410147299149,3cyclictest1095-21runrttasks21:40:052
4101469991410,3cyclictest3056-21PK-Backend00:19:161
410146599142,8cyclictest4175700-21apt22:40:130
410146999138,4cyclictest26-21ksoftirqd/120:25:001
410146599131,9cyclictest4165674-21diskstats22:10:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional