You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-11 - 12:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Fri Sep 11, 2026 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
326287022970,8sleep10-21swapper/119:10:011
332244521520,10sleep013-21ksoftirqd/021:55:170
326753321140,4sleep13056-21PK-Backend19:20:151
326383821080,2sleep30-21swapper/319:10:193
333489421010,3sleep00-21swapper/022:30:170
326137227952,17sleep00-21swapper/019:05:110
326269127839,13sleep30-21swapper/319:08:233
326266426645,13sleep20-21swapper/219:08:012
326296199451,33cyclictest3309174-21cat21:20:003
139099440,6rtkit-daemon0-21swapper/222:41:052
33121702410,4sleep13056-21PK-Backend21:25:221
326296199391,17cyclictest883-21dbus-daemon00:40:003
326296199380,15cyclictest0-21swapper/319:40:013
32887862360,1sleep33288787-21expr20:20:123
326296199360,16cyclictest0-21swapper/323:00:013
326296199360,15cyclictest3348009-21latency_hist23:10:013
326296199350,15cyclictest3274487-21sed19:40:013
326296199350,15cyclictest0-21swapper/323:15:013
326296199344,19cyclictest3285253-21cstates20:10:133
326296199340,20cyclictest3296929-21taskset20:44:133
326296199340,19cyclictest3339359-21runrttasks22:45:083
326296199340,15cyclictest0-21swapper/321:05:013
326296199330,18cyclictest0-21swapper/323:03:033
326296199330,18cyclictest0-21swapper/322:00:083
326296199330,17cyclictest0-21swapper/319:20:083
326296199320,23cyclictest3293548-21munin-run20:35:003
326296199320,22cyclictest3317842-21apt-config21:45:003
326296199320,21cyclictest3295284-21munin-run20:40:013
326296199320,19cyclictest3272152-21taskset19:33:113
326296199320,17cyclictest3375558-21timerandwakeup00:25:233
326296199320,17cyclictest0-21swapper/300:14:383
326296199320,16cyclictest3358667-21sed23:40:003
326296199320,16cyclictest3267022-21cat19:20:013
326296199312,14cyclictest0-21swapper/323:57:343
326296199310,21cyclictest3287278-21memory20:15:183
326296199310,20cyclictest3328945-21date22:15:003
326296199310,18cyclictest3323290-21systemd-run22:00:013
326296199310,18cyclictest0-21swapper/323:45:593
326296199310,18cyclictest0-21swapper/321:12:443
326296199310,18cyclictest0-21swapper/300:03:463
326296199310,15cyclictest3311137-21apt21:25:123
326296199310,15cyclictest0-21swapper/323:10:123
326296199310,14cyclictest0-21swapper/322:10:013
326296199310,13cyclictest0-21swapper/322:55:023
326295199319,3cyclictest3309175-21grep21:20:010
326296199301,18cyclictest0-21swapper/322:05:193
326296199300,20cyclictest0-21swapper/321:40:003
326296199300,18cyclictest0-21swapper/321:34:583
326296199300,18cyclictest0-21swapper/300:32:423
326296199300,17cyclictest3337066-21taskset22:36:043
326296199300,17cyclictest3302380-21latency_hist21:00:023
326296199300,17cyclictest3278514-21irqstats19:50:163
326296199300,17cyclictest0-21swapper/322:41:153
326296199300,16cyclictest0-21swapper/319:45:383
326296199300,14cyclictest3321675-21sed21:55:003
326296199300,14cyclictest0-21swapper/323:20:123
326296199297,10cyclictest42-21ksoftirqd/323:30:003
326296199291,16cyclictest0-21swapper/321:05:423
326296199290,17cyclictest0-21swapper/322:29:433
326296199290,17cyclictest0-21swapper/322:23:503
326296199290,17cyclictest0-21swapper/322:23:493
326296199290,17cyclictest0-21swapper/320:07:413
326296199290,17cyclictest0-21swapper/319:59:293
326296199290,17cyclictest0-21swapper/319:25:373
326296199290,17cyclictest0-21swapper/319:25:373
326296199290,17cyclictest0-21swapper/300:21:593
326296199290,17cyclictest0-21swapper/300:21:593
326296199290,17cyclictest0-21swapper/300:18:343
326296199290,16cyclictest0-21swapper/320:54:193
326296199280,16cyclictest0-21swapper/323:32:433
326296199280,16cyclictest0-21swapper/320:49:343
326296199280,15cyclictest3290637-21cat20:25:133
326296199280,15cyclictest3290637-21cat20:25:123
326296199271,13cyclictest3334191-21apt22:30:103
326296199270,15cyclictest3309930-21latency21:20:193
326296199270,14cyclictest3367859-21cstates00:05:123
3262959992722,4cyclictest34-21ksoftirqd/221:20:002
326296199264,20cyclictest3362652-21cstates23:50:133
326296199260,14cyclictest3358690-21/usr/sbin/munin23:40:113
3262959992511,12cyclictest3312632-21unin-run21:30:002
33281932240,3sleep13056-21PK-Backend22:10:221
326295999241,11cyclictest34-21ksoftirqd/200:35:002
326296199238,13cyclictest3282079-21cut20:00:183
3262959992314,7cyclictest3368014-21df00:05:132
326295999230,22cyclictest0-21swapper/220:27:072
326295999230,22cyclictest0-21swapper/220:27:072
326295999230,21cyclictest3345166-21memory23:00:182
3262951992318,4cyclictest13-21ksoftirqd/022:40:000
326295199230,4cyclictest3269057-21cstates19:25:110
326295199230,4cyclictest3269057-21cstates19:25:110
326296199221,19cyclictest3318712-21/usr/sbin/munin21:45:143
326295999228,2cyclictest141rcu_preempt21:05:012
3262959992220,1cyclictest0-21swapper/223:14:302
3262959992220,1cyclictest0-21swapper/221:32:382
3262959992220,1cyclictest0-21swapper/219:29:152
3262959992220,1cyclictest0-21swapper/219:29:142
326295999221,2cyclictest141rcu_preempt00:00:012
326295999220,21cyclictest0-21swapper/223:31:542
326295999220,21cyclictest0-21swapper/200:16:292
326295499220,20cyclictest3056-21PK-Backend21:04:481
326295199228,4cyclictest13-21ksoftirqd/023:00:010
326295199226,3cyclictest3289140-21grep20:20:170
3262951992219,2cyclictest3376991-21strings00:30:180
3262951992218,3cyclictest0-21swapper/020:51:100
32819612210,3sleep0161rcuc/020:00:170
326295999216,2cyclictest141rcu_preempt20:24:202
326295999214,3cyclictest141rcu_preempt22:05:202
3262959992119,1cyclictest0-21swapper/223:09:572
3262959992119,1cyclictest0-21swapper/222:32:402
3262959992119,1cyclictest0-21swapper/221:57:582
3262959992119,1cyclictest0-21swapper/221:53:592
3262959992119,1cyclictest0-21swapper/220:49:372
3262959992119,1cyclictest0-21swapper/220:31:332
3262959992119,1cyclictest0-21swapper/219:40:372
3262959992119,1cyclictest0-21swapper/219:31:302
3262959992113,6cyclictest3325003-21munin-run22:05:002
3262959992113,6cyclictest3280823-21sed19:55:202
3262959992112,7cyclictest34-21ksoftirqd/219:35:102
326295999210,20cyclictest3353060-21taskset23:22:492
326295999210,20cyclictest0-21swapper/221:23:442
326295999210,20cyclictest0-21swapper/220:09:152
326295999210,1cyclictest0-21swapper/219:21:382
3262954992118,2cyclictest3056-21PK-Backend21:37:471
326295499210,20cyclictest0-21swapper/119:15:041
326295499210,19cyclictest3056-21PK-Backend23:58:271
326295499210,19cyclictest3056-21PK-Backend23:15:071
326295499210,19cyclictest3056-21PK-Backend22:20:131
326295499210,19cyclictest3056-21PK-Backend22:20:121
326295499210,19cyclictest3056-21PK-Backend21:52:141
326295499210,19cyclictest3056-21PK-Backend21:21:041
326295499210,19cyclictest3056-21PK-Backend20:51:441
326295199219,10cyclictest13-21ksoftirqd/021:30:000
326295199218,2cyclictest141rcu_preempt23:15:010
326295199218,2cyclictest141rcu_preempt19:50:000
326295199211,4cyclictest141rcu_preempt23:15:040
3262951992114,5cyclictest3286421-21cron20:15:000
3262951992113,6cyclictest3361907-21apt-get23:50:020
326295199211,11cyclictest13-21ksoftirqd/000:15:160
326295199210,7cyclictest3279821-21sort19:55:110
3262959992018,1cyclictest0-21swapper/222:19:282
3262959992018,1cyclictest0-21swapper/221:46:152
3262959992018,1cyclictest0-21swapper/220:57:152
3262959992018,1cyclictest0-21swapper/200:37:392
3262959992018,1cyclictest0-21swapper/200:27:512
3262959992017,2cyclictest3056-21PK-Backend21:41:432
3262959992015,4cyclictest978-21snmpd23:46:162
3262959992015,3cyclictest978-21snmpd20:00:052
3262959992012,6cyclictest34-21ksoftirqd/223:50:042
3262959992012,6cyclictest34-21ksoftirqd/200:20:142
3262959992012,6cyclictest34-21ksoftirqd/200:20:132
3262959992012,6cyclictest3294474-21grep20:35:192
3262959992011,7cyclictest34-21ksoftirqd/222:45:232
3262959992011,7cyclictest34-21ksoftirqd/222:15:002
326295999200,19cyclictest0-21swapper/222:38:342
326295999200,19cyclictest0-21swapper/219:51:112
326295999200,19cyclictest0-21swapper/219:49:562
3262954992018,1cyclictest3056-21PK-Backend23:35:161
3262954992018,1cyclictest3056-21PK-Backend20:19:191
3262954992018,1cyclictest3056-21PK-Backend19:49:121
3262954992017,2cyclictest3056-21PK-Backend19:33:171
3262954992017,2cyclictest3056-21PK-Backend00:16:111
3262954992013,5cyclictest26-21ksoftirqd/121:45:221
326295499200,6cyclictest3376049-21dump-pmu-power00:30:001
326295499200,3cyclictest3056-21PK-Backend23:50:011
326295499200,1cyclictest3056-21PK-Backend19:52:431
326295499200,19cyclictest3056-21PK-Backend20:47:051
326295499200,19cyclictest3056-21PK-Backend19:41:061
326295499200,18cyclictest3056-21PK-Backend23:27:441
326295499200,18cyclictest3056-21PK-Backend21:31:521
326295499200,18cyclictest3056-21PK-Backend19:29:591
326295499200,18cyclictest3056-21PK-Backend19:29:591
326295199206,3cyclictest141rcu_preempt21:50:130
326295199205,2cyclictest141rcu_preempt20:40:170
326295199204,3cyclictest13-21ksoftirqd/019:40:010
326295199203,3cyclictest141rcu_preempt20:39:300
326295199203,3cyclictest141rcu_preempt20:15:410
3262951992015,3cyclictest13-21ksoftirqd/020:30:180
326295199200,6cyclictest3330730-21grep22:20:120
326295199200,6cyclictest3330730-21grep22:20:120
32980272190,3sleep03056-21PK-Backend20:45:190
326295999196,2cyclictest141rcu_preempt00:14:592
326295999192,2cyclictest141rcu_preempt22:20:182
326295999192,2cyclictest141rcu_preempt22:20:182
3262959991917,1cyclictest0-21swapper/222:29:432
3262959991915,3cyclictest34-21ksoftirqd/221:10:012
3262959991914,4cyclictest34-21ksoftirqd/223:15:002
3262959991912,5cyclictest3366972-21cat00:00:222
3262959991912,5cyclictest3299875-21swap20:50:222
3262959991912,5cyclictest3264991-21fw_packets19:15:032
3262959991911,6cyclictest34-21ksoftirqd/221:38:482
3262959991911,6cyclictest3295545-21dpkg20:40:112
326295999190,6cyclictest3360182-21/usr/sbin/munin23:45:002
326295999190,18cyclictest0-21swapper/223:36:272
326295999190,18cyclictest0-21swapper/222:53:362
326295999190,18cyclictest0-21swapper/220:12:072
3262954991917,1cyclictest3056-21PK-Backend23:31:311
3262954991917,1cyclictest3056-21PK-Backend23:21:501
3262954991917,1cyclictest3056-21PK-Backend22:01:051
3262954991917,1cyclictest3056-21PK-Backend21:14:361
3262954991917,1cyclictest3056-21PK-Backend20:38:591
3262954991917,1cyclictest3056-21PK-Backend19:35:321
3262954991917,1cyclictest0-21swapper/123:02:301
3262954991917,1cyclictest0-21swapper/122:35:201
3262954991917,1cyclictest0-21swapper/122:15:191
3262954991917,1cyclictest0-21swapper/122:05:451
3262954991917,1cyclictest0-21swapper/120:30:211
3262954991916,2cyclictest3056-21PK-Backend23:12:111
3262954991916,2cyclictest3056-21PK-Backend20:57:111
326295499190,1cyclictest3056-21PK-Backend23:41:101
326295499190,1cyclictest3056-21PK-Backend22:44:451
326295499190,1cyclictest3056-21PK-Backend22:34:321
326295499190,1cyclictest0-21swapper/121:41:331
326295499190,18cyclictest3056-21PK-Backend23:54:281
326295499190,18cyclictest3056-21PK-Backend20:07:291
326295199197,6cyclictest13-21ksoftirqd/019:15:020
326295199193,2cyclictest141rcu_preempt23:30:000
326295199192,3cyclictest141rcu_preempt00:15:000
326295199191,7cyclictest141rcu_preempt23:00:230
3262951991914,4cyclictest13-21ksoftirqd/021:05:120
3262951991913,5cyclictest3310990-21latency_hist21:25:010
326295199191,2cyclictest141rcu_preempt23:30:250
3262951991912,6cyclictest13-21ksoftirqd/019:30:170
326295199190,6cyclictest0-21swapper/021:35:130
326295199190,1cyclictest0-21swapper/019:19:100
326295199190,18cyclictest0-21swapper/019:54:420
326295199190,18cyclictest0-21swapper/000:38:250
139099190,13rtkit-daemon0-21swapper/222:56:082
3262959991816,1cyclictest0-21swapper/223:28:102
3262959991816,1cyclictest0-21swapper/221:12:092
3262959991810,6cyclictest3287625-21cat20:15:232
3262954991816,1cyclictest3056-21PK-Backend20:00:271
3262954991816,1cyclictest3056-21PK-Backend00:37:541
3262954991811,5cyclictest3378190-21cat00:35:011
326295499180,1cyclictest3056-21PK-Backend22:56:351
326295499180,1cyclictest3056-21PK-Backend20:11:081
326295199189,3cyclictest13-21ksoftirqd/021:00:110
326295199183,3cyclictest141rcu_preempt22:10:010
326295199182,3cyclictest141rcu_preempt22:19:590
326295199182,14cyclictest3367814-21cpuspeed_turbos00:05:130
326295199181,7cyclictest141rcu_preempt19:40:140
3262951991816,1cyclictest0-21swapper/021:45:240
3262951991814,2cyclictest13-21ksoftirqd/023:35:200
326295199181,3cyclictest141rcu_preempt22:29:590
326295199181,3cyclictest141rcu_preempt22:10:010
3262951991813,4cyclictest3365700-21systemd00:00:010
3262951991813,2cyclictest13-21ksoftirqd/023:20:200
326295199181,14cyclictest3317808-21munin-run21:45:000
3262951991810,2cyclictest13-21ksoftirqd/000:29:560
326295199180,5cyclictest0-21swapper/022:45:240
326295199180,17cyclictest0-21swapper/000:03:540
326295999179,2cyclictest34-21ksoftirqd/219:12:462
3262954991715,1cyclictest3056-21PK-Backend21:59:431
3262954991715,1cyclictest3056-21PK-Backend21:18:551
3262954991715,1cyclictest3056-21PK-Backend19:58:381
3262954991715,1cyclictest0-21swapper/119:12:271
3262954991714,2cyclictest33474742sleep123:06:161
3262954991714,2cyclictest3056-21PK-Backend20:24:201
326295199174,2cyclictest141rcu_preempt23:06:390
326295199172,3cyclictest141rcu_preempt21:10:460
326295199172,2cyclictest141rcu_preempt22:50:150
326295199172,12cyclictest3337658-21apt22:40:120
326295199171,3cyclictest141rcu_preempt21:00:010
3262951991712,3cyclictest13-21ksoftirqd/000:20:120
3262951991712,3cyclictest13-21ksoftirqd/000:20:120
3262951991711,5cyclictest3313378-21cut21:30:170
3262951991711,5cyclictest3268569-21cron19:24:590
3262951991711,4cyclictest3362617-21cut23:50:130
3262951991710,5cyclictest3359725-21grep23:40:210
3262954991614,1cyclictest3056-21PK-Backend22:46:211
3262954991614,1cyclictest3056-21PK-Backend20:27:451
3262954991614,1cyclictest3056-21PK-Backend20:27:441
3262954991614,1cyclictest3056-21PK-Backend00:11:241
326295499160,4cyclictest3056-21PK-Backend22:29:591
326295199161,3cyclictest141rcu_preempt20:25:120
326295199161,3cyclictest141rcu_preempt20:25:120
3262951991612,2cyclictest13-21ksoftirqd/022:00:180
326295499158,5cyclictest3304193-21cat21:05:111
326295499151,6cyclictest3056-21PK-Backend00:20:201
326295499151,6cyclictest3056-21PK-Backend00:20:191
3262954991513,1cyclictest3056-21PK-Backend22:54:251
3262954991512,2cyclictest3056-21PK-Backend20:42:381
3262951991511,2cyclictest13-21ksoftirqd/020:05:190
326295499145,2cyclictest141rcu_preempt00:00:371
326295499133,2cyclictest141rcu_preempt00:05:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional