You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-06 - 04:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Jul 06, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
319380622780,4sleep20-21swapper/219:05:142
328745421220,2sleep33056-21PK-Backend23:30:003
330928621080,3sleep00-21swapper/000:30:160
329229821000,2sleep20-21swapper/223:40:222
139099810,68rtkit-daemon0-21swapper/319:09:433
319477927153,13sleep00-21swapper/019:06:530
319357926643,15sleep10-21swapper/119:05:131
32242422500,3sleep30-21swapper/320:30:133
33102772470,4sleep33056-21PK-Backend00:35:003
32437082420,4sleep13056-21PK-Backend21:25:161
139099360,3rtkit-daemon3056-21PK-Backend22:00:123
139099360,3rtkit-daemon3056-21PK-Backend22:00:113
139099350,3rtkit-daemon3056-21PK-Backend22:30:063
319515999318,12cyclictest3300497-21irqstats00:05:180
319515999296,14cyclictest0-21swapper/020:05:360
32193672280,4sleep33056-21PK-Backend20:15:173
3195169992822,4cyclictest42-21ksoftirqd/300:10:013
319515999286,11cyclictest3233416-21python320:55:200
319515999283,11cyclictest3274376-21/usr/sbin/munin22:50:170
319515999275,9cyclictest3247439-21ntp_states21:35:190
319515999275,9cyclictest32346612sh21:00:140
319515999275,9cyclictest32346612sh21:00:140
319515999274,13cyclictest3197817-21/usr/sbin/munin19:15:200
3195165992620,4cyclictest2156-21gnome-shell22:09:082
319515999262,14cyclictest0-21swapper/022:40:120
3195159992611,13cyclictest3262314-21unin-run22:20:010
319515999255,12cyclictest3267547-21chrt22:32:400
319515999254,8cyclictest3305502-21expr00:20:130
319515999254,8cyclictest3238343-21memory21:10:160
319515999253,7cyclictest3248428-21apt21:40:110
139099250,4rtkit-daemon1389-21rtkit-daemon00:35:150
32975562240,3sleep13056-21PK-Backend23:55:221
319516999249,14cyclictest3056-21PK-Backend23:34:053
3195169992411,4cyclictest42-21ksoftirqd/320:55:223
3195165992417,5cyclictest3209697-21cron19:50:012
319516599241,21cyclictest3290194-21python323:35:202
319515999248,14cyclictest3259631-21cat22:10:180
319515999247,15cyclictest1095-21runrttasks20:15:090
319515999244,7cyclictest3298864-21memory00:00:160
319515999244,6cyclictest3236032-21apt21:05:120
319515999242,13cyclictest0-21swapper/020:10:190
319515999241,21cyclictest3265407-21python322:25:210
139099240,16rtkit-daemon3056-21PK-Backend21:52:523
3195169992319,2cyclictest3056-21PK-Backend22:47:113
3195165992321,1cyclictest0-21swapper/219:20:312
319516399230,21cyclictest3285268-21taskset23:21:001
3195159992321,1cyclictest0-21swapper/020:32:270
3195159992316,5cyclictest3290798-21cron23:40:010
319515999231,14cyclictest0-21swapper/022:40:010
139099230,20rtkit-daemon3194-21gmain00:29:210
139099230,15rtkit-daemon0-21swapper/322:40:223
32726832220,4sleep20-21swapper/222:45:212
319516999229,4cyclictest42-21ksoftirqd/319:45:373
3195169992220,1cyclictest0-21swapper/300:22:223
319516999221,9cyclictest3245439-21irqstats21:30:173
3195169992219,2cyclictest3056-21PK-Backend23:45:453
3195169992219,2cyclictest3056-21PK-Backend19:25:073
3195169992219,2cyclictest3056-21PK-Backend00:36:353
319516999220,20cyclictest3056-21PK-Backend23:38:153
319516999220,20cyclictest3056-21PK-Backend20:14:483
319516999220,20cyclictest3056-21PK-Backend20:09:553
3195165992220,1cyclictest0-21swapper/222:40:382
3195165992220,1cyclictest0-21swapper/220:15:332
3195165992214,6cyclictest3286443-21memory23:25:182
3195165992214,6cyclictest3269086-21users22:35:202
3195163992220,1cyclictest0-21swapper/122:41:451
3195163992220,1cyclictest0-21swapper/121:44:411
3195163992219,2cyclictest3056-21PK-Backend21:17:371
3195163992214,7cyclictest3272726-21sed22:45:201
319516399220,20cyclictest3056-21PK-Backend20:50:201
319515999227,13cyclictest3271673-21gmain22:45:020
319515999222,8cyclictest0-21swapper/020:30:000
3195159992220,1cyclictest0-21swapper/021:33:380
139099220,4rtkit-daemon1389-21rtkit-daemon00:15:540
139099220,15rtkit-daemon0-21swapper/220:32:162
139099220,14rtkit-daemon3056-21PK-Backend22:08:513
32934602210,9sleep126-21ksoftirqd/123:45:141
3195169992119,1cyclictest0-21swapper/321:37:563
3195169992119,1cyclictest0-21swapper/300:10:533
3195169992118,2cyclictest3056-21PK-Backend19:53:043
319516999210,20cyclictest3056-21PK-Backend22:29:053
319516999210,20cyclictest0-21swapper/320:45:373
319516999210,19cyclictest3056-21PK-Backend23:53:143
319516999210,19cyclictest3056-21PK-Backend23:09:353
319516999210,19cyclictest3056-21PK-Backend22:39:003
319516999210,19cyclictest3056-21PK-Backend20:21:163
3195165992119,1cyclictest0-21swapper/223:46:022
3195165992119,1cyclictest0-21swapper/223:17:032
3195165992119,1cyclictest0-21swapper/222:23:082
3195165992119,1cyclictest0-21swapper/222:18:292
3195165992119,1cyclictest0-21swapper/221:52:532
3195165992119,1cyclictest0-21swapper/221:47:062
3195165992119,1cyclictest0-21swapper/221:43:142
3195165992119,1cyclictest0-21swapper/221:12:522
3195165992119,1cyclictest0-21swapper/220:46:362
3195165992119,1cyclictest0-21swapper/220:27:222
3195165992119,1cyclictest0-21swapper/219:14:432
3195165992119,1cyclictest0-21swapper/200:19:082
3195165992113,6cyclictest3217825-21python320:10:222
3195165992112,7cyclictest3299736-21cron00:04:592
3195165992112,7cyclictest3231215-21fw_packets20:50:162
319516599210,20cyclictest0-21swapper/220:57:352
319516599210,20cyclictest0-21swapper/219:56:132
319516599210,1cyclictest0-21swapper/219:34:092
319516599210,1cyclictest0-21swapper/219:28:552
319516599210,19cyclictest3056-21PK-Backend20:05:332
3195163992119,1cyclictest3056-21PK-Backend21:05:401
3195163992119,1cyclictest3056-21PK-Backend20:34:521
3195163992119,1cyclictest3056-21PK-Backend00:05:411
3195163992119,1cyclictest0-21swapper/123:39:051
3195163992119,1cyclictest0-21swapper/122:52:411
3195163992119,1cyclictest0-21swapper/122:32:271
3195163992119,1cyclictest0-21swapper/122:05:091
3195163992119,1cyclictest0-21swapper/119:51:391
3195163992119,1cyclictest0-21swapper/100:31:511
3195163992116,3cyclictest3264532-21find22:25:121
3195163992114,6cyclictest3251024-21python321:45:221
3195163992113,6cyclictest3287862-21expr23:30:131
3195163992113,6cyclictest3209696-21cron19:50:011
319516399210,20cyclictest0-21swapper/123:02:141
319516399210,20cyclictest0-21swapper/120:17:211
319516399210,20cyclictest0-21swapper/119:32:051
319516399210,19cyclictest3056-21PK-Backend00:22:231
319516399210,19cyclictest3056-21PK-Backend00:02:441
319516399210,19cyclictest0-21swapper/120:57:361
319515999219,10cyclictest3254465-21/usr/sbin/munin21:55:200
319515999217,12cyclictest3301899-21cpuspeed_turbos00:10:140
319515999213,7cyclictest0-21swapper/022:55:010
3195159992119,1cyclictest0-21swapper/023:55:110
3195159992119,1cyclictest0-21swapper/023:40:120
3195159992119,1cyclictest0-21swapper/023:26:110
3195159992119,1cyclictest0-21swapper/022:06:360
3195159992119,1cyclictest0-21swapper/021:25:330
3195159992119,1cyclictest0-21swapper/020:46:180
3195159992119,1cyclictest0-21swapper/020:37:410
3195159992119,1cyclictest0-21swapper/019:53:470
319515999210,20cyclictest0-21swapper/019:25:350
319515999210,1cyclictest0-21swapper/023:20:300
319515999210,19cyclictest0-21swapper/023:01:460
139099210,15rtkit-daemon3056-21PK-Backend22:56:543
139099210,14rtkit-daemon3056-21PK-Backend19:33:573
139099210,14rtkit-daemon0-21swapper/221:03:532
139099210,14rtkit-daemon0-21swapper/221:03:522
139099210,14rtkit-daemon0-21swapper/219:36:382
139099210,13rtkit-daemon0-21swapper/221:37:032
32810172200,3sleep10-21swapper/123:10:131
3195169992018,1cyclictest3056-21PK-Backend23:00:493
3195169992018,1cyclictest3056-21PK-Backend22:18:053
3195169992018,1cyclictest3056-21PK-Backend21:56:493
3195169992018,1cyclictest0-21swapper/321:19:233
3195169992018,1cyclictest0-21swapper/320:36:593
3195169992018,1cyclictest0-21swapper/300:29:513
319516999201,2cyclictest141rcu_preempt23:15:203
3195169992012,6cyclictest3248797-21/usr/sbin/munin21:40:133
319516999200,19cyclictest3056-21PK-Backend20:28:343
319516999200,19cyclictest3056-21PK-Backend19:14:323
319516999200,19cyclictest0-21swapper/321:25:263
319516999200,19cyclictest0-21swapper/321:24:033
319516999200,19cyclictest0-21swapper/320:53:073
319516999200,19cyclictest0-21swapper/320:43:413
319516999200,19cyclictest0-21swapper/300:16:143
3195165992018,1cyclictest0-21swapper/223:52:112
3195165992018,1cyclictest0-21swapper/222:59:252
3195165992018,1cyclictest0-21swapper/222:54:322
3195165992018,1cyclictest0-21swapper/222:27:482
3195165992018,1cyclictest0-21swapper/221:30:432
3195165992018,1cyclictest0-21swapper/221:09:342
3195165992018,1cyclictest0-21swapper/219:51:332
3195165992018,1cyclictest0-21swapper/200:35:092
3195165992018,1cyclictest0-21swapper/200:12:582
319516599200,1cyclictest0-21swapper/221:59:552
319516599200,1cyclictest0-21swapper/220:22:502
319516599200,19cyclictest0-21swapper/223:14:042
319516599200,19cyclictest0-21swapper/221:26:162
319516599200,19cyclictest0-21swapper/221:23:332
319516599200,19cyclictest0-21swapper/220:42:102
3195163992018,1cyclictest3056-21PK-Backend23:54:061
3195163992018,1cyclictest3056-21PK-Backend21:10:411
3195163992018,1cyclictest0-21swapper/123:42:491
3195163992018,1cyclictest0-21swapper/122:35:451
3195163992018,1cyclictest0-21swapper/122:18:401
3195163992018,1cyclictest0-21swapper/122:01:171
3195163992018,1cyclictest0-21swapper/122:01:161
3195163992018,1cyclictest0-21swapper/121:57:241
3195163992018,1cyclictest0-21swapper/120:20:181
3195163992018,1cyclictest0-21swapper/119:24:281
3195163992018,1cyclictest0-21swapper/119:12:311
3195163992012,6cyclictest3207228-21ntpq19:40:201
319516399200,19cyclictest0-21swapper/123:09:041
319516399200,19cyclictest0-21swapper/122:56:471
319516399200,19cyclictest0-21swapper/122:13:271
319516399200,19cyclictest0-21swapper/120:27:351
319516399200,19cyclictest0-21swapper/120:01:191
319516399200,19cyclictest0-21swapper/119:57:061
319516399200,19cyclictest0-21swapper/119:39:551
319515999209,10cyclictest3196098-21runrttasks19:10:180
319515999208,10cyclictest3231689-21python320:50:210
3195159992018,1cyclictest0-21swapper/023:53:360
3195159992018,1cyclictest0-21swapper/022:02:430
3195159992018,1cyclictest0-21swapper/022:02:430
3195159992018,1cyclictest0-21swapper/021:52:300
3195159992018,1cyclictest0-21swapper/021:45:330
3195159992018,1cyclictest0-21swapper/021:16:130
3195159992018,1cyclictest0-21swapper/020:22:460
3195159992015,3cyclictest3206242-21grep19:40:000
319515999200,1cyclictest0-21swapper/019:40:150
319515999200,19cyclictest0-21swapper/020:40:450
139099200,12rtkit-daemon0-21swapper/222:13:482
319516999194,2cyclictest141rcu_preempt22:20:143
3195169991917,1cyclictest3056-21PK-Backend22:51:173
3195169991917,1cyclictest3056-21PK-Backend21:47:553
319516999191,2cyclictest141rcu_preempt23:40:223
3195169991911,6cyclictest3201043-21cron19:24:593
3195169991910,7cyclictest3298651-21df_inode00:00:123
319516999190,1cyclictest3056-21PK-Backend23:21:313
319516999190,1cyclictest0-21swapper/321:03:083
319516999190,1cyclictest0-21swapper/321:03:083
3195165991917,1cyclictest0-21swapper/223:23:312
3195165991917,1cyclictest0-21swapper/223:09:112
3195165991917,1cyclictest0-21swapper/219:41:262
3195165991917,1cyclictest0-21swapper/200:20:432
319516599190,1cyclictest0-21swapper/220:36:012
319516599190,1cyclictest0-21swapper/200:09:472
3195163991917,1cyclictest3056-21PK-Backend21:38:321
3195163991917,1cyclictest3056-21PK-Backend21:21:291
3195163991917,1cyclictest3056-21PK-Backend20:41:541
3195163991917,1cyclictest3056-21PK-Backend00:13:451
3195163991917,1cyclictest0-21swapper/123:28:031
3195163991917,1cyclictest0-21swapper/122:20:021
3195163991917,1cyclictest0-21swapper/119:28:411
3195163991916,2cyclictest3056-21PK-Backend00:39:071
3195163991916,2cyclictest3056-21PK-Backend00:25:221
3195163991913,4cyclictest3252403-21latency21:50:181
3195163991912,6cyclictest3215917-21ntpq20:05:181
3195163991912,5cyclictest3282670-21cat23:15:111
319515999199,8cyclictest3262712-21cut22:20:120
319515999190,18cyclictest0-21swapper/023:48:160
319515999190,18cyclictest0-21swapper/023:31:540
319515999190,18cyclictest0-21swapper/019:33:120
139099190,13rtkit-daemon0-21swapper/321:12:363
32458292181,3sleep126-21ksoftirqd/121:30:221
3195169991815,2cyclictest3056-21PK-Backend22:10:213
3195169991815,2cyclictest3056-21PK-Backend19:43:123
3195169991812,5cyclictest3297578-21ls23:55:213
3195169991810,6cyclictest42-21ksoftirqd/319:35:133
3195165991816,1cyclictest0-21swapper/223:59:072
3195165991816,1cyclictest0-21swapper/223:30:472
3195165991815,2cyclictest0-21swapper/200:25:162
3195165991812,5cyclictest3240361-21seq21:15:212
3195163991816,1cyclictest3056-21PK-Backend00:18:041
3195163991814,3cyclictest3218418-21sh20:14:591
319515999188,8cyclictest3279169-21expr23:05:130
319515999188,8cyclictest0-21swapper/023:15:110
319515999187,9cyclictest3199844-21cstates19:20:130
3195159991816,1cyclictest0-21swapper/019:48:190
3195159991813,3cyclictest3213651-21cstates20:00:130
3195159991812,5cyclictest3211820-21cat19:55:120
3195159991811,5cyclictest3242969-21/usr/sbin/munin21:25:010
139099180,12rtkit-daemon0-21swapper/223:01:192
139099180,12rtkit-daemon0-21swapper/222:32:312
319516999172,2cyclictest141rcu_preempt21:05:173
319516999170,5cyclictest3056-21PK-Backend19:55:163
3195165991715,1cyclictest0-21swapper/222:03:212
3195165991715,1cyclictest0-21swapper/222:03:212
319516399179,6cyclictest3234300-21apt21:00:131
319516399179,6cyclictest3234300-21apt21:00:121
3195163991715,1cyclictest3056-21PK-Backend20:46:411
3195163991712,3cyclictest3197291-21apt-get19:15:011
319516999161,12cyclictest3214305-21chrt20:00:213
319516999160,8cyclictest3056-21PK-Backend23:10:163
319516999160,3cyclictest3056-21PK-Backend19:15:033
319516599161,7cyclictest3309173-21/usr/sbin/munin00:30:162
3195165991614,1cyclictest0-21swapper/220:00:042
319515999168,7cyclictest3281142-21interrupts23:10:170
319516599133,9cyclictest0-21swapper/219:15:012
319516399139,3cyclictest3056-21PK-Backend20:36:271
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional