You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-04 - 09:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Tue Aug 04, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
293065821160,7sleep2341ksoftirqd/221:40:132
290502421030,4sleep23056-21PK-Backend20:25:192
287687628541,13sleep20-21swapper/219:06:112
139099820,65rtkit-daemon0-21swapper/319:05:263
287699127960,14sleep00-21swapper/019:07:300
29750602720,1sleep00-21swapper/023:45:140
287697726242,14sleep10-21swapper/119:07:181
29324272590,3sleep20-21swapper/221:45:152
29834222500,3sleep02983415-21perf00:10:000
29845872480,3sleep00-21swapper/000:10:200
29239562480,3sleep33056-21PK-Backend21:20:143
29904752450,3sleep342-21ksoftirqd/300:30:013
28909532410,3sleep10-21swapper/119:45:321
2877326993811,12cyclictest2919172-21seq21:05:191
29349172370,3sleep3411rcuc/321:50:203
287732699357,15cyclictest2156-21gnome-shell22:55:091
2877326993511,11cyclictest2981640-21seq00:04:501
2877323993519,4cyclictest13-21ksoftirqd/000:05:010
287732699349,9cyclictest2973001-21munin-run23:39:591
287732699349,9cyclictest2973001-21munin-run23:39:591
287732699338,8cyclictest2905372-21taskset20:25:231
2877326993312,11cyclictest2970935-21runrttasks23:31:021
2877326993310,9cyclictest2907505-21awk20:35:001
287732699327,12cyclictest0-21swapper/120:38:201
2877326993211,8cyclictest1-21systemd23:13:001
2877326993210,11cyclictest2988546-21chrt00:24:171
287732699318,9cyclictest2885089-21gpgv19:30:001
287732699317,14cyclictest2916694-21taskset21:00:051
287732699316,13cyclictest0-21swapper/120:45:261
287732699316,13cyclictest0-21swapper/120:45:251
2877326993116,12cyclictest2979129-21needreboot23:55:181
287732699308,9cyclictest0-21swapper/119:38:041
287732699308,10cyclictest0-21swapper/120:42:571
2877326993013,15cyclictest2983855-21cpuspeed_turbos00:10:121
2877326993010,6cyclictest2900567-21awk20:15:001
287732699299,11cyclictest2945758-21taskset22:22:461
287732699297,9cyclictest2895174-21taskset19:57:571
287732699297,10cyclictest2956601-21seq22:51:491
287732699296,8cyclictest2993636-21seq00:37:481
287732699294,8cyclictest0-21swapper/120:15:121
2877326992912,14cyclictest883-21dbus-daemon22:31:441
287732699286,9cyclictest2981914-21/usr/sbin/munin00:05:171
287732699286,9cyclictest2978325-21cat23:55:011
287732699284,14cyclictest0-21swapper/121:55:121
287732699276,8cyclictest0-21swapper/120:52:011
287732699273,14cyclictest0-21swapper/119:20:131
287733399268,4cyclictest42-21ksoftirqd/321:00:233
287733399261,2cyclictest3056-21PK-Backend20:35:013
287733399260,2cyclictest3056-21PK-Backend19:49:593
287732699266,9cyclictest2991066-21if_enp2s000:30:161
287732699265,13cyclictest0-21swapper/123:05:121
287732699264,9cyclictest0-21swapper/121:15:121
287732699264,8cyclictest0-21swapper/122:45:041
2877326992613,11cyclictest2931103-21kernelversion21:40:181
287733399258,4cyclictest42-21ksoftirqd/323:44:523
287733399258,4cyclictest42-21ksoftirqd/320:25:353
287733399258,4cyclictest42-21ksoftirqd/320:05:423
287733399258,4cyclictest42-21ksoftirqd/319:35:433
287733399258,4cyclictest42-21ksoftirqd/300:16:113
2877333992511,4cyclictest42-21ksoftirqd/322:23:273
2877333992511,4cyclictest42-21ksoftirqd/321:07:213
287732699257,6cyclictest2896848-21taskset20:02:091
287732699256,8cyclictest2930064-21runrttasks21:36:461
287732699254,8cyclictest0-21swapper/120:10:001
2877326992516,6cyclictest2933203-21python321:45:231
2877326992513,7cyclictest2920205-21apt21:10:111
29416902240,4sleep30-21swapper/322:10:183
287733399249,4cyclictest42-21ksoftirqd/323:55:433
287733399249,4cyclictest42-21ksoftirqd/323:47:163
287733399243,9cyclictest42-21ksoftirqd/300:14:033
287732699247,9cyclictest2969462-21runrttasks23:29:271
287732699246,6cyclictest2935276-21runrttasks21:51:571
287732699244,8cyclictest0-21swapper/123:40:121
2877326992416,5cyclictest2976471-21idleruntime-cro23:50:001
2877326992415,6cyclictest2877500-21/usr/sbin/munin19:10:171
2877326992415,5cyclictest2937554-21apt22:00:131
2877323992412,4cyclictest13-21ksoftirqd/020:05:110
287733399239,4cyclictest42-21ksoftirqd/323:10:243
2877333992310,3cyclictest42-21ksoftirqd/322:47:173
287733399230,2cyclictest141rcu_preempt19:15:003
287733399230,22cyclictest3056-21PK-Backend20:49:103
287733399230,22cyclictest3056-21PK-Backend20:49:103
2877328992310,3cyclictest141rcu_preempt22:15:012
2877326992315,6cyclictest2951392-21/usr/sbin/munin22:40:161
2877326992314,7cyclictest2986031-21missed_timers00:15:201
2877326992314,7cyclictest2925588-21grep21:25:141
2877326992311,9cyclictest2946193-21/usr/sbin/munin22:25:181
2877326992311,9cyclictest0-21swapper/123:15:121
2877326992310,11cyclictest2883339-21cpuspeed_turbos19:25:121
139099230,14rtkit-daemon0-21swapper/220:53:362
287733399228,4cyclictest42-21ksoftirqd/320:35:323
287733399227,4cyclictest42-21ksoftirqd/321:47:233
287733399222,7cyclictest42-21ksoftirqd/320:00:253
2877333992220,1cyclictest3056-21PK-Backend22:35:363
2877333992220,1cyclictest3056-21PK-Backend22:35:363
2877333992220,1cyclictest3056-21PK-Backend19:54:143
2877333992219,2cyclictest3056-21PK-Backend20:59:433
2877333992219,2cyclictest3056-21PK-Backend19:31:433
2877333992210,4cyclictest42-21ksoftirqd/319:40:403
287733399220,21cyclictest3056-21PK-Backend22:53:263
287733399220,20cyclictest3056-21PK-Backend22:40:553
287733399220,20cyclictest3056-21PK-Backend00:34:193
287733399220,12cyclictest2962683-21latency_hist23:10:013
2877328992220,1cyclictest0-21swapper/223:38:192
2877328992220,1cyclictest0-21swapper/223:38:192
2877328992220,1cyclictest0-21swapper/222:36:212
2877328992220,1cyclictest0-21swapper/222:36:202
2877328992220,1cyclictest0-21swapper/221:01:572
2877328992220,1cyclictest0-21swapper/219:55:322
2877328992220,1cyclictest0-21swapper/219:20:582
2877328992219,2cyclictest3056-21PK-Backend22:30:252
2877328992214,7cyclictest883-21dbus-daemon19:17:162
287732899220,21cyclictest0-21swapper/220:58:472
2877326992215,5cyclictest2928615-21perf21:35:001
2877326992215,4cyclictest436-20systemd-journal22:15:001
2877326992214,6cyclictest2990141-21sshd00:27:331
2877326992214,6cyclictest2916489-21unin-run21:00:001
2877326992214,5cyclictest2939297-21cpu22:05:131
2877326992213,7cyclictest2949861-21cpuspeed_turbos22:35:111
2877326992213,7cyclictest2949861-21cpuspeed_turbos22:35:111
2877326992213,7cyclictest2925160-21chrt21:25:001
2877326992213,7cyclictest2903352-21wc20:20:211
2877326992213,7cyclictest2880998-2101-ifupdown19:17:171
2877326992212,8cyclictest2967789-21idleruntime-cro23:25:011
2877326992212,8cyclictest2943654-21/usr/sbin/munin22:15:221
287732399227,4cyclictest13-21ksoftirqd/000:00:010
2877323992214,7cyclictest2989750-21python300:25:220
139099220,15rtkit-daemon3056-21PK-Backend00:10:142
139099220,14rtkit-daemon3056-21PK-Backend00:36:203
139099220,14rtkit-daemon0-21swapper/200:34:392
287733399218,3cyclictest42-21ksoftirqd/321:13:353
2877333992119,1cyclictest3056-21PK-Backend23:21:453
2877333992119,1cyclictest3056-21PK-Backend19:28:183
2877333992118,2cyclictest3056-21PK-Backend21:42:023
2877333992117,2cyclictest29005152sleep320:14:483
2877333992117,2cyclictest28952942sleep319:59:303
2877333992116,3cyclictest29813322sleep300:00:543
287733399210,9cyclictest42-21ksoftirqd/322:00:213
287733399210,6cyclictest3056-21PK-Backend22:30:183
287733399210,20cyclictest3056-21PK-Backend23:27:483
287733399210,20cyclictest3056-21PK-Backend21:33:163
287733399210,19cyclictest3056-21PK-Backend22:15:213
287732899216,2cyclictest141rcu_preempt19:15:002
2877328992119,1cyclictest0-21swapper/223:56:232
2877328992119,1cyclictest0-21swapper/223:49:132
2877328992119,1cyclictest0-21swapper/222:53:302
2877328992119,1cyclictest0-21swapper/222:27:282
2877328992119,1cyclictest0-21swapper/222:06:052
2877328992119,1cyclictest0-21swapper/221:36:042
2877328992119,1cyclictest0-21swapper/220:18:312
2877328992119,1cyclictest0-21swapper/219:51:542
2877328992119,1cyclictest0-21swapper/219:31:132
2877328992119,1cyclictest0-21swapper/200:36:262
2877328992119,1cyclictest0-21swapper/200:25:372
287732899210,20cyclictest2909213-21apt-get20:40:012
287732899210,20cyclictest0-21swapper/222:04:022
287732899210,20cyclictest0-21swapper/200:08:212
2877326992114,5cyclictest2960626-21unin-run23:05:001
2877326992114,4cyclictest2892518-21/usr/sbin/munin19:50:131
2877326992112,7cyclictest2888982-21cstates19:40:151
287732399213,2cyclictest141rcu_preempt19:30:010
2877323992115,4cyclictest13-21ksoftirqd/022:35:010
2877323992114,6cyclictest2906379-21df20:30:120
287732399210,20cyclictest0-21swapper/023:50:520
287732399210,20cyclictest0-21swapper/022:25:420
287732399210,20cyclictest0-21swapper/022:21:490
287732399210,20cyclictest0-21swapper/021:01:330
287732399210,1cyclictest0-21swapper/020:51:180
139099210,14rtkit-daemon3056-21PK-Backend23:30:313
139099210,14rtkit-daemon3056-21PK-Backend20:53:143
2877333992016,2cyclictest29603292sleep323:01:513
287733399200,5cyclictest3056-21PK-Backend19:18:083
287733399200,1cyclictest3056-21PK-Backend00:09:583
287733399200,19cyclictest3056-21PK-Backend21:15:053
287733399200,19cyclictest3056-21PK-Backend20:23:553
287733399200,19cyclictest3056-21PK-Backend19:24:063
287733399200,19cyclictest0-21swapper/321:25:393
2877328992018,1cyclictest0-21swapper/223:54:072
2877328992018,1cyclictest0-21swapper/222:56:272
2877328992018,1cyclictest0-21swapper/222:48:102
2877328992018,1cyclictest0-21swapper/221:18:072
2877328992018,1cyclictest0-21swapper/220:21:142
2877328992018,1cyclictest0-21swapper/220:01:072
2877328992018,1cyclictest0-21swapper/200:22:322
2877328992018,1cyclictest0-21swapper/200:15:502
287732899200,19cyclictest0-21swapper/223:43:472
287732899200,19cyclictest0-21swapper/223:31:432
287732899200,19cyclictest0-21swapper/222:22:082
287732899200,19cyclictest0-21swapper/221:56:402
287732899200,19cyclictest0-21swapper/221:24:152
287732899200,19cyclictest0-21swapper/220:11:562
287732899200,19cyclictest0-21swapper/219:47:222
287732399205,7cyclictest13-21ksoftirqd/020:45:220
287732399205,7cyclictest13-21ksoftirqd/020:45:220
287732399205,1cyclictest141rcu_preempt22:50:130
2877323992018,1cyclictest0-21swapper/023:31:530
2877323992018,1cyclictest0-21swapper/023:24:420
2877323992018,1cyclictest0-21swapper/023:17:460
2877323992018,1cyclictest0-21swapper/023:03:410
2877323992018,1cyclictest0-21swapper/021:53:030
2877323992018,1cyclictest0-21swapper/020:37:120
2877323992018,1cyclictest0-21swapper/020:03:400
2877323992018,1cyclictest0-21swapper/019:13:100
2877323992018,1cyclictest0-21swapper/000:23:240
2877323992014,5cyclictest13-21ksoftirqd/021:57:180
2877323992014,4cyclictest2974708-21cron23:44:590
2877323992013,6cyclictest2939661-21diskstats22:05:160
2877323992012,6cyclictest2943151-21idleruntime22:15:150
2877323992012,6cyclictest2919525-21seq21:05:230
2877323992012,6cyclictest2884689-21apt-get19:30:010
287732399200,19cyclictest0-21swapper/023:25:510
287732399200,19cyclictest0-21swapper/023:10:430
287732399200,19cyclictest0-21swapper/023:08:410
287732399200,19cyclictest0-21swapper/022:37:520
287732399200,19cyclictest0-21swapper/022:37:520
287732399200,19cyclictest0-21swapper/021:46:480
287732399200,19cyclictest0-21swapper/021:14:170
287732399200,19cyclictest0-21swapper/020:19:080
287732399200,19cyclictest0-21swapper/019:35:210
287732399200,19cyclictest0-21swapper/000:37:490
139099200,13rtkit-daemon0-21swapper/219:29:112
139099200,12rtkit-daemon0-21swapper/223:17:302
287733399194,2cyclictest141rcu_preempt23:40:003
287733399194,2cyclictest141rcu_preempt23:40:003
287733399190,9cyclictest42-21ksoftirqd/320:40:133
287733399190,7cyclictest42-21ksoftirqd/322:10:013
287733399190,1cyclictest3056-21PK-Backend21:56:213
2877328991917,1cyclictest3056-21PK-Backend23:22:502
2877328991917,1cyclictest0-21swapper/223:12:582
287732899190,1cyclictest0-21swapper/220:47:042
287732899190,1cyclictest0-21swapper/220:47:032
287732899190,1cyclictest0-21swapper/220:06:082
287732899190,18cyclictest0-21swapper/221:05:082
287732899190,18cyclictest0-21swapper/220:42:372
2877323991917,1cyclictest0-21swapper/021:44:250
2877323991917,1cyclictest0-21swapper/021:36:000
2877323991917,1cyclictest0-21swapper/019:54:280
2877323991912,5cyclictest2992067-21cron00:34:590
2877323991912,5cyclictest2954335-21sed22:45:240
2877323991912,5cyclictest2952866-21cron22:44:590
287732399190,1cyclictest0-21swapper/021:25:590
287732399190,18cyclictest0-21swapper/021:22:000
287732399190,18cyclictest0-21swapper/021:15:250
287732399190,18cyclictest0-21swapper/020:58:550
287732399190,18cyclictest0-21swapper/020:10:010
287732399190,18cyclictest0-21swapper/019:43:110
287732399190,18cyclictest0-21swapper/000:15:320
139099190,12rtkit-daemon0-21swapper/223:29:412
139099190,11rtkit-daemon0-21swapper/219:42:092
2877333991816,1cyclictest3056-21PK-Backend23:18:343
287733399180,5cyclictest3056-21PK-Backend23:50:163
287733399180,5cyclictest3056-21PK-Backend21:35:203
287733399180,3cyclictest3056-21PK-Backend22:25:233
2877328991816,1cyclictest0-21swapper/221:26:592
2877328991816,1cyclictest0-21swapper/220:32:442
2877328991816,1cyclictest0-21swapper/219:38:162
2877323991816,1cyclictest0-21swapper/022:12:170
2877323991816,1cyclictest0-21swapper/020:42:270
2877323991811,5cyclictest2904281-21apt20:25:110
287733399172,13cyclictest2900793-21/usr/sbin/munin20:15:173
2877333991715,1cyclictest3056-21PK-Backend22:57:183
287732899176,3cyclictest141rcu_preempt00:00:122
2877328991715,1cyclictest0-21swapper/221:51:192
2877328991711,4cyclictest2928598-21sh21:34:592
287732899170,7cyclictest2944229-21munin-run22:20:012
287732899170,6cyclictest2962315-21cron23:10:012
287732399176,3cyclictest13-21ksoftirqd/019:20:200
2877323991715,1cyclictest0-21swapper/019:48:400
2877323991711,5cyclictest2937534-21basename22:00:120
2877323991711,5cyclictest2903276-21ntpq20:20:200
2877323991711,5cyclictest2895433-21ls20:00:010
2877323991710,5cyclictest2880456-21ntpq19:15:200
29873032160,3sleep33056-21PK-Backend00:20:143
2877328991614,1cyclictest0-21swapper/223:00:542
2877328991612,3cyclictest2952894-21perf22:45:002
287732899160,6cyclictest2921704-21latency_hist21:15:002
287732399169,6cyclictest2971771-21cat23:35:140
287732399169,6cyclictest2971771-21cat23:35:130
287732399169,5cyclictest2957396-21cat22:55:120
287732399163,2cyclictest141rcu_preempt21:30:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional