You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-19 - 11:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Aug 19, 2026 00:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291503421070,4sleep02915021-21munin-run21:45:010
288262921040,3sleep13056-21PK-Backend20:10:181
295791021030,4sleep13056-21PK-Backend23:45:021
139099830,72rtkit-daemon0-21swapper/019:06:190
286006027537,14sleep20-21swapper/219:09:012
285869326545,14sleep30-21swapper/319:05:123
286013826241,14sleep10-21swapper/119:09:591
28719052530,4sleep13056-21PK-Backend19:40:191
28925862510,4sleep13056-21PK-Backend20:40:141
29329422500,7sleep02932871-21latency_hist22:35:010
29177002490,4sleep13056-21PK-Backend21:50:151
28943112480,1sleep20-21swapper/220:45:122
29181492450,2sleep20-21swapper/221:50:202
29729102410,3sleep13056-21PK-Backend00:25:211
2860312993414,17cyclictest436-20systemd-journal20:50:003
29425382330,6sleep30-21swapper/323:00:183
2860312993214,16cyclictest2019-21gmain19:30:003
2860312993011,16cyclictest2882628-21cat20:10:183
286031299292,4cyclictest0-21swapper/321:20:153
2860312992911,16cyclictest2923859-21sshd22:05:443
2860310992915,4cyclictest34-21ksoftirqd/221:45:012
28897872280,4sleep23056-21PK-Backend20:30:202
286031299280,8cyclictest2928892-21/usr/sbin/munin22:20:233
2860310992813,4cyclictest34-21ksoftirqd/220:40:012
2860304992817,5cyclictest141rcu_preempt23:05:000
286031299279,16cyclictest2936365-21idleruntime-cro22:45:003
286031299278,6cyclictest2940593-21cut22:55:163
286031299278,5cyclictest2877111-21expr19:55:143
286031299275,6cyclictest436-20systemd-journal19:49:553
286031299273,20cyclictest2945556-21apt23:10:133
286031299271,7cyclictest2916812-21apt-key21:50:003
286031299271,4cyclictest436-20systemd-journal23:30:003
2860312992710,15cyclictest42-21ksoftirqd/323:45:243
286031299270,25cyclictest2930226-21ls22:25:173
286031099279,10cyclictest34-21ksoftirqd/223:39:592
286031099274,5cyclictest34-21ksoftirqd/221:50:002
286031299269,3cyclictest2933885-21sh22:35:203
286031299268,5cyclictest2896897-21xargs20:50:223
286031299267,5cyclictest2875261-21cat19:50:133
286031299267,4cyclictest2911994-21cpuspeed_turbos21:35:143
286031299267,4cyclictest2902030-21missed_timers21:05:183
286031299261,5cyclictest436-20systemd-journal20:02:283
286031299261,5cyclictest2968639-21apt00:15:143
286031299260,9cyclictest2885288-21perf20:19:593
286031299260,24cyclictest2943796-21/usr/sbin/munin23:05:203
286031099265,4cyclictest34-21ksoftirqd/200:25:012
2860310992618,6cyclictest2961134-21cron23:55:002
139099260,17rtkit-daemon2897416-21apt-get20:55:023
286031299259,3cyclictest2975777-21cstates00:35:133
286031299257,4cyclictest2932879-21sh22:35:003
286031299256,4cyclictest2953564-21who23:30:243
286031299254,6cyclictest2865477-21cat19:20:233
286031299252,7cyclictest2917253-21apt-get21:50:033
286031299252,5cyclictest2963152-21/usr/sbin/munin00:00:013
286031299251,8cyclictest2950493-21/usr/sbin/munin23:25:003
286031299251,7cyclictest2927718-21latency_hist22:20:013
286031299251,6cyclictest2968487-21latency_hist00:15:013
286031299251,5cyclictest2880367-21apt20:05:173
286031299251,11cyclictest2890763-21apt20:35:133
2860312992511,12cyclictest2948044-21open_files23:15:203
2860312992511,12cyclictest2871505-21cat19:40:143
286031299250,8cyclictest2939835-21idleruntime-cro22:55:003
286031299250,8cyclictest2939835-21idleruntime-cro22:54:593
286031299250,8cyclictest2925340-21chrt22:10:223
286031299250,6cyclictest2905557-21needreboot21:15:183
286031299250,23cyclictest2974196-21df00:30:123
286031299250,11cyclictest978-21snmpd19:13:483
286031299250,10cyclictest2955492-21sshd23:37:453
286031099256,4cyclictest34-21ksoftirqd/222:35:012
2860310992521,3cyclictest34-21ksoftirqd/220:30:012
2860310992512,2cyclictest141rcu_preempt20:10:272
2860310992511,4cyclictest34-21ksoftirqd/200:00:012
2860304992515,3cyclictest13-21ksoftirqd/019:45:010
286031299249,13cyclictest2899658-21/usr/sbin/munin21:00:213
286031299242,8cyclictest2908476-21cut21:25:133
286031299242,4cyclictest2868331-21diskstats19:30:153
286031299241,7cyclictest2972246-21cpuspeed_turbos00:25:143
286031299241,6cyclictest978-21snmpd20:40:133
286031299241,6cyclictest2960625-21threads23:50:223
286031299241,5cyclictest2870427-21ntp_states19:35:203
286031299241,4cyclictest0-21swapper/323:40:113
286031299241,4cyclictest0-21swapper/321:10:133
286031299241,3cyclictest0-21swapper/319:15:123
2860312992410,12cyclictest29222132sleep322:01:483
2860310992414,2cyclictest141rcu_preempt20:00:012
286031099240,4cyclictest34-21ksoftirqd/219:20:012
139099240,3rtkit-daemon0-21swapper/220:57:412
286031299239,3cyclictest1095-21runrttasks00:22:593
286031299232,4cyclictest2914411-21python321:40:203
286031299231,7cyclictest2963480-21grep00:00:133
286031299231,7cyclictest2937202-21/usr/sbin/munin22:45:183
286031299231,7cyclictest2910060-21/usr/sbin/munin21:30:163
286031299231,5cyclictest2920484-21turbostat.cron21:59:593
2860310992319,3cyclictest34-21ksoftirqd/223:05:002
2860310992318,3cyclictest894-21irqbalance00:06:392
29758852220,3sleep20-21swapper/200:35:122
2860310992220,1cyclictest0-21swapper/223:12:122
2860310992220,1cyclictest0-21swapper/221:57:162
2860310992220,1cyclictest0-21swapper/220:05:352
2860310992220,1cyclictest0-21swapper/200:28:032
2860310992217,4cyclictest34-21ksoftirqd/222:42:292
2860310992214,6cyclictest2951824-21users23:25:222
286031099220,3cyclictest2945015-21sshd23:06:442
286031099220,20cyclictest2973948-21cpuspeed_turbos00:30:132
286030699221,4cyclictest141rcu_preempt23:05:001
286030699220,20cyclictest3056-21PK-Backend21:33:221
286030699220,20cyclictest3056-21PK-Backend21:22:341
29550802210,4sleep10-21swapper/123:35:211
286031299210,9cyclictest2886423-21python320:20:213
286031299210,19cyclictest3056-21PK-Backend20:25:053
286031299210,19cyclictest3056-21PK-Backend00:05:153
286031099216,2cyclictest141rcu_preempt23:16:222
286031099213,9cyclictest34-21ksoftirqd/219:50:012
2860310992119,1cyclictest0-21swapper/223:49:502
2860310992119,1cyclictest0-21swapper/222:26:292
2860310992119,1cyclictest0-21swapper/222:13:182
2860310992119,1cyclictest0-21swapper/221:15:182
2860310992119,1cyclictest0-21swapper/221:11:262
2860310992119,1cyclictest0-21swapper/200:10:582
2860310992116,4cyclictest34-21ksoftirqd/200:02:322
286031099210,20cyclictest0-21swapper/222:54:412
286031099210,20cyclictest0-21swapper/222:54:412
286031099210,20cyclictest0-21swapper/221:21:062
286031099210,20cyclictest0-21swapper/220:51:252
2860306992119,1cyclictest0-21swapper/120:29:141
2860306992116,3cyclictest26-21ksoftirqd/122:00:001
2860306992115,4cyclictest2885289-21sh20:20:001
2860306992112,7cyclictest26-21ksoftirqd/122:49:591
286030699210,20cyclictest0-21swapper/122:00:051
286030699210,19cyclictest3056-21PK-Backend23:51:251
286030699210,19cyclictest3056-21PK-Backend20:06:501
286030699210,19cyclictest3056-21PK-Backend19:45:471
286030699210,19cyclictest3056-21PK-Backend19:38:431
2860304992119,1cyclictest0-21swapper/022:25:110
2860304992119,1cyclictest0-21swapper/021:29:000
2860304992115,4cyclictest2901137-21cron21:05:000
2860304992114,5cyclictest13-21ksoftirqd/000:35:210
286030499210,8cyclictest28944711cstates20:45:120
286030499210,20cyclictest0-21swapper/000:17:530
139099210,15rtkit-daemon0-21swapper/221:26:322
139099210,14rtkit-daemon0-21swapper/220:43:382
139099210,14rtkit-daemon0-21swapper/200:15:032
286031099207,4cyclictest34-21ksoftirqd/221:35:002
286031099204,8cyclictest34-21ksoftirqd/221:00:232
2860310992018,1cyclictest0-21swapper/222:36:082
2860310992018,1cyclictest0-21swapper/221:37:352
2860310992018,1cyclictest0-21swapper/221:07:002
2860310992018,1cyclictest0-21swapper/219:55:482
2860310992018,1cyclictest0-21swapper/219:54:472
2860310992018,1cyclictest0-21swapper/219:26:152
2860310992018,1cyclictest0-21swapper/219:22:362
2860310992017,2cyclictest0-21swapper/219:10:062
286031099200,19cyclictest0-21swapper/222:45:352
286031099200,19cyclictest0-21swapper/222:05:332
286031099200,19cyclictest0-21swapper/222:01:422
286031099200,19cyclictest0-21swapper/219:44:262
286031099200,19cyclictest0-21swapper/219:38:592
2860306992018,1cyclictest3056-21PK-Backend22:52:031
2860306992018,1cyclictest3056-21PK-Backend22:52:031
2860306992018,1cyclictest3056-21PK-Backend21:06:581
2860306992018,1cyclictest3056-21PK-Backend00:24:091
2860306992017,2cyclictest3056-21PK-Backend20:36:291
2860306992017,2cyclictest3056-21PK-Backend19:30:061
2860306992015,4cyclictest26-21ksoftirqd/123:35:011
2860306992012,6cyclictest2879483-21python320:00:231
2860306992010,7cyclictest26-21ksoftirqd/122:42:291
286030699200,7cyclictest3056-21PK-Backend20:45:121
286030699200,1cyclictest3056-21PK-Backend22:59:471
286030699200,19cyclictest3056-21PK-Backend21:39:101
286030699200,19cyclictest3056-21PK-Backend21:13:071
286030699200,19cyclictest0-21swapper/123:57:271
286030699200,19cyclictest0-21swapper/120:22:111
286030699200,19cyclictest0-21swapper/100:02:541
2860304992018,1cyclictest0-21swapper/022:55:050
2860304992018,1cyclictest0-21swapper/021:16:490
2860304992018,1cyclictest0-21swapper/020:30:330
2860304992018,1cyclictest0-21swapper/019:58:500
2860304992018,1cyclictest0-21swapper/019:37:000
286030499200,1cyclictest0-21swapper/000:20:020
286030499200,19cyclictest0-21swapper/023:52:160
286030499200,19cyclictest0-21swapper/022:14:430
286030499200,19cyclictest0-21swapper/022:01:590
286030499200,19cyclictest0-21swapper/019:10:570
286030499200,19cyclictest0-21swapper/000:25:420
286030499200,19cyclictest0-21swapper/000:12:590
286030499200,18cyclictest2876600-21cron19:55:000
139099200,14rtkit-daemon3056-21PK-Backend20:32:543
286031099199,2cyclictest141rcu_preempt19:30:132
2860310991917,1cyclictest0-21swapper/223:44:572
2860310991917,1cyclictest0-21swapper/223:20:232
2860310991915,2cyclictest331rcuc/223:32:362
286031099190,1cyclictest0-21swapper/220:24:212
2860306991917,1cyclictest3056-21PK-Backend23:18:471
2860306991917,1cyclictest3056-21PK-Backend21:29:301
2860306991917,1cyclictest3056-21PK-Backend19:28:171
2860306991917,1cyclictest0-21swapper/123:20:091
2860306991917,1cyclictest0-21swapper/120:30:011
2860306991916,2cyclictest3056-21PK-Backend23:26:441
2860306991916,2cyclictest3056-21PK-Backend22:12:021
2860306991911,6cyclictest2923155-21basename22:05:161
286030699190,1cyclictest3056-21PK-Backend22:20:541
286030699190,1cyclictest3056-21PK-Backend21:18:081
286030699190,1cyclictest3056-21PK-Backend19:50:131
286030699190,1cyclictest3056-21PK-Backend00:17:541
286030699190,1cyclictest0-21swapper/100:35:191
286030499199,8cyclictest1-21systemd21:15:000
286030499192,13cyclictest2885559-21apt20:20:110
2860304991917,1cyclictest3056-21PK-Backend20:55:410
2860304991917,1cyclictest0-21swapper/023:17:010
2860304991917,1cyclictest0-21swapper/022:23:480
2860304991917,1cyclictest0-21swapper/021:49:340
2860304991917,1cyclictest0-21swapper/021:35:360
2860304991917,1cyclictest0-21swapper/021:34:140
2860304991917,1cyclictest0-21swapper/019:47:410
2860304991917,1cyclictest0-21swapper/019:31:320
2860304991917,1cyclictest0-21swapper/000:01:150
2860304991912,5cyclictest436-20systemd-journal21:58:440
2860304991912,5cyclictest2946167-21wc23:10:180
2860304991911,6cyclictest2897416-21apt-get20:55:000
286030499190,1cyclictest3056-21PK-Backend23:30:270
286030499190,1cyclictest0-21swapper/023:21:410
286030499190,1cyclictest0-21swapper/022:39:030
286030499190,1cyclictest0-21swapper/020:26:140
286030499190,1cyclictest0-21swapper/020:09:240
286030499190,18cyclictest0-21swapper/023:27:440
286030499190,18cyclictest0-21swapper/023:08:170
286030499190,18cyclictest0-21swapper/021:52:180
286030499190,18cyclictest0-21swapper/019:24:160
286030499190,18cyclictest0-21swapper/000:31:240
2860310991816,1cyclictest0-21swapper/222:16:362
2860306991816,1cyclictest0-21swapper/122:30:341
2860306991815,2cyclictest3056-21PK-Backend21:47:411
286030699181,4cyclictest2865355-21taskset19:20:221
2860306991811,5cyclictest2864223-21cron19:20:001
2860306991810,6cyclictest2965918-21ntpq00:05:231
286030699180,7cyclictest3056-21PK-Backend22:25:131
286030499187,5cyclictest141rcu_preempt21:20:160
2860304991816,1cyclictest3056-21PK-Backend23:57:100
2860304991816,1cyclictest0-21swapper/022:53:430
2860304991816,1cyclictest0-21swapper/022:53:430
2860304991816,1cyclictest0-21swapper/022:40:590
2860304991816,1cyclictest0-21swapper/022:16:050
2860304991816,1cyclictest0-21swapper/022:08:210
2860304991816,1cyclictest0-21swapper/020:37:500
2860304991812,5cyclictest2902316-21sed21:05:220
2860304991811,5cyclictest2884573-21ntpq20:15:210
286030499180,8cyclictest2956895-21chrt23:40:210
286030499180,17cyclictest0-21swapper/023:36:290
139099180,13rtkit-daemon3056-21PK-Backend20:20:002
286031099179,6cyclictest2940272-21expr22:55:122
286031099170,10cyclictest2928740-21ps22:20:212
2860306991715,1cyclictest3056-21PK-Backend19:55:411
2860306991715,1cyclictest0-21swapper/100:10:311
2860306991710,5cyclictest26-21ksoftirqd/120:55:001
286030699170,16cyclictest3056-21PK-Backend23:43:131
2860304991715,1cyclictest0-21swapper/020:44:330
2860304991715,1cyclictest0-21swapper/020:11:480
2860304991715,1cyclictest0-21swapper/019:27:540
2860304991710,6cyclictest2959405-21cron23:50:000
2860306991613,2cyclictest3056-21PK-Backend21:03:211
286030699160,1cyclictest3056-21PK-Backend22:16:271
286030499169,5cyclictest2878821-21cat20:00:140
286030499168,6cyclictest2966640-21munin-run00:10:000
286030699157,2cyclictest26-21ksoftirqd/119:10:221
2860306991513,1cyclictest3056-21PK-Backend23:11:361
2860306991513,1cyclictest3056-21PK-Backend21:44:231
286030699150,3cyclictest2934622-21awk22:40:011
286030699150,2cyclictest141rcu_preempt00:30:131
286030499158,5cyclictest2863455-21awk19:15:190
2860304991513,1cyclictest0-21swapper/022:46:000
286030699141,5cyclictest141rcu_preempt23:09:591
286030699139,2cyclictest26-21ksoftirqd/120:55:241
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional