You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-07 - 12:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Thu May 07, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
284804121210,3sleep13056-21PK-Backend22:50:211
284524921130,2sleep30-21swapper/322:45:023
276872327438,13sleep10-21swapper/119:08:391
139099730,63rtkit-daemon0-21swapper/019:09:560
276859726745,15sleep30-21swapper/319:07:013
276876626243,14sleep20-21swapper/219:09:122
27931192480,4sleep00-21swapper/020:15:170
28654502460,3sleep33056-21PK-Backend23:40:233
28170202440,4sleep12817012-21munin-run21:25:001
28170202440,4sleep12817012-21munin-run21:25:001
28693922420,2sleep10-21swapper/123:55:011
139099380,5rtkit-daemon1788643-21eadedCompositor20:09:160
28632082370,3sleep10-21swapper/123:35:161
139099360,4rtkit-daemon1788643-21eadedCompositor20:29:230
139099330,2rtkit-daemon13-21ksoftirqd/021:24:570
139099330,2rtkit-daemon13-21ksoftirqd/021:24:570
2768968993113,10cyclictest26-21ksoftirqd/120:15:001
2768972992813,4cyclictest34-21ksoftirqd/223:15:002
139099280,4rtkit-daemon1389-21rtkit-daemon21:27:520
139099270,4rtkit-daemon1389-21rtkit-daemon23:43:490
28522102260,4chrt0-21swapper/023:05:120
28514522260,4sleep33056-21PK-Backend23:00:213
2768972992614,10cyclictest2791964-21unin-run20:15:002
2768968992623,2cyclictest251rcuc/120:29:591
276896899261,7cyclictest2827406-21munin-run21:55:001
28745182250,6sleep12874523-21munin-plugin-st00:10:011
28137292250,3sleep02813733-21latency_hist21:15:000
276896899255,4cyclictest2774860-21apt19:25:121
276896899254,12cyclictest2855312-21seq23:13:461
276896899251,9cyclictest2829543-21grep22:00:131
276896899251,23cyclictest251rcuc/100:24:551
276896899250,7cyclictest26-21ksoftirqd/122:45:011
276896899250,6cyclictest26-21ksoftirqd/123:00:011
2768966992517,2cyclictest141rcu_preempt22:25:000
276896699250,2cyclictest141rcu_preempt20:15:000
276897699240,4cyclictest2874837-21cat00:10:113
276897299246,4cyclictest34-21ksoftirqd/223:50:012
276896899246,5cyclictest26-21ksoftirqd/122:00:011
2768968992421,2cyclictest251rcuc/123:08:031
139099240,20rtkit-daemon945-21thermald00:35:141
276897699230,1cyclictest3056-21PK-Backend20:00:013
276897299237,3cyclictest141rcu_preempt19:50:142
2768972992315,2cyclictest34-21ksoftirqd/200:25:002
276897299231,3cyclictest2783278-21munin-run19:50:002
2768972992313,7cyclictest883-21dbus-daemon22:11:352
276896899234,5cyclictest2865222-21python323:40:201
276896899233,3cyclictest26-21ksoftirqd/121:50:011
276896899232,6cyclictest26-21ksoftirqd/100:00:011
276896899231,20cyclictest0-21swapper/123:25:001
276896899231,20cyclictest0-21swapper/123:24:591
276896899231,12cyclictest2841285-21munin-run22:35:001
276896899230,21cyclictest2792711-21expr20:15:141
276896699230,21cyclictest2858009-21memory23:20:160
276896699230,21cyclictest2858009-21memory23:20:160
2768976992211,8cyclictest42-21ksoftirqd/322:55:193
276897299227,4cyclictest34-21ksoftirqd/200:30:012
276897299227,2cyclictest141rcu_preempt21:25:002
276897299227,2cyclictest141rcu_preempt21:25:002
276897299226,3cyclictest141rcu_preempt19:25:132
2768972992213,7cyclictest34-21ksoftirqd/223:50:192
2768972992213,7cyclictest34-21ksoftirqd/222:55:202
2768972992213,7cyclictest34-21ksoftirqd/220:00:192
2768972992213,7cyclictest34-21ksoftirqd/200:14:092
2768972992213,3cyclictest34-21ksoftirqd/221:30:162
276897299220,11cyclictest2865812-21cron23:45:002
276896899225,6cyclictest26-21ksoftirqd/120:00:201
276896899225,5cyclictest26-21ksoftirqd/120:30:111
276896899223,17cyclictest2813542-21turbostat.cron21:15:001
276896899223,10cyclictest26-21ksoftirqd/100:00:121
2768968992220,1cyclictest0-21swapper/119:48:231
2768968992220,1cyclictest0-21swapper/100:32:401
2768968992218,3cyclictest251rcuc/119:37:231
276896899221,7cyclictest26-21ksoftirqd/122:05:161
2768968992212,3cyclictest26-21ksoftirqd/120:53:311
276896899220,6cyclictest3056-21PK-Backend22:47:091
276896899220,6cyclictest2857172-21perf23:20:001
276896899220,21cyclictest0-21swapper/100:17:461
2768966992213,2cyclictest13-21ksoftirqd/019:45:190
276897699214,15cyclictest2873963-21sed00:05:203
2768976992113,6cyclictest2871106-21systemd00:00:013
276897699210,20cyclictest3056-21PK-Backend20:04:403
276897699210,20cyclictest0-21swapper/323:26:383
276897699210,19cyclictest3056-21PK-Backend21:26:193
276897699210,19cyclictest3056-21PK-Backend20:42:263
276897299217,3cyclictest141rcu_preempt21:55:002
276897299217,2cyclictest141rcu_preempt20:30:222
276897299215,4cyclictest34-21ksoftirqd/220:20:002
2768972992119,1cyclictest0-21swapper/200:00:062
2768972992113,7cyclictest34-21ksoftirqd/223:05:142
2768972992113,7cyclictest34-21ksoftirqd/200:05:112
2768972992113,6cyclictest34-21ksoftirqd/220:55:112
2768972992113,2cyclictest34-21ksoftirqd/220:25:002
2768972992113,2cyclictest34-21ksoftirqd/219:35:002
2768972992112,7cyclictest34-21ksoftirqd/223:30:122
2768972992112,7cyclictest34-21ksoftirqd/222:35:152
2768972992112,7cyclictest34-21ksoftirqd/222:20:172
2768972992112,7cyclictest34-21ksoftirqd/222:15:132
2768972992112,7cyclictest34-21ksoftirqd/222:00:202
2768972992112,7cyclictest34-21ksoftirqd/221:55:192
2768972992112,7cyclictest34-21ksoftirqd/221:45:132
2768972992112,7cyclictest34-21ksoftirqd/221:25:192
2768972992112,7cyclictest34-21ksoftirqd/221:15:162
2768972992112,7cyclictest34-21ksoftirqd/221:10:172
2768972992112,7cyclictest34-21ksoftirqd/200:30:172
2768972992112,7cyclictest34-21ksoftirqd/200:15:092
2768972992112,6cyclictest34-21ksoftirqd/222:45:042
2768972992112,3cyclictest34-21ksoftirqd/222:45:002
2768972992112,2cyclictest34-21ksoftirqd/219:15:022
276896899215,6cyclictest26-21ksoftirqd/119:42:531
276896899213,7cyclictest26-21ksoftirqd/120:40:101
2768968992119,1cyclictest3056-21PK-Backend21:40:031
2768968992119,1cyclictest0-21swapper/123:49:341
2768968992119,1cyclictest0-21swapper/123:04:461
2768968992118,2cyclictest251rcuc/119:33:021
2768968992117,3cyclictest251rcuc/119:59:291
276896899211,6cyclictest2769844-21cut19:10:161
2768968992114,5cyclictest2881439-21cron00:30:001
2768968992111,7cyclictest2804863-21cron20:50:001
276896899211,17cyclictest2795655-21dump-pmu-power20:25:001
276896899210,9cyclictest2822195-21sh21:40:001
276896899210,6cyclictest2784814-21sshd19:52:481
276896899210,20cyclictest0-21swapper/121:17:521
2768966992119,1cyclictest0-21swapper/019:41:290
2768966992116,3cyclictest2864102-21munin-run23:40:000
2768966992114,6cyclictest2827400-21cron21:55:000
2768966992112,3cyclictest13-21ksoftirqd/020:00:190
2768966992112,2cyclictest13-21ksoftirqd/023:50:210
276896699210,2cyclictest141rcu_preempt00:00:010
139099210,4rtkit-daemon1389-21rtkit-daemon00:06:280
2768976992018,1cyclictest3056-21PK-Backend21:54:313
2768976992018,1cyclictest0-21swapper/322:09:523
2768976992017,2cyclictest3056-21PK-Backend23:34:493
2768976992015,3cyclictest2883190-21dump-pmu-power00:35:003
276897699201,2cyclictest141rcu_preempt22:15:003
2768976992010,8cyclictest2835608-21cat22:15:213
276897699200,1cyclictest3056-21PK-Backend21:16:393
276897699200,1cyclictest3056-21PK-Backend19:29:053
276897699200,19cyclictest3056-21PK-Backend20:21:173
276897699200,19cyclictest3056-21PK-Backend20:05:423
276897699200,19cyclictest3056-21PK-Backend00:23:023
276897699200,19cyclictest0-21swapper/321:48:163
276897699200,19cyclictest0-21swapper/320:53:083
276897699200,19cyclictest0-21swapper/300:00:113
276897699200,18cyclictest3056-21PK-Backend19:41:353
276897299207,2cyclictest141rcu_preempt23:00:452
276897299202,3cyclictest141rcu_preempt20:40:152
2768972992018,1cyclictest0-21swapper/223:22:212
2768972992018,1cyclictest0-21swapper/223:22:212
2768972992018,1cyclictest0-21swapper/220:35:452
2768972992018,1cyclictest0-21swapper/219:23:532
2768972992018,1cyclictest0-21swapper/219:23:532
2768972992012,7cyclictest34-21ksoftirqd/200:00:012
2768972992012,6cyclictest34-21ksoftirqd/223:25:092
2768972992012,6cyclictest34-21ksoftirqd/222:50:112
2768972992012,6cyclictest34-21ksoftirqd/222:25:132
2768972992012,6cyclictest34-21ksoftirqd/220:55:002
2768972992012,6cyclictest34-21ksoftirqd/220:05:202
2768972992012,6cyclictest34-21ksoftirqd/219:55:202
2768972992012,6cyclictest34-21ksoftirqd/200:35:092
2768972992012,2cyclictest34-21ksoftirqd/220:25:112
2768972992011,3cyclictest34-21ksoftirqd/221:05:122
2768972992011,3cyclictest34-21ksoftirqd/219:35:212
276897299200,1cyclictest0-21swapper/220:47:282
276897299200,19cyclictest0-21swapper/222:34:562
276897299200,19cyclictest0-21swapper/221:43:522
276896899205,8cyclictest26-21ksoftirqd/123:25:211
276896899203,8cyclictest26-21ksoftirqd/119:15:141
2768968992018,1cyclictest3056-21PK-Backend22:23:291
2768968992018,1cyclictest0-21swapper/120:06:271
2768968992017,2cyclictest251rcuc/121:05:561
2768968992017,2cyclictest251rcuc/120:39:301
2768968992017,2cyclictest251rcuc/100:12:511
276896899200,6cyclictest3056-21PK-Backend22:35:191
276896899200,5cyclictest3056-21PK-Backend22:15:121
276896899200,1cyclictest3056-21PK-Backend22:27:081
276896899200,1cyclictest3056-21PK-Backend22:12:211
276896899200,19cyclictest3056-21PK-Backend21:02:111
276896899200,19cyclictest3056-21PK-Backend20:59:341
276896699203,3cyclictest2776330-21turbostat19:30:010
2768966992018,1cyclictest0-21swapper/023:15:090
2768966992018,1cyclictest0-21swapper/021:49:590
2768966992018,1cyclictest0-21swapper/000:01:260
276896699200,8cyclictest2795648-21cron20:25:010
276896699200,19cyclictest0-21swapper/023:26:250
276896699200,19cyclictest0-21swapper/021:03:490
276896699200,19cyclictest0-21swapper/020:51:460
276896699200,18cyclictest2811802-21perf21:10:000
139099200,3rtkit-daemon1389-21rtkit-daemon23:46:370
2768976991917,1cyclictest3056-21PK-Backend23:24:353
2768976991917,1cyclictest3056-21PK-Backend23:24:353
2768976991917,1cyclictest3056-21PK-Backend19:11:553
2768976991917,1cyclictest0-21swapper/322:21:153
2768976991917,1cyclictest0-21swapper/321:41:133
2768976991917,1cyclictest0-21swapper/321:00:583
2768976991916,2cyclictest3056-21PK-Backend23:50:103
2768976991916,2cyclictest3056-21PK-Backend23:15:363
2768976991916,2cyclictest3056-21PK-Backend21:21:053
2768976991916,2cyclictest3056-21PK-Backend21:21:053
2768976991916,2cyclictest3056-21PK-Backend20:30:563
2768976991916,2cyclictest3056-21PK-Backend19:35:203
276897699191,5cyclictest2840327-21wc22:30:163
2768976991911,6cyclictest2811200-21python321:05:213
276897699190,1cyclictest3056-21PK-Backend23:46:053
276897699190,1cyclictest3056-21PK-Backend21:57:083
276897699190,1cyclictest3056-21PK-Backend21:11:253
276897699190,1cyclictest3056-21PK-Backend00:27:563
276897699190,18cyclictest3056-21PK-Backend22:01:343
276897699190,18cyclictest3056-21PK-Backend19:45:143
276897699190,18cyclictest0-21swapper/322:36:153
276897699190,18cyclictest0-21swapper/320:18:403
276897299199,6cyclictest34-21ksoftirqd/221:35:162
2768972991917,1cyclictest0-21swapper/219:40:162
2768972991911,2cyclictest34-21ksoftirqd/223:39:092
2768972991911,2cyclictest34-21ksoftirqd/223:15:192
2768968991917,1cyclictest3056-21PK-Backend23:30:071
276896899190,7cyclictest26-21ksoftirqd/119:24:131
276896899190,7cyclictest26-21ksoftirqd/119:24:121
276896899190,1cyclictest0-21swapper/121:31:111
276896699195,2cyclictest141rcu_preempt22:50:120
2768966991917,1cyclictest0-21swapper/023:03:060
2768966991917,1cyclictest0-21swapper/021:40:200
2768966991917,1cyclictest0-21swapper/021:31:270
2768966991917,1cyclictest0-21swapper/020:46:180
2768966991917,1cyclictest0-21swapper/020:40:510
2768966991917,1cyclictest0-21swapper/020:39:490
2768966991917,1cyclictest0-21swapper/000:25:330
2768966991917,1cyclictest0-21swapper/000:15:320
2768966991917,1cyclictest0-21swapper/000:13:300
2768966991916,2cyclictest3056-21PK-Backend21:35:530
2768966991913,5cyclictest2834031-2101-ifupdown22:11:130
2768966991913,4cyclictest2786722-21cron20:00:000
2768966991911,6cyclictest2808346-21munin-run21:00:000
276896699190,1cyclictest0-21swapper/022:49:140
276896699190,1cyclictest0-21swapper/022:43:190
276896699190,1cyclictest0-21swapper/022:36:230
276896699190,1cyclictest0-21swapper/022:33:320
276896699190,1cyclictest0-21swapper/022:16:420
276896699190,1cyclictest0-21swapper/021:17:070
276896699190,1cyclictest0-21swapper/019:14:180
276896699190,1cyclictest0-21swapper/000:30:260
276896699190,18cyclictest0-21swapper/022:57:110
276896699190,18cyclictest0-21swapper/019:38:380
276896699190,18cyclictest0-21swapper/000:37:220
2768976991816,1cyclictest3056-21PK-Backend23:12:323
2768976991816,1cyclictest3056-21PK-Backend20:35:233
2768976991816,1cyclictest3056-21PK-Backend19:23:033
2768976991816,1cyclictest3056-21PK-Backend19:23:033
276897699181,3cyclictest0-21swapper/322:50:113
2768976991810,6cyclictest2844760-21apt-get22:45:003
2768972991816,1cyclictest0-21swapper/219:10:082
2768968991816,1cyclictest0-21swapper/121:25:571
276896699189,7cyclictest2799215-21anacron20:32:460
2768966991816,1cyclictest0-21swapper/022:27:030
2768966991816,1cyclictest0-21swapper/022:09:120
2768966991813,3cyclictest2830128-21ntp_offset22:00:190
2768966991813,3cyclictest2778060-21latency_hist19:35:000
2768966991811,6cyclictest2827648-21cat21:55:110
276897699177,8cyclictest42-21ksoftirqd/323:35:163
276897699177,2cyclictest141rcu_preempt20:15:003
2768976991715,1cyclictest3056-21PK-Backend20:29:073
2768976991715,1cyclictest0-21swapper/322:28:453
2768976991715,1cyclictest0-21swapper/321:36:473
2768976991715,1cyclictest0-21swapper/320:58:213
276897699170,6cyclictest3056-21PK-Backend19:15:253
2768972991711,4cyclictest34-21ksoftirqd/222:10:002
2768972991710,5cyclictest2809274-21strings21:00:182
2768966991713,3cyclictest2772177-21proc_pri19:15:180
2768966991712,3cyclictest2156-21gnome-shell19:23:090
2768966991712,3cyclictest2156-21gnome-shell19:23:090
2768966991710,5cyclictest2855302-21seq23:13:380
276896699170,8cyclictest13-21ksoftirqd/000:20:210
276896699170,6cyclictest0-21swapper/023:30:160
276897699166,8cyclictest42-21ksoftirqd/321:30:093
2768976991614,1cyclictest3056-21PK-Backend23:08:403
2768976991513,1cyclictest3056-21PK-Backend20:47:533
2768976991513,1cyclictest3056-21PK-Backend00:35:533
276896699159,5cyclictest2784325-21ps19:50:170
276897699140,6cyclictest3056-21PK-Backend19:50:143
276897699140,5cyclictest3056-21PK-Backend00:15:213
276897699140,2cyclictest3056-21PK-Backend19:35:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional