You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-04 - 10:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri Sep 04, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
255012170116,46sleep20-21swapper/219:05:292
256012162125,26sleep30-21swapper/319:06:463
255222140102,27sleep10-21swapper/119:05:431
255882138103,24sleep00-21swapper/019:06:360
180502800,2chrt0-21swapper/300:24:033
228972650,21sleep228-21ksoftirqd/223:39:592
139282550,1sleep10-21swapper/123:56:241
31942540,3sleep30-21swapper/300:36:183
2612499519,31cyclictest25672-21ssh00:29:550
2612499519,30cyclictest11308-21seq00:19:070
26124995111,24cyclictest18579-21rm21:13:380
26124994911,28cyclictest25691-21sh22:54:130
26124994911,27cyclictest547-21sh23:46:350
26124994911,27cyclictest547-21sh23:46:350
102202490,1sleep20-21swapper/222:42:292
228922480,1sleep00-21swapper/000:03:340
122402480,4chrt12225-21munin-run00:20:001
2612499473,21cyclictest18302-21apt-get22:25:130
26124994710,28cyclictest7329-21sh23:04:400
26124994710,28cyclictest7329-21sh23:04:400
26124994710,23cyclictest4864-21sh23:25:330
2612499469,27cyclictest15690-21sh00:21:430
26124994611,26cyclictest11408-21sh23:06:480
2612499446,21cyclictest30487-21sh21:21:570
294212430,1sleep10-21swapper/123:44:451
26124994311,20cyclictest29355-21sh00:31:500
2612499421,6cyclictest13104-21ssh23:31:580
2612499420,6cyclictest15715-21ssh21:59:330
228372420,1sleep30-21swapper/322:28:143
2612499415,20cyclictest21836-21awk23:15:120
2612499413,22cyclictest26601-21sh21:43:190
2612499410,6cyclictest636-21runrttasks20:59:370
26124994011,17cyclictest636-21runrttasks22:58:390
2612499390,6cyclictest4364-21seq20:47:010
26124993811,24cyclictest27854-21seq23:43:110
2612499373,19cyclictest0-21swapper/022:32:240
26124993714,21cyclictest2490-21memory21:25:200
52402360,2sleep30-21swapper/323:02:373
52402360,2sleep30-21swapper/323:02:373
2612499369,15cyclictest13189-21diskmemload22:06:530
2612499361,6cyclictest6654-21sh21:52:160
26124993615,19cyclictest19137-21sh21:37:390
2612499361,4cyclictest13189-21diskmemload00:08:280
26124993613,20cyclictest31555-21python322:10:270
26124993613,20cyclictest17816-21sh23:11:260
2612499360,9cyclictest11867-21ssh23:55:180
2612499360,7cyclictest27745-21runrttasks19:11:120
2612499360,2cyclictest18667-21ntpq20:05:220
2612499351,6cyclictest636-21runrttasks19:27:170
2612499348,22cyclictest25212-21ls20:20:200
2612499341,8cyclictest18338-21ssh23:35:290
26124993413,18cyclictest18398-21sh22:00:280
26124993413,18cyclictest12518-21switchtime21:05:230
26124993411,21cyclictest12238-21grep19:50:240
2612499332,25cyclictest28700-21apt-get20:30:110
2612499331,9cyclictest5597-21/usr/sbin/munin19:35:190
2612499331,8cyclictest31496-21/usr/sbin/munin20:35:180
26124993312,18cyclictest2361-21sh21:49:430
2612499330,7cyclictest2294-21ssh22:36:340
2612499330,5cyclictest7781-21ssh23:52:020
2612499330,3cyclictest7868-21ssh22:40:250
2612499327,22cyclictest15988-21seq22:46:290
2612499324,25cyclictest28941-21expr23:20:140
2612499320,5cyclictest8186-21/usr/sbin/munin19:40:260
2612499320,4cyclictest20981-21gunzip20:10:220
2612499320,3cyclictest15733-21ssh22:22:530
2612499318,21cyclictest30187-21cron19:24:560
2612499318,20cyclictest9036-21expr21:30:210
2612499312,9cyclictest8961-21/usr/sbin/munin19:45:280
2612499312,26cyclictest15891-21gltestperf20:00:160
2612499312,23cyclictest22250-21apt-get20:15:120
2612499311,9cyclictest5924-21/usr/sbin/munin20:50:220
2612499311,3cyclictest29813-21users19:15:250
2612499311,27cyclictest2369-21ssh00:35:290
2612499311,26cyclictest32395-21ssh00:10:180
2612499310,3cyclictest4890-21ssh22:15:140
2612499310,3cyclictest4890-21ssh22:15:130
2612499310,3cyclictest14158-21/usr/sbin/munin19:55:220
26125993021,7cyclictest28306-21ntp_states21:20:231
2612499306,22cyclictest21431-21ntpq21:15:190
2612499302,3cyclictest10126-21pmu-power21:00:230
2612499301,9cyclictest27324-21/usr/sbin/munin20:25:200
281952290,4sleep1532-21in:imuxsock23:20:011
2612499291,8cyclictest2988-21/usr/sbin/munin19:30:190
2612499291,23cyclictest556-21apt-get20:40:100
114882290,3sleep30-21swapper/323:30:293
26126992821,5cyclictest8306-21ls20:55:252
126862280,3sleep10-21swapper/123:08:051
2612799271,24cyclictest0-21swapper/319:17:253
26125992713,10cyclictest648-21ntpd23:10:251
2612599271,24cyclictest0-21swapper/119:24:511
2612599271,24cyclictest0-21swapper/119:18:501
254862270,1sleep10-21swapper/121:18:451
296152260,9sleep334-21ksoftirqd/320:30:203
2612799261,23cyclictest0-21swapper/319:23:033
2612799251,19cyclictest13194-21ssh22:45:123
2612699251,22cyclictest0-21swapper/219:20:022
2612699250,23cyclictest0-21swapper/219:15:232
218212250,10chrt28-21ksoftirqd/221:40:052
26127992422,1cyclictest0-21swapper/319:25:043
2612799241,7cyclictest23423-21apt-get22:05:153
2612699249,13cyclictest3843-21cat23:25:172
2612799239,4cyclictest25976-21wc00:05:213
2612799230,10cyclictest14262-21hddtemp_smartct21:10:173
26125992316,5cyclictest4445-21/usr/sbin/munin23:25:241
297692220,2sleep30-21swapper/323:20:233
2612799223,4cyclictest141rcu_preempt22:00:123
26126992220,1cyclictest0-21swapper/219:25:052
26126992215,5cyclictest28-21ksoftirqd/200:00:012
26126992214,6cyclictest19723-21cat22:25:252
26126992213,7cyclictest9624-21fw_packets21:00:182
26125992217,3cyclictest22-21ksoftirqd/123:05:001
26125992217,3cyclictest22-21ksoftirqd/123:04:591
26125992212,8cyclictest560-21dbus-daemon23:50:001
26125992212,8cyclictest560-21dbus-daemon23:50:001
26125992212,7cyclictest18017-21apt-get00:00:121
2612599221,14cyclictest24796-21ssh00:05:121
131482220,2sleep30-21swapper/323:55:373
90532210,3sleep213189-21diskmemload23:05:162
2612799219,8cyclictest11124-21apt-get19:50:143
2612799214,6cyclictest6358-21fschecks_count00:15:153
2612799214,3cyclictest13189-21diskmemload23:45:533
2612799214,3cyclictest13189-21diskmemload23:45:523
2612799214,3cyclictest13189-21diskmemload23:29:203
2612799213,3cyclictest13189-21diskmemload00:32:113
26127992119,1cyclictest52950irq/141-enp3s0-00:11:533
2612799211,6cyclictest3121-21apt21:50:143
26127992113,6cyclictest9173-21sed23:05:183
26127992113,6cyclictest5327-21cat23:50:213
26127992113,6cyclictest30382-21awk21:45:263
26127992113,6cyclictest21175-21ls22:50:243
26127992113,6cyclictest15138-21apt21:35:103
2612699218,8cyclictest53050irq/142-enp3s0-22:05:152
2612699218,4cyclictest30495-21expr22:10:172
2612699213,16cyclictest30358-21grep21:45:262
2612699211,8cyclictest10217-21df21:55:152
26126992114,4cyclictest23351-21sh23:40:122
26126992113,6cyclictest20641-21entropy22:50:202
26126992113,6cyclictest18545-21fw_packets00:00:162
26126992113,6cyclictest12019-21ntpq19:50:212
26126992112,6cyclictest20395-21grep00:25:192
26126992112,5cyclictest53050irq/142-enp3s0-23:10:162
26126992112,5cyclictest53050irq/142-enp3s0-21:20:132
26126992110,9cyclictest24815-21sh00:05:132
26125992114,4cyclictest636-21runrttasks21:43:471
26125992113,6cyclictest5145-21fw_packets19:35:131
26125992112,7cyclictest24466-21ntpq22:05:231
2612599211,17cyclictest27415-21uname20:25:201
2612799209,8cyclictest13189-21diskmemload21:15:113
2612799202,2cyclictest0-21swapper/323:35:233
2612799202,13cyclictest640-21snmpd22:10:163
2612799201,5cyclictest0-21swapper/322:40:173
26127992015,3cyclictest24372-21dump-pmu-power00:05:013
26127992012,6cyclictest26549-21sed22:30:273
26127992011,7cyclictest1505-21rm22:35:463
26127992010,7cyclictest560-21dbus-daemon20:39:593
2612699209,9cyclictest28309-21switchtime22:55:232
2612699208,8cyclictest19903-21apt-get20:10:112
2612699207,11cyclictest28159-21ssh00:30:382
2612699204,4cyclictest141rcu_preempt19:44:592
2612699201,6cyclictest31847-21apt-get22:35:132
26126992015,3cyclictest25492-21sh21:18:452
26126992015,3cyclictest23250-21proc_pri20:15:222
26126992015,3cyclictest17773-21proc_pri22:00:212
26126992015,3cyclictest14441-21sendmail_mailqu22:45:232
26126992015,3cyclictest1232-21kernelversion20:40:162
26126992014,4cyclictest6433-21sh23:03:472
26126992014,4cyclictest6433-21sh23:03:462
26126992014,4cyclictest30020-21sh23:45:072
26126992014,4cyclictest30020-21sh23:45:062
26126992014,4cyclictest15572-21sh21:10:392
26126992014,3cyclictest2906-21hwlatdetect19:30:172
26126992014,3cyclictest13189-21diskmemload23:24:532
26126992013,5cyclictest2470-21ls21:25:212
26126992012,6cyclictest4128-21ntp_kernel_err21:50:232
26126992012,6cyclictest12850-21ls00:20:122
26126992010,8cyclictest19368-21sshd20:06:332
26125992015,3cyclictest5760-21sh22:40:001
26125992014,4cyclictest30826-21acpi20:35:121
26125992013,5cyclictest19618-21seq20:09:451
26125992012,6cyclictest5055-21fw_packets22:15:151
26125992012,6cyclictest5055-21fw_packets22:15:141
26125992012,6cyclictest4220-21sed22:14:591
26125992012,6cyclictest29715-21which20:30:211
26125992011,6cyclictest52750irq/139-enp3s0-23:30:231
2612599201,13cyclictest9864-21ssh21:55:131
26125992010,7cyclictest20504-21cat20:10:161
2612799191,8cyclictest15137-21ssh23:10:163
2612799191,3cyclictest0-21swapper/300:27:423
26127991912,5cyclictest31492-21sh22:58:153
26127991912,5cyclictest18466-21sed20:05:193
26127991911,6cyclictest560-21dbus-daemon21:10:003
2612799191,16cyclictest27403-21grep20:25:193
26127991911,6cyclictest24465-21awk20:20:143
26127991910,6cyclictest3662-21sed19:30:263
2612699199,8cyclictest560-21dbus-daemon20:30:012
2612699192,12cyclictest8451-21apt-get21:30:182
2612699192,11cyclictest11539-21apt-get22:20:132
26126991914,3cyclictest22946-21open_inodes23:15:212
26126991914,3cyclictest12430-21sh23:31:192
26126991912,5cyclictest560-21dbus-daemon20:04:592
26126991912,5cyclictest4931-21forks22:15:152
26126991912,5cyclictest4931-21forks22:15:152
26126991912,5cyclictest11629-21df21:05:142
26126991911,6cyclictest608-21cat00:10:232
2612699191,16cyclictest16178-21/usr/sbin/munin21:35:202
2612699191,16cyclictest13189-21diskmemload22:31:442
26126991911,6cyclictest10480-21uptime19:45:272
2612699191,12cyclictest559-21apt-get00:35:142
2612599199,7cyclictest52750irq/139-enp3s0-21:25:141
2612599199,6cyclictest52750irq/139-enp3s0-00:20:121
2612599193,14cyclictest560-21dbus-daemon21:40:001
2612599191,9cyclictest206762sh23:37:481
26125991914,3cyclictest9518-21sh21:30:261
26125991914,3cyclictest29635-21missed_timers21:45:191
26125991912,5cyclictest27237-21expr22:55:121
26125991912,5cyclictest15325-21seq22:22:291
26125991912,5cyclictest1011-21tune2fs20:40:141
26125991911,6cyclictest7773-21ntpq19:40:221
26125991911,6cyclictest16500-21sendmail_mailqu20:00:221
26125991911,6cyclictest1495-21sed19:25:241
26125991911,6cyclictest1425-21sed00:35:201
2612599191,15cyclictest20834-21cut22:50:211
26125991910,7cyclictest11119-21sed21:05:011
2612799189,6cyclictest24157-21systemctl20:20:003
2612799187,9cyclictest0-21swapper/322:16:213
2612799187,9cyclictest0-21swapper/322:16:213
2612799187,9cyclictest0-21swapper/321:25:213
2612799186,8cyclictest4859-21apt-get20:50:123
2612799182,12cyclictest8986-21apt-get19:45:133
2612799181,6cyclictest22069-21/usr/sbin/munin21:40:153
26127991813,3cyclictest3816-21sendmail_mailst20:45:223
26127991813,3cyclictest23315-21/usr/sbin/munin23:40:143
26127991813,3cyclictest16511-21sh22:23:403
26127991812,4cyclictest23216-21wc23:15:253
26127991811,5cyclictest11231-21sed21:55:243
26127991810,5cyclictest13740-21grep19:55:173
2612699188,7cyclictest6987-21ntpq00:15:222
2612699181,5cyclictest15328-21latency_hist20:00:012
26126991813,3cyclictest27141-21pmu-power19:10:202
26126991812,4cyclictest53050irq/142-enp3s0-23:50:202
26126991810,6cyclictest32078-21ls20:35:252
2612599189,7cyclictest14092-21tune2fs21:10:171
2612599189,6cyclictest52750irq/139-enp3s0-00:30:221
2612599187,9cyclictest0-21swapper/123:50:481
26125991812,4cyclictest6974-21/usr/sbin/munin20:55:141
26125991812,4cyclictest5463-21latency_hist00:15:001
26125991812,4cyclictest19284-21sh22:01:201
26125991812,4cyclictest13189-21diskmemload22:25:561
26125991811,5cyclictest13385-21ls22:45:121
2612599181,12cyclictest5776-21apt-get22:40:041
26125991810,6cyclictest3755-21expr21:50:181
26125991810,6cyclictest3092-21cut20:45:151
26125991810,6cyclictest30878-21seq23:20:551
26125991810,6cyclictest2918-21ls19:30:181
26125991810,6cyclictest28252-21date19:15:011
26125991810,6cyclictest25971-21ls22:30:211
26125991810,6cyclictest12684-21grep21:05:251
2612799179,7cyclictest3897-1kworker/u9:221:30:103
2612799178,7cyclictest16921-21users20:00:253
2612799172,11cyclictest26164-21apt-get19:10:123
26127991711,4cyclictest10301-21tr21:00:243
26127991710,5cyclictest1320-21sed20:40:173
2612699179,5cyclictest4735-21cpuspeed_turbos19:35:122
26126991710,5cyclictest2931-21expr20:45:142
26126991710,5cyclictest28820-21grep20:30:112
2612599179,6cyclictest13746-21grep19:55:171
2612599176,8cyclictest5188-21df_inode20:50:141
2612599176,8cyclictest19420-21apt-get00:25:121
2612599172,11cyclictest21797-21apt-get20:15:061
26125991710,5cyclictest11393-21expr19:50:151
2612799168,6cyclictest19876-21/usr/sbin/munin20:10:153
2612799162,10cyclictest4230-21apt-get19:35:033
26127991611,3cyclictest31167-21sh21:22:383
2612699169,5cyclictest24408-21basename20:20:112
26126991611,3cyclictest4834-21/usr/sbin/munin20:50:212
26125991611,3cyclictest9389-21fschecks_time19:45:161
26125991610,4cyclictest25586-21switchtime20:20:241
26127991510,3cyclictest7170-21cstates20:55:133
26127991510,3cyclictest7018-21cstates19:40:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional