You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-13 - 12:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat Dec 13, 2025 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2796122590,7sleep00-21swapper/019:05:080
294402202164,25sleep30-21swapper/319:06:413
294492201164,25sleep20-21swapper/219:06:482
294432200164,24sleep10-21swapper/119:06:431
2786621710,1sleep10-21swapper/120:15:171
1255621510,3sleep32979699cyclictest19:40:203
1046321430,6sleep22979199cyclictest19:35:212
381721420,4sleep32979699cyclictest00:15:133
381721420,4sleep32979699cyclictest00:15:133
244832570,4sleep22979199cyclictest23:50:082
17292500,1sleep10-21swapper/120:30:141
122452450,4sleep230-21rcuc/220:55:112
2979199365,7cyclictest32733-40dnf00:08:002
2979199353,7cyclictest774-21dbus-daemon23:07:142
29791993511,3cyclictest23706-21/usr/sbin/munin22:35:122
2978199358,12cyclictest12161-21awk00:40:010
29791993411,4cyclictest20285-21grep21:10:242
29791993411,2cyclictest87150irq/126-enp1s0-20:15:112
29791993410,5cyclictest0-21swapper/223:25:212
29791993410,3cyclictest24698-21/usr/sbin/munin20:10:132
29791993410,3cyclictest21986-21ntp_states21:15:192
29781993410,11cyclictest21899-21basename23:45:010
2979199338,4cyclictest10988-21/usr/sbin/munin00:30:202
2979199332,6cyclictest32572-21cat22:55:082
2979199332,5cyclictest774-21dbus-daemon20:40:082
2979199332,5cyclictest774-21dbus-daemon20:40:082
29791993311,3cyclictest29882-21if_err_enp1s020:20:152
2978699334,16cyclictest18189-40dnf21:06:151
2978199332,24cyclictest811-21kworker/0:220:43:270
2978199332,24cyclictest811-21kworker/0:220:43:270
2979199329,4cyclictest31675-21idleruntime-cro21:40:012
2979199327,5cyclictest0-21swapper/221:55:192
2979199324,6cyclictest774-21dbus-daemon19:15:012
29791993211,3cyclictest17207-21sh21:05:162
2978199326,12cyclictest5323-21perf20:40:010
2978199320,23cyclictest15090-21kworker/0:119:11:270
2979199312,3cyclictest407-21/usr/sbin/munin19:15:232
2979199312,11cyclictest19924-21tail22:25:112
29791993110,3cyclictest2118-21wc00:10:202
29791993110,3cyclictest19212-21cut22:25:022
2978199311,22cyclictest6708-21kworker/0:019:46:290
2979199309,4cyclictest0-21swapper/220:30:242
2979199308,3cyclictest576-21systemd-journal00:39:132
2979199301,7cyclictest10895-21sh23:20:012
29791993012,3cyclictest29478-21perf21:35:012
29791993010,3cyclictest20736-21cut23:40:172
29791993010,3cyclictest17626-21date23:35:022
29791993010,3cyclictest16575-21grep19:50:152
29786993011,10cyclictest14739-21kworker/u8:221:31:361
2978199302,21cyclictest28119-21kworker/0:323:55:500
2978199302,21cyclictest25895-21kworker/0:122:47:090
2979199299,3cyclictest831-21runrttasks19:55:102
2979199299,3cyclictest4265-21expr21:50:112
2979199299,3cyclictest4265-21expr21:50:112
2979199298,4cyclictest576-21systemd-journal21:45:012
29791992910,3cyclictest26799-21expr23:55:082
2978199291,22cyclictest25014-21kworker/0:121:25:090
2979699281,8cyclictest27003-21grep20:15:103
2979699281,6cyclictest770-21rngd00:00:013
2979199289,3cyclictest14303-21grep19:45:152
2979199284,3cyclictest87150irq/126-enp1s0-23:00:112
2979199282,3cyclictest5516-21diskstats00:20:122
2979199282,3cyclictest23114-21/usr/sbin/munin21:20:112
2979199281,6cyclictest11962-21wc22:05:222
2979199281,5cyclictest23771-21sshd23:46:132
2979199281,5cyclictest21512-21/usr/sbin/munin22:30:122
2979199281,3cyclictest2723-21grep19:20:112
29791992810,3cyclictest7727-21/usr/sbin/munin20:45:092
29791992810,3cyclictest25896-21/usr/sbin/munin22:40:212
2978199282,18cyclictest26081-21kworker/0:200:11:420
2978199281,20cyclictest15090-21kworker/0:119:22:330
2979699272,9cyclictest32489-21/usr/sbin/munin22:57:233
29796992719,6cyclictest5563-21awk23:05:173
2979199271,5cyclictest7532-21sshd23:10:142
2979199271,5cyclictest11990-21latency23:20:172
29791992711,2cyclictest2972-21/usr/sbin/munin00:15:202
29791992711,2cyclictest2972-21/usr/sbin/munin00:15:202
29791992710,3cyclictest17713-21/usr/sbin/munin23:35:152
2978699271,17cyclictest7189-21ntpq21:55:161
2978199272,8cyclictest23869-21kworker/0:023:53:130
2978199272,13cyclictest25675-21munin-run22:40:000
2979699266,8cyclictest7395-21cpu00:25:083
2979699264,7cyclictest869-21sssd_nss23:46:133
2979699262,2cyclictest87150irq/126-enp1s0-21:10:133
2979699262,11cyclictest14922-21/usr/sbin/munin22:15:083
2979199261,4cyclictest7363-21/usr/sbin/munin00:25:192
2979199261,14cyclictest0-21swapper/222:19:052
29791992611,3cyclictest30065-21latency_hist22:50:012
2978699261,17cyclictest0-21swapper/123:45:211
2979699252,6cyclictest788-21NetworkManager22:14:383
2979199251,6cyclictest20929-21gltestperf20:00:152
29791992511,2cyclictest6289-21sed19:25:232
2979199250,14cyclictest0-21swapper/220:08:302
2978199252,9cyclictest15964-21kworker/0:121:03:150
2978199252,9cyclictest1212-21kworker/0:121:41:290
2978199252,9cyclictest10853-21kworker/0:023:24:330
2978199252,8cyclictest9646-21kworker/0:120:53:530
2978199252,8cyclictest3666-21kworker/0:221:52:570
2978199252,8cyclictest3666-21kworker/0:221:52:570
2978199252,8cyclictest15493-21kworker/0:119:58:170
2978199251,9cyclictest31982-21kworker/0:222:59:410
2978199251,9cyclictest1349-21kworker/0:123:00:010
113932250,1sleep00-21swapper/000:33:200
2979699244,9cyclictest87150irq/126-enp1s0-19:25:113
2979699243,11cyclictest1-21systemd20:40:083
2979699243,11cyclictest1-21systemd20:40:083
2979199241,5cyclictest30290-21/usr/sbin/munin22:50:112
2978699241,9cyclictest24698-21/usr/sbin/munin20:10:141
2978199242,9cyclictest25014-21kworker/0:121:31:190
2978199242,9cyclictest15493-21kworker/0:120:07:430
2978199242,10cyclictest15594-21kworker/0:022:18:410
2978199241,9cyclictest21609-21kworker/0:123:47:110
2978199241,9cyclictest15964-21kworker/0:121:10:110
2978199241,9cyclictest12441-21kworker/0:222:41:450
2978199241,8cyclictest2545-21kworker/0:000:24:470
2978199241,10cyclictest2545-21kworker/0:000:28:510
2978199240,10cyclictest4127-21kworker/0:023:07:550
2979699235,7cyclictest5440-21cat20:40:013
2979699235,7cyclictest25834-21latency_hist22:40:013
2979699232,10cyclictest12121-21/usr/sbin/munin20:55:253
2979699231,11cyclictest800-21snmpd00:34:143
2979699231,10cyclictest31306-21/usr/sbin/munin20:25:123
29796992310,3cyclictest6718-21gltestperf21:55:113
2979199237,3cyclictest0-21swapper/220:39:312
2979199236,3cyclictest0-21swapper/221:25:312
29791992310,3cyclictest0-21swapper/222:11:312
2978699232,18cyclictest7404-21hddtemp_smartct23:10:121
29786992312,3cyclictest0-21swapper/121:12:201
2978699231,18cyclictest11539-21perf00:35:011
2978199232,9cyclictest22735-21kworker/0:221:38:570
2978199231,8cyclictest6398-21kworker/0:123:10:490
2978199231,8cyclictest15493-21kworker/0:120:15:250
2978199231,8cyclictest1212-21kworker/0:122:08:090
113632230,8sleep341-21ksoftirqd/323:20:113
2979699229,6cyclictest39-21rcuc/319:10:133
2979699222,11cyclictest16132-21taskset21:03:003
29796992212,2cyclictest87450irq/129-enp1s0-19:55:123
2979699221,11cyclictest28090-21/usr/sbin/munin22:45:113
2979199224,3cyclictest0-21swapper/220:29:552
2978699228,6cyclictest0-21swapper/121:17:001
2978699226,14cyclictest221ktimersoftd/100:00:011
2978699225,9cyclictest21-21rcuc/121:00:141
29786992219,2cyclictest23-21ksoftirqd/119:35:011
29786992219,2cyclictest23-21ksoftirqd/119:35:011
29786992213,3cyclictest0-21swapper/121:41:311
29786992213,3cyclictest0-21swapper/120:40:041
29786992213,3cyclictest0-21swapper/120:40:041
2978699221,19cyclictest0-21swapper/123:39:081
2978699221,11cyclictest768-21in:imjournal22:20:011
2978699221,10cyclictest576-21systemd-journal22:45:141
29786992210,3cyclictest457-21munin-run00:10:011
2978199222,9cyclictest15493-21kworker/0:120:04:450
2978199222,8cyclictest6708-21kworker/0:019:30:370
2978199222,8cyclictest6708-21kworker/0:019:30:370
2978199222,6cyclictest30941-21kworker/0:020:32:370
2978199221,9cyclictest7409-21kworker/0:020:45:550
2978199221,9cyclictest32678-21kworker/0:100:17:190
2978199221,9cyclictest32678-21kworker/0:100:17:190
29781992217,3cyclictest25034-21df20:10:130
2979699214,3cyclictest29838-21pmu-power00:00:183
2979699213,2cyclictest87150irq/126-enp1s0-22:25:163
2979699212,7cyclictest25896-21/usr/sbin/munin22:40:153
2979699211,17cyclictest9941-21sed19:35:153
2979699210,9cyclictest25083-21latency_hist21:25:023
2979199218,3cyclictest0-21swapper/222:01:352
2979199217,3cyclictest0-21swapper/221:48:352
2979199215,3cyclictest0-21swapper/219:41:152
2979199212,2cyclictest0-21swapper/221:01:232
2978699215,8cyclictest24472-21munin-run20:10:011
2978699214,8cyclictest30819-21fschecks_time22:50:101
2978699212,8cyclictest821-21lldpd22:33:361
2978699211,3cyclictest4050-21chrt23:01:431
2978699211,18cyclictest0-21swapper/100:25:131
2978699211,17cyclictest0-21swapper/119:20:181
2978699211,15cyclictest0-21swapper/119:56:481
2978699211,14cyclictest0-21swapper/122:28:321
2978699211,11cyclictest23463-21diskstats21:20:121
2978699211,11cyclictest12199-21ntp_states23:20:201
2978699211,10cyclictest831-21runrttasks22:20:131
2978199218,3cyclictest15493-21kworker/0:120:26:470
2978199212,9cyclictest17714-21kworker/0:219:54:030
2978199212,8cyclictest20447-21kworker/0:021:21:210
2978199212,8cyclictest19373-21kworker/0:223:36:570
2978199212,8cyclictest18816-21kworker/0:122:28:530
2978199212,6cyclictest7746-21kworker/0:022:00:070
2978199212,5cyclictest11263-21munin-run19:40:010
2978199212,10cyclictest29743-21kworker/0:022:53:530
2978199212,10cyclictest12441-21kworker/0:222:30:570
2978199211,8cyclictest7409-21kworker/0:020:55:210
2978199211,8cyclictest6398-21kworker/0:123:17:070
2978199211,8cyclictest22735-21kworker/0:221:18:150
2978199211,8cyclictest1212-21kworker/0:121:49:210
2978199210,9cyclictest16496-21kworker/0:023:32:430
2979699207,10cyclictest13478-21grep23:25:103
2979699206,2cyclictest87150irq/126-enp1s0-19:15:163
2979699203,9cyclictest41-21ksoftirqd/319:50:163
2979699203,4cyclictest768-21in:imjournal21:50:013
2979699201,2cyclictest16198-21smartctl23:30:123
2979699201,12cyclictest0-21swapper/323:15:113
2979699201,12cyclictest0-21swapper/322:30:213
29791992012,5cyclictest28496-40updatedb00:00:032
2978699208,10cyclictest14739-21kworker/u8:200:10:341
2978699206,3cyclictest88550irq/134-enp2s0-19:45:111
2978699205,6cyclictest0-21swapper/121:45:111
2978699204,7cyclictest87450irq/129-enp1s0-20:20:141
2978699203,9cyclictest5238-21sort23:05:131
2978699201,17cyclictest0-21swapper/119:27:571
2978699201,16cyclictest221ktimersoftd/122:10:101
2978699201,15cyclictest0-21swapper/120:36:261
2978699201,14cyclictest0-21swapper/121:52:201
2978699201,14cyclictest0-21swapper/121:52:201
2978699201,14cyclictest0-21swapper/120:51:591
2978699201,11cyclictest25668-21latency_hist22:40:011
29786992010,3cyclictest9002-21switchtime20:45:211
2978699200,18cyclictest21-21rcuc/119:35:141
2978699200,15cyclictest0-21swapper/120:02:391
2978699200,13cyclictest0-21swapper/100:15:071
2978699200,13cyclictest0-21swapper/100:15:071
2978199208,3cyclictest7746-21kworker/0:021:57:310
2978199207,10cyclictest28496-40updatedb00:00:020
2978199205,2cyclictest15090-21kworker/0:119:17:050
2978199202,8cyclictest4495-21kworker/0:319:25:370
2978199201,9cyclictest26081-21kworker/0:200:05:210
2978199201,9cyclictest13082-21kworker/0:223:29:530
2978199201,7cyclictest12441-21kworker/0:222:21:130
2978199201,10cyclictest11147-21kworker/0:119:42:410
2978199200,9cyclictest15493-21kworker/0:120:20:250
2979699199,2cyclictest12050irq/123-ahci[0023:54:203
2979699195,11cyclictest831-21runrttasks21:54:443
2979699195,11cyclictest831-21runrttasks21:54:443
2979699194,7cyclictest87150irq/126-enp1s0-19:45:113
2979699193,3cyclictest12050irq/123-ahci[0021:30:213
2979699191,6cyclictest0-21swapper/323:10:153
2979699191,5cyclictest678-21gunzip21:40:223
2979699191,3cyclictest20922-21/usr/sbin/munin21:15:233
2979699190,6cyclictest22268-21latency_hist20:05:023
2978699197,2cyclictest28496-40updatedb00:00:021
2978699196,3cyclictest87350irq/128-enp1s0-20:55:231
2978699194,7cyclictest10188-21smart_sda23:15:191
2978699194,3cyclictest88550irq/134-enp2s0-00:20:111
29786991912,4cyclictest31946-21munin-run19:15:011
29786991912,2cyclictest0-21swapper/122:44:391
29786991912,2cyclictest0-21swapper/120:29:071
2978699191,17cyclictest0-21swapper/100:35:131
2978699191,16cyclictest0-21swapper/123:52:471
2978699191,16cyclictest0-21swapper/122:07:071
2978699191,16cyclictest0-21swapper/119:53:141
2978699191,10cyclictest9084-21ethtool22:00:131
2978699191,10cyclictest12159-21hddtemp_smartct19:40:141
2978699191,10cyclictest0-21swapper/123:25:231
2978699190,14cyclictest0-21swapper/121:25:511
2979699187,3cyclictest17119-21/usr/sbin/munin22:20:193
2979699183,5cyclictest7096-21/usr/sbin/munin19:30:133
2979699183,5cyclictest7096-21/usr/sbin/munin19:30:133
2979699183,12cyclictest29700-21/usr/sbin/munin21:35:183
2979699182,2cyclictest12509-21munin-run22:10:013
2979699181,8cyclictest31139-21expr00:05:093
2979699181,7cyclictest26898-21uptime21:25:263
2979699181,5cyclictest32039-21sshd22:52:143
29796991812,3cyclictest30612-21awk20:20:233
2979699181,15cyclictest0-21swapper/300:39:593
2979699181,14cyclictest24698-21/usr/sbin/munin20:10:113
29796991811,4cyclictest20617-21sort23:40:163
2979699181,14cyclictest12105-21latency_hist20:55:023
2979199185,3cyclictest0-21swapper/219:31:472
2979199185,3cyclictest0-21swapper/219:31:472
2979199183,3cyclictest0-21swapper/220:50:552
2978699183,9cyclictest23-21ksoftirqd/123:40:121
2978699181,9cyclictest1227-21runrttasks22:55:351
2978699181,5cyclictest407-21/usr/sbin/munin19:15:171
29786991814,3cyclictest0-21swapper/123:32:551
29786991811,2cyclictest0-21swapper/121:38:091
2978199185,2cyclictest18198-21kworker/0:221:08:590
2979699177,2cyclictest821-21lldpd20:33:053
2979699175,3cyclictest9144-21kernelversion22:00:143
2979699174,5cyclictest800-21snmpd20:47:503
2979699172,6cyclictest2706-21/usr/sbin/munin19:20:153
2979699171,3cyclictest87450irq/129-enp1s0-21:05:143
29796991712,3cyclictest18963-21mailstats23:35:213
2979699171,14cyclictest0-21swapper/300:21:213
2979699171,14cyclictest0-21swapper/300:10:073
2979699171,10cyclictest0-21swapper/320:05:113
29796991710,4cyclictest2536-21expr23:00:113
2978199176,2cyclictest12441-21kworker/0:222:11:430
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional