You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-09 - 19:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Aug 09, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
251542238163,25sleep20-21swapper/207:06:502
250532236162,25sleep00-21swapper/007:05:320
250612202167,23sleep30-21swapper/307:05:383
251532201164,24sleep10-21swapper/107:06:491
604021800,4sleep32549899cyclictest10:05:023
1255121350,5sleep32549899cyclictest09:05:133
25493999391,1cyclictest0-21swapper/210:15:012
25493997370,2cyclictest0-21swapper/208:10:332
25493997167,3cyclictest0-21swapper/211:41:532
273762530,5sleep22549399cyclictest12:05:162
25493994441,2cyclictest0-21swapper/211:28:552
2548699445,10cyclictest0-21swapper/111:30:161
303162410,2sleep10-21swapper/111:00:021
25486993816,19cyclictest8463-21vmstat11:20:271
25486993816,19cyclictest3840-21gltestperf12:25:161
191622370,1sleep00-21swapper/011:45:230
2548699351,2cyclictest28679-21perf12:10:001
2548699349,4cyclictest0-21swapper/110:05:231
2548699344,7cyclictest16648-40dnf10:25:361
2548699341,9cyclictest0-21swapper/111:07:311
2548699339,3cyclictest22114-40dnf09:25:231
2548699337,6cyclictest19332-21pmu-power08:05:251
2548699332,4cyclictest936-21diskstats07:25:131
2548699331,14cyclictest0-21swapper/107:59:061
137612330,3sleep230-21rcuc/211:35:142
2548699321,3cyclictest20779-21latency_hist09:25:011
2548699321,19cyclictest0-21swapper/110:32:421
25486993211,3cyclictest22269-21/usr/sbin/munin11:55:211
25486993210,3cyclictest5718-21ntpq11:15:211
2548199328,11cyclictest9179-21munin-run07:45:020
2548699312,11cyclictest23790-21cut09:30:161
2548699312,11cyclictest23790-21cut09:30:161
2548699311,4cyclictest2961-21awk12:25:011
2548699311,3cyclictest7321-21cat12:34:561
25486993112,3cyclictest27760-21expr09:40:141
2548699311,18cyclictest0-21swapper/107:53:221
25486993111,3cyclictest827-21runrttasks10:02:541
25486993111,3cyclictest2813-21/usr/sbin/munin07:30:291
25486993111,3cyclictest1845-21/usr/sbin/munin09:55:211
2548699311,13cyclictest0-21swapper/112:19:411
2548699311,11cyclictest29773-21latency12:10:181
2548699311,11cyclictest18743-21cat09:20:021
2548199318,11cyclictest2596-21awk07:30:020
25498993014,9cyclictest0-21swapper/311:47:203
2548699301,7cyclictest15988-21df11:40:121
25486993016,3cyclictest0-21swapper/111:39:541
25486993013,3cyclictest29217-21grep07:15:161
25486993011,3cyclictest30106-21ls09:45:131
2548699301,12cyclictest25050-21gunzip10:45:241
25486993010,4cyclictest29309-21grep08:30:141
25486993010,3cyclictest5389-21latency_hist08:50:021
25498992914,9cyclictest0-21swapper/312:10:033
2549899291,20cyclictest0-21swapper/311:57:163
2549899291,20cyclictest0-21swapper/310:35:353
2549899291,19cyclictest0-21swapper/310:11:243
2549899291,18cyclictest0-21swapper/312:23:533
2548699292,6cyclictest0-21swapper/109:13:381
25486992914,2cyclictest0-21swapper/109:39:461
25486992914,2cyclictest0-21swapper/108:21:111
25486992914,2cyclictest0-21swapper/107:42:551
2548699291,3cyclictest970-21latency_hist08:40:021
2548699291,3cyclictest31299-21fschecks_time07:20:141
2548699291,15cyclictest0-21swapper/110:15:341
25486992911,3cyclictest22889-21expr08:15:131
2548699291,13cyclictest0-21swapper/109:03:221
2548699291,12cyclictest22407-21cat08:15:011
25486992910,3cyclictest8822-21mailstats12:35:241
25486992910,3cyclictest21398-21latency_hist10:40:021
2548699290,4cyclictest0-21swapper/111:04:571
2548199292,17cyclictest29834-21kworker/0:011:06:550
2548199292,10cyclictest25748-21kworker/0:210:51:430
25498992814,4cyclictest0-21swapper/311:13:013
2549899281,19cyclictest0-21swapper/312:36:373
2549899281,19cyclictest0-21swapper/312:06:093
2549899281,19cyclictest0-21swapper/311:24:183
2549899281,19cyclictest0-21swapper/310:45:523
2549899281,19cyclictest0-21swapper/309:53:583
2549899281,18cyclictest0-21swapper/312:16:153
2549899281,18cyclictest0-21swapper/311:16:233
2549399288,11cyclictest0-21swapper/210:45:132
25493992825,2cyclictest0-21swapper/212:38:222
2548699286,2cyclictest22039-21latency_hist11:55:011
2548699281,5cyclictest23320-21sshd10:41:331
25486992814,2cyclictest0-21swapper/108:56:191
25486992812,3cyclictest0-21swapper/110:21:551
2548699281,12cyclictest27127-21expr08:25:141
2548699281,12cyclictest0-21swapper/110:52:441
2548699280,14cyclictest0-21swapper/108:40:051
2549899271,19cyclictest0-21swapper/309:13:053
2549899271,18cyclictest0-21swapper/310:17:083
2549899271,18cyclictest0-21swapper/309:17:113
2548699272,8cyclictest24491-21/usr/sbin/munin12:00:131
2548699271,6cyclictest9478-21fschecks_count11:25:161
2548699271,12cyclictest0-21swapper/110:12:021
2548699270,16cyclictest0-21swapper/107:13:311
2548699270,15cyclictest0-21swapper/111:12:091
2548699270,15cyclictest0-21swapper/107:38:181
2548199272,7cyclictest2532-21kworker/0:007:36:560
25498992614,3cyclictest0-21swapper/309:23:353
2549899261,18cyclictest0-21swapper/312:28:543
2549899261,17cyclictest0-21swapper/311:40:343
25486992613,2cyclictest0-21swapper/107:45:091
2549899252,16cyclictest87350irq/126-enp1s0-10:22:233
2549899251,18cyclictest0-21swapper/311:52:543
2549899251,17cyclictest0-21swapper/309:40:163
2549899251,16cyclictest19780-21kworker/u8:112:00:233
2549899251,15cyclictest0-21swapper/309:30:403
2549899251,15cyclictest0-21swapper/309:30:403
2548699251,4cyclictest808-21snmpd08:54:271
2548199258,15cyclictest3210-21perf08:45:010
2548199252,8cyclictest22130-21kworker/0:008:15:080
2548199252,4cyclictest29588-21munin-run09:45:010
280132241,19sleep328286-21munin-node07:15:033
25498992414,4cyclictest0-21swapper/312:33:013
2549899241,17cyclictest0-21swapper/311:03:533
2549899241,17cyclictest0-21swapper/310:56:513
2549899241,16cyclictest0-21swapper/311:30:563
2549899241,15cyclictest0-21swapper/310:43:543
2548199242,8cyclictest4227-21kworker/0:211:19:110
2548199241,20cyclictest0-21swapper/007:22:450
2548199241,11cyclictest25405-21/usr/sbin/munin09:35:130
2548199241,11cyclictest17235-21/usr/sbin/munin10:30:150
25498992313,2cyclictest87350irq/126-enp1s0-08:20:233
2549899231,16cyclictest0-21swapper/310:05:123
2549899231,15cyclictest0-21swapper/311:25:423
2549899230,6cyclictest0-21swapper/309:35:393
2548699231,9cyclictest0-21swapper/111:49:371
2548199239,3cyclictest23830-21/usr/sbin/munin10:45:230
2548199232,7cyclictest15682-21taskset09:10:240
2548199231,11cyclictest8091-21sshd10:08:330
2548199231,11cyclictest23211-21/usr/sbin/munin09:30:270
2548199231,11cyclictest23211-21/usr/sbin/munin09:30:270
2548199231,11cyclictest15146-21taskset11:36:330
2548199231,11cyclictest10642-21/usr/sbin/munin10:15:200
2549899222,11cyclictest20739-21df_inode08:10:133
25498992218,3cyclictest0-21swapper/309:25:143
2549899221,7cyclictest0-21swapper/311:37:343
2549899221,15cyclictest0-21swapper/309:46:103
2549899221,14cyclictest0-21swapper/310:30:543
2549899221,13cyclictest87350irq/126-enp1s0-09:55:163
2548699222,12cyclictest583-21ntp_kernel_pll_09:50:201
2548699221,15cyclictest0-21swapper/109:07:531
2548199228,3cyclictest2813-21/usr/sbin/munin07:30:260
2548199222,9cyclictest4296-21kworker/0:311:13:390
2548199222,6cyclictest0-21swapper/010:25:360
2548199222,5cyclictest7344-21munin-run12:35:020
2548199221,5cyclictest5142-21munin-run12:30:020
2548199221,10cyclictest808-21snmpd08:28:020
25481992210,3cyclictest87550irq/128-enp1s0-12:09:400
25481992210,2cyclictest30210-21awk11:00:000
250832220,4sleep07-21ksoftirqd/012:00:170
2549899214,4cyclictest24577-21sleep08:20:003
2549899212,10cyclictest11599-21/usr/sbin/munin07:50:163
2549899211,9cyclictest0-21swapper/310:26:533
2549899211,11cyclictest7205-21/usr/sbin/munin07:40:193
2549899211,11cyclictest3422-21/usr/sbin/munin08:45:173
2549399219,3cyclictest31535-21fschecks_count12:15:152
2549399211,6cyclictest0-21swapper/210:26:322
25493992112,3cyclictest7203-21grep08:50:282
2548199218,3cyclictest25160-21fschecks_count08:20:150
2548199216,3cyclictest87550irq/128-enp1s0-08:05:140
2548199211,10cyclictest0-21swapper/009:55:190
2548199210,19cyclictest0-21swapper/009:15:230
2548199210,19cyclictest0-21swapper/009:05:160
2548199210,19cyclictest0-21swapper/008:55:190
2548199210,19cyclictest0-21swapper/007:45:190
2549899209,3cyclictest808-21snmpd08:02:563
2549899202,10cyclictest26033-21/usr/sbin/munin10:50:123
2549899201,6cyclictest7818-21/usr/sbin/munin08:55:213
2549899201,6cyclictest1322-21ntp_states11:05:213
25498992016,3cyclictest0-21swapper/307:45:443
2549899201,10cyclictest1681-21ntp_states07:25:213
2549899201,10cyclictest15760-21latency_hist08:00:003
2549399201,5cyclictest0-21swapper/211:02:542
2548699200,14cyclictest0-21swapper/108:01:591
2548199205,3cyclictest27545-21sshd07:13:450
2548199205,3cyclictest17628-21munin-run11:45:000
2548199205,3cyclictest14063-21gzip10:20:230
2548199202,15cyclictest950-21/usr/sbin/munin12:20:180
2548199202,15cyclictest13797-21/usr/sbin/munin07:55:200
2548199201,8cyclictest32093-21kworker/0:311:03:470
2548199201,8cyclictest21257-21kworker/0:010:42:040
2548199201,5cyclictest21009-21/usr/sbin/munin09:25:180
2548199201,11cyclictest0-21swapper/009:50:210
2549899198,3cyclictest5007-21/usr/sbin/munin07:35:113
2549899193,6cyclictest6467-21grep08:50:193
2549899191,9cyclictest20298-21sed08:10:003
2549899191,6cyclictest30773-21/usr/sbin/munin07:20:253
2549899191,6cyclictest2813-21/usr/sbin/munin07:30:223
2549399196,8cyclictest32681-40dnf07:24:252
2549399195,6cyclictest630-21open_files09:50:222
2549399194,5cyclictest910-21systemd-logind08:30:012
2549399193,14cyclictest21763-21tr10:40:132
2549399191,9cyclictest0-21swapper/212:25:082
2549399191,7cyclictest0-21swapper/211:58:482
2549399191,6cyclictest18661-21irqstats11:45:172
2549399191,5cyclictest0-21swapper/210:37:192
2549399191,5cyclictest0-21swapper/209:31:032
2549399191,5cyclictest0-21swapper/209:31:032
2549399191,11cyclictest0-21swapper/209:13:202
2548199193,4cyclictest29642-21ntp_states07:15:210
2548199191,9cyclictest26464-21kworker/0:012:10:180
2548199191,8cyclictest808-21snmpd09:22:260
2548199191,5cyclictest5402-21awk08:50:010
2548199191,15cyclictest31388-21/usr/sbin/munin08:35:200
2548199191,15cyclictest31119-21/usr/sbin/munin12:15:210
109542191,2sleep00-21swapper/011:29:100
86862181,3sleep00-21swapper/011:23:160
2549899185,3cyclictest87650irq/129-enp1s0-08:30:253
2549899181,5cyclictest11528-21sed09:00:263
2549899181,4cyclictest473-21sed08:35:263
25498991812,3cyclictest26794-21pluginstate07:10:243
2549399186,9cyclictest14189-21diskmemload10:00:202
2549399182,14cyclictest0-21swapper/208:45:002
2549399181,6cyclictest0-21swapper/211:10:342
25493991813,3cyclictest17654-21grep10:30:162
25493991812,3cyclictest13457-21find09:05:252
2549399181,15cyclictest0-21swapper/208:32:522
2548199182,5cyclictest19432-21/usr/sbin/munin10:35:270
2548199182,4cyclictest22269-21/usr/sbin/munin11:55:130
2548199182,13cyclictest29972-21cpuspeed_turbos09:45:140
2548199181,9cyclictest8025-21kworker/0:110:14:360
2548199181,9cyclictest0-21swapper/011:52:060
2548199181,6cyclictest0-21swapper/011:30:170
2548199181,5cyclictest12670-21ntp_states07:50:220
2548199181,4cyclictest29195-21/usr/sbin/munin08:30:190
2548199181,4cyclictest15991-21/usr/sbin/munin08:00:180
2548199181,14cyclictest7568-21/usr/sbin/munin12:35:130
2549899172,3cyclictest1203-21/usr/sbin/munin08:40:123
2549899170,7cyclictest0-21swapper/308:25:133
2549399176,3cyclictest0-21swapper/211:17:092
2549399173,7cyclictest0-21swapper/212:12:362
2549399172,12cyclictest6858-21gltestperf10:05:152
2549399171,7cyclictest28837-21gzip09:40:262
2549399171,6cyclictest0-21swapper/207:25:112
2549399171,5cyclictest0-21swapper/212:03:522
2549399171,5cyclictest0-21swapper/209:23:332
2549399171,5cyclictest0-21swapper/207:30:262
2549399171,4cyclictest0-21swapper/211:21:462
2549399171,3cyclictest18215-21cpu08:05:132
25493991713,3cyclictest0-21swapper/208:20:442
25493991713,3cyclictest0-21swapper/207:55:532
25493991712,3cyclictest26703-21tail10:50:172
25493991711,3cyclictest1766-21unixbench_multi11:05:262
2549399171,10cyclictest0-21swapper/208:00:152
2549399171,10cyclictest0-21swapper/207:15:182
25493991710,4cyclictest0-21swapper/209:25:232
2549399167,2cyclictest25944-21fschecks_time09:35:152
2549399165,3cyclictest87350irq/126-enp1s0-09:15:162
2549399165,3cyclictest13441-21taskset10:20:152
2549399161,7cyclictest7818-21/usr/sbin/munin08:55:202
2549399161,6cyclictest9172-21latency_hist07:45:012
2549399161,6cyclictest11600-21sed10:15:192
2549399161,6cyclictest0-21swapper/209:57:072
25493991612,3cyclictest0-21swapper/211:51:382
25493991612,3cyclictest0-21swapper/209:45:382
25493991612,3cyclictest0-21swapper/207:38:352
2549399161,14cyclictest0-21swapper/212:20:452
25493991611,3cyclictest29323-21open_inodes10:55:232
25493991611,3cyclictest11942-21hwlatdetect11:30:182
25493991610,3cyclictest827-21runrttasks07:47:332
25493991610,3cyclictest7508-21cat12:35:012
2548199162,3cyclictest10201-21cstates09:00:130
25481991613,2cyclictest6031-21awk10:05:020
134322161,3sleep20-21swapper/207:53:262
2549399159,3cyclictest3577-21grep08:45:142
2549399155,3cyclictest87550irq/128-enp1s0-07:10:142
2549399154,3cyclictest23198-21gltestperf08:15:162
25493991510,3cyclictest31475-21cpuspeed_turbos08:35:142
25493991510,3cyclictest10267-21cat09:00:142
2548199151,6cyclictest5621-21/usr/sbin/munin08:50:130
2548199151,3cyclictest20394-21/usr/sbin/munin08:10:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional