You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-08 - 16:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Mon Jun 08, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
323822239167,23sleep20-21swapper/207:05:552
326292232162,57sleep00-21swapper/007:09:070
325332222154,55sleep10-21swapper/107:07:511
326372203166,24sleep30-21swapper/307:09:133
321991090,107cyclictest0-21swapper/209:37:352
321998077,2cyclictest0-21swapper/209:00:372
321998077,2cyclictest0-21swapper/208:01:482
321997669,6cyclictest0-21swapper/212:30:182
196012480,4sleep019604-21idleruntime07:50:160
291942450,1sleep10-21swapper/110:40:111
260422450,1sleep226045-21awk10:30:262
325993813,18cyclictest26423-21smart_sda11:45:253
30799389,14cyclictest14897-21munin-run08:55:010
32599378,20cyclictest19472-21unixbench_multi10:15:253
32199371,24cyclictest0-21swapper/207:28:462
321993515,7cyclictest0-21swapper/208:59:112
307993511,11cyclictest19316-21basename09:05:010
32599346,6cyclictest21498-21munin-run09:10:013
325993413,6cyclictest20240-40dnf11:31:253
32199341,22cyclictest0-21swapper/209:08:152
31599348,3cyclictest0-21swapper/107:29:211
32599332,18cyclictest87350irq/126-enp1s0-10:25:153
32599332,18cyclictest87350irq/126-enp1s0-10:25:153
32199331,18cyclictest15420-21memory07:40:192
32199330,21cyclictest0-21swapper/207:14:232
30799336,12cyclictest5504-21perf07:20:000
32599328,16cyclictest12275-21ls10:00:183
32599327,8cyclictest15412-21runrttasks10:08:453
325993215,8cyclictest773-21NetworkManager10:22:303
325993212,3cyclictest0-21swapper/309:41:183
325993211,10cyclictest1518-21wc09:35:243
32199321,20cyclictest0-21swapper/208:15:352
32199321,18cyclictest0-21swapper/211:58:352
32199321,15cyclictest0-21swapper/212:36:312
32199321,15cyclictest0-21swapper/210:39:432
32199321,13cyclictest0-21swapper/208:25:192
32199321,12cyclictest9448-21memory08:40:192
32199321,12cyclictest0-21swapper/211:05:422
31599321,23cyclictest0-21swapper/111:24:331
31599321,11cyclictest0-21swapper/107:24:031
31599320,25cyclictest0-21swapper/107:55:231
31599320,12cyclictest0-21swapper/110:23:171
30799328,11cyclictest26527-21idleruntime-cro10:35:010
32599319,5cyclictest87350irq/126-enp1s0-11:35:163
32599319,5cyclictest87350irq/126-enp1s0-11:35:153
32599319,10cyclictest10390-21find09:55:253
325993114,6cyclictest10689-21memory11:10:183
325993112,10cyclictest11456-21ntp_states12:25:213
325993111,12cyclictest24599-21switchtime08:00:233
325993110,12cyclictest25354-21irqstats10:30:183
321993113,10cyclictest0-21swapper/207:51:452
32199311,22cyclictest0-21swapper/211:43:552
32199310,14cyclictest0-21swapper/209:10:262
31599316,3cyclictest0-21swapper/107:53:011
31599312,19cyclictest0-21swapper/111:30:111
31599311,9cyclictest0-21swapper/109:30:151
31599311,23cyclictest0-21swapper/108:18:171
30799313,13cyclictest3905-21munin-run08:30:000
32599308,13cyclictest7926-21/usr/sbin/munin11:05:143
32599306,17cyclictest4945-21cat09:45:113
325993014,4cyclictest27705-21diskstats11:50:143
325993010,6cyclictest15152-21sed07:40:163
325993010,11cyclictest653-21latency_hist10:50:023
32199301,14cyclictest0-21swapper/210:50:312
32199301,12cyclictest0-21swapper/212:23:552
321993010,8cyclictest0-21swapper/211:49:512
32199300,22cyclictest0-21swapper/209:41:312
32599296,12cyclictest29663-21latency_hist08:15:023
321992912,7cyclictest0-21swapper/211:00:152
32199291,12cyclictest0-21swapper/208:22:272
32199291,10cyclictest0-21swapper/211:39:312
32199291,10cyclictest0-21swapper/211:39:312
321992910,9cyclictest21495-21diskmemload10:40:592
32199290,14cyclictest0-21swapper/209:25:352
32199290,13cyclictest0-21swapper/209:56:152
32199290,12cyclictest0-21swapper/208:50:232
315992916,2cyclictest16473-21munin-run07:45:011
31599291,26cyclictest0-21swapper/109:49:441
32599289,11cyclictest7964-21latency_hist12:20:013
32599288,9cyclictest17246-21ls11:25:163
32599287,12cyclictest14826-21cpu12:35:123
32599287,10cyclictest17055-21switchtime10:10:243
325992819,5cyclictest13229-21smart_sda11:15:223
325992817,8cyclictest31104-21gltestperf09:30:143
325992811,6cyclictest23637-21hddtemp_smartct11:40:153
32199281,20cyclictest0-21swapper/210:02:432
32199281,20cyclictest0-21swapper/209:33:272
32199281,20cyclictest0-21swapper/208:13:512
32199281,20cyclictest0-21swapper/207:48:072
321992811,6cyclictest0-21swapper/209:49:232
32199281,14cyclictest0-21swapper/209:17:102
32199281,14cyclictest0-21swapper/209:17:102
32199281,13cyclictest0-21swapper/212:27:072
32199281,13cyclictest0-21swapper/210:09:592
32199281,11cyclictest0-21swapper/209:54:192
32199280,13cyclictest0-21swapper/207:38:352
31599281,17cyclictest0-21swapper/109:26:131
315992811,4cyclictest12417-21ntp_states10:00:211
31599281,13cyclictest0-21swapper/108:30:151
31599281,10cyclictest0-21swapper/112:19:211
32599278,9cyclictest29621-21hwlatdetect10:40:163
32599276,18cyclictest1281-21tail07:10:163
325992718,5cyclictest5397-21smart_sda08:30:233
325992710,14cyclictest30385-21fschecks_time08:15:143
325992710,14cyclictest12721-21grep12:30:123
32599270,16cyclictest0-21swapper/312:00:393
32199271,15cyclictest0-21swapper/212:10:112
32199271,13cyclictest0-21swapper/211:17:262
32199271,13cyclictest0-21swapper/208:31:272
32199271,12cyclictest0-21swapper/212:01:032
32199271,11cyclictest0-21swapper/212:18:152
32199271,11cyclictest0-21swapper/209:23:432
32199270,13cyclictest0-21swapper/207:58:592
31599271,23cyclictest11403-21hddtemp_smartct08:45:151
31599271,17cyclictest0-21swapper/111:39:181
31599271,17cyclictest0-21swapper/111:39:181
31599271,16cyclictest15567-21latency_hist10:10:011
31599271,11cyclictest16010-21grep12:35:231
31599271,10cyclictest0-21swapper/110:26:431
31599271,10cyclictest0-21swapper/110:26:431
32599268,9cyclictest24717-21cat09:15:173
32599268,9cyclictest24717-21cat09:15:173
32599268,11cyclictest29002-21sed09:25:163
32599266,11cyclictest7242-21expr09:50:113
32599264,16cyclictest87350irq/126-enp1s0-10:50:133
32599263,10cyclictest0-21swapper/312:20:213
325992610,6cyclictest9901-21kworker/u8:009:00:223
325992610,6cyclictest7685-21cat08:35:253
32199269,7cyclictest49832chrt10:58:072
32199261,16cyclictest0-21swapper/207:23:072
32199261,12cyclictest0-21swapper/211:54:112
32199261,12cyclictest0-21swapper/208:08:112
32199261,12cyclictest0-21swapper/208:08:112
32199261,11cyclictest0-21swapper/211:10:542
32199260,13cyclictest0-21swapper/210:47:192
32199260,12cyclictest0-21swapper/212:06:352
31599269,3cyclictest0-21swapper/108:50:241
31599268,10cyclictest19609-21kworker/u8:010:30:371
31599266,3cyclictest0-21swapper/108:35:351
315992612,3cyclictest0-21swapper/112:13:441
31599261,15cyclictest0-21swapper/107:17:531
31599261,11cyclictest0-21swapper/108:27:551
31599261,10cyclictest0-21swapper/108:13:211
31599261,10cyclictest0-21swapper/108:06:571
31599261,10cyclictest0-21swapper/108:06:571
31599260,24cyclictest0-21swapper/107:14:101
32599258,9cyclictest27654-21memory10:35:203
32599258,5cyclictest6789-21ntp_states07:20:233
32599256,11cyclictest28141-21sed09:25:013
32599255,12cyclictest30896-21timerandwakeup11:55:253
32599255,10cyclictest0-21swapper/307:40:003
32599254,17cyclictest808-21snmpd07:15:363
325992516,5cyclictest2677-21ntp_states12:05:233
325992513,3cyclictest87350irq/126-enp1s0-08:20:133
32199251,18cyclictest0-21swapper/210:28:552
32199251,18cyclictest0-21swapper/210:28:552
32199251,18cyclictest0-21swapper/208:35:232
32199251,12cyclictest0-21swapper/210:18:552
32199251,11cyclictest0-21swapper/207:33:472
321992510,8cyclictest0-21swapper/211:27:072
31599258,3cyclictest24013-21idleruntime08:00:151
31599258,3cyclictest0-21swapper/107:33:251
31599251,9cyclictest0-21swapper/107:35:391
315992513,3cyclictest0-21swapper/111:25:141
31599251,15cyclictest21495-21diskmemload12:33:081
31599251,12cyclictest23733-21if_err_enp1s011:40:171
31599251,11cyclictest8537-21sh12:20:151
31599251,10cyclictest2464-21sort12:05:191
31599251,10cyclictest0-21swapper/112:00:351
31599251,10cyclictest0-21swapper/109:41:411
30799251,12cyclictest8301-21munin-run08:40:010
32599248,4cyclictest27463-21latency_hist08:10:023
32599244,9cyclictest3934-21if_enp2s010:55:183
32599243,8cyclictest0-21swapper/307:25:263
325992416,5cyclictest22988-21smart_sda09:10:243
325992415,6cyclictest6443-21ntp_states11:00:243
325992415,6cyclictest15229-21/usr/sbin/munin11:20:233
325992414,7cyclictest11184-21ntp_states07:30:213
32199241,12cyclictest0-21swapper/211:32:152
321992410,7cyclictest0-21swapper/211:21:592
31599248,3cyclictest0-21swapper/111:03:591
31599241,9cyclictest22026-21sadc09:10:131
31599241,9cyclictest0-21swapper/110:54:291
31599241,9cyclictest0-21swapper/109:04:071
31599241,5cyclictest0-21swapper/111:08:241
315992413,3cyclictest0-21swapper/108:22:571
31599241,10cyclictest17228-21grep10:10:251
31599240,15cyclictest0-21swapper/111:50:171
31599240,15cyclictest0-21swapper/111:48:311
31599240,15cyclictest0-21swapper/110:55:101
30799245,16cyclictest19922-21munin-run10:20:000
30799241,6cyclictest30292-21perf09:30:010
307992410,8cyclictest4885-21if_enp2s008:30:160
325992316,4cyclictest5064-21smart_sda12:10:233
325992316,4cyclictest15799-21hddtemp_smartct08:55:163
325992315,5cyclictest26765-21smart_sda08:05:253
325992314,6cyclictest9343-21irqstats08:40:183
325992314,6cyclictest18896-21/usr/sbin/munin07:50:223
325992314,6cyclictest12925-21/usr/sbin/munin08:50:133
325992314,6cyclictest10726-21/usr/sbin/munin08:45:133
32199231,8cyclictest0-21swapper/210:20:512
32199231,14cyclictest0-21swapper/207:15:192
32199231,12cyclictest0-21swapper/208:46:352
31599239,3cyclictest17962-21smart_sda07:45:231
31599238,3cyclictest0-21swapper/110:18:351
31599237,7cyclictest30467-21memory11:55:191
31599231,14cyclictest0-21swapper/109:36:031
31599231,11cyclictest31353-21expr10:45:111
31599231,11cyclictest0-21swapper/110:39:151
31599231,10cyclictest0-21swapper/109:08:391
31599231,10cyclictest0-21swapper/108:58:431
30799231,7cyclictest11386-21needreboot12:25:190
150862230,6sleep014167-21/usr/sbin/munin11:20:210
325992215,4cyclictest2612-21hddtemp_smartct08:25:163
325992215,4cyclictest17374-21hddtemp_smartct07:45:143
32199220,8cyclictest0-21swapper/210:15:012
31599223,12cyclictest87550irq/128-enp1s0-09:57:041
315992213,1cyclictest0-21swapper/112:27:311
31599221,15cyclictest0-21swapper/111:14:271
31599220,14cyclictest0-21swapper/108:41:031
30799221,4cyclictest22015-21sendmail_mailtr11:35:240
30799221,4cyclictest22015-21sendmail_mailtr11:35:240
325992114,5cyclictest21113-21/usr/sbin/munin07:55:123
31599211,11cyclictest0-21swapper/109:17:171
31599211,11cyclictest0-21swapper/109:17:171
31599211,10cyclictest0-21swapper/111:18:551
31599210,10cyclictest0-21swapper/109:53:311
30799212,16cyclictest16486-21awk07:45:010
30799211,6cyclictest0-21swapper/009:35:310
30799211,11cyclictest19628-21ntp_states11:30:220
30799210,6cyclictest4463-21kworker/u8:009:50:240
30799209,3cyclictest808-21snmpd09:23:000
30799202,10cyclictest29891-21/usr/sbin/munin08:15:260
30799201,7cyclictest11968-21/usr/sbin/munin11:15:170
30799201,6cyclictest0-21swapper/012:11:450
30799201,6cyclictest0-21swapper/010:59:500
30799201,4cyclictest0-21swapper/010:21:200
307992014,3cyclictest14334-40dnf12:32:160
30799201,3cyclictest0-21swapper/011:29:000
307992013,4cyclictest21707-21grep07:55:170
30799201,12cyclictest952-21/usr/sbin/munin10:50:170
30799201,11cyclictest9771-21/usr/sbin/munin11:10:130
130612201,2sleep00-21swapper/010:01:440
99742191,3sleep00-21swapper/012:22:410
31599199,2cyclictest27911-21chrt09:22:511
30799191,7cyclictest0-21swapper/012:39:440
30799191,6cyclictest14366-21irqstats10:05:180
30799191,6cyclictest0-21swapper/010:45:330
30799191,5cyclictest8564-1kworker/u9:112:07:060
30799191,5cyclictest0-21swapper/009:13:330
307991914,3cyclictest30910-21awk10:45:010
307991913,3cyclictest11366-21gltestperf08:45:150
30799191,10cyclictest32086-21/usr/sbin/munin08:20:130
307991910,5cyclictest235652chrt10:25:230
307991910,5cyclictest235652chrt10:25:230
30799185,3cyclictest31060-21grep11:55:260
30799181,6cyclictest0-21swapper/011:00:070
30799181,4cyclictest0-21swapper/011:42:270
30799181,4cyclictest0-21swapper/010:10:530
30799181,4cyclictest0-21swapper/009:34:530
30799181,3cyclictest0-21swapper/010:36:300
30799181,12cyclictest0-21swapper/009:46:450
256952180,2sleep00-21swapper/009:17:370
256952180,2sleep00-21swapper/009:17:370
30799175,3cyclictest26150-21strings11:45:200
30799171,9cyclictest0-21swapper/007:25:200
30799171,7cyclictest28459-21if_enp2s008:10:160
30799171,6cyclictest0-21swapper/012:00:250
30799171,6cyclictest0-21swapper/011:05:590
30799171,2cyclictest0-21swapper/011:53:230
307991711,3cyclictest2922-21sshd09:40:130
307991711,3cyclictest15795-21hddtemp_smartct08:55:160
30799165,3cyclictest87550irq/128-enp1s0-07:32:350
30799163,4cyclictest6397-21cut12:15:140
30799161,7cyclictest8528-21/usr/sbin/munin08:40:150
307991611,3cyclictest533-21/usr/sbin/munin07:10:200
307991611,3cyclictest24072-21if_err_enp1s008:00:160
30799161,12cyclictest16699-21/usr/sbin/munin07:45:150
307991610,4cyclictest1-21systemd07:38:140
307991610,3cyclictest7701-21basename07:25:010
30799159,3cyclictest20203-21hddtemp_smartct09:05:160
30799151,5cyclictest0-21swapper/009:55:160
307991510,3cyclictest25714-21cat08:05:140
307991510,3cyclictest25714-21cat08:05:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional