You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-09 - 12:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Wed Sep 09, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21412238162,25sleep20-21swapper/219:09:182
18622237161,25sleep00-21swapper/019:05:430
19872235163,24sleep30-21swapper/319:07:193
19032234165,22sleep10-21swapper/119:06:171
2515021680,4sleep2228899cyclictest22:25:182
2515021680,4sleep2228899cyclictest22:25:182
220921540,4sleep0227899cyclictest21:35:130
1248021460,4sleep2228899cyclictest23:10:182
2288991081,105cyclictest0-21swapper/221:49:302
2288998986,2cyclictest0-21swapper/222:46:292
2288998279,2cyclictest0-21swapper/222:54:352
324892570,1sleep30-21swapper/320:15:173
312352560,3sleep30-21swapper/322:40:163
218522560,4sleep121-21rcuc/119:50:231
73742530,1sleep10-21swapper/121:45:161
100212510,4sleep0227899cyclictest21:50:260
295892490,4sleep232-21ksoftirqd/223:50:152
2288994945,3cyclictest0-21swapper/220:27:242
321852470,1sleep10-21swapper/122:40:271
171682450,2sleep00-21swapper/019:40:190
2278993910,13cyclictest16435-21awk00:35:010
227899381,27cyclictest11614-21kworker/0:023:24:000
227899339,11cyclictest29955-21idleruntime-cro21:25:000
227899335,13cyclictest22729-21munin-run19:55:010
227899334,13cyclictest13927-21idleruntime-cro19:35:020
227899331,19cyclictest11838-21kworker/0:100:28:410
187402330,2sleep00-21swapper/000:40:010
227899328,11cyclictest8589-21idleruntime-cro21:50:020
229399311,14cyclictest808-21snmpd20:25:223
227899315,13cyclictest24578-21munin-run23:40:000
227899315,13cyclictest24578-21munin-run23:40:000
227899312,10cyclictest17981-21kworker/0:219:50:010
229399291,12cyclictest20731-21unixbench_singl21:00:273
228399293,12cyclictest23347-21diskmemload21:52:241
228399291,3cyclictest0-21swapper/119:25:121
229399282,12cyclictest20866-21ntp_states22:15:233
228399281,16cyclictest28380-21timerandwakeup23:45:261
228399281,16cyclictest0-21swapper/122:03:521
227899281,13cyclictest0-21swapper/020:55:000
21312280,1sleep30-21swapper/300:00:193
228399271,17cyclictest0-21swapper/121:10:181
228399271,16cyclictest6465-21if_err_enp2s000:10:181
228399271,16cyclictest0-21swapper/122:32:091
228399270,19cyclictest0-21swapper/122:56:011
227899273,11cyclictest3245-21idleruntime-cro00:05:020
227899272,12cyclictest26444-21kworker/0:023:47:000
227899271,17cyclictest17081-21kworker/0:222:17:290
2278992711,14cyclictest12365-21perf20:45:020
120332270,3sleep3229399cyclictest21:55:223
229399265,10cyclictest31162-21turbostat.cron23:55:013
2293992611,3cyclictest3397-21fschecks_time22:50:183
228399261,2cyclictest0-21swapper/119:57:231
228399260,7cyclictest0-21swapper/123:58:221
227899261,10cyclictest30282-21kworker/0:222:42:210
229399253,11cyclictest18551-21/usr/sbin/munin19:45:103
2283992515,2cyclictest0-21swapper/100:23:551
2283992514,3cyclictest0-21swapper/121:20:091
228399251,3cyclictest1345-21perf20:20:011
228399251,15cyclictest23347-21diskmemload22:06:281
228399250,16cyclictest0-21swapper/123:09:221
227899251,11cyclictest18952-21awk21:00:000
229399247,8cyclictest22953-21/usr/sbin/munin19:55:133
229399245,16cyclictest5833-21/usr/sbin/munin22:55:173
229399244,7cyclictest808-21snmpd23:46:153
228399242,11cyclictest14548-21munin-run20:50:021
228399241,4cyclictest30557-21munin-run22:40:011
2283992414,2cyclictest0-21swapper/120:42:291
2283992414,2cyclictest0-21swapper/100:06:341
228399241,3cyclictest4945-21ntp_states19:15:031
228399241,3cyclictest27262-21unixbench_multi21:15:271
228399241,3cyclictest27262-21unixbench_multi21:15:271
228399241,3cyclictest22373-21latency_hist23:35:011
228399241,3cyclictest22237-21ls22:20:131
2283992413,3cyclictest14889-21seq23:15:221
228399241,12cyclictest24869-21hddtemp_smartct22:25:161
228399241,12cyclictest24869-21hddtemp_smartct22:25:161
227899241,8cyclictest12643-21kworker/0:022:02:210
229399234,6cyclictest23871-21expr21:10:133
229399233,12cyclictest30508-21df_inode21:25:143
229399233,10cyclictest10862-21cat21:55:023
228899235,9cyclictest10770-21crond21:55:022
228399239,3cyclictest31525-21munin-run20:15:011
228399231,3cyclictest9196-21latency_hist23:05:021
228399231,3cyclictest3224-21latency_hist00:05:011
228399231,3cyclictest29871-21hddtemp_smartct23:50:161
228399231,3cyclictest19030-21cut21:00:011
228399231,16cyclictest0-21swapper/100:38:011
228399231,15cyclictest0-21swapper/123:20:221
228399230,6cyclictest0-21swapper/123:37:411
228399230,6cyclictest0-21swapper/123:37:411
227899234,16cyclictest19577-21munin-run22:15:010
227899232,9cyclictest28038-21kworker/0:122:39:510
227899232,12cyclictest17949-21kworker/0:123:26:200
227899232,12cyclictest11614-21kworker/0:023:13:040
227899232,10cyclictest30282-21kworker/0:222:55:330
227899231,15cyclictest742-21kworker/0:000:13:170
227899230,11cyclictest7360-21perf19:20:000
229399225,8cyclictest13926-21grep23:15:133
229399223,7cyclictest0-21swapper/322:05:013
229399223,4cyclictest13322-21ntp_states00:25:213
229399222,6cyclictest87350irq/126-enp1s0-19:20:133
229399222,12cyclictest763-21dbus-daemon20:52:163
229399221,11cyclictest22949-21df_inode23:35:153
229399221,11cyclictest22949-21df_inode23:35:153
228899226,8cyclictest87350irq/126-enp1s0-23:25:192
228899225,9cyclictest5274-21latency21:40:162
228899224,5cyclictest7982-21grep19:20:142
228899222,11cyclictest19516-21df_inode21:00:152
2283992213,3cyclictest0-21swapper/100:25:571
2283992213,2cyclictest0-21swapper/123:25:221
228399221,19cyclictest0-21swapper/119:24:181
228399221,15cyclictest0-21swapper/121:28:351
228399220,15cyclictest0-21swapper/121:35:221
227899222,11cyclictest9589-21kworker/0:200:20:490
227899222,11cyclictest32704-21kworker/0:022:45:250
227899222,11cyclictest25451-21kworker/0:021:17:000
227899222,11cyclictest25451-21kworker/0:021:17:000
227899221,9cyclictest24735-21kworker/0:120:27:410
258832211,16sleep0808-21snmpd22:26:340
258832211,16sleep0808-21snmpd22:26:340
229399217,7cyclictest87350irq/126-enp1s0-20:05:203
229399217,7cyclictest0-21swapper/323:26:113
229399215,2cyclictest39-21rcuc/319:10:153
229399215,13cyclictest1-21systemd22:20:163
229399211,9cyclictest0-21swapper/322:45:373
229399211,6cyclictest0-21swapper/300:32:483
229399211,4cyclictest0-21swapper/300:16:013
229399210,9cyclictest0-21swapper/321:20:063
228899215,4cyclictest15240-21fschecks_time20:50:152
228899211,7cyclictest312382cut22:40:172
228899211,6cyclictest3223-21sh00:05:022
228899211,11cyclictest8441-21perl00:15:162
228399218,6cyclictest655-21awk21:30:191
228399211,3cyclictest9206-21/usr/sbin/munin20:35:201
228399211,18cyclictest0-21swapper/122:17:361
228399211,18cyclictest0-21swapper/120:22:261
228399211,18cyclictest0-21swapper/119:37:301
228399211,18cyclictest0-21swapper/119:14:411
228399211,17cyclictest269442sleep120:02:481
228399211,14cyclictest0-21swapper/121:43:411
228399211,14cyclictest0-21swapper/100:32:561
228399211,11cyclictest3503-21hddtemp_smartct22:50:201
228399211,11cyclictest1851-21grep22:45:221
227899219,2cyclictest25806-21kworker/0:022:30:250
227899218,2cyclictest3079-21kworker/0:100:06:050
227899217,2cyclictest7257-21kworker/0:019:22:280
227899217,2cyclictest11614-21kworker/0:023:18:120
227899212,9cyclictest4140-21kworker/0:319:11:490
227899212,7cyclictest11614-21kworker/0:023:30:360
227899212,11cyclictest28716-21kworker/0:123:52:280
227899212,11cyclictest12643-21kworker/0:022:06:470
227899211,9cyclictest27030-21kworker/0:020:04:290
227899211,8cyclictest24735-21kworker/0:120:23:570
2278992114,4cyclictest11394-21dump-pmu-power23:10:020
227899210,9cyclictest24735-21kworker/0:120:14:090
229399207,5cyclictest0-21swapper/323:20:393
229399206,4cyclictest0-21swapper/321:45:243
229399206,2cyclictest401ktimersoftd/320:30:283
229399205,5cyclictest0-21swapper/322:35:223
229399204,5cyclictest10763-21idleruntime00:20:173
229399204,4cyclictest0-21swapper/323:30:143
229399204,2cyclictest87350irq/126-enp1s0-00:05:233
229399203,6cyclictest14159-21/usr/sbin/munin19:35:213
229399203,11cyclictest0-21swapper/322:28:013
229399203,11cyclictest0-21swapper/322:28:013
229399202,10cyclictest8341-21cpuspeed_turbos20:35:123
229399201,10cyclictest16354-21/usr/sbin/munin19:40:183
2293992010,4cyclictest0-21swapper/323:59:093
229399200,14cyclictest0-21swapper/323:05:183
228899206,3cyclictest87350irq/126-enp1s0-21:10:132
228899203,4cyclictest21716-21df_inode21:05:142
228899202,10cyclictest0-21swapper/222:59:282
228899201,8cyclictest22011-21vmstat23:30:272
228899201,7cyclictest32146-21awk21:30:012
2288992015,3cyclictest26771-21munin-plugin-st23:45:012
228899201,11cyclictest18416-21irqstats22:10:162
228899201,10cyclictest17621-21smart_sda19:40:242
228399208,3cyclictest16519-40dnf20:52:161
228399201,18cyclictest0-21swapper/120:07:341
228399201,17cyclictest0-21swapper/120:30:181
228399201,12cyclictest19385-21kworker/u8:022:13:111
228399201,10cyclictest5755-21latency_hist20:30:021
228399201,10cyclictest0-21swapper/121:56:181
228399200,2cyclictest87550irq/128-enp1s0-23:44:081
228399200,18cyclictest0-21swapper/119:30:521
228399200,14cyclictest0-21swapper/121:06:511
228399200,14cyclictest0-21swapper/100:15:091
227899208,2cyclictest7257-21kworker/0:019:26:210
227899207,2cyclictest5557-21kworker/0:220:32:550
227899206,3cyclictest18736-21kworker/0:121:01:500
227899202,8cyclictest30959-21kworker/0:223:56:280
227899202,11cyclictest8780-21kworker/0:121:56:130
227899202,10cyclictest20198-21kworker/0:223:40:160
227899201,9cyclictest9181-21kworker/0:119:35:530
227899201,11cyclictest6798-21kworker/0:123:01:490
227899201,10cyclictest30282-21kworker/0:222:50:210
244792191,3sleep20-21swapper/223:38:562
244792191,3sleep20-21swapper/223:38:562
229399198,4cyclictest0-21swapper/321:30:163
229399198,4cyclictest0-21swapper/300:35:183
229399198,3cyclictest26391-21/usr/sbin/munin22:30:123
229399195,2cyclictest17599-21/usr/sbin/munin22:10:143
229399193,8cyclictest12758-21cstates20:45:143
229399193,6cyclictest0-21swapper/323:42:353
229399193,3cyclictest0-21swapper/319:54:193
229399193,11cyclictest0-21swapper/323:12:473
229399193,11cyclictest0-21swapper/300:12:073
229399193,10cyclictest0-21swapper/321:36:123
229399192,10cyclictest0-21swapper/321:15:043
229399192,10cyclictest0-21swapper/321:15:043
229399191,11cyclictest0-21swapper/323:00:443
229399191,10cyclictest0-21swapper/321:40:383
228899196,3cyclictest87350irq/126-enp1s0-00:05:252
228899195,9cyclictest0-21swapper/200:30:512
228899195,3cyclictest87350irq/126-enp1s0-23:00:272
228899193,4cyclictest11870-21sed21:55:192
228899192,5cyclictest32096-21diskstats20:15:142
228899192,5cyclictest12824-21expr20:45:152
228899191,9cyclictest6732-21ntp_states00:10:222
228899191,6cyclictest0-21swapper/221:19:482
228899191,6cyclictest0-21swapper/221:19:482
2288991914,3cyclictest3161-21grep20:20:252
228899191,10cyclictest21147-21fschecks_count19:50:152
228899191,10cyclictest0-21swapper/221:20:092
228899190,4cyclictest0-21swapper/222:00:132
228399197,3cyclictest20519-21sh19:50:011
2283991915,3cyclictest0-21swapper/119:40:191
2283991912,2cyclictest0-21swapper/123:11:061
227899198,2cyclictest24735-21kworker/0:120:16:570
227899197,3cyclictest23216-21kworker/0:221:11:540
227899197,2cyclictest18736-21kworker/0:121:07:200
227899195,3cyclictest17981-21kworker/0:219:57:250
227899192,7cyclictest29668-21kworker/0:321:42:530
227899192,7cyclictest29668-21kworker/0:321:34:480
227899192,10cyclictest24735-21kworker/0:120:06:290
229399182,3cyclictest0-21swapper/320:21:243
229399181,6cyclictest0-21swapper/320:55:233
2293991811,4cyclictest11966-21/usr/sbin/munin19:30:153
229399181,10cyclictest23350-21grep21:10:013
229399181,10cyclictest0-21swapper/319:18:483
228899188,2cyclictest10324-21date20:40:022
228899183,3cyclictest2253-21cpu21:35:132
228899182,6cyclictest10913-21pmu-power19:25:222
228899182,4cyclictest18239-21users00:35:272
228899181,9cyclictest0-21swapper/223:58:282
228899181,9cyclictest0-21swapper/223:18:162
228899181,9cyclictest0-21swapper/221:30:502
228899181,8cyclictest0-21swapper/200:20:222
228899181,6cyclictest28102-21/usr/sbin/munin23:45:232
228899181,6cyclictest10405-21/usr/sbin/munin23:05:192
228899181,5cyclictest22909-21load22:20:182
228899181,5cyclictest15646-21expr22:05:122
228899181,3cyclictest27455-21ntp_states22:30:202
228899181,2cyclictest0-21swapper/219:15:172
228899181,10cyclictest0-21swapper/223:20:592
228899181,10cyclictest0-21swapper/222:38:412
228899181,10cyclictest0-21swapper/200:25:402
227899187,3cyclictest7863-21/usr/sbin/munin00:15:160
227899185,2cyclictest5557-21kworker/0:220:37:010
227899183,2cyclictest87550irq/128-enp1s0-21:25:260
229399176,3cyclictest87650irq/129-enp1s0-20:00:193
2293991712,3cyclictest16476-21ntpq22:05:213
229399171,14cyclictest0-21swapper/319:25:583
228899174,3cyclictest87350irq/126-enp1s0-20:40:132
228899172,8cyclictest0-21swapper/220:33:302
228899171,9cyclictest28032-21sshd20:05:152
228899171,9cyclictest0-21swapper/222:15:052
228899171,4cyclictest0-21swapper/219:46:342
228899171,3cyclictest3476-21strings19:10:212
2283991712,3cyclictest19173-21/usr/sbin/munin21:00:181
227899178,2cyclictest0-21swapper/022:20:170
2293991610,3cyclictest11707-21cat20:40:243
228899161,9cyclictest0-21swapper/220:56:212
228899161,9cyclictest0-21swapper/220:02:232
228899161,9cyclictest0-21swapper/219:58:102
228899161,9cyclictest0-21swapper/219:30:172
228899161,14cyclictest0-21swapper/219:35:222
2288991611,3cyclictest30622-21ntpq20:10:222
2278991611,3cyclictest14164-21users20:45:280
2293991510,3cyclictest30378-21kernelversion20:10:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional