You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 01:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue May 05, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
186092220144,25sleep20-21swapper/207:08:012
184372205169,23sleep10-21swapper/107:05:461
187382202164,26sleep00-21swapper/007:09:400
168042201165,24sleep30-21swapper/307:05:093
2998821640,4sleep31886399cyclictest08:45:173
77582560,1sleep00-21swapper/007:55:180
1886399501,6cyclictest901-21tune2fs11:35:133
1886399411,29cyclictest31454-21cat11:30:173
1886399411,29cyclictest31454-21cat11:30:173
18856994113,10cyclictest29114-21sed11:25:162
1886399401,26cyclictest16174-21memory10:55:183
1886399391,27cyclictest25035-21sh12:30:153
1884299398,15cyclictest30511-21munin-run07:35:000
1886399371,8cyclictest6762-21grep09:05:233
1886399371,4cyclictest20077-21sleep12:20:013
18856993715,12cyclictest9059-21crond11:55:012
18856993711,16cyclictest18720-21smart_sda11:00:242
1886399363,3cyclictest87350irq/126-enp1s0-12:05:133
1886399363,28cyclictest15598-40dnf08:13:303
1886399362,5cyclictest0-21swapper/310:05:263
18856993612,10cyclictest21361-21kworker/u8:112:00:022
1886399351,5cyclictest1840-21latency_hist10:25:013
1886399351,5cyclictest0-21swapper/312:25:563
18863993510,21cyclictest1-21systemd12:10:133
18856993515,11cyclictest0-21swapper/207:35:012
1886399349,14cyclictest23630-21sshd11:12:273
1886399343,3cyclictest13063-21/usr/sbin/munin10:50:123
1886399342,27cyclictest0-21swapper/311:15:023
1886399341,6cyclictest0-21swapper/311:09:263
1886399341,3cyclictest463-21/usr/sbin/munin07:40:133
1886399335,18cyclictest4048-21perf10:30:013
1886399332,5cyclictest0-21swapper/310:42:143
1886399332,5cyclictest0-21swapper/310:03:083
1886399331,7cyclictest14934-21fschecks_time09:25:163
1886399331,19cyclictest0-21swapper/309:34:283
1885699338,13cyclictest0-21swapper/212:35:112
1885699338,12cyclictest3689-21kworker/u8:010:38:532
18856993312,13cyclictest24753-21/usr/sbin/munin08:35:232
18856993310,13cyclictest11631-21kworker/u8:009:24:342
1886399322,3cyclictest4331-21/usr/sbin/munin10:30:113
1886399322,3cyclictest12548-21ntp_states12:00:203
1886399321,6cyclictest0-21swapper/311:56:113
1886399321,6cyclictest0-21swapper/309:12:493
1886399321,6cyclictest0-21swapper/309:12:493
1886399321,5cyclictest808-21snmpd08:17:473
1886399321,5cyclictest4699-21cat09:00:243
1886399321,5cyclictest29451-21memory07:30:183
1886399321,4cyclictest26722-21latency_hist08:40:023
1886399321,21cyclictest0-21swapper/311:50:293
1886399321,20cyclictest0-21swapper/309:41:553
1886399321,10cyclictest7556-21diskmemload09:45:333
18863993210,19cyclictest18346-21cat08:20:123
1886399320,7cyclictest0-21swapper/310:11:233
1885699329,9cyclictest22001-21kworker/u8:209:10:242
1885699329,9cyclictest22001-21kworker/u8:209:10:242
1886399317,21cyclictest13740-21grep09:20:263
1886399313,5cyclictest0-21swapper/311:40:123
1886399313,3cyclictest5750irq/9-acpi08:55:213
1886399313,21cyclictest0-21swapper/309:16:343
1886399313,19cyclictest0-21swapper/309:54:253
1886399312,2cyclictest87550irq/128-enp1s0-12:20:203
1886399311,5cyclictest25659-21mailstats09:55:253
1886399311,4cyclictest0-21swapper/309:35:203
1886399311,3cyclictest87350irq/126-enp1s0-07:35:173
1886399311,3cyclictest17642-21expr11:00:123
1886399311,3cyclictest0-21swapper/307:25:273
1886399311,2cyclictest0-21swapper/311:25:013
1886399311,18cyclictest0-21swapper/310:15:173
18863993111,17cyclictest10989-21grep10:45:113
18863993110,18cyclictest27745-21cat12:35:173
1885699319,12cyclictest22345-21kworker/u8:112:28:392
1885699318,12cyclictest0-21swapper/209:45:182
18856993112,11cyclictest0-21swapper/212:10:022
18856993112,10cyclictest21361-21kworker/u8:111:23:562
18856993111,11cyclictest7919-1kworker/u9:009:44:562
18850993119,4cyclictest24182-21munin-run09:55:011
1886399309,18cyclictest7347-21cstates07:55:163
1886399308,19cyclictest21183-21kernelversion08:25:183
1886399303,5cyclictest20883-40dnf07:13:223
1886399303,4cyclictest0-21swapper/311:46:163
1886399301,6cyclictest0-21swapper/307:50:253
1886399301,4cyclictest29691-21/usr/sbin/munin11:25:233
18863993014,13cyclictest579-21systemd-journal10:40:013
1885699309,10cyclictest0-21swapper/211:56:282
1885699308,4cyclictest12423-21sort08:05:192
18856993014,5cyclictest0-21swapper/209:00:012
18856993011,11cyclictest0-21swapper/207:55:252
1885699300,16cyclictest32066-21timerwakeupswit11:30:252
1885699300,16cyclictest32066-21timerwakeupswit11:30:252
1886399296,20cyclictest22978-21fschecks_count08:30:153
18863992914,12cyclictest31344-21/usr/sbin/munin08:50:243
1886399291,3cyclictest2766-21/usr/sbin/munin07:45:143
1885699295,16cyclictest7556-21diskmemload11:45:432
18856992911,11cyclictest30997-21chrt08:48:372
18856992911,10cyclictest0-21swapper/209:15:172
18856992910,8cyclictest7556-21diskmemload09:35:482
18856992910,12cyclictest24690-21/usr/sbin/munin12:30:142
1886399282,5cyclictest9702-21df_inode08:00:143
1886399282,5cyclictest9702-21df_inode08:00:143
1885699289,10cyclictest14155-21awk10:50:222
1885699288,5cyclictest87350irq/126-enp1s0-12:20:212
1885699288,11cyclictest0-21swapper/208:00:142
1885699288,11cyclictest0-21swapper/208:00:142
1885699287,7cyclictest16068-21kworker/u8:109:33:172
1885699284,11cyclictest87350irq/126-enp1s0-12:10:212
18856992816,10cyclictest13761-21/usr/sbin/munin08:10:192
18856992813,5cyclictest0-21swapper/211:15:132
18856992812,5cyclictest0-21swapper/207:45:022
18856992811,5cyclictest11631-21kworker/u8:009:29:592
1885699281,10cyclictest0-21swapper/209:52:322
18856992810,11cyclictest0-21swapper/208:25:012
1885699280,16cyclictest5840-21grep10:30:262
1886399271,3cyclictest27292-21df_inode08:40:143
1886399271,3cyclictest11914-21diskstats08:05:163
18863992713,11cyclictest22968-21netstat07:15:203
1885699279,7cyclictest7556-21diskmemload10:10:252
1885699279,13cyclictest1038-21kworker/u8:311:08:292
1885699278,11cyclictest0-21swapper/211:41:152
18856992714,10cyclictest5589-21/usr/sbin/munin09:05:162
18856992711,5cyclictest0-21swapper/207:35:172
18856992710,8cyclictest0-21swapper/207:14:272
18856992710,6cyclictest7556-21diskmemload09:55:352
18856992710,3cyclictest0-21swapper/212:15:192
18856992710,11cyclictest31362-21cat10:15:202
1885699270,15cyclictest22934-21ntp_states11:10:202
1885099272,18cyclictest15340-21kworker/u8:212:20:591
18850992711,10cyclictest0-21swapper/111:25:291
1884299275,3cyclictest87550irq/128-enp1s0-09:34:180
1884299271,16cyclictest0-21swapper/009:28:460
18842992710,15cyclictest17033-21tr12:10:220
1886399261,3cyclictest24491-21/usr/sbin/munin07:20:143
1885699269,7cyclictest0-21swapper/207:45:472
1885699265,7cyclictest1038-21kworker/u8:310:26:432
18856992611,7cyclictest1038-21kworker/u8:310:59:262
18856992610,9cyclictest87350irq/126-enp1s0-10:20:202
18856992610,6cyclictest0-21swapper/210:10:012
18856992610,10cyclictest774-21avahi-daemon10:44:112
1885099261,6cyclictest11829-21df_inode12:00:131
18850992611,3cyclictest0-21swapper/112:33:391
1884299262,14cyclictest11-21rcuc/011:50:230
1884299261,11cyclictest0-21swapper/009:45:230
1885699258,9cyclictest0-21swapper/207:27:382
1885699258,3cyclictest0-21swapper/210:03:312
1885699257,7cyclictest0-21swapper/207:15:132
18856992516,6cyclictest10863-21/usr/sbin/munin10:45:202
18856992510,5cyclictest0-21swapper/208:55:012
1885099255,14cyclictest755-21rngd09:32:271
1885099252,10cyclictest7556-21diskmemload09:20:131
1885099251,18cyclictest0-21swapper/109:37:261
1885099250,15cyclictest0-21swapper/110:50:211
1884299252,11cyclictest18084-21/usr/sbin/munin12:15:100
1885699249,4cyclictest0-21swapper/208:30:122
1885699247,6cyclictest0-21swapper/211:35:122
1885699247,11cyclictest87550irq/128-enp1s0-08:40:182
18856992416,5cyclictest3378-21/usr/sbin/munin09:00:182
18856992415,6cyclictest4961-21/usr/sbin/munin07:50:162
18856992415,6cyclictest24142-21/usr/sbin/munin07:20:172
18856992415,6cyclictest20363-21/usr/sbin/munin08:25:182
18856992415,6cyclictest15950-21taskset08:15:022
18850992416,3cyclictest0-21swapper/112:10:251
1885099241,19cyclictest23301-21irqstats12:25:181
1885099241,13cyclictest0-21swapper/110:13:061
1885099240,23cyclictest0-21swapper/110:38:331
1885099240,16cyclictest32245-21/usr/sbin/munin11:30:271
1885099240,16cyclictest32245-21/usr/sbin/munin11:30:261
1884299243,10cyclictest20576-21grep07:10:260
1884299242,7cyclictest9695-40dnf09:14:230
1884299242,7cyclictest9695-40dnf09:14:230
1884299241,21cyclictest0-21swapper/011:10:140
1884299241,12cyclictest22492-21/usr/sbin/munin12:25:200
1885099236,2cyclictest0-21swapper/110:20:011
1885099232,16cyclictest87550irq/128-enp1s0-09:28:381
18850992315,2cyclictest0-21swapper/111:15:011
18850992313,3cyclictest3765-21ntp_states11:40:221
18850992313,3cyclictest22293-21diskstats07:15:141
1885099231,19cyclictest17818-21/usr/sbin/munin11:00:141
1885099231,13cyclictest24400-21df11:15:111
1885099231,12cyclictest0-21swapper/109:45:181
1885099230,5cyclictest0-21swapper/111:56:311
1884299231,4cyclictest0-21swapper/012:20:380
1884299231,4cyclictest0-21swapper/010:38:340
1884299231,11cyclictest31541-21irqstats07:35:170
1884299231,11cyclictest24067-21/usr/sbin/munin11:15:140
1884299231,11cyclictest15961-21/usr/sbin/munin08:15:210
1884299230,20cyclictest21532-21ps08:25:230
1884299230,20cyclictest0-21swapper/007:50:190
1884299230,10cyclictest19500-21basename09:35:170
1884299230,10cyclictest0-21swapper/011:40:160
130852230,2sleep1221ktimersoftd/108:05:261
126172231,2sleep00-21swapper/008:05:220
1885099228,3cyclictest0-21swapper/109:44:141
1885099222,9cyclictest5330-21ntp_states10:30:201
18850992216,3cyclictest20572-21unixbench_multi07:10:261
18850992215,2cyclictest0-21swapper/109:05:271
18850992214,7cyclictest0-21swapper/107:49:401
1885099221,3cyclictest16308-21diskstats08:15:131
1885099221,11cyclictest14537-21latency12:05:181
1885099221,11cyclictest0-21swapper/111:51:551
1885099220,6cyclictest0-21swapper/110:04:371
1885099220,14cyclictest0-21swapper/112:38:381
1884299227,2cyclictest23623-21ntp_states08:30:220
1884299225,3cyclictest3242-21ps10:25:240
1884299221,6cyclictest16522-21/usr/sbin/munin10:55:240
1884299221,2cyclictest10759-21unixbench-2d11:55:270
1884299221,10cyclictest30637-21head10:15:110
1884299221,10cyclictest1318-21grep11:35:160
1885099217,3cyclictest27401-21cat10:05:131
1885099215,6cyclictest755-21rngd11:20:521
1885099211,7cyclictest0-21swapper/109:15:301
1885099211,6cyclictest0-21swapper/109:56:321
1885099211,3cyclictest29914-21cat08:45:161
1885099211,18cyclictest0-21swapper/107:55:031
1885099211,17cyclictest0-21swapper/111:35:211
18850992111,3cyclictest2003-21wc07:40:231
1885099211,11cyclictest0-21swapper/110:25:201
1885099211,10cyclictest19272-21pmu-power08:20:221
1884299216,3cyclictest27250-21cpu10:05:120
1884299216,3cyclictest25500-21irqstats12:30:180
1884299213,15cyclictest10638-21munin-run10:45:010
1884299211,6cyclictest26512-21sshd08:37:270
1884299211,6cyclictest11959-21munin-run09:20:010
1884299211,5cyclictest26863-21sshd10:03:270
1884299211,5cyclictest0-21swapper/011:08:170
1885099207,3cyclictest0-21swapper/108:53:431
1885099206,4cyclictest16570-21ls10:55:241
1885099206,12cyclictest4020-21kworker/u8:108:27:581
1885099201,6cyclictest0-21swapper/110:45:101
1885099201,17cyclictest0-21swapper/107:35:011
18850992011,2cyclictest23237-21smartctl08:30:161
1885099201,10cyclictest14043-21cstates08:10:131
1885099200,6cyclictest0-21swapper/110:22:421
1885099200,19cyclictest0-21swapper/107:40:001
1884299209,2cyclictest87550irq/128-enp1s0-10:11:440
1884299207,2cyclictest87550irq/128-enp1s0-12:01:550
1884299201,7cyclictest31344-21/usr/sbin/munin08:50:160
1884299201,7cyclictest0-21swapper/010:50:410
1884299201,6cyclictest21350-21sshd09:44:570
1884299201,5cyclictest7556-21diskmemload11:01:220
1884299201,16cyclictest9358-21/usr/sbin/munin08:00:130
1884299201,16cyclictest9358-21/usr/sbin/munin08:00:130
1884299201,12cyclictest0-21swapper/008:56:520
1885099193,2cyclictest87550irq/128-enp1s0-11:47:341
1885099193,11cyclictest20033-21kworker/u8:012:19:451
1885099192,3cyclictest9014-21diskstats10:40:151
1885099191,5cyclictest0-21swapper/111:06:351
1885099191,15cyclictest8315-21grep09:10:161
1885099191,15cyclictest8315-21grep09:10:151
1884299194,3cyclictest30648-21runrttasks11:30:040
1884299194,3cyclictest30648-21runrttasks11:30:040
1884299192,4cyclictest4601-21sshd07:48:270
1884299191,7cyclictest0-21swapper/008:40:220
1884299191,6cyclictest13691-21/usr/sbin/munin12:05:270
1884299191,6cyclictest0-21swapper/011:47:050
1884299191,6cyclictest0-21swapper/008:49:010
1884299191,6cyclictest0-21swapper/007:21:520
1884299191,5cyclictest22617-21hddtemp_smartct07:15:170
1884299191,5cyclictest13761-21/usr/sbin/munin08:10:180
1884299191,5cyclictest12185-21/usr/sbin/munin09:20:240
1884299191,4cyclictest0-21swapper/011:25:460
1884299191,15cyclictest10863-21/usr/sbin/munin10:45:160
192852180,2sleep081ktimersoftd/008:20:230
1885099181,5cyclictest1163-21/usr/sbin/munin08:55:231
18850991815,2cyclictest0-21swapper/107:25:421
1885099181,4cyclictest27289-21df08:40:141
18850991814,3cyclictest0-21swapper/109:00:431
18850991814,3cyclictest0-21swapper/107:52:531
18850991812,3cyclictest25394-21head08:35:131
18850991812,2cyclictest0-21swapper/108:00:271
18850991812,2cyclictest0-21swapper/108:00:271
1885099181,15cyclictest0-21swapper/107:21:101
1884299181,7cyclictest0-21swapper/009:56:000
1884299181,5cyclictest755-21rngd10:23:430
1884299181,5cyclictest5589-21/usr/sbin/munin09:05:250
1884299173,2cyclictest87350irq/126-enp1s0-10:30:180
1884299171,9cyclictest0-21swapper/012:38:500
1884299171,6cyclictest3378-21/usr/sbin/munin09:00:180
1884299171,6cyclictest0-21swapper/009:50:140
18842991711,4cyclictest26971-21basename11:20:150
18842991711,3cyclictest26449-21cpuspeed_turbos07:25:140
1884299161,5cyclictest862-21df_inode07:40:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional