You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-30 - 20:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Jun 30, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
193422239162,26sleep00-21swapper/007:09:290
174192234163,58sleep10-21swapper/107:05:031
192122233157,24sleep20-21swapper/207:07:512
193752224194,20sleep30-21swapper/307:09:573
19481999187,3cyclictest0-21swapper/212:05:012
276952490,3sleep21948199cyclictest08:40:122
1948999414,20cyclictest19965-21latency_hist10:50:013
19489994113,9cyclictest11005-21idleruntime-cro10:30:013
182142410,2sleep218205-21munin-run12:00:022
243952390,4sleep339-21rcuc/308:30:213
198112390,3sleep339-21rcuc/309:35:163
1948999379,12cyclictest31539-21fschecks_time11:15:143
19481993614,8cyclictest32356-21sh10:05:002
19481993613,10cyclictest11000-21perf10:30:022
111002360,4sleep11947599cyclictest11:40:251
19489993515,12cyclictest3470-21cut11:25:163
1947099359,13cyclictest16005-21munin-run11:55:020
19470993510,12cyclictest15393-21sshd10:40:020
19489993414,12cyclictest14033-21/usr/sbin/munin11:50:253
19489993411,20cyclictest0-21swapper/310:50:143
19489993314,16cyclictest22779-21sshd08:28:003
19489993310,13cyclictest0-21swapper/309:25:253
1947099338,12cyclictest7212-21munin-run11:35:000
1948999328,20cyclictest1244-21awk10:05:203
19489993214,14cyclictest25026-21/usr/sbin/munin12:15:103
19489993213,16cyclictest2829-21awk11:25:013
1947099327,11cyclictest0-21swapper/008:10:020
19489993113,15cyclictest17023-21if_err_enp2s011:55:183
19489993112,6cyclictest15771-21cpuspeed_turbos10:40:123
1947099311,19cyclictest3520-21kworker/0:009:12:000
19489993021,6cyclictest8080-21gltestperf11:35:163
19489993020,7cyclictest27367-21cpuspeed_turbos12:20:133
19489993015,12cyclictest13802-21latency_hist11:50:023
19489993013,14cyclictest9977-21/usr/sbin/munin08:00:173
19489993013,14cyclictest30235-21irqstats12:25:183
1948199309,9cyclictest14704-21hddtemp_smartct11:50:162
1947099305,12cyclictest30152-21munin-run10:00:010
1947099302,18cyclictest23268-21kworker/0:209:43:290
1947099301,11cyclictest13040-21kworker/0:110:57:560
1948999297,10cyclictest10492-21date09:15:023
19489992914,12cyclictest23139-21cat12:10:133
19489992912,10cyclictest18536-21sshd08:20:023
19481992921,6cyclictest0-21swapper/208:30:172
19481992920,6cyclictest25006-21/usr/sbin/munin09:45:232
1947099294,12cyclictest26733-21munin-run07:25:010
1947099292,25cyclictest24074-21kworker/0:011:15:160
1947099290,10cyclictest9113-21kworker/0:211:46:020
1948999288,6cyclictest7399-21cut10:20:143
1948999288,18cyclictest0-21swapper/309:45:143
19489992817,8cyclictest20293-21sed08:20:263
19489992814,11cyclictest29525-21/usr/sbin/munin09:55:273
19489992813,8cyclictest31146-21/usr/sbin/munin10:00:163
19489992813,12cyclictest32538-21memory12:30:203
19489992813,12cyclictest31968-21/usr/sbin/munin08:50:123
19489992813,12cyclictest12797-21/usr/sbin/munin09:20:213
19489992811,6cyclictest30556-21runrttasks11:12:163
19481992821,4cyclictest19576-21if_err_enp2s008:20:172
19481992820,5cyclictest19235-21/usr/sbin/munin12:00:162
19481992819,6cyclictest5873-21/usr/sbin/munin10:15:242
19481992819,6cyclictest4549-21/usr/sbin/munin07:45:232
19481992819,6cyclictest32279-21cut12:30:162
19481992819,6cyclictest13061-21/usr/sbin/munin11:45:242
19481992818,7cyclictest31008-21/usr/sbin/munin12:25:262
19481992814,11cyclictest12285-21runrttasks09:16:332
19481992814,10cyclictest1030-21/usr/sbin/munin10:05:172
19481992810,10cyclictest0-21swapper/210:12:032
19481992810,10cyclictest0-21swapper/210:12:032
1947099281,17cyclictest8557-21kworker/0:010:33:130
1947099281,10cyclictest24074-21kworker/0:011:01:560
1948999279,8cyclictest8172-21diskmemload12:04:123
1948999275,8cyclictest0-21swapper/308:35:243
19489992717,7cyclictest22347-21awk09:40:183
19489992716,8cyclictest11116-21grep11:40:273
19489992716,7cyclictest2688-21ls08:55:193
19489992716,7cyclictest12723-21sed10:30:253
19489992715,9cyclictest22375-21tr10:55:123
19489992715,8cyclictest3118-21sshd12:36:103
19489992715,8cyclictest3118-21sshd12:36:103
19489992713,11cyclictest21668-21/usr/sbin/munin12:05:193
19481992720,4cyclictest796-21/usr/sbin/munin11:20:122
19481992720,4cyclictest5768-21cstates07:50:142
19481992720,4cyclictest27652-21/usr/sbin/munin11:05:202
19481992720,4cyclictest2104-21/usr/sbin/munin07:40:192
19481992719,5cyclictest24752-21/usr/sbin/munin11:00:142
19481992719,5cyclictest19456-21/usr/sbin/munin09:35:132
19481992719,5cyclictest18157-21diskstats10:45:162
19481992719,5cyclictest14741-21/usr/sbin/munin08:10:122
19481992718,6cyclictest4294-21/usr/sbin/munin11:25:252
19481992718,6cyclictest26097-21/usr/sbin/munin12:15:182
19481992718,6cyclictest24760-21/usr/sbin/munin07:20:242
19481992718,6cyclictest22562-21/usr/sbin/munin07:15:182
19481992718,6cyclictest21258-21/usr/sbin/munin07:10:252
19481992718,6cyclictest12797-21/usr/sbin/munin09:20:192
19481992715,9cyclictest23734-21tail10:55:262
19481992713,11cyclictest16825-21runrttasks09:28:252
19481992713,11cyclictest13894-21runrttasks08:06:542
19481992711,5cyclictest0-21swapper/208:35:202
1947099274,11cyclictest2828-21sshd11:25:000
1947099272,9cyclictest21007-21kworker/0:009:38:410
1948999269,15cyclictest0-21swapper/309:50:163
1948999266,17cyclictest7262-21latency_hist11:35:023
19489992618,5cyclictest15031-21sed08:10:143
19489992617,6cyclictest32027-21chrt07:35:173
19489992617,6cyclictest27879-21/usr/sbin/munin11:05:233
19489992617,6cyclictest22562-21/usr/sbin/munin07:15:123
19489992617,6cyclictest20724-21/usr/sbin/munin07:10:193
19489992617,5cyclictest2665-21grep07:40:263
19489992616,7cyclictest7935-21grep07:55:123
19489992616,7cyclictest763-21dbus-daemon07:50:023
19489992616,7cyclictest30186-21sort08:45:143
19489992616,7cyclictest28045-21uniq08:40:153
19489992616,7cyclictest27324-21cut07:25:153
19489992616,7cyclictest24483-21tr11:00:123
19489992616,7cyclictest15389-21awk10:40:013
19481992619,4cyclictest9945-21/usr/sbin/munin09:10:272
19481992619,4cyclictest32301-21/usr/sbin/munin08:50:142
19481992619,4cyclictest30534-21/usr/sbin/munin08:45:162
19481992619,4cyclictest29242-21/usr/sbin/munin07:30:132
19481992619,4cyclictest28770-21/usr/sbin/munin09:55:172
19481992619,4cyclictest27982-21/usr/sbin/munin07:25:202
19481992619,4cyclictest23838-21/usr/sbin/munin12:10:192
19481992619,4cyclictest14235-21/usr/sbin/munin10:35:172
19481992619,4cyclictest10784-21/usr/sbin/munin08:00:162
19481992618,5cyclictest31702-21/usr/sbin/munin11:15:162
19481992618,5cyclictest27971-21/usr/sbin/munin12:20:162
19481992618,5cyclictest22620-21/usr/sbin/munin09:40:212
19481992618,5cyclictest20364-21/usr/sbin/munin10:50:142
19481992618,5cyclictest17209-21cpu09:30:112
19481992617,6cyclictest25987-21/usr/sbin/munin09:50:142
19481992617,6cyclictest20977-21/usr/sbin/munin08:25:262
19481992617,6cyclictest1805-21/usr/sbin/munin08:55:202
19481992617,6cyclictest15611-21/usr/sbin/munin10:40:212
19481992617,6cyclictest1442-21/usr/sbin/munin12:35:142
19481992617,6cyclictest1442-21/usr/sbin/munin12:35:142
19481992616,7cyclictest10315-21/usr/sbin/munin11:40:172
1947599261,17cyclictest28971-21if_err_enp2s009:55:181
1947599260,18cyclictest6471-21kworker/u8:110:27:231
1947099262,9cyclictest26692-21kworker/0:007:32:120
1947099262,11cyclictest2824-21kworker/0:207:51:530
1947099261,17cyclictest20424-21kworker/0:008:31:050
1947099261,13cyclictest4398-21munin-run10:15:010
1947099261,13cyclictest4398-21munin-run10:15:010
19470992611,3cyclictest31238-21kworker/0:012:29:150
1947099261,11cyclictest24074-21kworker/0:011:35:200
1947099261,10cyclictest23268-21kworker/0:210:01:050
1947099261,10cyclictest20380-21kworker/0:112:10:210
19489992517,5cyclictest7165-21/usr/sbin/munin07:50:273
19489992517,5cyclictest2778-21/usr/sbin/munin10:10:163
19489992517,5cyclictest2778-21/usr/sbin/munin10:10:163
19489992516,5cyclictest25408-21head07:20:163
19489992515,7cyclictest8183-21dump-pmu-power09:10:013
19489992515,7cyclictest5449-21/usr/sbin/munin10:15:193
19489992515,7cyclictest4671-21chrt09:00:153
19489992515,7cyclictest4671-21chrt09:00:153
19489992515,7cyclictest31120-21turbostat07:35:013
19489992515,7cyclictest12321-21grep08:05:133
19489992512,10cyclictest909-21sssd_nss09:18:533
19481992519,4cyclictest7603-21/usr/sbin/munin11:35:132
19481992519,4cyclictest11562-21/usr/sbin/munin10:30:142
19481992518,4cyclictest7863-21/usr/sbin/munin07:55:112
19481992518,4cyclictest6208-21/usr/sbin/munin09:05:172
19481992518,4cyclictest5249-21/usr/sbin/munin11:30:132
19481992518,4cyclictest4085-21/usr/sbin/munin09:00:132
19481992518,4cyclictest4085-21/usr/sbin/munin09:00:132
19481992518,4cyclictest31416-21/usr/sbin/munin07:35:132
19481992518,4cyclictest16936-21/usr/sbin/munin08:15:152
19481992515,7cyclictest28965-21grep11:10:142
1947599252,4cyclictest3159-21munin-run07:45:001
1947599251,16cyclictest0-21swapper/112:30:151
19475992511,3cyclictest654-21cut11:20:021
1947599251,12cyclictest0-21swapper/109:15:481
19475992510,7cyclictest23-21ksoftirqd/110:20:021
1947099252,9cyclictest24074-21kworker/0:011:25:080
1947099252,10cyclictest25519-21kworker/0:009:48:570
1947099251,22cyclictest15885-21kworker/0:212:01:490
1947099251,18cyclictest11380-21kworker/0:111:58:290
1947099251,13cyclictest26724-21kworker/0:012:17:450
19470992511,2cyclictest16920-21kworker/0:109:32:040
1947099251,10cyclictest24074-21kworker/0:011:13:520
19489992414,7cyclictest18405-21chrt09:30:223
19481992416,5cyclictest6829-21/usr/sbin/munin10:20:112
1947599241,6cyclictest0-21swapper/109:46:551
19475992413,3cyclictest4298-21memory07:45:181
1947599241,17cyclictest0-21swapper/109:22:301
1947599241,16cyclictest0-21swapper/109:51:231
1947599241,12cyclictest0-21swapper/111:34:581
1947599241,10cyclictest0-21swapper/111:57:511
1947099242,13cyclictest24074-21kworker/0:011:09:200
1947099242,10cyclictest6329-21kworker/0:110:26:170
1947099242,10cyclictest3520-21kworker/0:009:05:240
1947099241,10cyclictest23268-21kworker/0:209:53:450
1947099241,10cyclictest14623-21kworker/0:209:26:480
1947099241,10cyclictest1161-21kworker/0:212:36:170
1947099241,10cyclictest1161-21kworker/0:212:36:170
1947599237,3cyclictest0-21swapper/110:08:401
1947599233,9cyclictest3771-21kworker/u8:209:25:131
1947599232,15cyclictest87550irq/128-enp1s0-11:24:161
1947599231,5cyclictest0-21swapper/109:44:121
19475992313,2cyclictest490-21vmstat07:35:261
1947599231,10cyclictest0-21swapper/110:56:391
1947599230,12cyclictest10766-21kworker/u8:210:40:231
1947099239,7cyclictest20380-21kworker/0:112:20:530
1947099239,3cyclictest1200-21kworker/0:108:53:250
1947099239,2cyclictest13040-21kworker/0:110:50:320
1947099232,15cyclictest19433-21kworker/0:010:47:250
1947099232,12cyclictest6329-21kworker/0:110:24:290
1947099231,20cyclictest9113-21kworker/0:211:43:240
1947099231,16cyclictest18149-21kworker/0:012:08:450
19470992311,2cyclictest16571-21kworker/0:308:20:170
1947099231,11cyclictest2824-21kworker/0:207:46:530
1947099231,10cyclictest3520-21kworker/0:009:15:400
1947099231,10cyclictest1200-21kworker/0:108:55:170
1947599226,6cyclictest20963-21switchtime07:10:241
1947599222,2cyclictest87550irq/128-enp1s0-12:37:181
1947599222,2cyclictest87550irq/128-enp1s0-12:37:181
19475992217,2cyclictest27136-21tune2fs11:05:151
1947599221,2cyclictest8172-21diskmemload09:31:351
1947599221,19cyclictest0-21swapper/112:19:001
1947599221,19cyclictest0-21swapper/109:36:101
1947599221,18cyclictest31190-21irqstats10:00:161
1947599221,16cyclictest0-21swapper/108:48:281
1947599221,15cyclictest0-21swapper/112:24:181
1947599221,10cyclictest0-21swapper/110:47:381
1947099221,7cyclictest2824-21kworker/0:207:44:420
1947099221,19cyclictest12273-21kworker/0:109:20:400
1947099221,12cyclictest22562-21/usr/sbin/munin07:15:190
1947599219,3cyclictest25257-21latency11:00:191
1947599217,3cyclictest0-21swapper/111:35:411
1947599215,2cyclictest0-21swapper/110:38:041
1947599212,5cyclictest18245-1kworker/u9:212:25:081
19475992114,2cyclictest0-21swapper/110:20:181
1947599211,3cyclictest0-21swapper/111:46:111
19475992112,2cyclictest0-21swapper/110:30:131
1947599211,15cyclictest0-21swapper/112:06:511
1947599211,15cyclictest0-21swapper/108:25:581
1947599211,13cyclictest3771-21kworker/u8:209:11:261
1947599211,13cyclictest0-21swapper/111:12:471
1947599211,12cyclictest8644-21sed07:55:181
1947599211,11cyclictest27336-21latency_hist08:40:011
1947599211,11cyclictest24286-21sed08:30:221
1947599211,11cyclictest17494-21memory08:15:201
1947599211,10cyclictest5970-21crond09:05:011
1947599211,10cyclictest5970-21crond09:05:011
19475992110,3cyclictest7796-21users09:05:291
1947599210,15cyclictest0-21swapper/108:51:391
1947599210,15cyclictest0-21swapper/107:30:141
1947099212,11cyclictest29766-21/usr/sbin/munin08:45:230
1947099211,9cyclictest23268-21kworker/0:210:15:450
1947099211,9cyclictest1845-21kworker/0:110:09:410
19470992114,4cyclictest9358-21/usr/sbin/munin07:55:260
1947099211,11cyclictest9697-21kworker/0:308:00:210
285132200,2sleep10-21swapper/108:40:201
1947599209,3cyclictest9977-21/usr/sbin/munin08:00:111
1947599209,3cyclictest13318-21tr08:05:231
1947599201,8cyclictest0-21swapper/110:12:181
1947599201,8cyclictest0-21swapper/110:12:181
19475992013,3cyclictest0-21swapper/112:10:131
19475992013,2cyclictest0-21swapper/107:26:491
19475992012,3cyclictest0-21swapper/108:12:111
1947599201,14cyclictest0-21swapper/107:23:161
1947599201,14cyclictest0-21swapper/107:15:481
1947599201,11cyclictest3415-21fschecks_count11:25:161
1947599201,11cyclictest1991-21cstates08:55:141
1947599200,5cyclictest0-21swapper/112:00:041
1947599200,15cyclictest0-21swapper/111:52:341
1947099206,8cyclictest16024-21kworker/0:208:17:250
1947099202,10cyclictest16024-21kworker/0:208:11:250
1947099201,11cyclictest31238-21kworker/0:012:33:530
1947099201,11cyclictest15611-21/usr/sbin/munin10:40:240
1947599199,3cyclictest7658-21latency_hist07:55:011
1947599191,7cyclictest18245-1kworker/u9:210:50:091
1947599190,14cyclictest0-21swapper/108:24:511
1947099198,3cyclictest19673-21/usr/sbin/munin07:10:250
1947099192,14cyclictest22444-21grep08:25:250
1947099191,10cyclictest25373-21/usr/sbin/munin08:35:180
1947099187,2cyclictest27201-21kworker/0:008:40:330
19470991811,4cyclictest5527-21grep09:00:250
19470991811,4cyclictest5527-21grep09:00:250
19470991711,4cyclictest27620-21grep07:25:150
1947099171,11cyclictest0-21swapper/007:35:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional