You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-23 - 13:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Jun 23, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11145210677,9sleep00-21swapper/019:05:420
1125329668,9sleep20-21swapper/219:07:032
1126428868,15sleep30-21swapper/319:07:123
1138428465,15sleep10-21swapper/119:08:471
1156899290,4cyclictest10120-21sleep20:15:002
1156299280,0cyclictest0-21swapper/022:08:010
1156299280,0cyclictest0-21swapper/022:08:010
1156299270,2cyclictest0-21swapper/023:48:560
1156899262,1cyclictest4143-21diskmemload22:22:172
1156899251,7cyclictest552-21cron19:54:592
1156899251,23cyclictest0-21swapper/221:46:432
1156899250,2cyclictest0-21swapper/223:00:152
1156899250,2cyclictest0-21swapper/222:41:432
1156899250,2cyclictest0-21swapper/219:25:112
1156899250,2cyclictest0-21swapper/200:20:142
1156299250,23cyclictest0-21swapper/022:29:350
1156299250,1cyclictest0-21swapper/022:42:390
1156899241,3cyclictest22422-21grep23:35:272
1156899240,3cyclictest0-21swapper/200:08:512
1156899240,2cyclictest0-21swapper/223:16:232
1156899240,1cyclictest0-21swapper/221:33:122
1156899231,1cyclictest0-21swapper/221:10:022
1156899230,2cyclictest0-21swapper/223:46:182
1156899230,23cyclictest0-21swapper/222:32:502
1156899230,1cyclictest0-21swapper/221:23:362
1156899230,1cyclictest0-21swapper/220:40:132
1156899221,21cyclictest0-21swapper/222:54:282
1156899220,3cyclictest0-21swapper/219:40:002
1156899220,1cyclictest0-21swapper/223:05:372
11564992210,9cyclictest15455-21df_abs21:55:141
1156299220,1cyclictest0-21swapper/022:24:030
1156899211,3cyclictest4143-21diskmemload23:11:282
1156899210,3cyclictest0-21swapper/221:39:072
1156899210,21cyclictest0-21swapper/200:12:492
1156899210,1cyclictest0-21swapper/223:42:212
11562992121,0cyclictest0-21swapper/022:54:330
1156299210,2cyclictest0-21swapper/000:21:210
1156299210,2cyclictest0-21swapper/000:01:520
1156299210,18cyclictest0-21swapper/023:58:120
1156299210,0cyclictest0-21swapper/021:22:190
1156899201,4cyclictest0-21swapper/220:45:212
1156899200,4cyclictest17557-21/usr/sbin/munin22:35:162
1156899200,3cyclictest0-21swapper/222:25:122
1156899200,18cyclictest0-21swapper/221:42:112
1156899200,0cyclictest0-21swapper/200:04:492
1156499200,17cyclictest0-21swapper/121:25:261
1156299200,0cyclictest0-21swapper/021:37:450
11570991915,2cyclictest6478-21hddtemp_smartct21:30:173
1156899196,6cyclictest17618-21ssh23:52:252
1156899196,6cyclictest17618-21ssh23:52:242
1156899193,1cyclictest4143-21diskmemload00:33:362
1156899192,1cyclictest4143-21diskmemload22:02:272
1156899192,1cyclictest0-21swapper/223:23:562
1156899190,3cyclictest0-21swapper/221:15:092
1156899190,1cyclictest0-21swapper/222:07:282
1156899190,1cyclictest0-21swapper/222:07:272
1156899190,0cyclictest0-21swapper/223:58:402
1156899190,0cyclictest0-21swapper/222:12:082
1156499190,3cyclictest0-21swapper/123:20:101
1156499190,1cyclictest0-21swapper/123:17:111
1156299190,1cyclictest0-21swapper/023:00:040
1156299190,1cyclictest0-21swapper/000:39:200
1157099188,8cyclictest0-21swapper/319:35:133
1156899185,7cyclictest27593-21ssh00:36:302
1156899182,1cyclictest840-21apt-get19:55:142
1156899181,3cyclictest21955-21rm22:18:332
1156899180,1cyclictest0-21swapper/223:29:402
1156899180,0cyclictest0-21swapper/222:57:362
11564991817,1cyclictest0-21swapper/100:36:351
11562991818,0cyclictest0-21swapper/022:00:450
1156299180,1cyclictest0-21swapper/023:41:160
1156299180,1cyclictest0-21swapper/022:57:520
1156299180,1cyclictest0-21swapper/000:07:270
1156299180,17cyclictest0-21swapper/021:40:110
1156899173,4cyclictest16370-21grep20:25:232
1156899172,2cyclictest4143-21diskmemload21:55:142
1156899171,2cyclictest4350-21uname20:00:202
1156899171,15cyclictest0-21swapper/222:45:142
1156899170,3cyclictest18448-21unamelversion20:30:192
1156499179,5cyclictest27855-21kworker/u8:000:15:021
1156299176,6cyclictest7746-21perf20:10:010
1156299170,2cyclictest0-21swapper/023:23:280
1156299170,1cyclictest0-21swapper/021:48:090
1156299170,16cyclictest0-21swapper/023:16:080
1156299170,13cyclictest0-21swapper/022:12:050
1156299170,0cyclictest0-21swapper/021:15:140
1157099164,8cyclictest21457-21/usr/sbin/munin23:55:123
1156899167,4cyclictest10302-21ssh00:26:082
1156899160,4cyclictest0-21swapper/221:50:162
1156899160,4cyclictest0-21swapper/220:55:262
1156899160,1cyclictest0-21swapper/200:16:542
11564991616,0cyclictest0-21swapper/122:37:231
11564991614,2cyclictest25-21ksoftirqd/121:35:131
11564991614,2cyclictest0-21swapper/122:57:221
1156499160,1cyclictest0-21swapper/123:45:141
1156499160,1cyclictest0-21swapper/123:44:231
1156299161,8cyclictest26780-21munin-run20:50:010
1156299160,4cyclictest0-21swapper/023:05:520
1156299160,0cyclictest0-21swapper/000:16:180
11570991511,2cyclictest19545-21cut19:25:143
1157099150,14cyclictest0-21swapper/300:13:353
1156899159,5cyclictest17658-21grep19:20:192
1156899158,6cyclictest12545-21cat20:20:012
1156899154,1cyclictest28467-21apt-get19:45:142
1156899152,2cyclictest0-21swapper/220:50:252
1156899151,1cyclictest13556-21gltestperf20:20:162
1156899150,4cyclictest0-21swapper/223:32:412
1156899150,2cyclictest225-21systemd-journal20:36:082
1156899150,1cyclictest0-21swapper/221:25:122
1156499150,1cyclictest0-21swapper/122:53:471
1156499150,15cyclictest0-21swapper/123:36:051
1156299156,3cyclictest3246-21apt-get20:00:130
1156299151,7cyclictest16284-21grep00:10:280
1156299151,2cyclictest0-21swapper/021:31:310
11562991512,2cyclictest13-21ksoftirqd/023:30:130
1156299150,1cyclictest0-21swapper/022:34:390
239052140,0chrt0-21swapper/322:58:173
11570991412,2cyclictest39-21ksoftirqd/322:43:043
11570991412,2cyclictest39-21ksoftirqd/322:23:503
1156899148,5cyclictest3759-21users21:05:282
1156899148,5cyclictest21775-21expr19:30:152
1156899147,6cyclictest7083-21cat20:05:262
1156899147,6cyclictest14612-21cat19:15:142
1156899140,1cyclictest0-21swapper/219:40:252
1156499149,5cyclictest38150irq/131-snd_hda21:16:171
11564991412,1cyclictest29847-21apache_volume00:00:111
1156499141,13cyclictest4143-21diskmemload21:54:391
1156499140,2cyclictest0-21swapper/123:55:561
1156499140,13cyclictest0-21swapper/122:43:181
1156499140,12cyclictest0-21swapper/119:45:151
1156299141,5cyclictest7166-21apt-get00:25:020
1156299140,1cyclictest0-21swapper/023:28:220
1157099139,3cyclictest25425-21sed20:45:173
1157099139,3cyclictest17554-21cut22:35:163
1157099139,2cyclictest4004-21tr23:25:133
11570991311,1cyclictest0-21swapper/320:35:163
11570991310,2cyclictest9324-21sed00:25:273
11570991310,2cyclictest5986-21head23:45:203
11570991310,2cyclictest4564-21grep23:05:263
11570991310,2cyclictest14390-21grep20:20:253
1157099130,2cyclictest225-21systemd-journal20:45:013
1156899138,4cyclictest1410-21taskset21:00:382
11568991310,2cyclictest13335-21cat19:10:262
1156499138,5cyclictest16750-21apache200:34:431
1156499134,8cyclictest0-21swapper/123:01:581
11564991313,0cyclictest0-21swapper/122:27:161
1156499131,1cyclictest4143-21diskmemload23:11:041
11564991310,2cyclictest9104-21uname20:10:201
11564991310,2cyclictest7471-21grep21:30:261
11564991310,2cyclictest2803-21cut23:05:141
11564991310,2cyclictest17750-21tr20:30:151
11564991310,2cyclictest16275-21cron19:19:591
11564991310,2cyclictest15883-21sh00:10:241
1156299139,3cyclictest3273-21wc22:45:250
1156299139,3cyclictest31065-21cat20:55:260
1156299139,2cyclictest18938-21wc23:15:000
1156299135,4cyclictest28317-21idleruntime-cro23:40:010
11562991310,2cyclictest8423-21wc21:50:260
11562991310,2cyclictest25817-21perf19:40:000
11562991310,2cyclictest23031-21sed20:40:170
11562991310,2cyclictest12484-21latency_hist20:20:010
1156299130,0cyclictest0-21swapper/021:59:010
1157099129,2cyclictest5343-21cron20:05:003
1157099129,2cyclictest29565-21cut23:40:173
1157099129,2cyclictest28835-21wc23:20:203
1157099129,2cyclictest27182-21perf23:19:593
1157099129,2cyclictest24364-21sh00:35:143
1157099129,2cyclictest23952-21sort22:00:163
1157099129,2cyclictest21719-21tr23:35:203
1157099129,2cyclictest13405-21sed19:10:263
1157099128,3cyclictest29909-21tr20:55:143
1157099128,3cyclictest14699-21sort21:35:163
1157099128,2cyclictest7911-21cat20:10:013
1157099128,2cyclictest747-21tr22:25:123
1157099128,2cyclictest2101-21cut21:05:163
1157099128,2cyclictest16285-21tr20:25:223
1157099127,4cyclictest29621-21/usr/sbin/munin21:25:113
1157099120,11cyclictest29149-21basename21:24:593
1156499129,2cyclictest8417-21head22:10:181
1156499129,2cyclictest6352-21head20:05:181
1156499129,2cyclictest607-21cat00:20:251
1156499129,2cyclictest31905-21tr22:05:161
1156499129,2cyclictest31905-21tr22:05:151
1156499129,2cyclictest31599-21head19:50:161
1156499129,2cyclictest26854-21head19:40:161
1156499129,2cyclictest24582-21cat20:45:011
1156499129,2cyclictest22650-21ls21:40:141
1156499129,2cyclictest17017-21wc22:15:191
1156499128,3cyclictest22061-21grep19:30:161
1156499127,3cyclictest13364-21uname23:30:211
11564991212,0cyclictest0-21swapper/123:29:091
11564991212,0cyclictest0-21swapper/121:49:191
11564991210,1cyclictest0-21swapper/123:50:141
11564991210,1cyclictest0-21swapper/123:50:141
11564991210,1cyclictest0-21swapper/100:05:461
1156499120,5cyclictest0-21swapper/100:25:021
1156499120,11cyclictest3485-21grep22:45:261
1156499120,11cyclictest32365-21dump-pmu-power22:25:001
1156299129,2cyclictest8756-21cut20:10:190
1156299129,2cyclictest31597-21cut19:50:160
1156299129,2cyclictest2735-21grep21:05:190
1156299129,2cyclictest26707-21cut19:40:150
1156299129,2cyclictest23026-21head19:30:250
1156299129,2cyclictest13415-21cut20:20:170
1156299129,2cyclictest12784-21wc19:10:190
1156299128,3cyclictest6602-21sed21:10:280
1156299128,3cyclictest21970-21ssh22:37:360
1156299128,2cyclictest17693-21tr19:20:190
1157099118,2cyclictest9151-21ssh00:06:273
1157099118,2cyclictest8771-21ssh22:10:213
1157099118,2cyclictest7896-21ssh21:50:213
1157099118,2cyclictest5487-21grep21:10:183
1157099118,2cyclictest4916-21ssh22:08:193
1157099118,2cyclictest4916-21ssh22:08:183
1157099118,2cyclictest30939-21cut21:45:113
1157099118,2cyclictest29860-21ssh00:00:113
1157099118,2cyclictest28526-21grep23:00:273
1157099118,2cyclictest2389-21tail22:45:183
1157099118,2cyclictest23760-21ssh00:15:183
1157099118,2cyclictest23146-21seq19:31:343
1157099118,2cyclictest22951-21cat22:00:013
1157099118,2cyclictest22119-21ssh21:40:053
1157099118,2cyclictest17484-21ssh00:30:253
1157099118,2cyclictest15800-21ssh22:15:073
1157099118,2cyclictest15244-21ssh23:31:213
1157099118,2cyclictest14138-21ssh23:50:193
1157099118,2cyclictest14138-21ssh23:50:183
1157099118,2cyclictest13664-21cut21:15:173
1157099118,2cyclictest13322-21ssh22:32:203
1157099118,2cyclictest13078-21chrt22:51:193
1157099118,2cyclictest11486-21cat23:10:163
1157099118,1cyclictest17798-21tr20:30:143
1157099110,10cyclictest882-21ssh00:20:293
1156499119,1cyclictest16859-21grep19:20:141
1156499119,1cyclictest13312sleep119:55:151
1156499118,2cyclictest9813-21basename22:30:171
1156499118,2cyclictest7004-21ssh21:10:421
1156499118,2cyclictest31646-21taskset20:59:071
1156499118,2cyclictest2994-21sort21:05:211
1156499118,2cyclictest28062-21wc20:50:191
1156499118,2cyclictest24171-21expr19:35:161
1156499118,2cyclictest14879-21apt20:25:011
1156499117,3cyclictest12010-21sed20:15:251
1156499116,2cyclictest23252-21ssh22:00:111
1156499111,8cyclictest0-21swapper/121:00:121
1156499110,10cyclictest3275-21wget20:00:131
1156499110,10cyclictest15439-21cat20:25:151
1156299118,2cyclictest29744-21cut21:25:120
1156299118,2cyclictest2781-21taskset19:57:560
1156299118,2cyclictest27263-21tr20:50:130
1156299118,2cyclictest17561-21sensors22:15:240
1156299118,2cyclictest15857-21grep00:30:150
1156299118,2cyclictest15716-21wc19:15:260
1156299118,2cyclictest15077-21grep23:50:270
1156299118,2cyclictest15077-21grep23:50:260
1156299117,3cyclictest21044-21cron19:30:000
1156299117,3cyclictest1754-21cron21:04:590
1156299112,4cyclictest225-21systemd-journal20:33:220
1157099108,1cyclictest903-21wget19:55:143
1157099108,1cyclictest28287-21ps20:50:213
1157099108,1cyclictest14177-21apache_volume19:15:123
1157099107,2cyclictest32651-21cut21:00:173
1157099107,2cyclictest31123-21tr19:50:143
1157099107,2cyclictest16800-21tr19:20:143
1157099107,2cyclictest10610-21grep20:15:143
1157099107,2cyclictest10104-21chrt20:15:003
1157099101,2cyclictest0-21swapper/319:45:133
1156499108,1cyclictest26633-21taskset20:48:221
1156499108,1cyclictest12153-21expr19:10:151
1156499107,2cyclictest21545-21grep20:35:251
1156499107,2cyclictest20274-21mailstats19:25:201
1156499104,5cyclictest0-21swapper/121:20:161
1156299107,2cyclictest29716-21wc19:45:220
1156299107,2cyclictest21968-21chrt20:39:450
115629996,2cyclictest15179-21grep20:25:140
115709985,2cyclictest26761-21fschecks_time19:40:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional