You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-16 - 15:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Tue Jun 16, 2026 12:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21037992719,4cyclictest14516-21diskmemload09:39:510
2103799270,0cyclictest0-21swapper/010:37:220
2103799270,0cyclictest0-21swapper/010:37:220
2103799260,25cyclictest14516-21diskmemload09:18:480
2103799260,21cyclictest0-21swapper/009:24:520
2103799260,1cyclictest7046-21apt-get10:04:530
2103799260,0cyclictest0-21swapper/008:24:510
21042992522,2cyclictest26535-21sendmail_mailqu11:50:002
2104299250,24cyclictest7142-21ssh12:02:522
2104299250,1cyclictest0-21swapper/212:26:372
2104199250,24cyclictest0-21swapper/111:08:011
21037992521,4cyclictest14516-21diskmemload11:55:210
21037992521,3cyclictest0-21swapper/010:23:170
21037992518,6cyclictest3558-21users10:00:010
2103799250,24cyclictest0-21swapper/011:54:190
2103799250,24cyclictest0-21swapper/010:59:200
2103799250,1cyclictest0-21swapper/011:06:250
2103799250,0cyclictest0-21swapper/009:52:170
21043992423,1cyclictest14341-21/usr/sbin/munin10:40:003
21043992423,1cyclictest14341-21/usr/sbin/munin10:40:003
21043992422,1cyclictest25031-21mailstats09:20:013
2104399240,23cyclictest32044-21cron11:25:013
2104399240,23cyclictest17365-21sh09:44:233
2104399240,23cyclictest14516-21diskmemload09:33:263
21042992423,1cyclictest14516-21diskmemload10:05:092
2104299240,3cyclictest30864-21wc07:30:002
2104299240,24cyclictest14516-21diskmemload11:39:522
2104299240,23cyclictest14516-21diskmemload10:46:582
2104299240,23cyclictest14516-21diskmemload10:12:562
2104299240,23cyclictest14516-21diskmemload09:52:052
2104299240,1cyclictest0-21swapper/210:34:482
21041992420,3cyclictest31748-21turbostat.cron10:55:011
2104199240,24cyclictest0-21swapper/109:44:171
2104199240,1cyclictest0-21swapper/110:03:331
2104199240,0cyclictest0-21swapper/110:22:501
21037992420,3cyclictest14516-21diskmemload10:06:110
21037992420,3cyclictest0-21swapper/011:23:260
21037992418,5cyclictest12726-21basename08:00:000
2103799241,1cyclictest0-21swapper/010:26:510
2103799240,23cyclictest0-21swapper/011:44:020
2103799240,20cyclictest21905-21ssh12:16:080
2103799240,0cyclictest14516-21diskmemload11:32:440
2103799240,0cyclictest14516-21diskmemload10:50:440
2103799240,0cyclictest0-21swapper/012:26:160
2103799240,0cyclictest0-21swapper/010:48:170
2103799240,0cyclictest0-21swapper/010:11:000
21043992323,0cyclictest14516-21diskmemload12:04:173
21043992323,0cyclictest0-21swapper/312:29:103
21043992323,0cyclictest0-21swapper/311:54:513
21043992323,0cyclictest0-21swapper/310:57:093
21043992323,0cyclictest0-21swapper/310:52:093
21043992323,0cyclictest0-21swapper/310:40:353
21043992322,1cyclictest0-21swapper/311:45:473
21043992322,0cyclictest34-21ksoftirqd/309:37:293
21043992321,1cyclictest34-21ksoftirqd/310:35:003
21043992321,1cyclictest0-21swapper/311:58:553
21043992321,1cyclictest0-21swapper/309:56:413
21043992312,5cyclictest331rcuc/311:18:103
2104399230,2cyclictest22678-21latency_hist08:20:003
2104399230,23cyclictest0-21swapper/312:23:093
2104399230,23cyclictest0-21swapper/310:23:353
2104399230,22cyclictest8787-21plymouth10:05:013
2104399230,22cyclictest25412-21sh09:50:253
2104399230,22cyclictest16456-21ssh12:10:563
2104399230,22cyclictest16006-21sh09:11:363
2104399230,22cyclictest16006-21sh09:11:363
2104399230,22cyclictest14516-21diskmemload09:48:213
2104399230,22cyclictest0-21swapper/311:43:003
2104399230,22cyclictest0-21swapper/311:30:233
2104399230,1cyclictest0-21swapper/310:26:583
2104399230,0cyclictest14516-21diskmemload11:00:573
2104399230,0cyclictest0-21swapper/312:35:193
2104399230,0cyclictest0-21swapper/309:25:553
2104399230,0cyclictest0-21swapper/309:22:343
21042992323,0cyclictest14516-21diskmemload10:21:052
21042992323,0cyclictest0-21swapper/211:30:112
21042992323,0cyclictest0-21swapper/211:25:102
21042992323,0cyclictest0-21swapper/211:10:512
21042992323,0cyclictest0-21swapper/209:31:492
21042992321,1cyclictest13660-21ssh11:09:402
21042992321,1cyclictest0-21swapper/211:18:132
21042992321,1cyclictest0-21swapper/209:11:532
21042992321,1cyclictest0-21swapper/209:11:532
2104299230,23cyclictest0-21swapper/211:21:032
2104299230,23cyclictest0-21swapper/210:50:562
2104299230,23cyclictest0-21swapper/209:18:362
2104299230,22cyclictest27354-21sh12:21:152
2104299230,22cyclictest0-21swapper/212:30:282
2104299230,22cyclictest0-21swapper/211:56:502
2104299230,1cyclictest11406-21ssh10:37:312
2104299230,1cyclictest11406-21ssh10:37:312
2104299230,1cyclictest0-21swapper/209:36:412
2104299230,0cyclictest14516-21diskmemload12:38:252
2104299230,0cyclictest14516-21diskmemload10:17:172
2104299230,0cyclictest0-21swapper/210:03:332
2104299230,0cyclictest0-21swapper/209:45:382
2104299230,0cyclictest0-21swapper/209:40:582
2104299230,0cyclictest0-21swapper/209:28:432
21041992323,0cyclictest14516-21diskmemload12:06:141
21041992323,0cyclictest0-21swapper/111:55:131
21041992323,0cyclictest0-21swapper/111:37:351
21041992322,1cyclictest25241-21kworker/u8:009:51:391
21041992321,1cyclictest0-21swapper/112:22:101
21041992321,1cyclictest0-21swapper/111:45:011
21041992321,1cyclictest0-21swapper/111:30:511
21041992321,1cyclictest0-21swapper/111:12:491
2104199230,23cyclictest14516-21diskmemload10:26:461
2104199230,23cyclictest0-21swapper/112:04:081
2104199230,23cyclictest0-21swapper/111:26:251
2104199230,23cyclictest0-21swapper/110:47:391
2104199230,23cyclictest0-21swapper/110:34:571
2104199230,23cyclictest0-21swapper/109:35:281
2104199230,23cyclictest0-21swapper/108:24:501
2104199230,22cyclictest6106-21ssh11:01:481
2104199230,22cyclictest22407-21sh12:16:591
2104199230,22cyclictest21500-21ssh11:15:241
2104199230,22cyclictest20652-21ssh10:15:111
2104199230,22cyclictest19098-21/usr/sbin/munin12:14:571
2104199230,21cyclictest2555-21/usr/sbin/munin09:29:541
2104199230,1cyclictest7028-21/usr/sbin/munin09:34:531
2104199230,1cyclictest0-21swapper/110:14:241
2104199230,0cyclictest0-21swapper/112:27:141
2104199230,0cyclictest0-21swapper/110:06:161
2104199230,0cyclictest0-21swapper/109:56:411
21037992323,0cyclictest0-21swapper/012:36:460
21037992323,0cyclictest0-21swapper/012:06:170
21037992321,1cyclictest0-21swapper/011:45:010
21037992321,1cyclictest0-21swapper/010:32:500
21037992320,2cyclictest11322-21ssh11:35:320
21037992320,2cyclictest10388-21cron07:55:010
21037992317,5cyclictest18170-21diskstats09:44:520
2103799230,23cyclictest0-21swapper/009:10:480
2103799230,23cyclictest0-21swapper/009:10:480
2103799230,22cyclictest3433-21sendmail-msp07:40:010
2103799230,22cyclictest1248-21ssh11:27:000
2103799230,22cyclictest0-21swapper/012:03:300
2103799230,20cyclictest0-21swapper/010:40:270
2103799230,1cyclictest0-21swapper/011:16:440
2103799230,0cyclictest14516-21diskmemload12:24:460
2103799230,0cyclictest14516-21diskmemload09:25:100
2103799230,0cyclictest0-21swapper/012:32:340
2103799230,0cyclictest0-21swapper/009:46:580
21043992222,0cyclictest14516-21diskmemload12:05:333
21043992222,0cyclictest0-21swapper/311:11:263
21043992219,2cyclictest17683-21perf08:10:003
2104399220,22cyclictest0-21swapper/312:32:333
2104399220,21cyclictest21984-21ssh10:17:193
2104399220,21cyclictest14516-21diskmemload10:10:503
2104399220,21cyclictest0-21swapper/311:26:223
2104399220,21cyclictest0-21swapper/311:05:363
2104399220,21cyclictest0-21swapper/310:46:173
2104399220,1cyclictest32065-21turbostat.cron08:40:013
2104399220,1cyclictest0-21swapper/311:35:023
2104399220,1cyclictest0-21swapper/307:35:003
21042992222,0cyclictest14516-21diskmemload10:26:042
21042992222,0cyclictest0-21swapper/209:57:082
21042992220,1cyclictest32751-21ssh10:56:052
21042992219,3cyclictest0-21swapper/209:21:322
21042992219,2cyclictest3413-21latency_hist07:40:012
21042992219,2cyclictest14817-21cat12:09:572
2104299220,2cyclictest6352-21ssh11:02:082
2104299220,22cyclictest27105-21ssh11:50:112
2104299220,0cyclictest0-21swapper/211:41:222
21041992221,1cyclictest15505-21ssh09:10:481
21041992221,1cyclictest15505-21ssh09:10:481
21041992220,1cyclictest7043-21cron08:55:001
21041992220,1cyclictest27663-21rm11:51:071
2104199220,22cyclictest0-21swapper/109:47:201
2104199220,21cyclictest990-21runrttasks10:57:091
2104199220,21cyclictest28121-21apt-get08:34:521
2104199220,21cyclictest14516-21diskmemload11:40:491
2104199220,21cyclictest14428-21apt-get10:40:031
2104199220,21cyclictest0-21swapper/112:35:191
2104199220,21cyclictest0-21swapper/112:30:361
2104199220,21cyclictest0-21swapper/110:36:091
2104199220,21cyclictest0-21swapper/110:36:091
2104199220,21cyclictest0-21swapper/109:15:261
2104199220,1cyclictest0-21swapper/109:20:271
2104199220,1cyclictest0-21swapper/107:20:001
21037992222,0cyclictest0-21swapper/011:11:420
21037992220,2cyclictest0-21swapper/011:03:330
21037992220,1cyclictest0-21swapper/008:49:520
2103799220,0cyclictest0-21swapper/010:18:240
21043992120,1cyclictest7760-21apt-get08:59:513
21043992120,1cyclictest21948-21apt-get07:14:493
2104399210,21cyclictest14516-21diskmemload10:06:423
2104399210,21cyclictest0-21swapper/312:15:083
21042992120,0cyclictest0-21swapper/208:29:512
21042992119,1cyclictest4388-21awk08:50:002
21042992119,1cyclictest10178-21apt-get09:04:522
2104299210,20cyclictest0-21swapper/208:59:512
2104299210,20cyclictest0-21swapper/207:49:522
2104299210,1cyclictest16177-21ssh12:10:322
21041992121,0cyclictest0-21swapper/108:14:511
21041992121,0cyclictest0-21swapper/107:29:511
21041992120,1cyclictest11299-21apt-get07:59:511
21041992119,1cyclictest22632-21users08:20:001
2104199210,0cyclictest0-21swapper/108:39:521
21037992121,0cyclictest0-21swapper/008:39:510
21037992119,1cyclictest0-21swapper/008:30:010
2103799210,21cyclictest0-21swapper/008:35:010
2103799210,21cyclictest0-21swapper/007:14:490
2103799210,20cyclictest0-21swapper/008:15:010
2103799210,0cyclictest0-21swapper/009:05:020
2103799210,0cyclictest0-21swapper/008:19:510
21043992019,1cyclictest26764-21apt-get07:24:523
21043992018,1cyclictest11580-21perf09:05:003
2104399200,20cyclictest0-21swapper/308:49:523
2104399200,20cyclictest0-21swapper/307:44:503
2104399200,0cyclictest0-21swapper/307:59:513
21042992020,0cyclictest0-21swapper/207:24:522
21042992016,3cyclictest25909-21cron07:20:002
2104299200,20cyclictest0-21swapper/207:44:502
2104299200,0cyclictest0-21swapper/208:34:522
21041992020,0cyclictest0-21swapper/109:05:021
21041992020,0cyclictest0-21swapper/108:09:511
21041992020,0cyclictest0-21swapper/107:54:521
2104199200,19cyclictest13718-21apt-get08:04:521
21037992020,0cyclictest0-21swapper/008:09:510
21037992019,1cyclictest4111-21apt-get07:44:500
21037992019,0cyclictest0-21swapper/007:29:510
21043991919,0cyclictest24348-21apt-get07:19:513
21043991919,0cyclictest0-21swapper/308:24:503
21042991919,0cyclictest0-21swapper/208:09:512
21042991918,1cyclictest8905-21apt-get07:54:522
21042991918,0cyclictest0-21swapper/207:15:002
21042991917,1cyclictest22645-21systemctl08:20:002
21037991919,0cyclictest0-21swapper/007:49:520
21037991919,0cyclictest0-21swapper/007:24:520
21043991818,0cyclictest0-21swapper/308:29:513
21043991818,0cyclictest0-21swapper/308:04:523
2104399180,16cyclictest29712-21latency_hist08:35:013
21042991818,0cyclictest0-21swapper/208:24:502
2104299180,18cyclictest0-21swapper/208:15:002
21041991818,0cyclictest0-21swapper/108:59:511
21041991818,0cyclictest0-21swapper/107:44:501
21041991818,0cyclictest0-21swapper/107:14:491
21041991816,1cyclictest11553-21idleruntime-cro09:05:001
21041991816,1cyclictest0-21swapper/108:50:001
21041991816,1cyclictest0-21swapper/108:45:011
21037991812,5cyclictest9611-21grep09:00:010
2103799180,4cyclictest0-21swapper/008:54:590
2103799180,4cyclictest0-21swapper/008:04:520
21043991715,1cyclictest30848-21cron07:30:003
2104399170,14cyclictest0-21swapper/308:41:153
21042991717,0cyclictest0-21swapper/208:54:512
2103799170,2cyclictest0-21swapper/009:04:520
21043991614,1cyclictest7077-21idleruntime-cro08:55:003
21043991614,1cyclictest12422-21grep09:05:023
2104399160,13cyclictest1583-21apt-get07:39:513
21042991615,1cyclictest31607-21apt-get07:34:522
21041991616,0cyclictest0-21swapper/107:39:511
2104199160,1cyclictest0-21swapper/107:35:011
21037991616,0cyclictest0-21swapper/007:34:520
21043991515,0cyclictest0-21swapper/308:14:513
21043991515,0cyclictest0-21swapper/307:54:523
21042991515,0cyclictest0-21swapper/208:39:512
2104299150,4cyclictest11570-21apt-get09:05:022
21041991515,0cyclictest0-21swapper/107:24:521
21037991514,1cyclictest0-21swapper/007:19:340
21043991414,0cyclictest0-21swapper/307:45:253
2104299140,14cyclictest0-21swapper/208:40:432
2104299140,14cyclictest0-21swapper/208:00:112
2104299140,14cyclictest0-21swapper/207:56:312
2103799140,14cyclictest0-21swapper/008:40:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional