You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-02 - 22:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Sun Aug 02, 2026 12:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
516799272,3cyclictest16150-21munin-run09:25:013
5167992720,3cyclictest0-21swapper/309:10:173
5167992620,5cyclictest24362-21rm10:30:343
516799260,26cyclictest0-21swapper/310:59:393
516799260,25cyclictest31760-21diskmemload11:05:543
516799260,25cyclictest0-21swapper/312:17:533
516799260,1cyclictest0-21swapper/312:29:093
516299260,25cyclictest15872-21runrttasks09:24:472
5157992624,1cyclictest0-21swapper/009:48:500
5167992522,2cyclictest0-21swapper/311:53:153
5167992520,5cyclictest0-21swapper/308:08:483
516799250,3cyclictest313-21systemd-journal07:40:003
516799250,24cyclictest31760-21diskmemload11:04:193
516799250,24cyclictest31760-21diskmemload10:50:263
516799250,24cyclictest31760-21diskmemload09:59:133
516799250,24cyclictest2604-21sh12:36:323
516799250,24cyclictest0-21swapper/311:47:563
516799250,24cyclictest0-21swapper/311:35:143
516799250,21cyclictest31760-21diskmemload11:56:273
516799250,21cyclictest0-21swapper/312:01:183
516299251,21cyclictest151rcu_preempt11:30:232
516299250,25cyclictest0-21swapper/209:25:312
516299250,24cyclictest14696-21munin-run11:20:002
516299250,24cyclictest0-21swapper/211:56:222
5159992525,0cyclictest0-21swapper/112:01:081
5159992523,1cyclictest0-21swapper/111:20:001
515999252,22cyclictest0-21swapper/109:55:001
515799250,24cyclictest0-21swapper/012:08:490
5167992423,1cyclictest7821-21sh10:44:443
5167992423,1cyclictest31760-21diskmemload09:48:573
5167992422,1cyclictest0-21swapper/310:12:343
5167992420,3cyclictest31760-21diskmemload11:44:083
5167992420,3cyclictest0-21swapper/310:04:133
5167992420,3cyclictest0-21swapper/309:31:523
5167992419,4cyclictest0-21swapper/312:33:443
516799241,22cyclictest9234-21sh10:46:213
516799240,23cyclictest0-21swapper/312:10:303
516799240,23cyclictest0-21swapper/311:32:063
516799240,23cyclictest0-21swapper/311:26:063
516799240,21cyclictest0-21swapper/312:05:423
516799240,21cyclictest0-21swapper/309:27:333
516799240,20cyclictest0-21swapper/310:37:573
516799240,20cyclictest0-21swapper/310:28:423
516799240,0cyclictest0-21swapper/311:24:143
516799240,0cyclictest0-21swapper/309:50:553
5162992423,1cyclictest28-21ksoftirqd/212:28:492
516299241,0cyclictest28-21ksoftirqd/210:31:522
516299240,23cyclictest31760-21diskmemload10:52:292
516299240,23cyclictest0-21swapper/212:04:172
516299240,23cyclictest0-21swapper/210:43:002
5159992423,1cyclictest22-21ksoftirqd/109:25:311
515999240,24cyclictest31760-21diskmemload10:53:291
515999240,24cyclictest31760-21diskmemload10:02:401
515999240,24cyclictest23800-1kworker/u9:209:13:181
515999240,23cyclictest0-21swapper/111:52:321
515999240,21cyclictest151rcu_preempt10:36:171
5157992421,2cyclictest0-21swapper/009:15:250
515799240,23cyclictest4335-21ssh10:13:300
515799240,23cyclictest12981-21ssh09:52:050
515799240,0cyclictest0-21swapper/011:16:170
5167992322,1cyclictest0-21swapper/311:11:383
5167992320,3cyclictest0-21swapper/310:22:383
5167992320,2cyclictest0-21swapper/309:38:033
516799230,23cyclictest0-21swapper/310:16:343
516799230,0cyclictest0-21swapper/310:07:423
516799230,0cyclictest0-21swapper/309:18:413
5162992323,0cyclictest0-21swapper/210:15:562
5162992323,0cyclictest0-21swapper/209:31:462
5162992323,0cyclictest0-21swapper/209:16:322
516299232,21cyclictest28-21ksoftirqd/210:01:242
516299232,21cyclictest271rcuc/211:08:442
5162992322,0cyclictest0-21swapper/209:13:482
5162992321,2cyclictest28-21ksoftirqd/208:05:012
5162992321,1cyclictest11251-21cstates10:48:442
5162992321,1cyclictest0-21swapper/211:03:112
5162992320,2cyclictest1192-21ssh09:41:012
5162992320,2cyclictest11746-21sh11:46:312
5162992313,9cyclictest26573-21grep08:58:542
516299230,2cyclictest9742-21ssh11:43:582
516299230,2cyclictest20848-21ssh11:25:372
516299230,23cyclictest0-21swapper/212:11:292
516299230,23cyclictest0-21swapper/210:25:512
516299230,23cyclictest0-21swapper/210:12:122
516299230,22cyclictest31760-21diskmemload12:16:232
516299230,22cyclictest28-21ksoftirqd/211:37:172
516299230,22cyclictest13462-21rm10:20:362
516299230,22cyclictest0-21swapper/208:43:432
516299230,0cyclictest0-21swapper/212:33:382
516299230,0cyclictest0-21swapper/212:05:382
516299230,0cyclictest0-21swapper/211:50:432
516299230,0cyclictest0-21swapper/210:55:122
516299230,0cyclictest0-21swapper/210:37:122
5159992323,0cyclictest0-21swapper/112:16:181
5159992323,0cyclictest0-21swapper/110:31:521
5159992323,0cyclictest0-21swapper/109:22:481
5159992323,0cyclictest0-21swapper/109:15:321
5159992322,1cyclictest22-21ksoftirqd/111:08:441
5159992322,0cyclictest0-21swapper/110:55:121
5159992321,1cyclictest313-21systemd-journal11:35:001
5159992321,1cyclictest0-21swapper/112:32:081
5159992321,1cyclictest0-21swapper/112:10:201
5159992321,1cyclictest0-21swapper/111:22:001
5159992320,2cyclictest8519-21ssh10:16:341
5159992320,2cyclictest28970-21sh10:05:381
5159992320,2cyclictest20476-21rm11:25:081
515999231,21cyclictest22-21ksoftirqd/110:21:111
515999230,2cyclictest22280-21sh11:55:511
515999230,23cyclictest0-21swapper/112:39:591
515999230,23cyclictest0-21swapper/111:04:121
515999230,23cyclictest0-21swapper/109:56:161
515999230,22cyclictest963-21sh12:05:451
515999230,22cyclictest9511-21sh10:46:531
515999230,22cyclictest5500-21ssh11:12:271
515999230,22cyclictest4707-21cut10:13:441
515999230,22cyclictest32074-21ssh11:35:371
515999230,1cyclictest24172-21ssh09:33:161
515999230,0cyclictest0-21swapper/111:42:111
515999230,0cyclictest0-21swapper/110:41:091
5157992323,0cyclictest31760-21diskmemload10:31:300
5157992323,0cyclictest0-21swapper/011:25:330
5157992322,1cyclictest0-21swapper/009:39:360
5157992321,1cyclictest62550irq/127-eno109:25:210
5157992321,1cyclictest12101-21cron08:30:000
5157992321,1cyclictest0-21swapper/011:38:480
515799231,21cyclictest5166-21apt-get11:40:020
515799230,2cyclictest23838-21ssh09:32:500
515799230,23cyclictest0-21swapper/012:24:390
515799230,23cyclictest0-21swapper/011:55:350
515799230,23cyclictest0-21swapper/010:45:110
515799230,23cyclictest0-21swapper/009:23:200
515799230,22cyclictest8903-21ssh10:17:040
515799230,22cyclictest5414-21apt-get08:18:430
515799230,22cyclictest31760-21diskmemload12:34:140
515799230,22cyclictest23231-21ssh10:00:280
515799230,22cyclictest19056-21rm10:25:510
515799230,22cyclictest17370-21rm11:51:410
515799230,1cyclictest27443-21ssh11:03:270
515799230,0cyclictest0-21swapper/010:39:310
515799230,0cyclictest0-21swapper/009:55:060
515799230,0cyclictest0-21swapper/009:13:070
5167992220,1cyclictest25050-21apt-get07:50:023
5167992220,1cyclictest17998-21ssh12:21:363
5167992220,1cyclictest0-21swapper/308:30:003
516799220,21cyclictest0-21swapper/309:08:443
516799220,21cyclictest0-21swapper/308:03:453
5162992222,0cyclictest0-21swapper/210:05:242
5162992220,1cyclictest0-21swapper/211:11:252
5162992220,1cyclictest0-21swapper/209:50:262
5162992220,1cyclictest0-21swapper/209:03:442
516299220,22cyclictest0-21swapper/209:56:182
516299220,21cyclictest0-21swapper/211:20:252
516299220,21cyclictest0-21swapper/209:45:082
516299220,21cyclictest0-21swapper/209:36:372
516299220,21cyclictest0-21swapper/207:50:022
516299220,1cyclictest28-21ksoftirqd/208:28:442
516299220,1cyclictest271rcuc/212:21:232
516299220,0cyclictest0-21swapper/212:35:522
5159992222,0cyclictest0-21swapper/112:21:511
5159992222,0cyclictest0-21swapper/110:25:511
5159992221,0cyclictest0-21swapper/108:43:431
5159992220,1cyclictest24437-21apt-get07:50:021
5159992219,2cyclictest211rcuc/108:05:011
515999220,22cyclictest0-21swapper/109:35:491
515999220,21cyclictest31782-21turbostat09:10:001
515999220,1cyclictest6289-21sh09:45:151
515999220,1cyclictest0-21swapper/109:43:121
515999220,0cyclictest0-21swapper/111:45:181
5157992222,0cyclictest31760-21diskmemload10:55:200
5157992222,0cyclictest31760-21diskmemload10:20:130
5157992220,2cyclictest28973-21ssh10:05:380
5157992220,2cyclictest28960-21ssh12:02:280
5157992220,2cyclictest0-21swapper/011:06:200
5157992220,1cyclictest25065-21apt-get08:58:450
5157992220,1cyclictest22826-21ssh12:25:350
515799220,22cyclictest31760-21diskmemload11:30:590
515799220,22cyclictest0-21swapper/009:40:110
515799220,21cyclictest6017-21rm12:10:090
515799220,21cyclictest1590-21ssh12:35:070
515799220,21cyclictest10481-21apt-get07:23:440
515799220,21cyclictest0-21swapper/008:35:010
515799220,1cyclictest5665-21apt-get07:13:440
515799220,1cyclictest0-21swapper/012:17:070
515799220,1cyclictest0-21swapper/011:47:470
516799210,21cyclictest0-21swapper/308:23:453
516799210,20cyclictest21716-21cron08:50:013
516799210,20cyclictest0-21swapper/308:33:443
5162992121,0cyclictest0-21swapper/208:33:442
5162992119,1cyclictest9660-21cron08:25:002
516299210,20cyclictest0-21swapper/208:05:022
516299210,20cyclictest0-21swapper/207:38:432
515999210,21cyclictest0-21swapper/112:25:091
515999210,21cyclictest0-21swapper/107:38:431
515999210,0cyclictest22275-21apt-get08:53:431
5157992121,0cyclictest0-21swapper/011:10:080
5157992120,1cyclictest14585-21ssh10:51:060
515799210,20cyclictest27483-21apt-get09:03:440
515799210,0cyclictest0-21swapper/011:20:230
515799210,0cyclictest0-21swapper/008:53:430
5167992020,0cyclictest0-21swapper/308:38:443
5167992018,1cyclictest9906-21perf07:20:003
5159992020,0cyclictest0-21swapper/108:28:441
5159992020,0cyclictest0-21swapper/108:05:021
515999200,0cyclictest0-21swapper/107:48:431
5157992020,0cyclictest0-21swapper/009:08:440
5157992020,0cyclictest0-21swapper/007:43:410
515799200,19cyclictest25374-21apt-get07:53:450
5167991919,0cyclictest0-21swapper/308:13:423
5167991918,1cyclictest0-21swapper/308:18:433
5162991919,0cyclictest0-21swapper/208:48:412
5162991917,1cyclictest16955-21awk08:40:012
5159991919,0cyclictest0-21swapper/108:33:441
5159991919,0cyclictest0-21swapper/107:43:411
5159991917,1cyclictest606-21dbus-daemon08:25:001
5159991917,1cyclictest22-21ksoftirqd/108:58:531
5157991919,0cyclictest22581-21apt-get07:48:430
5157991919,0cyclictest0-21swapper/007:38:430
5157991918,0cyclictest32079-21apt-get08:05:020
5167991818,0cyclictest0-21swapper/308:43:433
5167991818,0cyclictest0-21swapper/307:13:443
516799180,17cyclictest14720-21perf07:30:003
5162991818,0cyclictest0-21swapper/208:13:432
5162991818,0cyclictest0-21swapper/207:48:432
5162991818,0cyclictest0-21swapper/207:33:432
5162991816,1cyclictest12336-21idleruntime-cro07:25:002
516299181,2cyclictest151rcu_preempt08:19:042
5159991818,0cyclictest0-21swapper/107:28:431
5157991818,0cyclictest15069-21apt-get08:38:440
5157991817,1cyclictest19877-21apt-get08:48:410
5167991717,0cyclictest0-21swapper/307:33:423
5167991714,2cyclictest0-21swapper/308:58:453
516799170,16cyclictest24438-21perf07:50:013
5162991717,0cyclictest20184-21apt-get07:43:402
5162991717,0cyclictest0-21swapper/207:58:462
516299170,0cyclictest0-21swapper/208:53:432
5159991717,0cyclictest0-21swapper/108:48:411
5159991716,0cyclictest3009-21apt-get08:13:431
5159991715,1cyclictest0-21swapper/108:40:011
5157991716,1cyclictest30186-21apt-get08:03:450
5167991615,0cyclictest0-21swapper/307:23:443
516799160,0cyclictest0-21swapper/307:55:593
516299165,11cyclictest28-21ksoftirqd/207:13:462
516299160,15cyclictest31105-21awk09:08:512
5159991615,1cyclictest22-21ksoftirqd/107:15:011
5159991614,1cyclictest15270-21apt-get07:33:431
5159991614,1cyclictest12335-21latency_hist07:25:001
5157991614,1cyclictest17114-21turbostat.cron07:35:000
515799160,15cyclictest0-21swapper/008:43:530
5167991512,2cyclictest611-21irqbalance08:52:353
516799150,15cyclictest0-21swapper/309:02:123
5162991515,0cyclictest0-21swapper/207:28:422
5162991514,1cyclictest28-21ksoftirqd/207:18:492
5157991514,1cyclictest0-21swapper/007:57:390
5159991414,0cyclictest0-21swapper/107:15:141
515999140,0cyclictest0-21swapper/108:16:171
5157991414,0cyclictest0-21swapper/007:25:180
515799140,0cyclictest0-21swapper/008:11:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional