You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 23:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Mar 24, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31769211197,9sleep00-21swapper/007:09:220
3160929380,9sleep20-21swapper/207:07:192
3148728742,41sleep30-21swapper/307:05:443
3155328668,14sleep10-21swapper/107:06:341
3191199291,6cyclictest13955-21apt-get12:15:132
3191199280,28cyclictest0-21swapper/209:32:442
3191199280,19cyclictest0-21swapper/211:50:262
3191199280,19cyclictest0-21swapper/211:50:252
3191199270,24cyclictest0-21swapper/210:53:052
3191199261,25cyclictest0-21swapper/211:13:112
3191199260,4cyclictest0-21swapper/209:18:322
3191199260,3cyclictest0-21swapper/211:00:122
3191199260,0cyclictest0-21swapper/211:44:372
3191199260,0cyclictest0-21swapper/210:33:092
3191199250,18cyclictest0-21swapper/212:33:522
3191199231,3cyclictest24486-21diskmemload09:40:202
3191199230,5cyclictest0-21swapper/210:01:072
3191199230,5cyclictest0-21swapper/210:01:072
3190399230,2cyclictest0-21swapper/012:37:210
3190399230,23cyclictest0-21swapper/012:02:500
3191199223,2cyclictest0-21swapper/210:46:292
3191199221,4cyclictest0-21swapper/209:54:462
3191199220,22cyclictest0-21swapper/210:57:522
3191199220,21cyclictest0-21swapper/209:58:432
3191199220,20cyclictest0-21swapper/209:22:362
3191199220,1cyclictest0-21swapper/210:09:202
3191199220,1cyclictest0-21swapper/209:26:112
31913992121,0cyclictest0-21swapper/309:26:473
3191199214,1cyclictest24486-21diskmemload11:39:112
3191199214,1cyclictest24486-21diskmemload10:27:492
3191199213,9cyclictest24486-21diskmemload10:10:542
3191199213,17cyclictest16554-21awk10:40:202
3191199212,15cyclictest9213-21grep08:35:202
31911992121,0cyclictest0-21swapper/210:15:012
3191199210,2cyclictest31897-21cyclictest12:29:222
3191199210,20cyclictest0-21swapper/212:12:532
3191199210,16cyclictest0-21swapper/209:45:512
3191199204,1cyclictest24486-21diskmemload11:15:302
3191199202,13cyclictest12430-21apache_accesses11:55:112
3191199200,5cyclictest0-21swapper/211:45:182
3191199200,3cyclictest0-21swapper/211:28:452
3190799207,13cyclictest8484-21apache210:21:561
31907992016,2cyclictest0-21swapper/109:32:461
3190799200,20cyclictest0-21swapper/109:58:531
3190399200,2cyclictest0-21swapper/009:21:220
3191399193,14cyclictest18226-21sh10:03:203
3191399193,14cyclictest18226-21sh10:03:193
3191199191,1cyclictest20533-21sed07:50:262
3191199190,5cyclictest26311-21head11:05:212
3191199190,3cyclictest0-21swapper/211:31:272
3191199190,3cyclictest0-21swapper/209:10:132
3191199190,3cyclictest0-21swapper/208:45:122
3191199190,17cyclictest0-21swapper/209:35:492
3190399192,3cyclictest0-21swapper/009:45:490
3191199182,2cyclictest29829-21ssh12:05:142
3191199180,5cyclictest0-21swapper/212:35:212
3191199180,16cyclictest0-21swapper/210:39:482
3190799181,13cyclictest0-21swapper/112:04:281
3190399187,6cyclictest6261-21munin-run10:35:000
3190399180,16cyclictest0-21swapper/010:44:520
31913991711,5cyclictest23304-21apt-get08:00:013
3191199174,2cyclictest0-21swapper/210:20:502
31907991717,0cyclictest0-21swapper/111:53:201
31907991717,0cyclictest0-21swapper/111:53:191
31907991716,1cyclictest0-21swapper/112:35:121
31907991716,1cyclictest0-21swapper/110:53:591
31907991715,1cyclictest5196-21ssh12:10:011
3190799170,1cyclictest0-21swapper/111:16:181
3190799170,16cyclictest0-21swapper/110:00:571
3190799170,16cyclictest0-21swapper/110:00:561
3190399175,6cyclictest16197-21perf07:45:000
31913991615,1cyclictest0-21swapper/310:40:463
3191399160,0cyclictest0-21swapper/312:10:583
3191199161,1cyclictest0-21swapper/208:20:132
3190799160,1cyclictest0-21swapper/110:30:451
3190399160,16cyclictest0-21swapper/009:32:460
3191399156,8cyclictest8486-21apache212:30:493
3191399154,10cyclictest30734-21apache_volume10:30:133
3191399153,10cyclictest15343-21/usr/sbin/munin12:35:143
31913991513,1cyclictest0-21swapper/309:50:323
31913991511,3cyclictest7713-21ls10:35:183
31913991511,3cyclictest21096-21head09:00:213
31913991511,3cyclictest11220-21apt09:40:133
3191399150,1cyclictest0-21swapper/309:15:133
3191399150,0cyclictest0-21swapper/311:13:443
3191199159,5cyclictest15457-21grep08:50:152
3191199154,1cyclictest24486-21diskmemload11:20:372
3191199152,4cyclictest24486-21diskmemload12:20:232
3190799156,6cyclictest14889-21munin-run12:34:591
31907991510,5cyclictest8484-21apache211:46:501
3190799150,14cyclictest9342-21ssh09:38:591
3190399151,6cyclictest225-21systemd-journal10:54:590
31913991412,1cyclictest0-21swapper/309:32:103
31913991411,2cyclictest15198-21awk07:40:203
31913991410,3cyclictest14952-21cron08:50:003
31913991410,2cyclictest7090-21cat08:30:233
31913991410,2cyclictest26751-21grep11:25:153
3191199149,4cyclictest7319-21tail08:30:242
3191199149,4cyclictest22614-21tr09:05:142
3191199149,3cyclictest10260-21wc07:30:192
3191199148,5cyclictest8544-21sed07:25:252
3191199148,5cyclictest2768-21cut07:15:172
3191199148,5cyclictest22992-21taskset07:56:172
3191199148,5cyclictest18014-21sort07:45:262
3191199146,7cyclictest1838-21grep07:15:002
3191199141,12cyclictest28272-21cron08:10:002
31911991411,2cyclictest13775-21cron07:39:592
3190799142,11cyclictest0-21swapper/110:55:361
3190799141,7cyclictest0-21swapper/111:01:541
31907991411,2cyclictest13968-21ssh12:15:131
31907991410,3cyclictest3731-21rm12:09:111
3190799140,1cyclictest0-21swapper/111:56:531
3190799140,13cyclictest0-21swapper/110:13:361
3190399148,2cyclictest19039-21munin-run11:40:010
31903991413,1cyclictest0-21swapper/010:05:460
3190399140,1cyclictest0-21swapper/011:59:040
3191399139,3cyclictest2387-21wc08:20:253
3191399139,3cyclictest17326-21sh08:55:003
3191399132,10cyclictest0-21swapper/311:01:453
31913991310,2cyclictest4899-21grep08:25:273
31913991310,2cyclictest4781-21tr07:20:143
31913991310,2cyclictest28773-21grep08:10:153
31913991310,2cyclictest26939-21grep11:05:273
31913991310,2cyclictest2235-21grep07:15:153
31913991310,2cyclictest13309-21head11:55:163
3191399130,12cyclictest12542-21grep11:35:233
3191199136,6cyclictest5471-21sed07:20:192
3191199136,6cyclictest23774-21ssh12:01:062
3191199131,2cyclictest4667-21sshd08:25:242
31911991310,2cyclictest615-21sh08:20:012
31911991310,2cyclictest615-21sh08:20:002
31911991310,2cyclictest19220-21grep08:55:262
3190799139,3cyclictest28541-21ssh09:50:161
3190799138,4cyclictest11949-21ssh10:18:511
3190799136,2cyclictest0-21swapper/111:39:111
31907991311,1cyclictest0-21swapper/109:45:131
31907991310,2cyclictest23625-21wc09:05:221
31907991310,2cyclictest18864-21cut11:20:161
3190399139,3cyclictest16174-21cut08:50:190
3190399139,3cyclictest12058-21tail09:40:180
3190399139,3cyclictest0-21swapper/011:09:450
3190399131,1cyclictest1699-21ssh12:07:360
31903991310,2cyclictest990-21cat09:15:120
31903991310,2cyclictest9370-21tr07:30:130
31903991310,2cyclictest4967-21sed08:25:280
31903991310,2cyclictest11208-21grep08:40:160
3190399130,12cyclictest30516-21cat12:25:000
3191399129,2cyclictest31570-21wc10:50:003
3191399129,2cyclictest28221-21chrt11:45:133
3191399129,2cyclictest26861-21grep09:10:273
3191399129,2cyclictest23625-21cut10:25:193
3191399129,2cyclictest108742sleep309:59:063
3191399128,3cyclictest19427-21ssh11:40:093
3191399124,7cyclictest30891-21sendmail_mailqu10:10:213
3191399124,6cyclictest11642-21/usr/sbin/munin07:35:133
3191399122,8cyclictest8486-21apache209:47:333
3191399121,2cyclictest0-21swapper/310:07:453
3191399120,12cyclictest0-21swapper/310:18:093
3191399120,11cyclictest18560-21awk07:50:003
3191199127,4cyclictest14032-21chrt07:40:082
3191199127,4cyclictest12514-21chrt08:44:162
3191199126,5cyclictest20865-21if_enp2s009:00:182
3191199122,2cyclictest28615-21aten2_r7power_e08:10:152
3191199121,2cyclictest225-21systemd-journal08:00:022
3190799129,2cyclictest30650-21sh08:15:011
3190799129,2cyclictest28752-21ssh11:25:331
3190799129,2cyclictest25017-21tr08:00:231
3190799129,2cyclictest23315-21perf08:00:011
3190799129,2cyclictest23170-21grep10:25:161
3190799129,2cyclictest20476-21grep07:50:261
3190799129,2cyclictest19044-21tail09:25:181
3190799129,2cyclictest1817-21cron07:15:001
3190799129,2cyclictest18143-21head07:45:271
3190799129,2cyclictest15899-21grep10:40:151
3190799129,2cyclictest13787-21tr08:45:181
3190799129,2cyclictest1265-21cut11:10:161
3190799129,2cyclictest10112-21tr09:20:141
3190799128,3cyclictest7076-21grep10:35:151
3190799127,4cyclictest20028-21cut09:00:141
3190799126,5cyclictest10027-21head07:30:161
3190799122,3cyclictest0-21swapper/109:15:131
3190799122,1cyclictest26690-21python11:05:261
3190799120,10cyclictest0-21swapper/112:23:511
3190399129,2cyclictest6264-21cut10:15:170
3190399129,2cyclictest5912-21grep08:30:130
3190399129,2cyclictest30028-21cat08:10:270
3190399129,2cyclictest25660-21grep10:45:300
3190399129,2cyclictest23288-21cron08:00:000
3190399129,2cyclictest21603-21grep09:00:260
3190399129,2cyclictest18360-21sed08:55:180
3190399129,2cyclictest16179-21cat11:00:000
3190399129,2cyclictest1405-21grep11:10:160
3190399128,3cyclictest8301-21tr12:30:210
3190399128,3cyclictest685-21rm09:15:000
3190399128,3cyclictest4745-21grep11:50:150
3190399128,3cyclictest4745-21grep11:50:150
3190399128,3cyclictest21482-21grep11:40:230
3190399127,4cyclictest8057-21pmu-power07:25:210
3190399122,4cyclictest6249-21cron10:39:560
3190399121,2cyclictest225-21systemd-journal10:29:550
31903991211,1cyclictest0-21swapper/011:29:140
3190399120,6cyclictest4244-21perf07:19:590
3190399120,12cyclictest0-21swapper/010:02:580
3190399120,12cyclictest0-21swapper/010:02:570
3191399118,2cyclictest9370-21taskset10:55:223
3191399118,2cyclictest6210-21ssh11:31:413
3191399118,2cyclictest3439-21grep09:35:183
3191399118,2cyclictest30670-21ssh12:05:213
3191399118,2cyclictest26158-21sh12:21:453
3191399118,2cyclictest2556-21ssh12:27:023
3191399118,2cyclictest22438-21ssh12:00:233
3191399118,2cyclictest20762-21ssh11:21:003
3191399118,2cyclictest20326-21ssh10:23:483
3191399118,2cyclictest19698-21cron08:59:593
3191399118,2cyclictest19549-21grep07:50:163
3191399118,2cyclictest11388-21grep11:15:263
3191399113,4cyclictest225-21systemd-journal09:24:593
3191399111,3cyclictest225-21systemd-journal12:15:143
3191399111,2cyclictest0-21swapper/310:50:143
3191399110,9cyclictest8163-21cut08:35:143
3191399110,8cyclictest0-21swapper/311:50:043
3191399110,8cyclictest0-21swapper/311:50:033
3190799118,2cyclictest970-21ssh12:25:511
3190799118,2cyclictest8607-21sh11:33:281
3190799118,2cyclictest27882-21grep08:05:261
3190799118,2cyclictest25465-21cat09:10:161
3190799118,2cyclictest21687-21cut10:05:161
3190799118,2cyclictest19503-21ssh11:40:101
3190799118,2cyclictest16597-21basename08:50:221
3190799118,2cyclictest14427-21ssh09:41:221
3190799118,2cyclictest12387-21head07:35:161
3190799113,5cyclictest38150irq/131-snd_hda07:30:001
3190799113,4cyclictest225-21systemd-journal08:25:001
3190799111,3cyclictest25211-21awk10:45:261
3190399118,2cyclictest7860-21ssh12:10:400
3190399118,2cyclictest6101-21grep07:20:260
3190399118,2cyclictest5963-21grep09:55:260
3190399118,2cyclictest31229-21cat08:15:170
3190399118,2cyclictest31229-21cat08:15:170
3190399118,2cyclictest29163-21ssh10:10:040
3190399118,2cyclictest2848-21ssh12:27:160
3190399118,2cyclictest28171-21tr11:45:140
3190399118,2cyclictest28087-21sh09:50:140
3190399118,2cyclictest17989-21ssh11:20:110
3190399118,2cyclictest17219-21head07:45:170
3190399118,2cyclictest16869-21cut11:00:150
3190399118,2cyclictest15881-21sh11:18:470
3190399118,2cyclictest1504-21cut08:20:170
3190399118,2cyclictest14369-21expr12:15:160
3190399118,2cyclictest13905-21taskset10:20:110
3190399118,2cyclictest10198-21sh08:40:000
3190399117,3cyclictest4043-21grep11:30:210
3190399110,7cyclictest0-21swapper/009:25:170
3190399110,11cyclictest0-21swapper/009:39:530
3190399110,10cyclictest12396-21grep07:35:160
3191399108,1cyclictest6829-21chrt07:25:073
3191399108,1cyclictest10804-21cat08:40:153
3191399107,2cyclictest27429-21wc08:05:233
3191399107,2cyclictest22918-21sed09:05:153
3191399106,3cyclictest24471-21grep08:00:183
3190799108,1cyclictest79802sleep108:35:011
3190799108,1cyclictest5848-21grep08:30:131
3190799108,1cyclictest5582-21tr07:20:191
3190799108,1cyclictest327302sleep108:16:151
3190799108,1cyclictest327302sleep108:16:141
3190799108,1cyclictest2418-21expr07:15:151
3190799108,1cyclictest160692sleep107:43:481
3190799108,1cyclictest124262sleep108:43:061
3190799107,2cyclictest3351-21tr08:25:151
3190799107,2cyclictest19571-21chrt08:58:261
3190399108,1cyclictest23670-21rm08:00:010
3190399107,2cyclictest26652-21taskset08:05:150
3190399107,2cyclictest19157-21expr07:50:140
3190399107,2cyclictest15040-21cat08:50:010
3190399107,2cyclictest1468-21taskset07:10:330
3190399100,9cyclictest22685-21expr09:05:150
319139997,1cyclictest9348-21cut07:30:143
319139997,1cyclictest745-21sed07:10:183
319139997,1cyclictest31147-21cut08:15:183
319139997,1cyclictest31147-21cut08:15:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional