You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-04 - 08:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Apr 04, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2036521080,0sleep10-21swapper/122:43:301
625210265,33sleep00-21swapper/019:07:480
58528362,6sleep10-21swapper/119:07:171
3109228270,8sleep30-21swapper/319:05:063
50928168,9sleep20-21swapper/219:06:292
88699320,26cyclictest13798-21ssh22:20:143
88099300,1cyclictest0-21swapper/021:56:570
88499293,4cyclictest225-21systemd-journal20:50:001
88699280,5cyclictest22322-21date22:44:593
88699275,1cyclictest25883-21diskmemload22:08:213
88699275,1cyclictest25883-21diskmemload22:08:203
88699264,4cyclictest22692-21/usr/sbin/munin22:45:173
88599260,26cyclictest0-21swapper/222:48:152
88499261,24cyclictest32084-21timerandwakeup22:30:251
88099262,1cyclictest13786-21apt-get22:20:130
88099260,2cyclictest9759-21grep23:55:140
88699252,5cyclictest0-21swapper/321:57:253
88699240,8cyclictest11384-21grep00:35:153
88699240,3cyclictest0-21swapper/321:10:123
88699240,21cyclictest0-21swapper/321:48:193
880992423,1cyclictest0-21swapper/023:12:140
88099242,2cyclictest25883-21diskmemload22:44:590
88099240,6cyclictest0-21swapper/021:24:450
88699232,6cyclictest7463-21grep22:55:163
886992319,1cyclictest141rcu_preempt00:28:293
88699231,1cyclictest0-21swapper/321:31:583
88699230,5cyclictest0-21swapper/322:19:323
88699230,1cyclictest0-21swapper/323:02:203
88599230,23cyclictest0-21swapper/223:29:072
88599230,22cyclictest0-21swapper/223:08:112
88499237,8cyclictest25-21ksoftirqd/123:53:301
88499237,4cyclictest0-21swapper/121:41:311
88099237,2cyclictest312-21timerandwakeup22:50:250
88699225,1cyclictest25883-21diskmemload23:57:083
88699223,1cyclictest25883-21diskmemload00:20:133
88599222,1cyclictest141rcu_preempt00:20:482
88499220,2cyclictest0-21swapper/121:37:161
88499220,18cyclictest0-21swapper/121:13:521
88099228,1cyclictest25883-21diskmemload22:09:410
88099228,1cyclictest25883-21diskmemload22:09:400
88099220,22cyclictest0-21swapper/021:51:270
88699212,2cyclictest0-21swapper/323:23:503
885992121,0cyclictest0-21swapper/221:53:012
88599211,2cyclictest19386-21/usr/sbin/munin00:00:242
88599210,21cyclictest0-21swapper/221:48:452
88599210,1cyclictest0-21swapper/222:24:122
88499212,19cyclictest0-21swapper/119:30:131
88499210,20cyclictest0-21swapper/122:01:491
88499210,13cyclictest0-21swapper/123:33:531
88099213,2cyclictest25308-21gltestperf23:25:160
88099210,1cyclictest0-21swapper/023:30:580
885992020,0cyclictest0-21swapper/223:56:312
88599201,16cyclictest0-21swapper/222:40:182
88599200,20cyclictest0-21swapper/221:12:422
88499201,19cyclictest0-21swapper/123:28:051
88099202,2cyclictest25883-21diskmemload00:36:180
88099201,3cyclictest8252-21taskset00:33:100
88099200,1cyclictest0-21swapper/023:17:300
88099200,1cyclictest0-21swapper/000:08:310
88099200,17cyclictest0-21swapper/021:25:380
88699194,1cyclictest0-21swapper/321:28:573
88699191,2cyclictest0-21swapper/321:41:253
88699190,3cyclictest0-21swapper/323:11:253
885991918,1cyclictest0-21swapper/221:25:142
885991914,1cyclictest32-21ksoftirqd/222:57:262
88599190,19cyclictest0-21swapper/222:54:392
88499199,1cyclictest0-21swapper/121:15:421
88099190,19cyclictest0-21swapper/021:18:420
88699183,2cyclictest23751-21apache_volume23:05:143
88699180,1cyclictest0-21swapper/321:37:383
88699180,1cyclictest0-21swapper/321:19:473
88599180,18cyclictest0-21swapper/223:39:382
88599180,17cyclictest0-21swapper/222:26:342
88499189,8cyclictest1892-21apache200:27:501
88499182,4cyclictest0-21swapper/123:03:221
884991815,2cyclictest0-21swapper/122:35:141
88499180,14cyclictest25883-21diskmemload23:17:581
880991815,2cyclictest0-21swapper/022:10:140
88699176,10cyclictest0-21swapper/322:34:323
88699175,2cyclictest0-21swapper/300:34:083
88699172,6cyclictest27309-21cron20:05:003
88699171,7cyclictest0-21swapper/323:38:593
88699171,4cyclictest0-21swapper/323:16:183
88699170,3cyclictest0-21swapper/321:21:523
885991714,2cyclictest32-21ksoftirqd/219:55:002
88599170,17cyclictest0-21swapper/223:34:072
88499170,6cyclictest0-21swapper/123:12:561
88499170,16cyclictest0-21swapper/123:47:121
88099179,7cyclictest18168-21sendmail00:19:590
88099172,2cyclictest27307-21cron20:05:010
880991712,4cyclictest874-21cyclictest22:32:030
88099170,5cyclictest0-21swapper/023:03:530
88699169,2cyclictest22921-21cat21:00:253
88699169,1cyclictest20620-21sh23:41:433
88699163,6cyclictest0-21swapper/323:45:113
88699162,8cyclictest25883-21diskmemload22:00:123
88699161,0cyclictest0-21swapper/322:26:123
885991614,1cyclictest0-21swapper/223:47:472
885991613,2cyclictest32-21ksoftirqd/221:35:152
88499162,14cyclictest0-21swapper/122:19:481
88499160,15cyclictest0-21swapper/122:24:531
88499160,14cyclictest0-21swapper/121:28:201
88099169,6cyclictest23020-21grep21:00:250
88099168,7cyclictest19012-21tr00:20:160
88099163,1cyclictest23343-21ntp_states22:25:210
88099161,6cyclictest25883-21diskmemload22:46:520
880991610,5cyclictest6225-21wc20:25:250
880991610,5cyclictest13768-21ssh22:59:060
88699159,2cyclictest1517-21expr23:50:153
88699159,1cyclictest9302-21cat19:25:173
88699159,1cyclictest25775-21sh23:25:203
88699159,1cyclictest21766-21sh00:02:013
88699159,1cyclictest18044-21wc20:50:253
88699159,1cyclictest16285-21sh00:18:303
88699159,1cyclictest14526-21tr20:45:133
88699158,2cyclictest30639-21ssh00:07:303
88699158,2cyclictest29934-21cat20:10:103
88699158,2cyclictest2850-21rm21:54:373
88699158,1cyclictest30000-21sed22:10:183
88599159,5cyclictest7394-21cron23:15:002
88599155,9cyclictest0-21swapper/221:00:122
88599150,1cyclictest0-21swapper/200:27:422
88599150,14cyclictest0-21swapper/223:40:122
88599150,0cyclictest0-21swapper/221:30:482
88499151,13cyclictest1235-21ntpd22:05:101
88499151,13cyclictest1235-21ntpd22:05:091
88499150,14cyclictest0-21swapper/122:52:591
88099159,5cyclictest21665-21grep21:45:260
88099157,2cyclictest27385-21ssh21:30:140
88099156,2cyclictest31508-21ssh23:10:000
88099154,5cyclictest0-21swapper/023:40:160
880991514,1cyclictest0-21swapper/020:43:450
88099151,1cyclictest0-21swapper/023:45:140
880991511,2cyclictest13178-21apt-config22:20:010
880991511,2cyclictest10430-21wc23:35:240
88099150,6cyclictest9244-21perf20:35:000
88699149,1cyclictest5437-21ssh23:32:493
88699149,1cyclictest4188-21chrt20:22:123
88699149,1cyclictest3454-21ls00:10:193
88699149,1cyclictest28915-21systemctl20:05:243
88699148,2cyclictest13090-21cut20:40:213
88699148,1cyclictest6687-21head19:20:183
88699148,1cyclictest499-21sed22:50:263
88699148,1cyclictest16499-21wc19:40:203
88699147,2cyclictest21200-21cat21:00:003
88699142,4cyclictest12816-21awk19:35:003
88699140,9cyclictest3097-21seq19:13:263
88699140,9cyclictest12470-21rm22:38:483
885991411,2cyclictest24693-21cron20:00:012
885991410,3cyclictest16698-21grep20:50:132
885991410,1cyclictest32-21ksoftirqd/200:09:592
88599140,3cyclictest0-21swapper/223:24:132
88599140,1cyclictest0-21swapper/223:51:122
88599140,13cyclictest0-21swapper/222:35:032
884991411,3cyclictest13754-21sh22:59:061
88499141,12cyclictest21149-21ssh00:20:491
884991410,1cyclictest0-21swapper/123:22:171
88099148,2cyclictest15075-21basename20:45:170
88099147,2cyclictest7939-21chrt19:23:300
88099147,2cyclictest20713-21head20:55:250
88099147,2cyclictest17565-21grep20:50:180
88099147,1cyclictest7446-21chrt21:37:090
88099147,1cyclictest5155-21cat19:15:250
88099144,2cyclictest225-21systemd-journal00:05:000
88099143,3cyclictest225-21systemd-journal19:10:590
88099143,2cyclictest225-21systemd-journal21:45:000
88099142,7cyclictest32070-21munin-run20:15:010
88099141,3cyclictest225-21systemd-journal00:12:210
880991410,3cyclictest3271-21wc20:20:200
880991410,3cyclictest28342-21grep21:10:260
880991410,2cyclictest16352-21grep23:20:140
880991410,2cyclictest11200-21grep20:35:260
88099140,4cyclictest225-21systemd-journal20:10:000
88699138,1cyclictest25428-21grep21:05:263
886991310,2cyclictest4127-21cut19:15:163
88599139,3cyclictest9773-21cat19:25:232
88599138,4cyclictest311rcuc/221:59:262
885991312,1cyclictest0-21swapper/222:12:112
88599131,12cyclictest0-21swapper/222:19:272
885991310,2cyclictest32-21ksoftirqd/200:35:052
885991310,2cyclictest21006-21wget22:05:152
885991310,2cyclictest21006-21wget22:05:142
885991310,2cyclictest14528-21ssh21:41:132
88599130,2cyclictest0-21swapper/222:03:412
88599130,1cyclictest141rcu_preempt00:15:282
88599130,13cyclictest0-21swapper/223:03:402
88499139,1cyclictest10300-21kworker/1:221:49:201
88499137,5cyclictest12069-21grep21:20:191
88499134,5cyclictest225-21systemd-journal00:08:581
88499131,5cyclictest0-21swapper/100:38:181
884991311,1cyclictest0-21swapper/120:20:141
88099138,1cyclictest14898-21chrt19:36:250
88099137,1cyclictest17397-21chrt19:42:460
88099136,2cyclictest7806-21cut22:35:230
88099136,2cyclictest23484-21cat19:55:170
88099135,2cyclictest27231-21tr00:25:140
88099132,2cyclictest225-21systemd-journal21:09:590
88099131,2cyclictest225-21systemd-journal19:34:590
88699129,2cyclictest24885-21cat20:00:013
88699129,2cyclictest1551-21grep20:15:263
88699128,1cyclictest19256-21awk19:45:233
88599129,2cyclictest9821-21grep23:15:242
88599129,2cyclictest8755-21sed20:30:262
88599129,2cyclictest6159-21cat20:25:242
88599129,2cyclictest30709-21head20:10:152
88599129,2cyclictest11726-21cat20:40:002
88599120,1cyclictest0-21swapper/221:17:072
88599120,12cyclictest0-21swapper/200:11:102
88499129,2cyclictest9936-21sed19:25:251
88499129,2cyclictest31054-21grep20:10:191
88499129,2cyclictest2633-21cat19:10:271
88499129,2cyclictest25482-21tr20:00:141
88499129,2cyclictest10216-21tail20:35:161
88499128,3cyclictest23592-21tail22:45:171
88499127,4cyclictest25883-21diskmemload23:06:411
88499125,6cyclictest2241-21tr00:10:121
88499125,2cyclictest0-21swapper/121:05:131
884991211,1cyclictest0-21swapper/122:12:451
88499120,11cyclictest19827-21seq19:48:321
88499120,11cyclictest14390-21cut20:45:141
88499120,11cyclictest12863-21rm23:37:071
88499120,0cyclictest0-21swapper/121:55:511
88099128,1cyclictest8351-21grep19:25:130
88099126,2cyclictest19833-21chrt19:48:360
88099123,2cyclictest225-21systemd-journal20:20:000
88099121,2cyclictest225-21systemd-journal23:55:000
88099120,3cyclictest225-21systemd-journal22:01:310
88699119,1cyclictest10994-21timerandwakeup20:35:253
88599118,2cyclictest5419-21chrt19:16:552
88599118,2cyclictest4379-21taskset20:24:392
88599118,2cyclictest32-21ksoftirqd/219:50:012
88599118,2cyclictest2282-21ssh22:32:192
88599118,2cyclictest15848-21expr19:40:142
88599118,2cyclictest1564-21expr19:10:162
88599118,2cyclictest13982-21cron20:45:012
88599118,2cyclictest13490-21cat19:35:142
88599118,2cyclictest10688-21ssh21:20:072
88599113,2cyclictest225-21systemd-journal21:06:222
885991111,0cyclictest0-21swapper/200:32:542
88499118,2cyclictest8323-21cut20:30:211
88499118,2cyclictest3743-21grep19:15:141
88499118,2cyclictest3475-21ssh00:30:151
88499118,2cyclictest30236-21ssh21:50:361
88499118,2cyclictest21103-21cron21:00:001
88499118,2cyclictest17884-21cut00:00:141
88499118,2cyclictest11966-21cut20:40:131
88499117,3cyclictest537-21snmpd23:40:111
88499117,3cyclictest22876-21wc21:00:241
88499117,3cyclictest22106-21cut22:25:141
88499117,3cyclictest10193-21ssh00:15:031
88499116,5cyclictest10929-21grep23:55:221
88499116,4cyclictest25883-21diskmemload21:32:141
88499116,4cyclictest24355-21taskset19:55:431
88499110,10cyclictest56632sleep120:25:181
88099118,2cyclictest22343-21perf19:55:000
88699109,1cyclictest6470-21awk20:25:263
88699108,1cyclictest21145-21uname19:50:203
88699108,1cyclictest14727-21grep19:35:253
88599108,1cyclictest7502sleep220:15:172
88599108,1cyclictest28887-21rt-features20:05:242
88599108,1cyclictest25197-21apache_processe20:00:112
88599108,1cyclictest10840-21wget19:30:122
88599107,2cyclictest7650-21grep19:20:272
88599107,2cyclictest32-21ksoftirqd/220:55:152
88599107,2cyclictest15727-21threads20:45:242
88499108,2cyclictest28053-21expr20:05:161
88499108,1cyclictest21207-21wc19:50:201
88499108,1cyclictest154342sleep119:40:061
88499107,2cyclictest711-21sort20:15:171
88499107,2cyclictest6731-21tail19:20:171
88499107,2cyclictest13104-21cut19:35:141
8869997,1cyclictest7510-21cat20:30:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional