You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-19 - 17:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Mar 19, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2152428674,8sleep00-21swapper/007:09:550
2129928373,6sleep10-21swapper/107:06:591
2143428270,8sleep20-21swapper/207:08:442
2135827966,9sleep30-21swapper/307:07:453
2162199280,27cyclictest0-21swapper/212:15:002
21621992710,4cyclictest32-21ksoftirqd/209:29:142
2161699270,27cyclictest0-21swapper/110:54:501
2161699270,25cyclictest0-21swapper/109:25:261
2162199267,7cyclictest32-21ksoftirqd/209:50:172
21621992623,2cyclictest32-21ksoftirqd/209:11:302
2161699250,1cyclictest0-21swapper/112:09:121
2162399240,23cyclictest0-21swapper/310:04:043
2162399240,1cyclictest0-21swapper/312:03:243
21621992420,1cyclictest32-21ksoftirqd/211:35:332
2162199240,24cyclictest0-21swapper/210:16:222
21616992421,2cyclictest8496-21apt-key07:49:591
2161699240,1cyclictest0-21swapper/110:36:001
2162399231,1cyclictest0-21swapper/311:18:513
2162199238,2cyclictest141rcu_preempt10:33:582
21621992315,4cyclictest32-21ksoftirqd/210:22:382
21621992311,6cyclictest32-21ksoftirqd/209:15:042
2162199230,23cyclictest0-21swapper/211:51:042
2162199230,1cyclictest0-21swapper/211:10:482
2161699230,23cyclictest0-21swapper/110:04:041
21612992320,2cyclictest13-21ksoftirqd/011:05:000
2162199220,22cyclictest0-21swapper/212:17:402
2162199220,21cyclictest0-21swapper/212:32:152
2162199220,1cyclictest0-21swapper/210:58:282
2162199220,0cyclictest0-21swapper/211:40:562
2161699220,18cyclictest0-21swapper/109:38:121
2161299220,1cyclictest141rcu_preempt10:39:360
2162199218,3cyclictest141rcu_preempt11:26:292
2162199213,2cyclictest141rcu_preempt10:09:272
21621992120,1cyclictest55550irq/126-enp2s0-11:18:372
21621992119,2cyclictest32-21ksoftirqd/212:39:072
2162199210,2cyclictest0-21swapper/211:33:122
2162199210,21cyclictest0-21swapper/212:01:302
21616992120,1cyclictest0-21swapper/112:01:361
21623992019,1cyclictest0-21swapper/311:51:183
2162399200,13cyclictest0-21swapper/312:25:353
2162199209,5cyclictest141rcu_preempt10:12:102
2162199200,19cyclictest0-21swapper/211:20:512
2161699200,5cyclictest0-21swapper/110:24:021
2161699200,1cyclictest0-21swapper/110:07:051
2161699200,18cyclictest0-21swapper/110:30:131
2161299200,20cyclictest0-21swapper/012:15:000
2162399194,3cyclictest26056-21apache_accesses12:30:103
2162399191,18cyclictest20905-21kworker/u8:012:37:063
2162399190,1cyclictest0-21swapper/312:21:463
2162199195,1cyclictest141rcu_preempt09:31:232
2162199192,3cyclictest32-21ksoftirqd/210:44:332
2162199190,5cyclictest0-21swapper/210:45:132
2161699190,5cyclictest0-21swapper/109:00:131
2161699190,18cyclictest0-21swapper/111:00:351
2161299192,1cyclictest141rcu_preempt10:13:460
2161299191,9cyclictest141rcu_preempt12:02:140
21612991918,1cyclictest13-21ksoftirqd/012:17:410
21612991916,3cyclictest0-21swapper/011:55:120
21612991913,3cyclictest13-21ksoftirqd/009:20:010
21612991911,3cyclictest13-21ksoftirqd/012:05:090
2161299191,10cyclictest141rcu_preempt09:40:130
2161299190,1cyclictest141rcu_preempt11:21:320
2162199183,1cyclictest14193-21diskmemload10:39:362
21621991816,2cyclictest32-21ksoftirqd/210:53:062
2162199181,4cyclictest0-21swapper/211:02:112
21621991812,1cyclictest55850irq/129-enp2s0-09:36:172
2161699180,18cyclictest0-21swapper/111:51:191
2161699180,16cyclictest0-21swapper/109:50:101
2161299181,1cyclictest141rcu_preempt11:40:240
2161299180,5cyclictest2691-21ssh11:37:020
2161299180,3cyclictest141rcu_preempt10:21:500
2161299180,1cyclictest141rcu_preempt10:55:520
2162399179,2cyclictest0-21swapper/309:38:123
2162399175,2cyclictest14193-21diskmemload11:12:513
21623991716,1cyclictest0-21swapper/310:07:143
21623991715,1cyclictest0-21swapper/310:23:453
2162199179,7cyclictest32-21ksoftirqd/211:05:232
2162199170,0cyclictest0-21swapper/212:09:152
2161699172,6cyclictest0-21swapper/107:55:121
21616991716,1cyclictest19978-21kworker/1:111:11:591
21616991716,1cyclictest0-21swapper/107:50:261
2161299177,2cyclictest13-21ksoftirqd/009:21:360
2161299173,1cyclictest13-21ksoftirqd/011:51:170
21612991715,2cyclictest13-21ksoftirqd/010:48:570
2162399162,1cyclictest0-21swapper/311:46:293
2162399160,3cyclictest0-21swapper/309:41:143
2162399160,1cyclictest0-21swapper/311:05:373
2162399160,1cyclictest0-21swapper/311:00:353
2162199169,2cyclictest321ksoftirqd/209:42:152
21621991615,1cyclictest0-21swapper/208:23:342
2161699160,1cyclictest0-21swapper/112:14:291
2161699160,16cyclictest0-21swapper/109:55:451
2161699160,16cyclictest0-21swapper/109:41:131
2161299163,6cyclictest0-21swapper/011:10:140
2161299162,1cyclictest141rcu_preempt09:54:080
2161299161,7cyclictest0-21swapper/011:30:160
21612991613,1cyclictest13-21ksoftirqd/009:49:370
2161299160,1cyclictest0-21swapper/009:35:590
21623991515,0cyclictest0-21swapper/309:51:123
21623991514,1cyclictest0-21swapper/309:24:493
2162399150,2cyclictest0-21swapper/311:37:283
2162399150,15cyclictest0-21swapper/310:34:423
2162399150,15cyclictest0-21swapper/310:29:123
21621991513,2cyclictest10503-21kworker/u8:109:09:542
2162199151,11cyclictest32-21ksoftirqd/211:59:212
2162199150,15cyclictest0-21swapper/209:45:212
2162199150,14cyclictest0-21swapper/212:24:132
2161699159,5cyclictest0-21swapper/109:48:311
2161699157,4cyclictest0-21swapper/111:38:531
2161699155,1cyclictest820-21apt-get12:15:111
21616991514,1cyclictest0-21swapper/112:32:151
21616991514,1cyclictest0-21swapper/111:24:541
21616991512,3cyclictest0-21swapper/111:18:281
2161699150,2cyclictest0-21swapper/110:40:531
2161299157,3cyclictest13-21ksoftirqd/011:07:050
2161299155,6cyclictest13-21ksoftirqd/009:57:200
2161299152,2cyclictest141rcu_preempt12:23:200
21612991510,2cyclictest141rcu_preempt12:32:140
2161299150,2cyclictest0-21swapper/009:12:240
2162399149,3cyclictest0-21swapper/312:13:083
21623991412,1cyclictest0-21swapper/312:05:113
2162399140,12cyclictest0-21swapper/310:59:573
2162199149,4cyclictest32-21ksoftirqd/209:23:512
21621991413,1cyclictest32-21ksoftirqd/211:45:122
21621991413,1cyclictest0-21swapper/207:40:122
21621991412,1cyclictest0-21swapper/207:15:142
2161699140,6cyclictest14453-21runrttasks10:45:371
2161699140,2cyclictest0-21swapper/111:40:351
2161299149,3cyclictest20764-21cat10:30:180
2161299148,2cyclictest11117-21ssh10:44:320
2161299147,5cyclictest23179-21sendmail_mailst07:10:230
2161299146,2cyclictest141rcu_preempt07:55:210
2161299143,11cyclictest13-21ksoftirqd/007:25:140
2161299142,1cyclictest141rcu_preempt12:25:190
2161299141,3cyclictest141rcu_preempt10:27:160
21612991413,1cyclictest13-21ksoftirqd/010:18:220
2161299141,2cyclictest141rcu_preempt10:51:310
2161299141,0cyclictest141rcu_preempt09:33:550
21612991410,3cyclictest13-21ksoftirqd/008:36:400
2162399136,3cyclictest0-21swapper/312:17:223
2162399136,1cyclictest8268-21apt-get07:50:023
2162399134,8cyclictest0-21swapper/307:55:113
2162399134,7cyclictest26629-21latency_hist10:15:013
2162399131,9cyclictest537-21snmpd09:13:593
2162399131,6cyclictest12165-21/usr/sbin/munin10:45:263
2162399130,3cyclictest0-21swapper/310:40:033
2162399130,2cyclictest0-21swapper/310:52:193
2162399130,1cyclictest0-21swapper/309:28:443
2162199135,8cyclictest32-21ksoftirqd/207:50:122
2162199135,2cyclictest225-21systemd-journal08:39:342
2162199134,1cyclictest141rcu_preempt10:25:582
2162199133,6cyclictest0-21swapper/207:10:132
2162199131,12cyclictest55550irq/126-enp2s0-10:00:152
21621991311,1cyclictest311rcuc/208:10:152
21621991310,2cyclictest18101-21wc08:10:002
2162199130,4cyclictest0-21swapper/209:57:182
2162199130,11cyclictest0-21swapper/208:45:132
2162199130,11cyclictest0-21swapper/208:45:122
2161699139,3cyclictest2200-21taskset08:44:421
2161699137,5cyclictest0-21swapper/107:40:131
2161699133,6cyclictest30449-21runrttasks11:34:541
2161699130,4cyclictest0-21swapper/111:08:471
2161699130,12cyclictest23993-21wc08:20:171
2161299138,4cyclictest0-21swapper/008:45:130
2161299138,4cyclictest0-21swapper/008:45:130
2161299132,5cyclictest7061-21awk08:55:010
21612991312,1cyclictest13-21ksoftirqd/008:00:240
21612991310,2cyclictest16193-21wc08:05:140
2161299130,2cyclictest0-21swapper/008:56:340
2161299130,1cyclictest141rcu_preempt12:37:060
2162399129,2cyclictest32202-21taskset11:55:123
2162399129,2cyclictest31750-21taskset10:17:403
2162399129,2cyclictest31241-21grep08:35:203
2162399129,2cyclictest28717-21sed10:35:163
2162399129,2cyclictest27851-21tr08:30:123
2162399129,2cyclictest27509-21cron08:30:003
2162399129,2cyclictest25288-21cat07:15:203
2162399129,2cyclictest23973-21grep09:15:153
2162399129,2cyclictest22958-21grep11:30:133
2162399129,2cyclictest14218-21tail08:00:163
2162399129,2cyclictest11380-21sh11:42:133
2162399124,6cyclictest14365-21apache_accesses11:25:113
21623991211,1cyclictest0-21swapper/307:25:123
21623991210,1cyclictest3715-21ls08:45:213
21623991210,1cyclictest3715-21ls08:45:213
2162399120,11cyclictest0-21swapper/311:24:423
2162399120,11cyclictest0-21swapper/309:00:133
2162199129,2cyclictest8033-21tail08:55:192
2162199129,2cyclictest29444-21grep08:30:252
2162199128,4cyclictest311rcuc/212:27:042
2162199127,2cyclictest32-21ksoftirqd/207:25:202
2161699129,2cyclictest38150irq/131-snd_hda11:56:261
2161699129,2cyclictest29417-21cat07:25:151
2161699129,2cyclictest27683-21cat07:20:191
2161699129,2cyclictest25885-21grep07:15:271
2161699129,2cyclictest25649-21tr08:25:141
2161699129,2cyclictest21311-21ssh09:13:521
2161699129,2cyclictest20381-21awk08:14:591
2161699129,2cyclictest17301-21cat11:45:261
2161699129,2cyclictest15871-21basename08:05:111
2161699129,2cyclictest15673-21ssh11:25:211
2161699129,2cyclictest13896-21ssh10:25:581
21616991210,1cyclictest0-21swapper/112:39:041
2161699120,12cyclictest0-21swapper/109:31:591
2161299129,2cyclictest7823-21grep07:45:270
2161299129,2cyclictest31918-21grep11:15:260
2161299129,2cyclictest27111-21grep08:25:260
2161299129,2cyclictest1846-21grep08:40:260
2161299129,1cyclictest14588-21ssh09:29:130
2161299123,4cyclictest0-21swapper/011:47:360
2161299123,2cyclictest23002-21/usr/sbin/munin08:20:150
2161299122,5cyclictest16447-21ssh10:08:530
21612991211,1cyclictest13-21ksoftirqd/007:50:170
21612991210,2cyclictest13-21ksoftirqd/011:25:120
2162399119,2cyclictest914-21seq07:33:263
2162399119,2cyclictest2763-21sed07:35:253
2162399119,2cyclictest24854-21taskset08:21:243
2162399119,2cyclictest18591-21rm09:30:483
2162399119,1cyclictest8571-21ssh09:45:013
2162399119,1cyclictest7283-21basename08:55:133
2162399119,1cyclictest5567-21seq07:41:473
2162399119,1cyclictest25364-21ssh09:55:043
2162399119,1cyclictest22619-21seq08:18:303
2162399119,1cyclictest16372-21grep08:05:153
2162399119,1cyclictest13503-21sed09:05:263
2162399119,1cyclictest1091-21cat08:40:193
2162399118,2cyclictest28707-21seq07:24:473
2162399118,2cyclictest18883-21uniq08:10:163
2162399110,7cyclictest225-21systemd-journal08:52:463
2162399110,10cyclictest6052-21taskset07:45:043
2162399110,10cyclictest22199-21expr07:10:153
2162199119,1cyclictest7663-21wc07:45:252
2162199118,2cyclictest32098-21tail07:30:152
2162199118,2cyclictest20699-21grep08:15:142
2162199118,2cyclictest10260-21sed09:00:152
2162199115,2cyclictest32-21ksoftirqd/207:40:002
2162199111,6cyclictest32-21ksoftirqd/208:45:002
2162199110,7cyclictest12334-21tr07:55:192
2161699119,2cyclictest28192-21expr08:30:141
2161699119,2cyclictest24840-21grep09:15:211
2161699119,1cyclictest7694-21expr08:55:171
2161699119,1cyclictest5205-21ls08:50:141
2161699119,1cyclictest512-21ssh10:57:041
2161699119,1cyclictest32728-21ssh10:18:251
2161699119,1cyclictest23503-21tail07:10:261
2161699119,1cyclictest21029-21expr08:15:151
2161699118,2cyclictest30494-21expr08:35:151
2161699118,2cyclictest19508-21grep12:25:251
2161699118,2cyclictest18419-21ssh10:10:081
2161699116,2cyclictest2505-21apt-get08:45:131
2161699116,2cyclictest2505-21apt-get08:45:121
2161699116,2cyclictest225-21systemd-journal08:05:001
21616991111,0cyclictest0-21swapper/112:22:571
2161299119,1cyclictest3535-21sed10:00:270
2161299119,1cyclictest25947-21grep07:15:270
2161299119,1cyclictest12195-21basename09:05:140
2161299116,1cyclictest13-21ksoftirqd/008:30:190
2161299111,2cyclictest537-21snmpd08:17:400
2162199108,1cyclictest48382sleep208:50:092
2162199108,1cyclictest27360-21tail07:20:162
2162199108,1cyclictest26664-21wc08:25:212
2162199107,2cyclictest13642-21cut08:00:132
2161699109,1cyclictest1634-21cat07:35:151
2161699109,1cyclictest13392-21sendmail_mailqu09:05:221
2161699108,1cyclictest779-21taskset07:31:391
2161699108,1cyclictest31530-21ssh09:20:061
2161299108,2cyclictest26927-21cat07:20:140
2161299108,1cyclictest4469-21tail07:40:160
2161299108,1cyclictest10106-21cat09:00:140
2161299107,2cyclictest161rcuc/008:10:130
2161299102,4cyclictest32081-21gltestperf07:30:150
216129997,2cyclictest1670-21expr07:35:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional