You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-03 - 20:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Apr 03, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1742429178,9sleep00-21swapper/007:09:120
1733528758,6sleep10-21swapper/107:08:031
1737328367,12sleep30-21swapper/307:08:333
1714528168,9sleep20-21swapper/207:05:352
17580992714,2cyclictest32-21ksoftirqd/210:00:132
17577992523,1cyclictest0-21swapper/112:36:411
1758599230,2cyclictest0-21swapper/308:10:133
1758599230,20cyclictest0-21swapper/311:44:423
17580992318,2cyclictest32-21ksoftirqd/209:34:072
1758099231,2cyclictest141rcu_preempt11:44:112
1757799221,2cyclictest141rcu_preempt12:13:581
17575992221,1cyclictest0-21swapper/011:37:560
17585992117,2cyclictest12882-21apache_processe09:50:133
1758599210,2cyclictest0-21swapper/311:46:193
1758599210,18cyclictest0-21swapper/310:44:333
17580992016,3cyclictest32-21ksoftirqd/210:14:542
1758099200,19cyclictest0-21swapper/209:15:322
17577992018,1cyclictest284782chrt09:40:021
17585991915,4cyclictest0-21swapper/312:16:123
1758099192,16cyclictest32-21ksoftirqd/211:35:122
1758099191,2cyclictest141rcu_preempt09:50:142
1758099190,0cyclictest0-21swapper/212:28:512
17577991916,2cyclictest25-21ksoftirqd/109:50:181
1757799190,0cyclictest0-21swapper/109:49:101
1757799190,0cyclictest0-21swapper/109:32:261
1757599190,19cyclictest0-21swapper/009:58:090
17585991818,0cyclictest0-21swapper/309:26:133
1758099182,16cyclictest0-21swapper/211:13:242
17580991817,1cyclictest0-21swapper/209:58:242
1758099180,18cyclictest0-21swapper/210:54:192
17577991816,2cyclictest25-21ksoftirqd/111:00:311
1757799180,2cyclictest0-21swapper/111:52:021
1757799180,18cyclictest0-21swapper/111:13:371
17575991813,4cyclictest13-21ksoftirqd/010:27:490
1757599181,1cyclictest0-21swapper/009:45:250
17585991716,1cyclictest39-21ksoftirqd/310:10:273
1758599170,2cyclictest0-21swapper/309:11:513
1758099178,6cyclictest32-21ksoftirqd/211:34:332
17580991713,1cyclictest141rcu_preempt10:23:172
17580991711,5cyclictest14550-21sh09:11:342
1757799170,2cyclictest14135-21ssh10:29:451
1757799170,1cyclictest141rcu_preempt09:27:321
1757799170,17cyclictest0-21swapper/111:20:291
1757799170,16cyclictest0-21swapper/111:15:301
1757599178,8cyclictest18930-21munin-run09:14:590
1757599175,5cyclictest225-21systemd-journal12:09:590
1758599165,4cyclictest5016-21apache211:03:503
1758599160,14cyclictest0-21swapper/309:46:443
1758099164,2cyclictest32-21ksoftirqd/212:09:192
17580991615,1cyclictest32-21ksoftirqd/210:20:012
17580991615,1cyclictest32-21ksoftirqd/210:20:002
17577991616,0cyclictest0-21swapper/111:39:141
17577991615,1cyclictest0-21swapper/109:17:391
1757799160,15cyclictest0-21swapper/110:12:381
1757599160,1cyclictest0-21swapper/010:34:250
1758599152,1cyclictest0-21swapper/312:33:133
1758599152,11cyclictest18157-21apache_processe11:30:123
17585991515,0cyclictest0-21swapper/310:39:103
1758099157,3cyclictest141rcu_preempt12:32:532
1758099152,0cyclictest141rcu_preempt11:27:092
17580991510,1cyclictest32-21ksoftirqd/210:29:032
1757799153,1cyclictest141rcu_preempt10:02:151
17577991514,1cyclictest11877-21kworker/u8:111:34:501
17577991513,1cyclictest0-21swapper/112:17:511
1757799150,14cyclictest0-21swapper/111:45:561
1757599150,9cyclictest10571-21cron09:10:000
1758599141,13cyclictest0-21swapper/310:53:543
17585991411,2cyclictest14484-21cron10:30:003
17585991411,2cyclictest12680-21sort08:05:163
17585991410,3cyclictest28205-21sendmail09:40:003
1758599140,14cyclictest0-21swapper/312:21:583
1758599140,12cyclictest7350-21grep07:55:143
1758099149,3cyclictest9426-21wc10:45:272
17580991413,1cyclictest32-21ksoftirqd/212:10:222
17580991412,2cyclictest17569-21cyclictest09:22:252
17580991412,1cyclictest32-21ksoftirqd/212:22:562
17580991412,1cyclictest32-21ksoftirqd/210:08:152
17580991411,2cyclictest16312-21cut10:30:192
1758099140,2cyclictest0-21swapper/208:25:222
1758099140,2cyclictest0-21swapper/208:25:212
1757799141,1cyclictest25-21ksoftirqd/112:30:121
1757799140,1cyclictest10578-21diskmemload12:08:491
17575991411,2cyclictest4839-21wc09:25:170
1757599141,11cyclictest3985-21runrttasks09:44:550
17575991410,3cyclictest9183-21grep11:05:150
1758599139,3cyclictest19979-21head08:20:163
1758599137,4cyclictest2376-21sh09:24:223
1758599131,2cyclictest0-21swapper/311:35:263
17585991310,2cyclictest27820-21grep07:30:133
17585991310,2cyclictest1987-21cron07:44:593
17585991310,2cyclictest17140-21tr11:10:143
17585991310,2cyclictest1630-21grep11:20:143
17585991310,2cyclictest12229-21tr09:30:133
17585991310,2cyclictest11909-21grep12:05:143
1758599130,12cyclictest0-21swapper/310:55:483
1758599130,12cyclictest0-21swapper/310:15:133
1758599130,12cyclictest0-21swapper/310:15:123
1758099139,3cyclictest660-21cut07:40:172
1758099139,3cyclictest29524-21ssh10:58:032
1758099139,3cyclictest26865-21sh11:54:592
1758099139,2cyclictest1070-21grep08:50:142
1758099137,4cyclictest32-21ksoftirqd/212:17:402
1758099134,6cyclictest32-21ksoftirqd/210:40:112
1758099134,2cyclictest225-21systemd-journal08:15:002
1758099134,2cyclictest141rcu_preempt08:40:222
1758099131,3cyclictest141rcu_preempt10:39:572
17580991312,1cyclictest32-21ksoftirqd/208:20:122
17580991311,2cyclictest32-21ksoftirqd/211:16:292
17580991310,2cyclictest31054-21grep12:35:252
17580991310,2cyclictest20721-21tr09:35:142
17580991310,2cyclictest17815-21basename07:10:132
1758099130,13cyclictest0-21swapper/211:01:072
1758099130,12cyclictest0-21swapper/211:57:582
1757799139,3cyclictest0-21swapper/108:35:131
1757799139,2cyclictest3540-21tr12:00:141
1757799138,4cyclictest25-21ksoftirqd/112:23:241
1757799138,4cyclictest11038-21ssh09:10:111
1757799131,4cyclictest0-21swapper/107:30:111
17577991312,1cyclictest0-21swapper/111:59:101
17577991311,1cyclictest25-21ksoftirqd/110:24:391
17577991310,3cyclictest38150irq/131-snd_hda10:36:441
17577991310,2cyclictest22732-21tr10:15:141
17577991310,2cyclictest22732-21tr10:15:131
17577991310,2cyclictest16124-21grep08:10:271
1757799130,1cyclictest0-21swapper/110:54:381
1757599139,3cyclictest22138-21grep12:30:180
17575991310,2cyclictest998-21tail11:00:190
17575991310,2cyclictest28523-21cat08:40:000
17575991310,2cyclictest28293-21sed07:30:140
17575991310,2cyclictest26006-21basename11:35:000
17575991310,2cyclictest24947-21taskset08:30:190
17575991310,2cyclictest21972-21munin-run10:15:000
17575991310,2cyclictest20856-21grep08:20:250
17575991310,2cyclictest16758-21sed10:50:170
17575991310,2cyclictest10653-21grep11:25:170
1758599129,2cyclictest9728-21grep11:05:193
1758599129,2cyclictest39-21ksoftirqd/309:45:013
1758599129,2cyclictest31035-21tr07:35:193
1758599129,2cyclictest29364-21wc12:35:133
1758599129,2cyclictest27428-21grep11:55:143
1758599129,2cyclictest27343-21cat07:29:593
1758599129,2cyclictest24187-21tr08:30:133
1758599129,2cyclictest23411-21cut07:20:153
1758599129,2cyclictest20186-21sort09:15:173
1758599129,2cyclictest15728-21head10:30:163
1758599129,2cyclictest11809-21grep11:25:263
1758599128,3cyclictest6332-21sed07:50:243
1758599128,3cyclictest5018-21apt10:05:013
1758599128,3cyclictest30342-21grep08:40:253
1758599128,2cyclictest5861-21tr10:05:163
1758099129,3cyclictest32-21ksoftirqd/211:20:212
1758099129,2cyclictest4512-21cut09:25:162
1758099129,2cyclictest32-21ksoftirqd/211:05:132
1758099125,7cyclictest32-21ksoftirqd/211:45:582
1758099120,1cyclictest0-21swapper/212:03:152
1757799129,2cyclictest9871-21cut11:05:191
1757799129,2cyclictest5654-21rm10:05:161
1757799129,2cyclictest32413-21grep10:40:151
1757799129,2cyclictest27278-21awk07:30:001
1757799129,2cyclictest15708-21cut10:30:141
1757799125,3cyclictest225-21systemd-journal11:44:591
1757799124,4cyclictest225-21systemd-journal08:10:011
17577991210,1cyclictest0-21swapper/112:27:591
1757599129,2cyclictest9267-21grep08:00:000
1757599129,2cyclictest7789-21grep10:45:140
1757599129,2cyclictest5279-21grep12:00:260
1757599129,2cyclictest32117-21taskset12:16:540
1757599129,2cyclictest2712-21tr11:40:160
1757599129,2cyclictest25446-21grep07:25:130
1757599129,2cyclictest24930-21wc11:15:010
1757599129,2cyclictest22533-21cat07:20:000
1757599129,2cyclictest10997-21ssh11:45:130
1757599128,2cyclictest19380-21cat09:15:110
1757599121,3cyclictest0-21swapper/010:02:510
17575991212,0cyclictest0-21swapper/010:23:550
1757599120,1cyclictest0-21swapper/009:33:400
1757599120,11cyclictest2967-21grep07:45:160
1758599119,1cyclictest7736-21ssh10:45:153
1758599119,1cyclictest281152sleep308:36:053
1758599118,2cyclictest9233-21grep09:05:163
1758599118,2cyclictest5572-21ssh12:00:363
1758599118,2cyclictest4045-21sort08:55:153
1758599118,2cyclictest32609-21ssh10:20:263
1758599118,2cyclictest28511-21ssh11:16:213
1758599118,2cyclictest23069-21ssh12:11:213
1758599118,2cyclictest21779-21expr07:15:243
1758599118,2cyclictest21302-21ssh09:55:133
1758599118,2cyclictest19160-21cut11:50:133
1758599118,2cyclictest12732-21cut12:25:143
1758599110,10cyclictest31812-21head08:45:143
1758099118,2cyclictest9523-21wc09:05:202
1758099118,2cyclictest3643-21tr08:55:132
1758099118,2cyclictest31807-21ssh09:41:172
1758099118,2cyclictest28726-21pluginstate07:30:192
1758099118,2cyclictest26276-21cut07:25:182
1758099118,2cyclictest24795-21taskset07:23:482
1758099118,2cyclictest22500-21sed07:19:592
1758099111,9cyclictest32-21ksoftirqd/207:35:132
1758099111,1cyclictest141rcu_preempt07:45:252
17580991110,1cyclictest32-21ksoftirqd/209:45:012
1757799119,1cyclictest6707-21basename09:00:171
1757799119,1cyclictest603-21head07:40:161
1757799119,1cyclictest28568-21grep09:20:151
1757799118,2cyclictest4911-21grep07:50:131
1757799118,2cyclictest31824-21head08:45:141
1757799118,2cyclictest26210-21head10:55:261
1757799118,2cyclictest23999-21ssh09:56:021
1757799118,2cyclictest2377-21basename07:45:141
1757799118,2cyclictest12252-21ssh10:47:331
1757799118,2cyclictest11285-21chrt08:01:281
1757799116,2cyclictest18804-21taskset08:18:401
1757799115,4cyclictest14405-21ssh11:27:251
1757799115,2cyclictest225-21systemd-journal09:00:001
17577991111,0cyclictest0-21swapper/109:36:351
1757599119,1cyclictest28502-21uniq09:20:160
1757599118,2cyclictest8118-21cat09:04:590
1757599118,2cyclictest758-21cron08:50:000
1757599118,2cyclictest4291-21cat08:55:170
1757599118,2cyclictest30328-21ssh12:35:180
1757599118,2cyclictest2849-21taskset08:51:130
1757599118,2cyclictest2807-21ssh10:41:400
1757599118,2cyclictest27569-21ssh10:56:290
1757599118,2cyclictest26563-21ssh11:54:460
1757599118,2cyclictest26124-21tail11:15:150
1757599118,2cyclictest26124-21sh10:36:050
1757599118,2cyclictest24524-21ssh10:15:350
1757599118,2cyclictest24524-21ssh10:15:340
1757599118,2cyclictest2391-21basename11:20:180
1757599118,2cyclictest23628-21ssh09:36:230
1757599118,2cyclictest22888-21gzip08:25:220
1757599118,2cyclictest22888-21gzip08:25:210
1757599118,2cyclictest17282-21df08:15:130
1757599118,2cyclictest15169-21ssh12:25:530
1757599118,2cyclictest12883-21ssh09:50:120
1757599117,3cyclictest10696-21grep08:00:210
1757599113,7cyclictest30840-21munin-run08:45:010
1757599113,7cyclictest28561-21missed_timers11:55:180
1758599108,1cyclictest7207-21run-parts09:00:233
1758599108,1cyclictest23431-21chrt08:26:513
1758599108,1cyclictest23431-21chrt08:26:513
1758599107,2cyclictest18954-21ntpq07:10:203
1758599107,2cyclictest17011-21cat08:15:133
1758599107,2cyclictest11427-21taskset08:03:183
1758099108,1cyclictest6710-21chrt07:53:562
1758099108,1cyclictest5883-21seq09:00:142
1758099108,1cyclictest32227-21grep08:45:182
1758099108,1cyclictest25401-21cat08:30:242
1758099108,1cyclictest18682-21chrt08:17:052
1758099108,1cyclictest12392-21expr08:05:142
1758099107,2cyclictest10504-21cat08:00:182
1758099100,9cyclictest27194-21cat08:35:172
1757799108,1cyclictest90442sleep107:58:281
1757799108,1cyclictest8575-21awk09:05:141
1757799108,1cyclictest31689-21taskset07:35:531
1757799108,1cyclictest21893-21ls08:25:151
1757799108,1cyclictest21893-21ls08:25:141
1757799107,2cyclictest2713-21awk08:50:261
1757799107,2cyclictest23500-21grep07:20:151
1757799107,2cyclictest18047-21cut07:10:141
1757799103,4cyclictest23956-21apt-get08:30:121
1757799103,4cyclictest141rcu_preempt07:15:131
1757799103,3cyclictest225-21systemd-journal08:40:011
1757799100,9cyclictest20580-21basename08:20:221
1757599108,1cyclictest5548-21ssh10:05:140
1757599108,1cyclictest23078-21expr07:20:130
1757599108,1cyclictest21078-21ssh12:10:190
1757599107,2cyclictest5083-21expr12:20:170
1757599107,2cyclictest4876-21tr07:50:140
1757599107,2cyclictest32658-21expr07:40:140
1757599107,2cyclictest19914-21idleruntime-cro07:14:590
1757599100,9cyclictest11968-21cat08:05:000
175859997,1cyclictest2924-21seq08:52:133
175859997,1cyclictest2716-21expr07:45:143
175809997,1cyclictest7064-21seq07:55:112
175759996,2cyclictest16592-21cat08:15:010
175759995,3cyclictest30671-21head07:35:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional