You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-18 - 16:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Sep 18, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1052729671,9sleep30-21swapper/307:06:133
848528773,9sleep00-21swapper/007:05:020
1078328665,6sleep10-21swapper/107:09:331
1054628573,8sleep20-21swapper/207:06:282
31742640,0sleep30-21swapper/312:23:413
1091399291,7cyclictest0-21swapper/210:45:012
1090699270,27cyclictest0-21swapper/012:38:060
1090699240,23cyclictest0-21swapper/009:56:200
1091399230,1cyclictest141rcu_preempt11:50:592
1091399220,1cyclictest0-21swapper/210:19:462
1091099211,1cyclictest141rcu_preempt10:05:401
1091099211,17cyclictest3702-21diskmemload10:31:221
1091399201,18cyclictest0-21swapper/207:45:132
1091399200,20cyclictest0-21swapper/209:51:012
10910992018,2cyclictest94-21kswapd009:35:141
1091099200,17cyclictest0-21swapper/112:20:101
1091699190,18cyclictest15529-21cat09:55:163
1091399194,1cyclictest0-21swapper/212:13:042
1091399194,1cyclictest0-21swapper/212:13:032
1091099191,2cyclictest25-21ksoftirqd/112:26:311
1090699190,19cyclictest0-21swapper/009:19:530
10913991815,2cyclictest32-21ksoftirqd/210:34:052
1091099181,1cyclictest11742-21ssh09:33:571
1091099180,0cyclictest0-21swapper/110:19:461
1090699187,6cyclictest23007-21perf07:35:000
1091699170,16cyclictest0-21swapper/308:35:133
10913991716,1cyclictest0-21swapper/210:03:442
10913991713,2cyclictest16440-21cstates07:20:142
10913991712,1cyclictest32-21ksoftirqd/209:45:342
1091099176,7cyclictest3702-21diskmemload10:45:011
10916991613,2cyclictest39-21ksoftirqd/312:10:193
10916991613,2cyclictest39-21ksoftirqd/312:10:183
1091699160,2cyclictest0-21swapper/310:50:123
10913991612,3cyclictest32-21ksoftirqd/209:45:002
10913991612,1cyclictest32-21ksoftirqd/210:38:502
1091399160,15cyclictest0-21swapper/211:25:332
10910991614,1cyclictest25-21ksoftirqd/109:29:171
10910991613,1cyclictest241rcuc/111:00:151
1091099160,16cyclictest0-21swapper/111:20:231
1091099160,16cyclictest0-21swapper/110:48:351
1091099160,15cyclictest15895-21/usr/sbin/munin09:55:201
1091099160,15cyclictest0-21swapper/107:30:131
1091099160,14cyclictest3702-21diskmemload11:48:341
1090699164,6cyclictest2404-21munin-run11:24:590
10906991616,0cyclictest0-21swapper/009:26:500
10906991611,4cyclictest16028-21munin-plugin-st10:35:010
1090699160,1cyclictest0-21swapper/011:11:350
1090699160,15cyclictest0-21swapper/012:20:030
1091699159,4cyclictest28863-21kernelversion11:40:193
10916991514,1cyclictest39-21ksoftirqd/311:30:193
1091399157,2cyclictest1142-21runrttasks10:06:172
10913991513,2cyclictest32-21ksoftirqd/210:27:372
10913991511,2cyclictest13116-21df09:15:132
10910991510,4cyclictest38150irq/131-snd_hda11:30:251
1091099150,1cyclictest141rcu_preempt11:35:271
1091099150,1cyclictest0-21swapper/111:50:401
1090699151,14cyclictest0-21swapper/010:14:150
53452140,4sleep10-21swapper/110:28:101
1091699141,11cyclictest0-21swapper/309:35:163
10916991410,3cyclictest5600-21tr12:25:133
10916991410,3cyclictest27908-21cut08:50:183
10916991410,2cyclictest26499-21wc10:40:233
1091699140,12cyclictest28155-21wget07:45:133
1091399149,3cyclictest27553-21head08:50:152
1091399149,1cyclictest32-21ksoftirqd/210:48:252
10913991412,1cyclictest32-21ksoftirqd/211:22:452
10913991411,2cyclictest14858-21ntpq07:15:212
1091099147,6cyclictest17609-21cat07:20:251
10910991410,3cyclictest2868-21grep08:00:141
10910991410,2cyclictest19459-21head11:15:171
1091099140,12cyclictest0-21swapper/110:55:291
1090699145,4cyclictest225-21systemd-journal11:05:000
1090699142,6cyclictest32752-21munin-run10:45:000
10906991411,2cyclictest21371-21grep11:35:270
1090699141,11cyclictest0-21swapper/009:36:260
10906991410,3cyclictest2162-21cut11:05:130
10906991410,2cyclictest19840-21dump-pmu-power11:55:000
1090699140,8cyclictest24407-21awk10:40:010
1090699140,8cyclictest0-21swapper/011:40:130
1090699140,3cyclictest0-21swapper/011:16:370
1091699139,3cyclictest30016-21grep09:25:163
1091699139,3cyclictest15885-21grep10:15:143
1091699135,1cyclictest3015-21irqstats11:05:193
1091699133,3cyclictest225-21systemd-journal11:25:003
10916991310,2cyclictest8804-21ps08:10:213
10916991310,2cyclictest8053-21grep10:10:173
10916991310,2cyclictest5802-21grep08:05:183
10916991310,2cyclictest5729-21sh09:50:003
10916991310,2cyclictest411-21df10:25:143
10916991310,2cyclictest3504-21grep08:00:163
10916991310,2cyclictest31981-21grep10:05:163
10916991310,2cyclictest26532-21cron08:50:003
10916991310,2cyclictest26532-21cron08:50:003
10916991310,2cyclictest25681-21cut07:40:123
10916991310,2cyclictest23637-21sort10:00:163
10916991310,2cyclictest11613-21grep08:15:263
1091699130,1cyclictest141rcu_preempt12:19:413
1091699130,13cyclictest0-21swapper/310:34:053
1091399139,3cyclictest28925-21cron12:20:012
1091399137,5cyclictest13079-21apt09:34:592
1091399134,3cyclictest141rcu_preempt09:39:322
10913991312,1cyclictest0-21swapper/210:50:242
10913991310,2cyclictest3207-21grep09:05:272
10913991310,2cyclictest24025-21tr10:20:132
10913991310,2cyclictest20666-21tr11:55:152
1091399130,2cyclictest0-21swapper/209:55:292
1091399130,1cyclictest141rcu_preempt10:59:562
1091099139,3cyclictest22216-21grep11:55:261
1091099139,2cyclictest26906-21grep08:50:121
1091099137,4cyclictest28138-21runrttasks12:19:231
1091099136,6cyclictest25-21ksoftirqd/109:51:301
10910991311,2cyclictest0-21swapper/110:50:511
10910991310,2cyclictest23066-21head08:40:201
10910991310,2cyclictest11062-21tr08:15:191
10910991310,2cyclictest0-21swapper/111:06:251
1091099130,1cyclictest0-21swapper/108:55:151
1091099130,13cyclictest0-21swapper/109:42:331
1091099130,13cyclictest0-21swapper/109:20:441
1090699139,3cyclictest889-21cut10:45:120
1090699139,2cyclictest6912-21grep09:30:240
1090699139,2cyclictest18682-21grep08:30:230
1090699139,1cyclictest1142-21runrttasks07:35:140
10906991310,2cyclictest6298-21wc12:05:250
10906991310,2cyclictest1294-21ssh09:46:310
10906991310,2cyclictest10657-21head08:15:160
1090699130,12cyclictest22711-21sed09:20:250
1091699129,2cyclictest3702-21diskmemload09:18:263
1091699129,2cyclictest29472-21head12:00:163
1091699129,2cyclictest25910-21grep10:20:243
1091699129,2cyclictest2199-21grep10:45:203
1091699129,2cyclictest20627-21grep11:15:263
1091699129,2cyclictest13094-21sed08:20:173
1091699129,2cyclictest11924-21head07:10:163
1091699128,3cyclictest17416-21sed10:35:173
10916991210,1cyclictest24028-21sort08:45:003
1091399129,2cyclictest7544-21tr10:10:142
1091399129,2cyclictest6156-21grep09:10:272
1091399129,2cyclictest5045-21open_inodes11:45:202
1091399129,2cyclictest4865-21tr12:05:142
1091399129,2cyclictest4134-21ssh12:24:252
1091399129,2cyclictest20912-21cut07:30:132
1091399129,2cyclictest20737-21tr11:35:212
1091399129,2cyclictest14012-21tr12:30:142
1091399127,4cyclictest32-21ksoftirqd/212:38:482
1091399126,2cyclictest141rcu_preempt11:01:122
1091399124,7cyclictest0-21swapper/208:15:122
10913991210,2cyclictest32-21ksoftirqd/211:30:182
10913991210,1cyclictest311rcuc/209:20:262
10913991210,1cyclictest12065-21uname07:10:182
1091399120,12cyclictest0-21swapper/211:10:552
1091099129,2cyclictest31726-21cron10:25:001
1091099129,2cyclictest29030-21grep11:40:201
1091099129,2cyclictest24820-21tail10:00:251
1091099129,2cyclictest24686-21grep08:45:151
1091099129,2cyclictest24686-21grep08:45:151
1091099129,2cyclictest23274-21ntpq12:35:201
1091099129,2cyclictest21145-21grep08:35:261
1091099129,2cyclictest15415-21tail08:25:141
1091099129,2cyclictest13951-21grep08:20:261
1091099127,2cyclictest12128-21ssh12:09:491
1091099124,5cyclictest1671-21aten2_r7power_p09:05:141
1091099124,1cyclictest25-21ksoftirqd/109:20:011
10910991211,1cyclictest0-21swapper/110:13:201
1090699129,2cyclictest7157-21ssh12:25:240
1090699129,2cyclictest5080-21sed11:45:200
1090699129,2cyclictest2297-21grep09:05:170
1090699129,2cyclictest22084-21cut08:40:130
1090699129,2cyclictest21927-21grep12:15:160
1090699129,2cyclictest18366-21sh10:55:160
1090699129,2cyclictest16914-21sed07:20:170
1090699129,2cyclictest13068-21tail11:30:250
1090699128,3cyclictest26701-21wc07:40:180
1090699125,6cyclictest15888-21sendmail-msp07:20:010
1090699120,7cyclictest3702-21diskmemload12:04:590
1090699120,11cyclictest3948-21uname11:25:180
1091699118,2cyclictest7830-21ssh09:30:543
1091699118,2cyclictest7127-21tail09:50:183
1091699118,2cyclictest6551-21ssh11:26:443
1091699118,2cyclictest6397-21ssh09:10:353
1091699118,2cyclictest502-21fgrep09:00:213
1091699118,2cyclictest4526-21sed11:45:173
1091699118,2cyclictest4348-21ssh12:05:033
1091699118,2cyclictest31329-21cut07:50:183
1091699118,2cyclictest28461-21ssh11:00:473
1091699118,2cyclictest21906-21wc07:30:193
1091699118,2cyclictest21884-21ssh12:35:103
1091699118,2cyclictest20663-21ssh10:56:133
1091699118,2cyclictest20251-21ssh11:55:093
1091699118,2cyclictest19406-21ssh11:35:113
1091699118,2cyclictest16702-21chrt07:20:143
1091699118,2cyclictest16680-21ssh11:52:333
1091699118,2cyclictest15065-21cut12:30:203
1091699118,2cyclictest14747-21cut08:25:103
1091699118,2cyclictest10421-21aten2_r7power_e11:10:143
1091699115,4cyclictest18722-21tr07:25:143
1091399119,1cyclictest7798-21expr08:10:142
1091399119,1cyclictest13006-21grep08:20:172
1091399118,2cyclictest961-21taskset09:00:432
1091399118,2cyclictest7081-21cat12:25:242
1091399118,2cyclictest32762-21ssh12:02:052
1091399118,2cyclictest31134-21rm09:25:262
1091399118,2cyclictest30923-21chrt08:55:312
1091399118,2cyclictest27607-21ssh11:40:052
1091399118,2cyclictest26520-21if_err_enp0s31f07:40:162
1091399118,2cyclictest22018-21ssh11:16:302
1091399118,2cyclictest18641-21grep07:25:142
1091399118,2cyclictest1781-21taskset11:05:032
1091099119,1cyclictest1318-21latency_hist09:05:011
1091099118,2cyclictest8263-21head08:10:161
1091099118,2cyclictest5388-21ssh11:25:501
1091099118,2cyclictest417-21aten2_r7power_e07:55:131
1091099118,2cyclictest31877-21rm12:01:231
1091099118,2cyclictest31039-21grep09:45:161
1091099118,2cyclictest30613-21tr07:50:131
1091099118,2cyclictest241rcuc/110:35:171
1091099118,2cyclictest19533-21expr07:25:191
1091099118,2cyclictest13956-21grep07:15:151
1091099118,2cyclictest13298-21http07:15:001
1091099118,2cyclictest10223-21ssh11:10:091
1091099110,10cyclictest19607-21ssh12:33:331
1090699118,2cyclictest5183-21apt-get08:05:140
1090699118,2cyclictest4429-21ssh10:27:250
1090699118,2cyclictest31006-21ssh10:05:030
1090699118,2cyclictest30060-21taskset07:49:190
1090699118,2cyclictest27127-21ssh10:02:100
1090699118,2cyclictest25418-21ssh09:41:370
1090699118,2cyclictest24732-21sh10:20:160
1090699118,2cyclictest20049-21cut08:35:160
1090699118,2cyclictest17723-21ssh10:15:330
1090699118,2cyclictest13666-21grep12:10:170
1090699118,2cyclictest13666-21grep12:10:170
1090699118,2cyclictest13583-21ps08:20:220
1090699118,2cyclictest11332-21ssh09:14:260
1090699118,2cyclictest1115-21tail07:55:160
1090699117,3cyclictest29934-21grep08:55:190
1090699116,4cyclictest26134-21sed08:45:270
1090699116,4cyclictest26134-21sed08:45:260
1090699116,4cyclictest14854-21ls12:30:190
1090699114,3cyclictest225-21systemd-journal09:53:020
1090699111,2cyclictest10005-21diskstats10:50:160
1091699108,1cyclictest2950-21mailstats09:05:243
1091699108,1cyclictest23938-21cut07:35:163
1091699108,1cyclictest22438-21ssh09:40:123
1091699108,1cyclictest21376-21expr09:20:153
1091699108,1cyclictest15490-21chrt07:15:393
1091699107,2cyclictest29138-21taskset08:55:053
1091699107,2cyclictest17057-21basename08:30:103
1091699105,2cyclictest225-21systemd-journal07:55:593
1091399108,1cyclictest24178-21cut07:35:172
1091399108,1cyclictest235852sleep208:40:272
1091399108,1cyclictest191192sleep208:34:192
1091399107,2cyclictest32-21ksoftirqd/208:45:142
1091399107,2cyclictest32-21ksoftirqd/208:45:142
1091399107,2cyclictest31168-21head07:50:162
1091399107,2cyclictest3028-21tr08:00:132
1091399107,2cyclictest15637-21sed08:25:162
1091099108,1cyclictest4193-21chrt09:10:121
1091099108,1cyclictest28129-21awk07:45:141
1091099108,1cyclictest12630-21ssh12:10:021
1091099108,1cyclictest12630-21ssh12:10:011
1091099107,2cyclictest25093-21chrt07:36:431
1091099107,2cyclictest17366-21tr08:30:121
1091099105,1cyclictest141rcu_preempt08:05:161
1090699108,1cyclictest48052sleep008:03:480
1090699108,1cyclictest30687-21wc07:50:130
1090699108,1cyclictest21101-21ssh11:55:170
1090699108,1cyclictest13347-21cat07:15:000
1090699107,2cyclictest9421-21chrt08:12:050
1090699107,2cyclictest16242-21wc08:25:220
1090699104,2cyclictest452-21systemd-logind09:03:480
1090699100,1cyclictest0-21swapper/008:50:120
109139997,1cyclictest733-21expr07:55:142
109139997,1cyclictest5486-21cat08:05:162
109139997,1cyclictest19452-21apt08:35:132
109109996,2cyclictest25620-21cat07:40:101
109069997,1cyclictest20283-21chrt07:25:560
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional