You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-15 - 23:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Apr 15, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1393729985,9sleep00-21swapper/007:07:270
1376228975,9sleep20-21swapper/207:05:272
1411028158,6sleep10-21swapper/107:09:401
1385828067,9sleep30-21swapper/307:06:243
255112590,0sleep00-21swapper/011:33:440
96942530,0sleep339-21ksoftirqd/310:45:353
145572480,0sleep314556-21ssh11:08:153
14226992727,0cyclictest0-21swapper/109:44:231
1422699260,3cyclictest0-21swapper/112:17:581
14231992523,2cyclictest25253-21kworker/3:210:00:023
14231992516,8cyclictest25253-21kworker/3:209:22:223
1422999250,23cyclictest0-21swapper/211:48:382
14231992422,2cyclictest25253-21kworker/3:209:26:023
14231992421,2cyclictest25253-21kworker/3:209:52:323
1422999240,23cyclictest0-21swapper/210:59:412
14231992321,2cyclictest25253-21kworker/3:209:12:023
14231992320,2cyclictest19578-21/usr/sbin/munin09:55:233
14231992315,8cyclictest4612-21kworker/3:111:10:433
14231992315,8cyclictest26262-21kworker/3:212:39:123
1422999230,0cyclictest0-21swapper/212:17:002
14226992321,2cyclictest0-21swapper/112:39:431
14231992214,8cyclictest4612-21kworker/3:111:33:223
14226992220,2cyclictest25-21ksoftirqd/109:55:531
200712210,1sleep10-21swapper/110:33:181
14231992119,2cyclictest4612-21kworker/3:111:23:223
14231992119,2cyclictest3524-21kworker/3:011:01:223
14231992119,2cyclictest3524-21kworker/3:010:37:233
1423199210,20cyclictest3524-21kworker/3:010:50:323
1423199210,20cyclictest3524-21kworker/3:010:32:423
1422999219,5cyclictest0-21swapper/211:20:002
1422699210,21cyclictest0-21swapper/110:26:191
1422699210,21cyclictest0-21swapper/110:26:181
14231992017,2cyclictest25253-21kworker/3:209:46:323
1423199200,19cyclictest4612-21kworker/3:111:56:233
14226992018,1cyclictest0-21swapper/107:40:121
14231991917,2cyclictest3524-21kworker/3:010:27:523
14231991917,2cyclictest3524-21kworker/3:010:27:523
1422999190,19cyclictest0-21swapper/211:34:132
1422999190,13cyclictest0-21swapper/210:19:002
1422699190,18cyclictest0-21swapper/111:54:281
14231991816,2cyclictest25253-21kworker/3:209:39:223
14231991815,2cyclictest3524-21kworker/3:010:41:423
14226991817,1cyclictest0-21swapper/111:40:381
1422699180,18cyclictest0-21swapper/111:55:251
1422699180,17cyclictest0-21swapper/109:14:591
1422699180,14cyclictest7026-21diskmemload12:10:141
14231991715,2cyclictest4612-21kworker/3:112:20:023
14231991715,2cyclictest4612-21kworker/3:112:00:223
14231991715,2cyclictest3524-21kworker/3:010:16:233
14231991715,2cyclictest3524-21kworker/3:010:10:223
14231991715,2cyclictest26262-21kworker/3:212:32:233
14231991715,2cyclictest25253-21kworker/3:209:40:033
14231991715,2cyclictest25253-21kworker/3:209:15:433
14231991715,1cyclictest3800-21apt-get11:40:133
1423199171,4cyclictest56550irq/124-enp0s3112:06:253
1423199171,4cyclictest56550irq/124-enp0s3112:06:243
14229991715,2cyclictest32-21ksoftirqd/209:26:352
14229991713,3cyclictest311rcuc/210:05:252
1422999170,17cyclictest0-21swapper/212:09:082
1422999170,17cyclictest0-21swapper/212:09:072
1422699170,3cyclictest0-21swapper/110:05:421
1422299171,11cyclictest1142-21runrttasks10:34:090
1422299170,0cyclictest0-21swapper/007:55:350
1422999169,6cyclictest475-21in:imuxsock10:40:112
14229991614,2cyclictest32-21ksoftirqd/212:35:402
1422999160,2cyclictest0-21swapper/212:24:112
1422999160,14cyclictest0-21swapper/211:29:012
1422999160,14cyclictest0-21swapper/211:08:012
1422699166,8cyclictest2248-21ntpq09:25:191
14226991611,4cyclictest0-21swapper/109:50:371
1422699160,3cyclictest0-21swapper/109:24:201
1422699160,1cyclictest0-21swapper/111:38:071
1422699160,13cyclictest0-21swapper/111:04:031
14222991613,2cyclictest23917-21cron10:55:010
1423199150,1cyclictest225-21systemd-journal10:08:513
1422999152,3cyclictest141rcu_preempt09:30:132
14229991510,2cyclictest24923-21sh08:40:002
1422699152,7cyclictest1142-21runrttasks11:06:061
14226991511,1cyclictest24683-21seq08:37:011
1422299154,8cyclictest25937-21taskset09:58:450
1422299152,10cyclictest15178-21taskset09:33:230
1422299151,7cyclictest0-21swapper/012:30:110
1422299150,1cyclictest0-21swapper/008:19:190
14229991410,1cyclictest32-21ksoftirqd/209:56:392
1422999140,13cyclictest0-21swapper/211:35:152
1422999140,13cyclictest0-21swapper/209:23:252
1422699145,8cyclictest20623-21cat10:15:001
1422699142,3cyclictest225-21systemd-journal12:10:011
1422699142,3cyclictest225-21systemd-journal12:10:001
14226991413,1cyclictest38150irq/131-snd_hda11:45:011
1422699140,1cyclictest0-21swapper/110:24:201
1422699140,13cyclictest0-21swapper/111:22:181
1422299143,9cyclictest15754-21apt-get09:15:000
14222991413,1cyclictest0-21swapper/009:26:470
1423199139,4cyclictest39-21ksoftirqd/310:59:193
1423199139,3cyclictest31351-21tail12:15:183
14231991310,2cyclictest845-21cron07:50:003
14231991310,2cyclictest27062-21grep11:15:183
1423199130,12cyclictest4218-21grep09:00:253
1422999139,3cyclictest9732-21tr08:05:212
14229991310,3cyclictest23786-21apache211:43:112
14229991310,2cyclictest32-21ksoftirqd/212:12:382
14229991310,2cyclictest22499-21ssh11:50:262
1422999130,13cyclictest0-21swapper/209:18:292
14226991312,1cyclictest25-21ksoftirqd/109:01:101
14226991311,2cyclictest25-21ksoftirqd/109:32:471
14226991310,2cyclictest7575-21grep08:00:281
14226991310,2cyclictest7382-21sed12:20:171
14226991310,2cyclictest7006-21sh09:10:011
14226991310,2cyclictest19349-21grep11:10:261
1422699130,11cyclictest7026-21diskmemload10:47:241
1422299138,2cyclictest16099-21/usr/sbin/munin12:25:190
1422299133,7cyclictest13-21ksoftirqd/011:40:130
1422299133,5cyclictest21573-21/usr/sbin/munin07:25:210
1422299133,3cyclictest0-21swapper/011:16:430
1422299132,8cyclictest21242-21apache_volume10:15:100
14222991311,2cyclictest0-21swapper/010:43:530
14222991310,2cyclictest30838-21grep12:15:140
14222991310,2cyclictest17265-21grep08:20:250
14222991310,2cyclictest13625-21grep11:45:220
1423199129,2cyclictest7597-21sed08:00:273
1423199129,2cyclictest5666-21head09:05:183
1423199129,2cyclictest5417-21grep10:25:003
1423199129,2cyclictest1777-21sed08:55:263
1423199129,2cyclictest15558-21strings07:10:203
1423199129,2cyclictest13311-21grep08:15:143
1423199129,2cyclictest10191-21tr09:30:163
1423199128,3cyclictest29807-21wc08:50:003
1423199128,3cyclictest26246-21ssh11:53:103
1423199128,3cyclictest19780-21grep07:20:143
1423199128,0cyclictest0-21swapper/307:35:133
1423199127,2cyclictest225-21systemd-journal12:15:003
1423199121,8cyclictest0-21swapper/312:27:133
14231991210,2cyclictest39-21ksoftirqd/307:52:213
1422999129,2cyclictest6041-21grep08:00:142
1422999129,2cyclictest3069-21df09:00:142
1422999129,2cyclictest29714-21basename08:50:002
1422999129,2cyclictest25143-21head07:30:172
1422999129,2cyclictest11992-21grep09:50:162
1422999128,3cyclictest9696-21ssh09:49:262
1422999128,3cyclictest6398-21ssh11:22:182
1422999128,3cyclictest32602-21sort07:45:272
1422999128,3cyclictest27282-21chrt08:44:372
1422999128,3cyclictest25772-21ssh10:36:102
1422999128,3cyclictest21902-21ssh12:29:152
1422999128,3cyclictest15413-21sh10:11:022
1422999127,5cyclictest7026-21diskmemload10:34:502
1422999126,5cyclictest21805-21sort07:25:132
1422999126,2cyclictest29157-21apt-get11:55:122
1422999125,6cyclictest7026-21diskmemload10:00:382
14229991210,1cyclictest32-21ksoftirqd/210:45:012
1422999120,11cyclictest9034-21sh12:01:502
1422999120,11cyclictest30861-21cat10:20:162
1422999120,11cyclictest28505-21ssh09:40:262
1422999120,11cyclictest24670-21tr12:30:192
1422999120,11cyclictest20543-21grep07:20:202
1422999120,11cyclictest19235-21ssh09:35:182
1422699129,2cyclictest28892-21cut10:00:151
1422699129,2cyclictest28514-21cron07:40:001
1422699129,2cyclictest21486-21cat07:25:001
1422699129,2cyclictest18434-21head09:15:261
1422699129,2cyclictest15640-21sort12:25:161
1422699128,3cyclictest25685-21sed07:30:241
1422699128,3cyclictest25510-21cat10:55:181
1422699128,3cyclictest18618-21cron11:30:011
1422699128,3cyclictest15525-21cat08:20:001
1422699124,4cyclictest225-21systemd-journal08:09:551
1422699120,1cyclictest0-21swapper/112:31:201
1422299129,2cyclictest6121-21grep09:05:210
1422299129,2cyclictest28947-21grep10:00:160
1422299129,2cyclictest2753-21grep11:20:160
1422299129,2cyclictest16518-21tr09:15:130
1422299129,2cyclictest13821-21wget12:05:130
1422299129,2cyclictest13821-21wget12:05:130
1422299129,2cyclictest13467-21cut10:10:170
1422299129,1cyclictest28683-21rm11:35:180
1422299127,4cyclictest2269-21ssh11:58:090
1422299127,3cyclictest13-21ksoftirqd/010:38:510
1422299126,3cyclictest6054-21grep10:05:240
1422299126,2cyclictest225-21systemd-journal07:48:230
1422299125,4cyclictest15460-21idleruntime-cro10:50:000
14222991210,1cyclictest25279-21grep09:20:140
14222991210,1cyclictest24912-21taskset12:11:040
14222991210,1cyclictest13-21ksoftirqd/012:20:370
1422299120,4cyclictest0-21swapper/011:05:130
1423199118,2cyclictest3815-21cat07:55:173
1423199118,2cyclictest30730-21sh11:36:153
1423199118,2cyclictest29149-21expr07:40:143
1423199118,2cyclictest23530-21chrt07:27:483
1423199118,2cyclictest20123-21ls11:50:013
1423199118,2cyclictest19804-21chrt08:26:023
1423199118,2cyclictest17608-21taskset08:23:163
1423199117,2cyclictest30079-21basename08:50:123
1422999119,2cyclictest8016-21tr09:10:142
1422999119,1cyclictest18660-21tr11:10:212
1422999118,2cyclictest8217-21ssh10:25:422
1422999118,2cyclictest8217-21ssh10:25:412
1422999118,2cyclictest5047-21egrep09:05:152
1422999118,2cyclictest3372sleep211:00:092
1422999118,2cyclictest30872-21taskset07:44:362
1422999118,2cyclictest19662-21grep08:25:262
1422999118,2cyclictest18668-21chrt07:16:372
1422999118,2cyclictest16256-21tr10:50:132
1422999118,2cyclictest1450-21wc08:55:232
1422999118,2cyclictest10923-21grep08:10:142
1422999117,3cyclictest26367-21date07:35:012
1422999111,1cyclictest15379-21basename08:20:002
1422999110,10cyclictest55932sleep207:59:522
1422999110,10cyclictest16511-21chrt07:14:292
1422999110,10cyclictest1176-21basename07:50:132
1422699118,2cyclictest849-21cron07:49:591
1422699118,2cyclictest32073-21turbostat.cron08:55:001
1422699118,2cyclictest31729-21sort10:40:141
1422699118,2cyclictest2831-21sed07:50:251
1422699118,2cyclictest28155-21rm11:15:391
1422699118,2cyclictest27901-21ssh10:37:481
1422699118,2cyclictest27271-21chrt08:44:271
1422699118,2cyclictest2447-21ssh09:45:091
1422699118,2cyclictest23713-21latency_hist07:30:001
1422699118,2cyclictest21904-21cat08:30:261
1422699118,2cyclictest21015-21ssh10:15:011
1422699118,2cyclictest20530-21grep11:30:211
1422699118,2cyclictest20500-21ssh09:35:481
1422699118,2cyclictest16627-21ssh10:50:161
1422699110,10cyclictest0-21swapper/112:00:201
1422299119,2cyclictest6477-21ssh12:00:200
1422299119,1cyclictest31605-21grep08:50:240
1422299119,1cyclictest30164-21expr10:20:130
1422299119,1cyclictest27179-21head07:35:170
1422299119,1cyclictest2629-21tr09:00:130
1422299119,1cyclictest2628-21ssh09:45:140
1422299119,1cyclictest24040-21ssh09:38:300
1422299119,1cyclictest131762sleep010:29:340
1422299119,1cyclictest131762sleep010:29:330
1422299118,3cyclictest13-21ksoftirqd/012:35:450
1422299118,2cyclictest9507-21cat08:05:190
1422299118,2cyclictest5938-21wget08:00:150
1422299118,2cyclictest27703-21basename08:45:140
1422299118,2cyclictest26300-21sh09:40:030
1422299118,2cyclictest25572-21cut11:15:000
1422299118,2cyclictest24829-21expr10:55:130
1422299118,2cyclictest20426-21ls07:20:200
1422299118,2cyclictest15110-21sh11:27:190
1422299114,3cyclictest0-21swapper/008:25:130
1423199108,1cyclictest9422-21cut08:05:183
1423199108,1cyclictest18666-21taskset07:16:353
1423199108,1cyclictest121502sleep311:25:223
1423199107,2cyclictest25146-21cat08:40:003
1423199107,2cyclictest24445-21cat07:30:133
1423199102,3cyclictest225-21systemd-journal08:42:043
1423199100,9cyclictest21462-21cat08:30:193
1422999107,2cyclictest20484-21basename08:30:152
1422699108,1cyclictest32561-21basename08:55:151
1422699108,1cyclictest19327-21grep08:25:221
1422699108,1cyclictest18595-21chrt07:15:411
1422699108,1cyclictest14910-21expr07:10:151
1422699107,2cyclictest4217-21grep07:55:181
1422699107,2cyclictest15607-21taskset08:20:081
1422699100,9cyclictest294892sleep108:47:271
1422299108,2cyclictest2060-21grep11:00:240
1422299108,1cyclictest32694-21egrep08:55:150
1422299108,1cyclictest26011-21taskset07:33:220
1422299108,1cyclictest21872-21grep11:50:190
1422299108,1cyclictest10952-21hostname08:10:140
1422299107,2cyclictest24263-21cat08:35:230
1422299107,2cyclictest11166-21ssh09:50:120
1422299103,5cyclictest18960-21munin-run07:20:010
1422299100,10cyclictest14874-21expr07:10:150
142319996,2cyclictest11594-21head08:10:163
142299996,2cyclictest16299-21grep08:20:172
142299990,3cyclictest225-21systemd-journal08:52:492
142269997,1cyclictest12030-21open_files08:10:211
142229998,1cyclictest2294-21ntpq07:50:210
142229997,2cyclictest21597-21cut08:30:210
142229994,4cyclictest26544-21/usr/sbin/munin08:40:220
142229986,1cyclictest29041-21grep07:40:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional