You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-23 - 15:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Jul 23, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
410729164,9sleep00-21swapper/007:05:370
441228665,6sleep10-21swapper/107:09:351
441928569,9sleep30-21swapper/307:09:413
426928265,13sleep20-21swapper/207:07:432
454099253,12cyclictest0-21swapper/310:40:143
454099250,24cyclictest0-21swapper/308:05:133
453399243,12cyclictest0-21swapper/107:30:121
185932240,0sleep20-21swapper/209:21:512
4540992321,1cyclictest0-21swapper/312:25:123
454099230,21cyclictest0-21swapper/311:14:333
4533992323,0cyclictest0-21swapper/109:12:011
4540992220,1cyclictest0-21swapper/307:40:133
454099220,22cyclictest0-21swapper/309:26:473
453699220,1cyclictest0-21swapper/211:06:152
4533992221,1cyclictest0-21swapper/109:57:191
453399220,18cyclictest225-21systemd-journal10:19:441
453399220,18cyclictest225-21systemd-journal10:19:441
4540992121,0cyclictest0-21swapper/309:30:043
4536992117,3cyclictest27101-21kworker/u8:209:10:002
453399210,4cyclictest0-21swapper/109:40:101
453099210,20cyclictest0-21swapper/012:22:060
4540992018,1cyclictest20405-21kworker/u8:209:59:123
454099201,2cyclictest141rcu_preempt10:47:463
4533992019,1cyclictest0-21swapper/107:15:021
453399200,16cyclictest0-21swapper/111:18:291
454099190,0cyclictest0-21swapper/311:25:343
4536991918,1cyclictest0-21swapper/208:45:132
453699191,3cyclictest32-21ksoftirqd/209:00:202
453699191,1cyclictest141rcu_preempt09:15:532
453399190,14cyclictest29756-21diskmemload09:51:311
453099190,18cyclictest0-21swapper/009:10:380
453699184,1cyclictest141rcu_preempt11:52:592
4536991814,1cyclictest6591-21/usr/sbin/munin11:10:252
453699181,16cyclictest0-21swapper/210:37:142
453699180,2cyclictest0-21swapper/209:25:142
453699180,17cyclictest0-21swapper/210:44:552
453099180,2cyclictest0-21swapper/010:15:350
453099180,2cyclictest0-21swapper/010:15:340
4540991713,3cyclictest39-21ksoftirqd/310:05:083
454099170,15cyclictest0-21swapper/310:10:313
454099170,0cyclictest0-21swapper/312:33:103
453699170,14cyclictest0-21swapper/210:06:412
453399178,9cyclictest0-21swapper/108:20:121
453399170,16cyclictest0-21swapper/110:27:401
4530991716,1cyclictest0-21swapper/012:39:160
454099161,14cyclictest0-21swapper/309:49:143
454099160,16cyclictest0-21swapper/310:24:553
453699166,3cyclictest225-21systemd-journal11:04:552
453699161,3cyclictest141rcu_preempt09:42:482
453099165,6cyclictest916-21basename08:10:000
453099160,16cyclictest0-21swapper/009:48:370
454099150,4cyclictest0-21swapper/310:28:503
4536991513,1cyclictest32-21ksoftirqd/212:29:592
4536991511,3cyclictest16614-21ls11:55:212
453399155,6cyclictest452-21systemd-logind11:02:561
453399150,2cyclictest0-21swapper/111:25:141
453099154,6cyclictest6895-21idleruntime-cro07:14:590
453099150,12cyclictest0-21swapper/010:43:520
454099149,4cyclictest758-21tr12:05:213
4540991410,3cyclictest32017-21grep11:45:203
454099140,4cyclictest0-21swapper/312:35:213
454099140,1cyclictest0-21swapper/312:10:153
4536991412,1cyclictest32-21ksoftirqd/212:03:102
453699140,12cyclictest0-21swapper/210:20:482
453399141,7cyclictest0-21swapper/107:55:141
453399141,1cyclictest17246-21ssh10:38:521
453399141,12cyclictest12013-21ssh11:52:541
4533991410,3cyclictest5203-21grep08:15:261
453399140,13cyclictest0-21swapper/110:00:481
453399140,13cyclictest0-21swapper/109:25:091
453099142,1cyclictest141rcu_preempt11:32:300
453099141,7cyclictest28573-21munin-run08:00:000
453099141,7cyclictest22612-21munin-run08:55:000
4530991414,0cyclictest0-21swapper/009:51:270
4530991413,1cyclictest0-21swapper/010:53:550
4530991411,2cyclictest20110-21head07:40:170
453099141,10cyclictest0-21swapper/009:27:260
4530991410,3cyclictest23184-21cat12:00:000
4530991410,2cyclictest17153-21sed08:40:250
453099140,2cyclictest0-21swapper/009:20:470
454099139,3cyclictest23484-21cut08:55:193
454099139,3cyclictest15280-21grep07:30:163
454099135,3cyclictest17241-21sed08:40:253
4540991313,0cyclictest0-21swapper/311:36:143
4540991312,1cyclictest0-21swapper/310:55:123
4540991310,2cyclictest32155-21wc09:10:253
4540991310,2cyclictest31419-21sh12:05:013
4540991310,2cyclictest30947-21cron08:05:003
4540991310,2cyclictest14854-21grep08:35:273
454099130,5cyclictest0-21swapper/309:35:223
454099130,1cyclictest0-21swapper/311:53:203
454099130,13cyclictest0-21swapper/310:52:513
453699139,3cyclictest21916-21munin-plugin-st11:40:002
453699135,2cyclictest32-21ksoftirqd/210:10:422
453699132,10cyclictest8628-21taskset12:10:172
453699131,11cyclictest18804-21ntpq12:35:202
4536991310,2cyclictest6471-21grep11:30:162
4536991310,2cyclictest6242-21grep08:20:132
4536991310,2cyclictest31583-21head11:45:172
4536991310,2cyclictest17711-21grep08:45:002
453699130,4cyclictest0-21swapper/212:06:002
453699130,1cyclictest0-21swapper/210:29:372
453399138,4cyclictest28971-21sh12:03:041
4533991310,2cyclictest8147-21tr12:10:151
4533991310,2cyclictest4523-21grep10:50:171
4533991310,2cyclictest32688-21latency_hist12:25:001
4533991310,2cyclictest30270-21turbostat11:45:001
4533991310,2cyclictest29471-21grep11:05:171
4533991310,2cyclictest29252-21wc10:45:261
453399130,12cyclictest22128-21grep08:50:261
453099139,3cyclictest8956-21ssh09:55:140
453099139,3cyclictest29622-21grep11:05:170
453099139,3cyclictest24312-21cat12:20:010
453099139,2cyclictest7040-21grep11:50:150
453099134,8cyclictest2981-21cron10:50:000
453099133,9cyclictest14032-21/usr/sbin/munin11:15:180
453099132,5cyclictest11940-21perf07:24:590
453099131,7cyclictest28300-21awk11:04:590
453099131,6cyclictest30271-21munin-run11:45:010
4530991312,1cyclictest0-21swapper/012:25:260
4530991310,2cyclictest7494-21cut09:35:140
4530991310,2cyclictest24368-21grep07:50:150
4530991310,2cyclictest17613-21cut07:35:170
4530991310,2cyclictest15316-21sed07:30:160
4530991310,2cyclictest12423-21grep08:30:260
174762130,1sleep10-21swapper/111:55:381
454099129,2cyclictest9005-21sort08:25:163
454099129,2cyclictest8293-21uname07:15:203
454099129,2cyclictest822-21grep09:50:173
454099129,2cyclictest30357-21grep11:05:233
454099129,2cyclictest29756-21diskmemload09:09:593
454099129,2cyclictest26924-21grep09:00:263
454099129,2cyclictest22062-21wc08:50:253
454099129,2cyclictest18141-21grep12:15:253
454099129,2cyclictest15967-21wc09:20:183
454099129,2cyclictest13546-21latency_hist11:35:003
454099129,2cyclictest11420-21sed10:15:283
454099129,2cyclictest11420-21sed10:15:273
454099127,4cyclictest6772-21grep09:15:153
454099124,4cyclictest17917-21cat10:00:193
454099123,5cyclictest225-21systemd-journal07:40:003
4540991212,0cyclictest0-21swapper/310:37:103
453699129,2cyclictest4396-21grep10:50:162
453699129,2cyclictest28118-21grep07:55:262
453699129,2cyclictest24733-21/usr/sbin/munin12:20:182
453699129,2cyclictest13875-21head08:35:192
453699129,2cyclictest13662-21ssh10:17:102
453699129,2cyclictest13662-21ssh10:17:102
453699124,4cyclictest225-21systemd-journal08:00:002
4536991210,1cyclictest32-21ksoftirqd/209:14:382
453699120,1cyclictest0-21swapper/209:32:542
453399129,2cyclictest3081-21cut10:30:161
453399129,2cyclictest27049-21basename10:45:001
453399129,2cyclictest24991-21head09:45:181
453399129,2cyclictest1956-21tr12:25:191
453399129,2cyclictest19152-21awk12:35:231
453399129,2cyclictest18340-21grep08:45:131
453399129,2cyclictest18078-21grep10:20:131
453399129,2cyclictest17179-21tr07:35:161
453399128,3cyclictest140832sleep111:35:131
453399128,2cyclictest7512-21ls12:10:001
453399128,2cyclictest25-21ksoftirqd/110:10:201
453399127,4cyclictest25-21ksoftirqd/112:30:111
453399123,1cyclictest0-21swapper/111:48:141
453099129,2cyclictest9800-21sed12:10:270
453099129,2cyclictest21774-21grep07:45:140
453099129,2cyclictest21760-21cut11:20:140
453099129,2cyclictest17434-21ssh09:40:250
453099128,3cyclictest835-21grep12:05:220
453099128,2cyclictest7175-21tr08:20:200
4530991211,1cyclictest0-21swapper/009:33:590
4530991210,1cyclictest29756-21diskmemload10:10:080
454099118,2cyclictest8048-21chrt08:24:583
454099118,2cyclictest4309-21grep08:15:173
454099118,2cyclictest29227-21ssh11:44:123
454099118,2cyclictest23930-21taskset11:20:503
454099118,2cyclictest1991-21rm10:30:023
454099118,2cyclictest15659-21ssh09:40:093
454099117,3cyclictest28113-21sed07:55:263
454099117,3cyclictest25073-21grep12:20:143
454099117,2cyclictest21936-21tr07:45:163
454099114,3cyclictest225-21systemd-journal11:17:253
454099110,8cyclictest22112-21awk11:00:263
454099110,10cyclictest20363-21cat08:50:013
454099110,10cyclictest16849-21ssh11:55:233
453699119,1cyclictest32-21ksoftirqd/209:50:012
453699119,1cyclictest26197-21munin-run07:55:002
453699119,1cyclictest12888-21sh09:38:092
453699118,2cyclictest9459-21cron07:19:592
453699118,2cyclictest9447-21cstates12:30:132
453699118,2cyclictest5649-21ssh10:31:322
453699118,2cyclictest5473-21expr07:10:172
453699118,2cyclictest4313-21head08:15:172
453699118,2cyclictest31410-21ssh10:46:532
453699118,2cyclictest25079-21cat09:45:192
453699118,2cyclictest23912-21ssh12:19:542
453699118,2cyclictest23674-21tail11:40:202
453699118,2cyclictest22102-21ssh10:03:022
453699118,2cyclictest20043-21grep07:40:162
453699118,2cyclictest16974-21cut07:35:152
453699118,2cyclictest15960-21chrt11:16:072
453699118,2cyclictest13985-21grep10:55:262
453699118,2cyclictest13144-21cat07:25:192
453699117,3cyclictest30896-21sh11:25:202
453699110,2cyclictest0-21swapper/211:24:382
453699110,1cyclictest0-21swapper/209:57:512
453399119,1cyclictest17654-21awk08:45:001
453399118,2cyclictest8782-21expr08:25:141
453399118,2cyclictest6562-21ssh11:10:251
453399118,2cyclictest25995-21grep09:00:171
453399118,2cyclictest25584-21expr10:05:121
453399118,2cyclictest23251-21expr07:45:271
453399118,2cyclictest22603-21wc11:20:191
453399118,2cyclictest19397-21ssh09:22:291
453399118,2cyclictest1841-21ssh09:31:031
453399118,2cyclictest1748-21df08:10:161
453399118,2cyclictest16000-21rm12:15:041
453399118,2cyclictest11408-21sh09:36:591
453399117,3cyclictest14920-21taskset08:36:011
453399117,3cyclictest10933-21tr08:30:141
453399114,3cyclictest8447-21grep09:15:281
453399110,1cyclictest0-21swapper/111:31:351
453399110,10cyclictest29934-21cut09:10:001
453099118,2cyclictest9459-21cat12:30:130
453099118,2cyclictest8363-21grep08:25:140
453099118,2cyclictest31546-21cut11:45:170
453099118,2cyclictest30991-21grep11:25:190
453099118,2cyclictest30946-21cron08:05:010
453099118,2cyclictest26035-21sed09:00:170
453099118,2cyclictest25946-21ssh12:00:420
453099118,2cyclictest2564-21ssh10:30:140
453099118,2cyclictest16872-21cut10:00:140
453099118,2cyclictest16710-21ssh11:35:570
453099118,2cyclictest13813-21sort08:35:190
453099118,2cyclictest12918-21grep10:55:160
453099118,2cyclictest11115-21ssh11:13:540
453099118,2cyclictest10771-21ssh10:35:110
453099118,2cyclictest10157-21ssh09:16:480
453099117,3cyclictest3846-21wc08:15:140
453099114,4cyclictest27398-21irqstats10:25:200
453099110,10cyclictest18028-21cat10:20:130
454099108,1cyclictest6825-21chrt07:14:343
454099108,1cyclictest12959-21head07:25:173
454099107,2cyclictest24073-21chrt07:50:113
454099107,2cyclictest1308-21cut08:10:153
454099107,2cyclictest12123-21readlink08:30:223
454099107,2cyclictest10521-21grep07:20:173
453699108,1cyclictest23256-21cat08:55:172
453699108,1cyclictest22013-21cat07:45:162
453699107,2cyclictest8618-21tr08:25:142
453699107,2cyclictest598-21chrt08:06:172
453699107,2cyclictest20847-21expr08:50:142
453699107,2cyclictest1525-21expr08:10:162
453699107,2cyclictest11319-21wc07:20:262
453699107,2cyclictest11019-21expr08:30:142
453399108,1cyclictest24096-21grep08:55:231
453399108,1cyclictest12052-21cat07:25:001
453399107,2cyclictest24788-21grep07:50:171
453399107,2cyclictest20827-21cat07:40:251
453399107,2cyclictest12329-21grep07:25:151
453399107,2cyclictest12112-21aten2_r7power_c10:55:131
453399106,3cyclictest0-21swapper/107:10:221
453399101,5cyclictest0-21swapper/108:05:131
4533991010,0cyclictest0-21swapper/108:00:121
453099108,1cyclictest29562sleep008:10:560
453099108,1cyclictest27679-21aten2_r7power_e09:05:130
453099108,1cyclictest22823-21chrt08:55:030
453099107,2cyclictest27352-21ssh10:05:370
453099107,2cyclictest14293-21sh07:30:000
45369997,1cyclictest16289-21awk07:30:262
45309997,1cyclictest8170-21grep07:15:200
45309996,2cyclictest18146-21wget08:45:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional