You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-21 - 07:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Mar 21, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1932728464,16sleep20-21swapper/219:06:482
1947928067,9sleep00-21swapper/019:08:460
1915228067,9sleep30-21swapper/319:05:293
1929327857,6sleep10-21swapper/119:06:231
1966699280,27cyclictest0-21swapper/300:00:223
1966299280,27cyclictest0-21swapper/222:08:372
1966299280,26cyclictest0-21swapper/200:06:492
1966699273,18cyclictest12492-21diskmemload22:35:593
1966699272,24cyclictest12492-21diskmemload22:58:083
1966699271,23cyclictest0-21swapper/322:05:133
1966699270,26cyclictest0-21swapper/323:55:183
1966699270,25cyclictest0-21swapper/321:30:433
1966699270,25cyclictest0-21swapper/321:30:423
1966299272,19cyclictest1009-21grep22:00:172
1966299270,2cyclictest0-21swapper/200:00:222
1966299270,26cyclictest0-21swapper/223:55:182
1966299270,26cyclictest0-21swapper/221:05:132
1966299270,25cyclictest0-21swapper/222:11:392
1966699262,19cyclictest12492-21diskmemload21:52:233
1966099261,1cyclictest22548-21tr19:15:151
1966099260,1cyclictest0-21swapper/122:59:521
1966699252,22cyclictest7705-21sh22:42:343
1966699252,22cyclictest12492-21diskmemload00:35:073
1966699251,23cyclictest0-21swapper/321:05:143
1966699251,20cyclictest31531-21cpuspeed_turbos21:40:143
1966699250,2cyclictest0-21swapper/300:05:463
1966299252,22cyclictest0-21swapper/223:25:202
1966299251,22cyclictest0-21swapper/222:17:382
1966299250,2cyclictest0-21swapper/223:01:362
1966299250,24cyclictest0-21swapper/222:28:062
1966099250,0cyclictest0-21swapper/122:10:181
1966699243,20cyclictest12492-21diskmemload00:11:563
1966699242,20cyclictest12492-21diskmemload22:15:563
1966699241,22cyclictest12492-21diskmemload23:18:233
1966699240,1cyclictest0-21swapper/321:14:043
1966299243,16cyclictest12492-21diskmemload21:27:292
1966299242,21cyclictest0-21swapper/221:15:252
1966299240,3cyclictest0-21swapper/221:50:312
1966299240,2cyclictest0-21swapper/222:20:382
1966299240,1cyclictest0-21swapper/222:45:122
1966299240,1cyclictest0-21swapper/222:42:352
1966099240,1cyclictest0-21swapper/122:02:411
1966099240,0cyclictest0-21swapper/122:43:581
1966099240,0cyclictest0-21swapper/122:22:191
1966699232,20cyclictest12492-21diskmemload23:48:453
1966699231,21cyclictest7310-21sh21:45:113
1966699231,1cyclictest0-21swapper/300:21:463
1966699231,19cyclictest974-21ssh22:00:183
1966699230,22cyclictest0-21swapper/321:29:563
1966699230,22cyclictest0-21swapper/321:15:253
1966699230,1cyclictest0-21swapper/320:20:143
19662992316,6cyclictest10550irq/122-ahci[0019:50:012
1966299231,21cyclictest12492-21diskmemload23:07:132
1966299230,2cyclictest0-21swapper/223:20:062
1966299230,2cyclictest0-21swapper/223:20:062
1966299230,2cyclictest0-21swapper/200:10:122
1966299230,0cyclictest0-21swapper/221:35:412
19660992323,0cyclictest0-21swapper/123:30:241
1966099230,0cyclictest0-21swapper/100:35:081
1966699223,18cyclictest12492-21diskmemload23:13:263
1966699223,18cyclictest12492-21diskmemload22:50:313
1966699222,1cyclictest12492-21diskmemload23:23:383
1966699222,1cyclictest12492-21diskmemload23:23:373
1966699222,19cyclictest0-21swapper/321:20:163
1966699221,2cyclictest12492-21diskmemload22:10:503
1966699221,1cyclictest0-21swapper/322:24:233
1966699220,2cyclictest0-21swapper/323:07:123
1966699220,20cyclictest0-21swapper/322:47:563
1966299224,17cyclictest12492-21diskmemload21:20:172
1966299222,1cyclictest12492-21diskmemload21:57:302
1966299221,5cyclictest12492-21diskmemload00:21:452
1966299221,1cyclictest0-21swapper/221:45:112
1966299220,2cyclictest12492-21diskmemload23:40:072
1966299220,21cyclictest0-21swapper/222:33:272
1966299220,21cyclictest0-21swapper/219:10:112
1966299220,21cyclictest0-21swapper/200:29:162
1966299220,1cyclictest0-21swapper/223:18:232
1966299220,1cyclictest0-21swapper/200:35:082
1966099220,0cyclictest0-21swapper/122:06:171
1966099220,0cyclictest0-21swapper/100:33:501
1966699211,19cyclictest12492-21diskmemload00:19:183
1966699210,5cyclictest16259-21chrt22:28:403
1966699210,2cyclictest0-21swapper/323:03:043
1966699210,20cyclictest0-21swapper/323:53:463
1966699210,1cyclictest0-21swapper/323:43:183
1966699210,1cyclictest0-21swapper/323:30:133
1966299219,5cyclictest8884-21cron19:55:002
1966299213,17cyclictest12492-21diskmemload22:51:122
1966299212,18cyclictest0-21swapper/221:00:142
19662992120,1cyclictest0-21swapper/200:30:022
1966299211,1cyclictest12492-21diskmemload21:31:252
1966299211,1cyclictest12492-21diskmemload21:31:242
1966299210,2cyclictest0-21swapper/223:13:262
1966299210,19cyclictest0-21swapper/222:58:082
1966099210,2cyclictest0-21swapper/123:21:071
1966099210,2cyclictest0-21swapper/123:21:071
1965699217,7cyclictest17909-21munin-run00:25:000
1965699210,20cyclictest0-21swapper/022:09:590
1965699210,0cyclictest0-21swapper/000:36:550
153852210,2sleep00-21swapper/022:47:050
1966699203,17cyclictest0-21swapper/323:29:463
1966699200,2cyclictest0-21swapper/322:31:133
1966699200,1cyclictest0-21swapper/323:38:423
1966699200,1cyclictest0-21swapper/300:25:163
1966699200,19cyclictest0-21swapper/321:55:123
1966699200,16cyclictest0-21swapper/300:33:123
1966299200,2cyclictest32679-21head21:40:222
1966299200,1cyclictest0-21swapper/223:48:452
1966299200,1cyclictest0-21swapper/222:37:342
1966299200,1cyclictest0-21swapper/220:30:132
1966299200,16cyclictest0-21swapper/223:53:422
1966099200,1cyclictest0-21swapper/123:05:471
1966099200,0cyclictest0-21swapper/121:41:091
1966699195,13cyclictest1769-21munin-run19:40:013
1966699195,13cyclictest1769-21munin-run19:40:003
1966699190,2cyclictest0-21swapper/321:37:363
1966699190,14cyclictest0-21swapper/320:30:133
1966299192,2cyclictest13144-21aten2_r7power_v21:10:142
1966299191,2cyclictest2062-21aten2_r7power_c19:40:132
19660991919,0cyclictest0-21swapper/121:34:461
19660991919,0cyclictest0-21swapper/121:34:451
1966099190,2cyclictest0-21swapper/123:13:231
1966099190,2cyclictest0-21swapper/100:09:351
1966099190,0cyclictest0-21swapper/123:45:291
1966699180,16cyclictest0-21swapper/319:10:113
1966299184,13cyclictest24391-21munin-run19:20:012
1966299183,15cyclictest0-21swapper/200:18:002
1966299183,14cyclictest1776-21perf19:40:012
1966299183,14cyclictest1776-21perf19:40:002
19662991816,1cyclictest0-21swapper/220:10:132
1966299180,2cyclictest0-21swapper/223:35:282
1966299180,2cyclictest0-21swapper/223:30:222
19660991818,0cyclictest0-21swapper/122:38:441
1966099180,1cyclictest0-21swapper/122:15:201
1966099180,0cyclictest0-21swapper/121:25:061
1966699170,1cyclictest0-21swapper/320:41:343
1966699170,1cyclictest0-21swapper/319:50:223
1966099170,3cyclictest0-21swapper/121:57:301
1966099170,1cyclictest0-21swapper/122:51:581
1966099170,0cyclictest0-21swapper/122:47:221
1965699170,0cyclictest0-21swapper/022:59:570
1966699163,12cyclictest0-21swapper/320:50:003
1966699161,14cyclictest24425-21idleruntime-cro19:20:013
1966699161,14cyclictest11274-21sendmail20:00:013
1966699160,2cyclictest0-21swapper/320:15:213
1966699160,1cyclictest19647-21cyclictest20:10:253
1966699160,1cyclictest0-21swapper/321:00:143
1966299161,2cyclictest11289-21awk20:00:002
1966099161,1cyclictest12492-21diskmemload22:28:351
1966099160,2cyclictest0-21swapper/100:27:001
1966099160,0cyclictest0-21swapper/123:28:281
1966099160,0cyclictest0-21swapper/121:36:551
1965699160,2cyclictest141rcu_preempt21:39:330
1965699160,2cyclictest0-21swapper/022:29:340
1965699160,15cyclictest0-21swapper/023:51:590
1966699158,6cyclictest12598-21cat20:00:193
1966699150,11cyclictest0-21swapper/319:40:123
1966699150,0cyclictest0-21swapper/319:30:123
1966299156,7cyclictest30449-21cut19:30:172
1966299150,2cyclictest0-21swapper/220:05:202
1966099152,6cyclictest12492-21diskmemload21:10:231
1966099150,2cyclictest0-21swapper/123:36:151
1966099150,1cyclictest11387-21ssh23:42:071
1966099150,14cyclictest0-21swapper/100:15:241
1966099150,13cyclictest0-21swapper/119:30:131
19656991515,0cyclictest0-21swapper/021:16:380
1965699150,2cyclictest0-21swapper/000:15:010
1965699150,1cyclictest0-21swapper/022:20:210
1965699150,14cyclictest0-21swapper/022:13:360
1965699150,11cyclictest32260-21ssh23:55:130
1966699148,5cyclictest4825-21chrt20:51:423
1966699148,5cyclictest25393-21chrt20:27:583
1966699147,6cyclictest30235-21chrt20:38:293
1966699147,6cyclictest26909-21cat19:25:003
1966299148,5cyclictest2773-21date20:50:012
1966299147,6cyclictest22976-21chrt20:22:242
1966299146,7cyclictest24956-21tr19:20:132
1966299142,11cyclictest0-21swapper/220:45:002
1966299142,10cyclictest27714-21fschecks_count19:25:152
19662991411,2cyclictest25574-21awk20:30:002
19662991410,2cyclictest7672-21cat21:00:012
1966099141,6cyclictest11288-21perf20:00:001
19660991410,3cyclictest25875-21tr00:10:131
1966099140,1cyclictest12492-21diskmemload21:47:421
1966099140,10cyclictest0-21swapper/121:05:131
1965699145,3cyclictest20973-21perf21:15:010
1965699143,7cyclictest6247-21kworker/0:222:35:160
19656991411,2cyclictest24680-21ssh21:55:160
19656991411,2cyclictest20567-21idleruntime-cro23:10:010
19656991410,3cyclictest3602-21grep22:40:150
1965699140,14cyclictest0-21swapper/022:18:010
1966699138,4cyclictest16011-21chrt20:09:513
1966699137,5cyclictest6492-21cat20:55:203
1966299139,2cyclictest28513-21grep20:35:162
1966299138,4cyclictest19839-21ntp_states20:15:202
1966299130,1cyclictest0-21swapper/220:50:212
1966099139,3cyclictest12254-21cat23:05:001
1966099139,2cyclictest2710-21cron20:50:001
1966099130,2cyclictest0-21swapper/121:22:481
1966099130,1cyclictest0-21swapper/123:56:411
1966099130,1cyclictest0-21swapper/122:32:061
1965699139,3cyclictest14992-21grep23:25:170
19656991310,2cyclictest9613-21seq00:00:220
19656991310,2cyclictest849-21grep19:35:210
19656991310,2cyclictest849-21grep19:35:210
19656991310,2cyclictest7512-21munin-plugin-st21:00:000
19656991310,2cyclictest5547-21strings23:00:180
19656991310,2cyclictest31403-21grep20:40:170
19656991310,2cyclictest2214-21grep20:45:270
19656991310,2cyclictest20054-21grep22:50:150
19656991310,2cyclictest125362sleep021:28:540
1965699130,9cyclictest0-21swapper/021:32:220
1965699130,9cyclictest0-21swapper/021:32:220
1965699130,7cyclictest16522-21fschecks_count21:50:180
1965699130,5cyclictest0-21swapper/000:10:520
1965699130,11cyclictest1010-21sed22:00:170
1966699122,7cyclictest225-21systemd-journal19:25:013
1966699121,3cyclictest1142-21runrttasks19:45:033
1966299128,3cyclictest13414-21systemd-cgroups20:01:572
1966099129,2cyclictest29009-21sed20:35:181
1966099129,2cyclictest23609-21cat21:55:011
1966099129,2cyclictest15625-21sed20:05:271
1966099129,2cyclictest13693-21cat20:05:011
1966099128,3cyclictest22754-21tail20:20:261
1966099121,6cyclictest0-21swapper/119:45:131
1966099120,11cyclictest0-21swapper/121:15:561
1966099120,10cyclictest0-21swapper/123:53:541
1966099120,0cyclictest0-21swapper/100:04:481
1965699129,2cyclictest31596-21tr21:40:150
1965699129,2cyclictest24417-21tr19:20:000
1965699129,2cyclictest16945-21basename00:05:130
1965699125,6cyclictest13473-21munin-run23:25:010
1965699125,6cyclictest13473-21munin-run23:25:000
1965699120,11cyclictest27901-21grep00:30:180
1966099118,2cyclictest20782-21cron20:19:591
1966099118,2cyclictest17458-21ntpq20:10:211
1966099118,2cyclictest11689-21ls00:20:241
1966099116,4cyclictest31579-21ssh23:15:301
1966099110,1cyclictest0-21swapper/120:30:131
1965699119,1cyclictest91852sleep021:00:260
1965699119,1cyclictest31446-21grep23:35:140
1965699118,2cyclictest4896-21cmp20:52:350
1965699118,2cyclictest3072-21sh23:18:320
1965699118,2cyclictest2784-21ssh21:22:540
1965699118,2cyclictest25368-21grep19:20:160
1965699118,2cyclictest24202-21sed20:25:170
1965699118,2cyclictest22150-21ssh23:10:170
1965699118,2cyclictest19928-21sh22:30:210
1965699118,2cyclictest18817-21cat00:25:150
1965699118,2cyclictest17044-21ssh23:45:210
1965699118,2cyclictest15121-21ls20:05:220
1965699118,2cyclictest14340-21ssh23:44:250
1965699118,2cyclictest12387-21taskset21:08:500
1965699113,3cyclictest26756-21apache219:55:120
1965699110,2cyclictest0-21swapper/023:30:150
1966099108,1cyclictest3807-21taskset19:40:461
1966099108,1cyclictest31480-21basename20:40:181
1966099108,1cyclictest10088-21dpkg21:05:001
1966099107,2cyclictest5587-21cpuspeed_turbos20:55:131
1966099107,2cyclictest3106-21aten2_r7power_v20:50:131
1966099107,2cyclictest25619-21latency19:20:171
1966099107,2cyclictest25308-21taskset20:26:521
1966099107,2cyclictest20621-21head19:10:151
1966099101,3cyclictest225-21systemd-journal19:40:011
1966099101,3cyclictest225-21systemd-journal19:40:001
1965699108,1cyclictest7900-21nfs4_client19:50:200
1965699108,1cyclictest7692-21expr21:45:130
1965699108,1cyclictest19088-21expr20:15:150
1965699107,2cyclictest30007-21expr19:30:150
1965699107,2cyclictest2799-21grep19:40:160
1965699107,2cyclictest27924-21ls19:25:160
1965699107,2cyclictest16636-21expr20:10:140
1965699100,9cyclictest21015-21ntp_kernel_pll_19:10:180
1965699100,9cyclictest135982sleep020:04:200
196609997,1cyclictest7213-21expr19:50:141
196609997,1cyclictest27529-21cat19:25:141
196569997,1cyclictest4523-21aten2_r7power_v19:45:130
196569997,1cyclictest28538-21tr20:35:150
196569997,1cyclictest21538-21expr20:20:150
196569985,2cyclictest25932-21tr20:30:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional