You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-16 - 15:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Jul 16, 2026 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
457729985,9sleep00-21swapper/007:09:190
438529583,8sleep20-21swapper/207:06:482
447829067,18sleep30-21swapper/307:08:023
249528669,13sleep10-21swapper/107:05:021
472199260,26cyclictest0-21swapper/211:20:252
472199260,26cyclictest0-21swapper/211:20:252
4721992512,2cyclictest311rcuc/210:57:032
472199250,2cyclictest0-21swapper/209:28:432
471799251,1cyclictest29954-21diskmemload10:52:301
472199245,2cyclictest0-21swapper/212:22:522
471799240,1cyclictest0-21swapper/110:05:541
4717992321,1cyclictest25-21ksoftirqd/110:27:251
4724992220,1cyclictest39-21ksoftirqd/310:25:053
4721992220,1cyclictest1430-21munin-run10:10:002
472199220,1cyclictest0-21swapper/212:16:052
472199220,0cyclictest0-21swapper/211:52:312
4717992216,2cyclictest25-21ksoftirqd/110:38:511
4724992120,1cyclictest0-21swapper/310:30:133
472499210,0cyclictest0-21swapper/311:19:343
4721992116,4cyclictest0-21swapper/209:35:022
471799210,5cyclictest0-21swapper/110:10:571
471799210,4cyclictest24895-21taskset12:38:561
471799210,1cyclictest0-21swapper/111:52:001
4724992019,1cyclictest0-21swapper/311:54:423
472199209,9cyclictest29954-21diskmemload11:44:142
472199209,11cyclictest0-21swapper/210:52:302
472199208,8cyclictest0-21swapper/209:52:192
472199205,1cyclictest0-21swapper/209:13:042
472199205,14cyclictest0-21swapper/208:15:142
4721992014,4cyclictest29954-21diskmemload10:24:582
4721992012,7cyclictest0-21swapper/209:59:442
472199200,2cyclictest0-21swapper/210:45:182
472199200,0cyclictest0-21swapper/211:07:282
471799204,3cyclictest25-21ksoftirqd/112:30:201
471799200,0cyclictest141rcu_preempt10:23:191
471599206,7cyclictest31422-21munin-run09:30:000
471599204,6cyclictest3523-21apt-get10:50:020
471599201,18cyclictest0-21swapper/009:36:150
4721991918,1cyclictest29954-21diskmemload12:10:012
472199190,3cyclictest0-21swapper/211:12:432
4717991917,1cyclictest38150irq/131-snd_hda09:55:101
471599197,6cyclictest20720-21awk11:00:000
472499180,16cyclictest0-21swapper/312:10:493
472199182,2cyclictest0-21swapper/211:18:172
4721991813,5cyclictest29954-21diskmemload12:27:102
4721991812,5cyclictest7599-21/usr/sbin/munin09:15:192
4721991811,1cyclictest29954-21diskmemload10:19:392
472199180,2cyclictest10914-21ls12:30:172
4717991817,0cyclictest251ksoftirqd/111:28:391
471799180,16cyclictest0-21swapper/112:12:121
472499173,9cyclictest10273-21cpuspeed_turbos12:30:153
472499170,1cyclictest0-21swapper/312:35:133
472499170,17cyclictest0-21swapper/311:26:573
472199174,13cyclictest0-21swapper/210:11:252
4721991716,1cyclictest55550irq/126-enp2s0-12:35:132
4721991714,1cyclictest32-21ksoftirqd/208:13:092
4721991713,2cyclictest9467-21awk07:20:002
472199170,1cyclictest0-21swapper/209:21:472
471799175,2cyclictest141rcu_preempt11:46:541
4717991712,1cyclictest25-21ksoftirqd/109:38:591
471799171,1cyclictest29954-21diskmemload09:49:501
471799170,2cyclictest0-21swapper/110:47:001
471799170,1cyclictest0-21swapper/110:57:321
472499162,3cyclictest141rcu_preempt11:07:003
4724991614,1cyclictest29954-21diskmemload11:49:343
472499161,14cyclictest0-21swapper/310:41:163
472199162,12cyclictest0-21swapper/211:55:182
4721991616,0cyclictest29954-21diskmemload10:03:582
472199161,2cyclictest32-21ksoftirqd/209:42:312
4721991611,4cyclictest29954-21diskmemload11:03:332
472199160,1cyclictest0-21swapper/209:30:582
471799164,4cyclictest0-21swapper/110:30:131
471799164,1cyclictest141rcu_preempt10:40:111
471799162,1cyclictest141rcu_preempt09:13:121
4717991615,1cyclictest0-21swapper/111:20:251
4717991615,1cyclictest0-21swapper/111:20:251
4717991614,2cyclictest25-21ksoftirqd/110:15:121
471799160,0cyclictest0-21swapper/111:36:591
4715991616,0cyclictest0-21swapper/012:29:460
4715991614,2cyclictest0-21swapper/010:22:360
471599161,14cyclictest31938-21ssh09:30:140
471599160,1cyclictest0-21swapper/012:21:320
471599160,16cyclictest0-21swapper/011:44:530
471599160,10cyclictest25227-21basename12:20:000
472499152,12cyclictest537-21snmpd10:21:043
4724991513,2cyclictest0-21swapper/310:54:403
472199158,6cyclictest0-21swapper/211:45:082
472199156,3cyclictest0-21swapper/208:20:142
472199150,7cyclictest0-21swapper/210:41:532
472199150,13cyclictest0-21swapper/211:27:062
471799151,6cyclictest25-21ksoftirqd/109:23:341
471799151,2cyclictest141rcu_preempt08:45:161
4717991512,3cyclictest25-21ksoftirqd/111:00:201
4717991511,2cyclictest25-21ksoftirqd/110:00:161
4717991510,3cyclictest25-21ksoftirqd/109:16:461
4717991510,1cyclictest25-21ksoftirqd/111:07:031
471799150,1cyclictest0-21swapper/112:29:331
4724991414,0cyclictest0-21swapper/311:44:223
472499141,1cyclictest29954-21diskmemload09:21:083
4724991411,2cyclictest14642-21cat09:20:013
4724991411,2cyclictest14640-21apt-config11:35:003
4724991410,3cyclictest19175-21ssh12:15:253
472199149,1cyclictest29954-21diskmemload10:35:122
472199144,9cyclictest0-21swapper/209:46:412
472199143,1cyclictest0-21swapper/211:39:252
472199142,6cyclictest0-21swapper/212:04:152
4721991413,1cyclictest32-21ksoftirqd/208:53:112
4721991413,1cyclictest32-21ksoftirqd/207:48:342
4721991411,2cyclictest12615-21wc08:30:252
4721991411,1cyclictest29954-21diskmemload10:30:042
472199140,9cyclictest0-21swapper/212:11:372
472199140,1cyclictest0-21swapper/211:32:182
4717991413,1cyclictest241rcuc/107:40:011
4717991410,3cyclictest7079-21chrt07:15:001
471799140,1cyclictest0-21swapper/109:44:121
471799140,14cyclictest0-21swapper/109:52:051
471599145,6cyclictest12107-21apt-get07:25:130
4715991414,0cyclictest0-21swapper/011:12:060
4715991410,3cyclictest32762-21sed11:45:150
472499139,3cyclictest9479-21wc08:25:183
472499139,3cyclictest31674-21tr08:05:163
472499139,3cyclictest17165-21head08:40:223
472499139,2cyclictest25363-21cut12:00:173
472499137,5cyclictest12892-21apache212:21:063
4724991310,2cyclictest8675-21head09:35:173
4724991310,2cyclictest858-21tr12:05:163
4724991310,2cyclictest22634-21wc11:00:223
4724991310,2cyclictest15263-21head07:30:163
4724991310,2cyclictest12023-21wc07:25:003
472499130,12cyclictest0-21swapper/310:46:053
472199139,3cyclictest11851-21sleep07:25:012
472199136,6cyclictest0-21swapper/210:28:002
4721991310,2cyclictest26367-21cron07:55:002
471799139,3cyclictest16635-21cron07:35:001
471799139,3cyclictest1491-21sed09:30:281
4717991310,3cyclictest38150irq/131-snd_hda08:25:181
4717991310,2cyclictest9432-21cron07:20:001
4717991310,2cyclictest24442-21cut11:40:181
4717991310,2cyclictest20911-21grep07:40:261
471799130,2cyclictest0-21swapper/111:12:431
471799130,2cyclictest0-21swapper/109:25:091
471799130,12cyclictest650-21grep08:05:281
471599139,3cyclictest25917-21grep09:45:210
471599139,3cyclictest21404-21cron07:45:000
471599136,6cyclictest30589-21/usr/sbin/munin09:10:210
4715991310,2cyclictest3988-21uname10:30:180
4715991310,2cyclictest30401-21sed11:05:170
4715991310,2cyclictest24064-21cat11:20:260
4715991310,2cyclictest24064-21cat11:20:260
4715991310,2cyclictest18732-21grep12:35:160
4715991310,2cyclictest15434-21cat11:35:130
4715991310,2cyclictest15272-21grep07:30:150
471599130,7cyclictest16835-21perf12:15:000
471599130,13cyclictest0-21swapper/011:51:160
472499129,2cyclictest8401-21dump-pmu-power09:55:003
472499129,2cyclictest5697-21grep07:10:173
472499129,2cyclictest31160-21ssh09:29:493
472499129,2cyclictest28103-21tr09:05:143
472499129,2cyclictest21427-21grep08:50:163
472499129,2cyclictest11357-21grep08:30:143
472499128,3cyclictest12383-21ssh11:14:153
472499126,3cyclictest9449-21munin-run07:20:003
472499120,11cyclictest0-21swapper/310:04:313
472199129,2cyclictest28063-21wc07:55:282
472199129,2cyclictest28063-21wc07:55:272
472199129,2cyclictest20053-21tail07:40:162
472199129,2cyclictest14262-21cron07:30:002
472199121,10cyclictest0-21swapper/208:35:182
4721991210,1cyclictest28568-21grep09:05:162
471799129,2cyclictest27733-21grep07:55:231
471799129,2cyclictest27733-21grep07:55:221
471799129,2cyclictest26474-21head12:20:161
471799129,2cyclictest22796-21munin-plugin-st08:55:011
471799129,2cyclictest13786-21tr08:35:151
471799129,2cyclictest13674-21wc07:25:261
471799129,2cyclictest10840-21cut07:20:221
471799127,5cyclictest25-21ksoftirqd/112:06:291
471799126,3cyclictest25-21ksoftirqd/111:15:061
471799124,5cyclictest25-21ksoftirqd/111:56:251
471799121,1cyclictest4065-21expr08:15:161
471599129,2cyclictest7204-21wc08:20:200
471599129,2cyclictest31150-21ls08:05:000
471599129,2cyclictest29152-21fgrep09:05:230
471599129,2cyclictest26257-21cut10:05:140
471599129,2cyclictest23016-21tr07:45:200
471599129,2cyclictest18983-21grep08:45:160
471599129,2cyclictest17380-21wc08:40:260
471599129,2cyclictest15257-21tr09:20:140
471599129,2cyclictest15165-21sed08:35:260
471599129,2cyclictest14511-21dump-pmu-power11:35:000
471599129,2cyclictest14492-21systemctl09:20:000
471599129,2cyclictest1384-21grep09:50:150
471599129,2cyclictest11652-21head10:15:200
471599128,3cyclictest21677-21sh11:00:140
471599128,3cyclictest14208-21tr11:15:130
471599124,4cyclictest225-21systemd-journal08:50:010
195102120,0chrt0-21swapper/011:56:250
472499118,2cyclictest853-21ssh09:30:213
472499118,2cyclictest7721-21cat08:20:283
472499118,2cyclictest6078-21cat08:20:003
472499118,2cyclictest30595-21ssh09:48:323
472499118,2cyclictest29524-21sed08:00:153
472499118,2cyclictest26212-21grep10:05:143
472499118,2cyclictest23804-21ssh11:20:223
472499118,2cyclictest23804-21ssh11:20:223
472499118,2cyclictest23696-21chrt07:46:163
472499118,2cyclictest2192-21expr10:10:153
472499118,2cyclictest2164-21taskset08:10:173
472499118,2cyclictest18607-21ssh09:40:433
472499118,2cyclictest17647-21ssh11:35:463
472499118,2cyclictest17457-21df07:35:153
472499118,2cyclictest16809-21ssh10:38:093
472499118,2cyclictest1589-21ssh09:11:293
472499118,2cyclictest13828-21tr08:35:143
472499118,2cyclictest13128-21tr10:55:143
472499118,2cyclictest12301-21ssh10:15:273
472499118,2cyclictest117562sleep309:56:073
472499117,3cyclictest16211-21cat11:55:103
472499116,3cyclictest23448-21expr08:55:153
472199118,2cyclictest6511-21timerwakeupswit07:10:262
472199118,2cyclictest31075-21chrt08:04:132
472199118,2cyclictest19521-21grep08:45:212
472199118,1cyclictest10756-21latency_hist08:30:002
472199111,7cyclictest32-21ksoftirqd/207:35:342
471799118,2cyclictest7403-21grep08:20:231
471799118,2cyclictest2989-21sed08:10:271
471799118,2cyclictest27893-21cut09:05:131
471799118,2cyclictest20757-21ssh12:16:341
471799118,2cyclictest16306-21cat08:40:151
471799110,10cyclictest12079-21ssh11:33:091
471599118,2cyclictest9870-21grep07:20:170
471599118,2cyclictest8548-21ls07:15:210
471599118,2cyclictest7081-21sleep07:15:000
471599118,2cyclictest31204-21grep11:25:140
471599118,2cyclictest3066-21sh12:05:490
471599118,2cyclictest26904-21tr07:55:160
471599118,2cyclictest26904-21tr07:55:160
471599118,2cyclictest26502-21cat09:00:200
471599118,2cyclictest2647-21tune2fs10:10:180
471599118,2cyclictest26398-21sed12:00:260
471599118,2cyclictest24715-21tail08:55:280
471599118,2cyclictest20550-21grep10:40:170
471599118,2cyclictest1952-21entropy08:10:170
471599118,2cyclictest15962-21chrt09:59:190
471599118,2cyclictest15495-21ssh12:33:080
471599118,2cyclictest14915-21ssh10:36:390
471599117,3cyclictest1122-21cat08:10:000
471599113,4cyclictest225-21systemd-journal10:00:230
472499108,1cyclictest28411-21taskset07:55:463
472499108,1cyclictest28411-21taskset07:55:463
472499108,1cyclictest26034-21chrt09:00:173
472499108,1cyclictest24869-21grep07:50:143
472499107,2cyclictest4040-21ssh12:25:443
472499107,2cyclictest19899-21fschecks_count07:40:163
472499107,2cyclictest18957-21cut08:45:163
472499107,2cyclictest12362-21tr07:25:143
472199108,1cyclictest31628-21ls08:05:162
472199108,1cyclictest26769-21sendmail_mailqu09:00:232
472199108,1cyclictest24370-21grep08:55:242
472199107,2cyclictest16244-21cstates08:40:152
472199100,9cyclictest15622-21needreboot07:30:182
471799108,1cyclictest310342sleep108:03:401
471799108,1cyclictest25619-21cut07:50:201
471799108,1cyclictest13221-21seq08:34:281
471799107,2cyclictest25720-21ssh12:00:191
471799101,5cyclictest24277-21sed08:55:231
471799101,3cyclictest25-21ksoftirqd/107:45:121
471599108,1cyclictest8553-21wget08:25:140
471599108,1cyclictest17123-21ls07:35:150
471599108,1cyclictest16561-21grep09:40:160
471599107,2cyclictest28090-21ssh10:45:120
471599107,2cyclictest26924-21ssh10:25:100
471599107,2cyclictest24526-21expr07:50:130
471599107,2cyclictest11454-21expr08:30:140
47179990,4cyclictest0-21swapper/109:00:141
47159996,2cyclictest4125-21expr08:15:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional