You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-20 - 07:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5s.osadl.org (updated Thu Aug 20, 2026 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
552999270,26cyclictest0-21swapper/100:03:051
5529992622,1cyclictest211rcuc/122:36:191
552999260,25cyclictest32758-21diskmemload21:37:421
553999250,1cyclictest0-21swapper/323:47:223
5532992520,4cyclictest0-21swapper/222:36:282
553299250,24cyclictest0-21swapper/223:08:502
553299250,24cyclictest0-21swapper/221:23:062
552699250,24cyclictest0-21swapper/023:17:470
552699250,24cyclictest0-21swapper/000:36:570
5539992422,1cyclictest0-21swapper/322:44:283
5539992421,2cyclictest8212-21perf23:40:013
553999240,23cyclictest32758-21diskmemload22:59:323
553999240,23cyclictest32758-21diskmemload21:53:433
553999240,23cyclictest0-21swapper/322:26:403
5532992421,2cyclictest0-21swapper/221:42:402
553299240,24cyclictest32758-21diskmemload23:37:302
553299240,23cyclictest3482-21ls22:10:102
5529992423,1cyclictest32758-21diskmemload21:50:381
552999240,2cyclictest27663-21sendmail23:00:011
552999240,24cyclictest22-21ksoftirqd/123:12:411
552999240,24cyclictest22-21ksoftirqd/122:29:271
552999240,24cyclictest0-21swapper/122:19:001
552999240,24cyclictest0-21swapper/122:01:071
552999240,23cyclictest0-21swapper/122:40:341
552999240,1cyclictest0-21swapper/123:07:331
552999240,0cyclictest32758-21diskmemload21:55:311
552999240,0cyclictest0-21swapper/123:45:261
5526992423,1cyclictest0-21swapper/000:20:400
5526992422,1cyclictest8400-21/usr/sbin/munin23:10:180
5526992421,2cyclictest0-21swapper/022:45:150
552699240,24cyclictest62550irq/127-eno122:18:280
552699240,24cyclictest32758-21diskmemload22:26:020
552699240,23cyclictest0-21swapper/022:20:280
552699240,0cyclictest0-21swapper/022:51:420
5539992323,0cyclictest32758-21diskmemload22:38:393
5539992323,0cyclictest32758-21diskmemload21:18:203
5539992322,1cyclictest34-21ksoftirqd/300:32:223
5539992322,1cyclictest32758-21diskmemload21:20:023
5539992321,1cyclictest0-21swapper/300:14:463
5539992321,1cyclictest0-21swapper/300:00:203
5539992320,2cyclictest21108-21ssh23:50:133
553999230,2cyclictest8168-21ssh00:06:463
553999230,2cyclictest2629-21sh21:10:213
553999230,23cyclictest0-21swapper/322:15:293
553999230,23cyclictest0-21swapper/322:14:143
553999230,23cyclictest0-21swapper/321:45:013
553999230,23cyclictest0-21swapper/300:18:543
553999230,22cyclictest27632-21sh22:30:323
553999230,22cyclictest25947-21sh23:25:553
553999230,1cyclictest313-21systemd-journal22:10:013
553999230,1cyclictest0-21swapper/323:55:103
553999230,1cyclictest0-21swapper/322:48:463
553999230,0cyclictest3816-21ssh23:06:253
553999230,0cyclictest32758-21diskmemload22:52:283
553999230,0cyclictest0-21swapper/323:10:133
5532992323,0cyclictest0-21swapper/222:01:062
5532992323,0cyclictest0-21swapper/200:27:502
5532992321,1cyclictest28198-21apt-get19:55:092
5532992321,1cyclictest14383-21rm23:15:232
5532992320,2cyclictest32758-21diskmemload23:50:482
5532992320,2cyclictest25569-21ssh21:30:432
5532992320,2cyclictest17600-21ssh21:53:442
553299230,2cyclictest22953-21ssh22:55:122
553299230,2cyclictest20728-21ssh21:26:452
553299230,23cyclictest0-21swapper/223:10:462
553299230,22cyclictest611-21irqbalance00:34:402
553299230,22cyclictest32758-21diskmemload23:46:422
553299230,22cyclictest32527-21ssh22:06:042
553299230,22cyclictest12224-21ssh21:49:012
553299230,22cyclictest0-21swapper/221:11:272
553299230,0cyclictest0-21swapper/223:56:562
553299230,0cyclictest0-21swapper/221:37:222
553299230,0cyclictest0-21swapper/200:13:002
553299230,0cyclictest0-21swapper/200:08:142
5529992323,0cyclictest32758-21diskmemload23:51:421
5529992323,0cyclictest32758-21diskmemload21:45:341
5529992323,0cyclictest0-21swapper/123:17:001
5529992323,0cyclictest0-21swapper/100:35:271
5529992323,0cyclictest0-21swapper/100:32:361
5529992322,1cyclictest22-21ksoftirqd/100:25:401
5529992321,1cyclictest19996-21basename20:45:001
5529992321,1cyclictest0-21swapper/122:53:071
5529992320,2cyclictest22751-21awk19:45:011
552999230,22cyclictest8922-21sh23:40:081
552999230,22cyclictest32758-21diskmemload21:11:101
552999230,22cyclictest30629-21rm22:05:111
552999230,22cyclictest0-21swapper/123:26:381
552999230,1cyclictest0-21swapper/122:30:141
552999230,0cyclictest0-21swapper/121:33:101
552999230,0cyclictest0-21swapper/121:29:081
5526992323,0cyclictest0-21swapper/022:13:230
5526992322,0cyclictest0-21swapper/023:35:070
5526992321,2cyclictest0-21swapper/021:11:270
5526992321,1cyclictest32758-21diskmemload00:33:380
5526992321,1cyclictest12158-21apt-get21:20:090
5526992321,1cyclictest0-21swapper/023:45:520
5526992321,1cyclictest0-21swapper/023:00:570
5526992321,1cyclictest0-21swapper/022:35:010
5526992321,1cyclictest0-21swapper/021:52:300
552699230,23cyclictest32758-21diskmemload21:40:090
552699230,23cyclictest0-21swapper/023:41:160
552699230,23cyclictest0-21swapper/022:07:270
552699230,23cyclictest0-21swapper/021:32:510
552699230,22cyclictest8672-21ssh21:15:410
552699230,22cyclictest11561-21sh00:10:140
552699230,22cyclictest0-21swapper/023:33:370
552699230,22cyclictest0-21swapper/023:33:370
552699230,22cyclictest0-21swapper/021:58:490
552699230,22cyclictest0-21swapper/020:00:080
552699230,1cyclictest22298-21ssh22:55:050
552699230,1cyclictest0-21swapper/021:26:320
552699230,1cyclictest0-21swapper/000:17:550
5539992222,0cyclictest24468-21ssh22:00:043
5539992220,1cyclictest30853-21ntp_states23:30:153
5539992220,1cyclictest30853-21ntp_states23:30:153
5539992220,1cyclictest14577-21apt-get22:20:093
553999220,21cyclictest8960-21apt-get23:40:093
553999220,21cyclictest32758-21diskmemload23:16:503
553999220,21cyclictest2992-21apt-get20:10:093
553999220,21cyclictest26941-21rm21:32:393
553999220,21cyclictest22079-21sh00:20:043
553999220,21cyclictest2166-21apt-get21:40:103
553999220,21cyclictest20232-21ssh23:20:403
553999220,21cyclictest15754-21apt-get19:30:103
553999220,21cyclictest15754-21apt-get19:30:103
553999220,21cyclictest0-21swapper/300:35:053
553999220,1cyclictest27822-21apt-get00:25:103
553999220,1cyclictest24781-21latency_hist20:55:003
553999220,1cyclictest0-21swapper/321:25:163
553999220,0cyclictest0-21swapper/323:01:323
5532992222,0cyclictest32758-21diskmemload00:00:582
5532992222,0cyclictest0-21swapper/223:41:432
5532992222,0cyclictest0-21swapper/222:54:272
5532992222,0cyclictest0-21swapper/200:20:072
5532992220,1cyclictest8147-21apt-get19:15:102
5532992220,1cyclictest15423-21ssh22:20:132
5532992219,2cyclictest8679-21rm22:42:332
553299220,22cyclictest32758-21diskmemload22:15:452
553299220,22cyclictest0-21swapper/200:17:572
553299220,21cyclictest8987-21ssh21:16:042
553299220,21cyclictest32758-21diskmemload23:00:082
553299220,21cyclictest29996-21apt-get21:05:022
553299220,21cyclictest15497-21munin-run19:30:002
553299220,21cyclictest0-21swapper/223:30:182
553299220,21cyclictest0-21swapper/223:30:182
553299220,21cyclictest0-21swapper/221:56:302
553299220,21cyclictest0-21swapper/200:37:142
553299220,1cyclictest32758-21diskmemload22:30:382
553299220,1cyclictest0-21swapper/222:45:472
5529992222,0cyclictest0-21swapper/123:01:471
5529992222,0cyclictest0-21swapper/121:44:361
5529992222,0cyclictest0-21swapper/100:10:091
5529992221,1cyclictest22-21ksoftirqd/121:20:361
5529992220,1cyclictest0-21swapper/123:20:231
552999220,21cyclictest6564-21/usr/sbin/munin00:05:201
552999220,21cyclictest32758-21diskmemload23:30:171
552999220,21cyclictest32758-21diskmemload23:30:171
552999220,21cyclictest32758-21diskmemload22:10:031
552999220,21cyclictest0-21swapper/123:35:361
552999220,21cyclictest0-21swapper/122:45:471
552999220,21cyclictest0-21swapper/119:50:081
552999220,21cyclictest0-21swapper/100:20:051
552999220,1cyclictest7566-21sendmail-msp20:20:001
552999220,1cyclictest27202-21apt-get21:00:001
552999220,1cyclictest11041-21sh21:18:571
552999220,0cyclictest0-21swapper/122:23:021
552999220,0cyclictest0-21swapper/119:35:011
552999220,0cyclictest0-21swapper/119:35:011
552999220,0cyclictest0-21swapper/100:18:281
5526992222,0cyclictest0-21swapper/000:01:090
5526992221,1cyclictest32758-21diskmemload00:08:270
5526992221,1cyclictest0-21swapper/021:00:100
5526992220,1cyclictest32758-21diskmemload00:25:120
552699220,22cyclictest0-21swapper/021:47:450
552699220,22cyclictest0-21swapper/021:38:100
552699220,21cyclictest20313-21apt-get23:50:100
552699220,21cyclictest1193-21apt-get23:05:090
552699220,21cyclictest0-21swapper/021:05:020
552699220,21cyclictest0-21swapper/020:30:100
552699220,1cyclictest28709-21ssh22:31:530
552699220,1cyclictest23407-21apt-get19:45:080
552699220,1cyclictest13085-21turbostat.cron19:25:000
5539992120,0cyclictest0-21swapper/321:05:103
5539992119,1cyclictest22746-21turbostat.cron19:45:013
5539992119,1cyclictest10603-21apt-get20:25:093
553999210,20cyclictest25056-21apt-get20:55:083
553999210,0cyclictest0-21swapper/319:15:103
5532992121,0cyclictest0-21swapper/223:20:142
5532992120,1cyclictest28-21ksoftirqd/222:25:122
5532992119,1cyclictest25802-21apt-get19:50:092
5532992119,1cyclictest23796-21/usr/sbin/munin23:25:092
553299210,20cyclictest30612-21apt-get20:00:072
553299210,20cyclictest20267-21apt-get20:45:092
5529992121,0cyclictest0-21swapper/119:15:101
5529992120,0cyclictest0-21swapper/120:20:101
552999210,21cyclictest0-21swapper/120:30:101
552999210,0cyclictest0-21swapper/123:55:191
552999210,0cyclictest0-21swapper/119:25:091
5526992121,0cyclictest0-21swapper/022:40:520
5526992121,0cyclictest0-21swapper/020:15:100
5526992120,1cyclictest0-21swapper/023:25:290
5526992119,1cyclictest10277-21cron19:20:000
552699210,20cyclictest0-21swapper/019:40:090
5539992020,0cyclictest5388-21apt-get20:15:103
5539992019,0cyclictest27202-21apt-get21:00:023
553999200,19cyclictest32755-21munin-run20:05:003
5532992020,0cyclictest0-21swapper/220:55:082
553299200,0cyclictest0-21swapper/221:00:022
553299200,0cyclictest0-21swapper/219:30:102
553299200,0cyclictest0-21swapper/219:30:102
5529992020,0cyclictest0-21swapper/121:05:101
5529992020,0cyclictest0-21swapper/120:50:081
5529992020,0cyclictest0-21swapper/119:45:081
5529992017,2cyclictest30325-21turbostat20:00:001
5526992020,0cyclictest0-21swapper/020:50:080
5526992020,0cyclictest0-21swapper/019:10:130
552699200,1cyclictest28165-21/usr/sbin/munin19:55:110
5539991919,0cyclictest0-21swapper/319:55:093
5539991919,0cyclictest0-21swapper/319:35:073
5532991919,0cyclictest0-21swapper/219:10:132
5532991918,1cyclictest0-21swapper/220:25:112
5529991919,0cyclictest0-21swapper/121:00:101
5529991919,0cyclictest0-21swapper/120:45:091
5529991919,0cyclictest0-21swapper/120:05:111
5529991919,0cyclictest0-21swapper/119:20:021
5526991918,1cyclictest5128-21perf20:15:000
552699190,18cyclictest0-21swapper/020:20:100
5539991818,0cyclictest0-21swapper/319:50:083
5539991818,0cyclictest0-21swapper/319:20:093
5539991818,0cyclictest0-21swapper/319:10:123
5532991818,0cyclictest0-21swapper/220:40:092
5532991817,1cyclictest20619-21apt-get19:40:092
5532991816,1cyclictest5124-21dump-pmu-power20:15:002
5529991818,0cyclictest0-21swapper/119:10:131
5529991816,1cyclictest0-21swapper/120:10:081
5526991816,1cyclictest27187-21sh21:00:000
552699180,0cyclictest0-21swapper/020:05:110
5539991717,0cyclictest13356-21apt-get19:25:093
5539991715,1cyclictest16896-21cat20:35:173
553999170,16cyclictest8447-21expr20:20:123
5532991717,0cyclictest0-21swapper/220:20:102
552999170,16cyclictest22-21ksoftirqd/120:00:071
5526991717,0cyclictest0-21swapper/019:55:000
5526991717,0cyclictest0-21swapper/019:35:070
553999160,15cyclictest17872-21apt-get20:40:093
5532991616,0cyclictest10295-21apt-get19:20:022
5532991616,0cyclictest0-21swapper/219:35:072
5532991615,0cyclictest22659-21apt-get20:50:082
5529991614,2cyclictest11783-21awk20:25:161
5526991616,0cyclictest0-21swapper/020:35:080
5526991614,1cyclictest22408-21munin-plugin-st20:50:000
552699160,15cyclictest9938-21apt-get20:25:020
553999150,5cyclictest0-21swapper/320:30:103
553999150,14cyclictest793-21lldpd20:09:573
5532991514,1cyclictest0-21swapper/219:45:022
553299150,10cyclictest15400-21apt-get20:35:082
5529991514,1cyclictest22-21ksoftirqd/120:35:091
5529991513,1cyclictest20312-21sendmail19:40:011
552699150,5cyclictest0-21swapper/020:40:080
5539991414,0cyclictest0-21swapper/320:45:393
553999140,14cyclictest0-21swapper/319:47:343
5532991414,0cyclictest0-21swapper/220:05:112
553299140,0cyclictest0-21swapper/220:30:252
553299140,0cyclictest0-21swapper/220:15:392
5526991412,1cyclictest15938-21cpuspeed_turbos19:30:100
5526991412,1cyclictest15938-21cpuspeed_turbos19:30:100
552699140,0cyclictest0-21swapper/019:29:370
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional