You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-26 - 03:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri Jun 26, 2026 00:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2501521160,2sleep00-21swapper/022:11:130
290932900,2sleep30-21swapper/322:16:083
2694999891,74cyclictest0-21swapper/023:15:190
2694999883,68cyclictest0-21swapper/021:55:350
2694999881,74cyclictest0-21swapper/023:32:270
2694999881,74cyclictest0-21swapper/023:32:270
2694999871,72cyclictest0-21swapper/021:35:230
2694999861,71cyclictest7305-21latency_hist19:45:090
2694999861,66cyclictest10085-21/usr/sbin/munin21:15:310
241742840,4sleep324177-21latency20:30:283
2694999827,42cyclictest0-21swapper/022:45:210
249072820,5sleep22695199cyclictest21:32:222
26949998133,20cyclictest0-21swapper/019:25:350
287212800,2sleep00-21swapper/022:55:300
2694999792,59cyclictest0-21swapper/020:30:080
2694999782,66cyclictest21139-21unixbench_multi20:20:350
2694999782,15cyclictest0-21swapper/020:50:210
2694999782,15cyclictest0-21swapper/020:50:210
2694999772,64cyclictest0-21swapper/022:03:300
2694999766,62cyclictest0-21swapper/000:10:320
2694999761,3cyclictest0-21swapper/023:12:260
136072760,2sleep22695199cyclictest20:00:322
2694999752,56cyclictest0-21swapper/021:10:250
26949997524,38cyclictest0-21swapper/000:05:210
26949997521,35cyclictest0-21swapper/023:29:250
26949997520,42cyclictest7950irq/56-dwc_otg_23:46:040
2694999751,62cyclictest0-21swapper/023:35:180
26949997513,45cyclictest29934-21fschecks_count00:15:260
2694999743,18cyclictest7369-21/usr/sbin/munin19:45:310
2694999731,19cyclictest0-21swapper/020:10:260
2694999731,19cyclictest0-21swapper/020:10:260
26949997311,4cyclictest6793-21rm22:29:310
34072720,2sleep339-21ksoftirqd/323:04:473
34072720,2sleep339-21ksoftirqd/323:04:473
2695099727,47cyclictest7950irq/56-dwc_otg_20:10:281
2695099727,47cyclictest7950irq/56-dwc_otg_20:10:281
2694999721,59cyclictest0-21swapper/022:32:210
26949997213,45cyclictest0-21swapper/019:59:110
2695099716,45cyclictest0-21swapper/123:55:251
26950997112,40cyclictest0-21swapper/120:50:191
26950997112,40cyclictest0-21swapper/120:50:191
2694999713,61cyclictest0-21swapper/022:16:550
2694999713,53cyclictest0-21swapper/020:45:260
2694999713,12cyclictest0-21swapper/023:01:230
2694999713,12cyclictest0-21swapper/023:01:230
2694999711,58cyclictest0-21swapper/023:05:340
2694999711,58cyclictest0-21swapper/000:26:140
2694999702,56cyclictest27354-21aten_rbpower_vo20:40:200
2694999701,6cyclictest2605-21unixbench_singl00:20:380
2694999701,3cyclictest0-21swapper/021:26:520
2694999701,15cyclictest26190-21taskset21:34:440
2695199691,54cyclictest2395-21/usr/sbin/munin00:20:362
2694999699,27cyclictest0-21swapper/023:50:190
2694999696,50cyclictest0-21swapper/023:42:440
2694999691,53cyclictest4636-21ssh21:46:450
2695099682,55cyclictest12236-21ssh23:15:211
2695099681,4cyclictest750-21snmpd21:11:131
26949996816,3cyclictest0-21swapper/020:15:320
26949996813,40cyclictest0-21swapper/023:55:340
26949996812,10cyclictest30644-21cut19:20:220
2694999681,16cyclictest0-21swapper/022:41:190
2694999681,14cyclictest13440-21ssh22:36:320
2695199672,50cyclictest32117-21ssh00:17:462
2695199671,51cyclictest12682-21sed23:55:082
2695099679,32cyclictest0-21swapper/122:28:011
2694999676,14cyclictest0-21swapper/023:20:210
2694999671,52cyclictest26293-21ssh22:52:260
26949996711,45cyclictest0-21swapper/022:21:190
2695199662,53cyclictest0-21swapper/200:05:542
2695199662,50cyclictest7950irq/56-dwc_otg_22:18:042
2695199661,49cyclictest5400-21diskmemload23:03:442
2695199661,49cyclictest5400-21diskmemload23:03:442
2695199661,45cyclictest27780-21ssh00:12:252
2695199661,44cyclictest8273-21rm23:49:432
2695199661,40cyclictest4381-21munin-run23:45:082
2695099661,52cyclictest22913-21ssh22:09:561
2695099661,52cyclictest22913-21ssh22:09:561
2695199651,50cyclictest24324-21ntp_states20:30:322
2695099657,46cyclictest28891-21grep23:35:221
2695099654,6cyclictest0-21swapper/119:20:191
2695099652,47cyclictest8050irq/56-dwc_otg_22:15:381
2694999655,46cyclictest4799-21grep19:35:370
2694999652,14cyclictest0-21swapper/019:14:220
2695199642,12cyclictest8050irq/56-dwc_otg_21:25:192
2695199641,49cyclictest21738-21ssh22:46:402
2695199641,48cyclictest0-21swapper/222:26:212
2695199641,46cyclictest848-21ps19:25:322
2695199641,40cyclictest22794-21rm22:09:432
2695199641,40cyclictest22794-21rm22:09:432
2695099642,49cyclictest9152-21ssh21:52:271
26950996411,38cyclictest7950irq/56-dwc_otg_00:13:241
2694999642,16cyclictest0-21swapper/020:35:480
2695299634,7cyclictest7950irq/56-dwc_otg_22:50:233
2695199634,48cyclictest12198-21sshd23:15:182
2695199631,48cyclictest5860-21seq23:06:412
2695199631,43cyclictest8701-21rm00:28:562
2695199631,43cyclictest17689-21ssh21:24:162
2695199631,40cyclictest0-21swapper/220:10:192
2695199631,40cyclictest0-21swapper/220:10:192
26949996327,6cyclictest8050irq/56-dwc_otg_21:05:200
2694999631,52cyclictest0-21swapper/000:30:440
2695199621,40cyclictest27002-21aten_rbpower_vo21:35:232
2695199621,39cyclictest23412-21wc20:30:082
2695199621,39cyclictest13551-21ssh21:57:562
2695199621,38cyclictest17391-21seq22:41:132
2695099627,43cyclictest7950irq/56-dwc_otg_21:15:331
2695099627,42cyclictest19001-21runrttasks23:23:021
2695099626,46cyclictest6213-21ssh21:49:371
2695099624,47cyclictest30840-21ssh22:58:131
2695099624,45cyclictest8179-21ssh23:49:331
2695099624,44cyclictest10429-21ssh23:12:301
2695099624,41cyclictest0-21swapper/100:05:211
2695099624,40cyclictest0-21swapper/121:35:231
2694999622,52cyclictest0-21swapper/019:33:580
2694999622,47cyclictest0-21swapper/020:05:220
2694999621,4cyclictest22666-21ssh22:09:300
2694999621,4cyclictest22666-21ssh22:09:300
26949996213,10cyclictest16021-21ssh21:21:130
2694999621,18cyclictest0-21swapper/000:03:420
2694999621,18cyclictest0-21swapper/000:03:420
26949996210,41cyclictest0-21swapper/019:53:340
2695199611,47cyclictest9073-21rm21:52:192
2695199611,47cyclictest22390-21ssh23:26:412
2695199611,47cyclictest12990-21ssh21:18:192
2695199611,46cyclictest30640-21aten_rbpower_en19:20:222
2695199611,46cyclictest16282-21ssh00:37:412
2695199611,44cyclictest5992-21ssh21:49:132
2695199611,43cyclictest17883-21ssh23:20:582
2695099618,32cyclictest0-21swapper/123:02:151
2695099618,32cyclictest0-21swapper/123:02:151
2695099617,42cyclictest0-21swapper/123:05:291
26949996113,7cyclictest30107-21chrt19:17:110
65462600,2sleep30-21swapper/323:46:343
26952996010,9cyclictest7950irq/56-dwc_otg_21:45:193
2695199603,44cyclictest28718-21sshd20:44:192
2695199602,46cyclictest0-21swapper/221:10:142
2695199602,41cyclictest0-21swapper/200:01:012
2695199602,41cyclictest0-21swapper/200:01:012
2695199601,48cyclictest29322-21sed22:55:392
2695199601,45cyclictest12801-21sed22:35:372
2695199601,38cyclictest29584-21/usr/sbin/munin23:35:322
2695099606,41cyclictest23429-21basename22:10:221
2695099605,43cyclictest8224-21/usr/sbin/munin19:45:321
2695099603,4cyclictest7850irq/56-dwc_otg21:00:081
2695099603,4cyclictest7850irq/56-dwc_otg21:00:081
2695099601,7cyclictest18761-21diskstats21:25:251
2694999602,15cyclictest0-21swapper/020:33:210
2694999601,14cyclictest1683-21ssh21:43:550
2694999601,14cyclictest16516-21rm00:38:060
2694999601,13cyclictest0-21swapper/021:53:370
2695199595,43cyclictest8212-21/usr/sbin/munin19:45:322
2695199591,46cyclictest0-21swapper/223:12:042
2695199591,44cyclictest0-21swapper/200:30:362
2695199591,38cyclictest611-21ssh22:20:492
2695199591,35cyclictest7145-21latency_hist19:45:082
2695099598,32cyclictest0-21swapper/122:34:141
2695099595,5cyclictest0-21swapper/120:05:091
2695099591,4cyclictest0-21swapper/122:35:351
2694999591,55cyclictest14531-21latency_hist20:05:090
2695299589,16cyclictest0-21swapper/300:25:213
2695199581,47cyclictest16777-21ssh22:01:172
2695199581,37cyclictest25626-21taskset22:12:202
2695199581,33cyclictest0-21swapper/223:34:542
2695199581,33cyclictest0-21swapper/223:34:542
2695099586,40cyclictest0-21swapper/123:25:111
2695099584,39cyclictest26288-21ssh22:52:251
2695099582,7cyclictest671-21sh22:20:551
2695099581,6cyclictest12264-21users21:55:371
2695099581,4cyclictest0-21swapper/119:25:331
2695199575,41cyclictest12883-21ssh23:55:112
2695199571,39cyclictest5172-21chrt19:39:312
2695199571,39cyclictest10363-21/usr/sbin/munin19:50:372
2695199571,38cyclictest9725-21ssh22:32:192
2695099574,3cyclictest0-21swapper/100:18:121
2695099572,4cyclictest0-21swapper/100:00:511
2695099572,4cyclictest0-21swapper/100:00:511
2694999571,52cyclictest0-21swapper/021:01:190
266292573,19sleep00-21swapper/019:09:270
266292573,19sleep00-21swapper/019:09:270
2695299566,13cyclictest8050irq/56-dwc_otg_21:20:203
26952995611,11cyclictest8050irq/56-dwc_otg_00:15:203
2695199561,36cyclictest21539-21chrt20:24:482
2695099561,43cyclictest0-21swapper/121:41:061
2695299552,8cyclictest7850irq/56-dwc_otg00:20:203
26951995523,30cyclictest3591-21latency_hist21:05:082
2695199551,44cyclictest301372chrt19:17:332
2695199551,39cyclictest32711-21ssh21:41:202
2695199551,38cyclictest32232-21sshd20:53:032
2695199551,38cyclictest32232-21sshd20:53:032
2695199551,36cyclictest0-21swapper/219:11:322
2695199551,34cyclictest5121-21chrt21:06:582
2695099556,4cyclictest0-21swapper/123:34:491
2695099556,4cyclictest0-21swapper/123:34:491
2695099552,6cyclictest0-21swapper/100:20:391
26950995525,11cyclictest8050irq/56-dwc_otg_23:50:191
2695099551,9cyclictest8050irq/56-dwc_otg_23:42:511
2695099551,5cyclictest25211-21ssh21:32:561
2695099551,5cyclictest15966-21ssh21:21:071
2695099551,4cyclictest16113-21ssh00:37:221
2695099551,4cyclictest0-21swapper/120:25:341
2694999552,36cyclictest0-21swapper/020:55:240
2694999552,36cyclictest0-21swapper/020:55:240
2695099541,6cyclictest1725650ssh22:40:581
2695099541,44cyclictest23339-21rm22:49:371
2695099541,42cyclictest0-21swapper/100:26:391
2695299533,8cyclictest8050irq/56-dwc_otg_20:20:183
2695199534,40cyclictest7950irq/56-dwc_otg_20:18:532
2695199531,32cyclictest0-21swapper/222:52:232
2695099532,5cyclictest11679-21ssh00:31:491
2695099531,10cyclictest8050irq/56-dwc_otg_22:03:431
2695199521,32cyclictest26863-21runrttasks20:38:372
2695099525,43cyclictest0-21swapper/121:10:081
26950995213,36cyclictest18448-21fschecks_count20:15:251
2695199511,37cyclictest1015-21zcat20:55:312
2695199511,37cyclictest1015-21zcat20:55:312
2695199511,37cyclictest0-21swapper/220:07:372
2695099519,5cyclictest3791-21cut21:05:081
2695099514,5cyclictest0-21swapper/119:15:201
2695099512,6cyclictest23657-21wget20:30:221
2695099511,4cyclictest0-21swapper/120:21:421
2695199501,36cyclictest3122-21taskset19:31:292
2695099504,6cyclictest125-21systemd-journal19:45:081
2695099501,8cyclictest727-21runrttasks20:05:251
2695099501,7cyclictest4817-21grep19:35:371
2695099501,7cyclictest0-21swapper/119:34:161
2695099501,6cyclictest27037-21wget19:10:211
2695099501,5cyclictest0-21swapper/120:46:231
2695199491,33cyclictest0-21swapper/220:49:562
2695199491,32cyclictest0-21swapper/219:56:192
2695099499,4cyclictest11609-21missed_timers19:55:281
2695099492,6cyclictest27342-21aten_rbpower_vo20:40:201
2695099491,7cyclictest10318-21sed19:50:361
2695099481,7cyclictest25461-21grep20:35:221
2695299476,25cyclictest25491-21ssh00:10:233
2695299457,24cyclictest0-21swapper/320:40:193
26952994524,18cyclictest727-21runrttasks20:50:173
26952994524,18cyclictest727-21runrttasks20:50:173
2695299447,24cyclictest0-21swapper/321:55:213
2695299441,12cyclictest0-21swapper/319:10:253
26952994313,19cyclictest0-21swapper/319:50:193
2695299427,23cyclictest0-21swapper/323:50:203
2695299427,22cyclictest0-21swapper/320:55:203
2695299427,22cyclictest0-21swapper/320:55:203
2695299422,9cyclictest0-21swapper/322:25:213
2695299417,23cyclictest0-21swapper/320:25:213
2695299413,31cyclictest8050irq/56-dwc_otg_21:40:193
2695299405,23cyclictest26802-21ssh21:35:153
2695299404,23cyclictest17172-21ssh00:00:273
2695299404,23cyclictest17172-21ssh00:00:273
2695299381,16cyclictest0-21swapper/322:40:203
26952993716,5cyclictest1014850wget21:15:223
2695299367,16cyclictest0-21swapper/319:35:223
2695299366,6cyclictest7850irq/56-dwc_otg23:55:253
2695299361,6cyclictest21499-21diskstats00:05:263
56882350,5sleep35693-21apt-key21:10:093
2695299359,14cyclictest0-21swapper/322:30:293
2695299357,15cyclictest0-21swapper/323:40:153
2695299355,21cyclictest5770-21ssh21:10:133
2695299351,4cyclictest8050irq/56-dwc_otg_22:35:193
2695299351,25cyclictest3618-21ls21:05:083
26952993512,4cyclictest20498-21missed_timers22:45:303
2695299341,4cyclictest0-21swapper/322:00:483
2695299341,27cyclictest14947-21ls20:05:263
2695299333,5cyclictest37-21rcuc/319:30:283
2695299332,4cyclictest0-21swapper/323:10:283
26952993318,4cyclictest727-21runrttasks23:05:333
2695299331,7cyclictest3130-21ssh22:25:093
26952993317,5cyclictest30466-21sh22:57:313
26952993314,4cyclictest5400-21diskmemload21:30:293
26952993310,13cyclictest23353-21apt-get22:10:213
2695299323,21cyclictest13508-21latency_hist00:35:083
2695299321,19cyclictest0-21swapper/319:21:193
26952993210,10cyclictest0-21swapper/322:05:213
26952993210,10cyclictest0-21swapper/322:05:213
253362326,14sleep10-21swapper/119:05:231
253362326,14sleep10-21swapper/119:05:231
2695299318,6cyclictest15030-21sh23:18:203
2695299317,20cyclictest0-21swapper/320:15:213
2695299317,13cyclictest8050irq/56-dwc_otg_19:25:133
2695299316,14cyclictest0-21swapper/319:45:203
2695299311,6cyclictest10632-21munin-run21:55:083
2695299311,18cyclictest27031-21ls20:40:083
2695299303,18cyclictest0-21swapper/323:31:463
2695299303,18cyclictest0-21swapper/323:31:463
2695299303,11cyclictest8050irq/56-dwc_otg_21:25:253
2695299302,18cyclictest0-21swapper/320:12:213
2695299302,18cyclictest0-21swapper/320:12:213
26952993019,8cyclictest32729-21latency_hist23:40:083
256212306,14sleep20-21swapper/219:05:282
256212306,14sleep20-21swapper/219:05:282
2695299298,5cyclictest39-21ksoftirqd/300:35:283
2695299291,3cyclictest0-21swapper/323:23:363
2695299286,6cyclictest8050irq/56-dwc_otg_19:40:213
26952992822,4cyclictest20603-21aten_rbpower_po23:25:233
2695299281,15cyclictest0-21swapper/320:04:143
2695299276,6cyclictest7950irq/56-dwc_otg_19:55:183
2695299271,5cyclictest29299-21fschecks_count20:45:253
263092263,14sleep30-21swapper/319:05:373
263092263,14sleep30-21swapper/319:05:373
2695299251,6cyclictest29848-21sed19:15:353
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional