You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 03:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Tue Nov 18, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2757122920,2sleep1221ktimersoftd/119:10:041
1768421720,5sleep22779299cyclictest22:49:232
277909911714,99cyclictest101ktimersoftd/021:30:330
2779099982,65cyclictest0-21swapper/020:15:160
95782900,1sleep30-21swapper/322:38:293
2779099907,54cyclictest0-21swapper/022:25:190
2779099886,62cyclictest16048-21latency_hist22:05:040
27992810,5sleep12779199cyclictest23:54:281
2779099792,63cyclictest0-21swapper/000:05:160
300512780,2sleep30-21swapper/323:05:223
2779099777,15cyclictest0-21swapper/021:40:040
27790997510,49cyclictest29879-21grep00:30:270
31932740,2sleep30-21swapper/321:47:493
2779099745,55cyclictest6973-21wc00:00:040
2779099723,39cyclictest29841-21ssh23:05:180
2779099715,54cyclictest0-21swapper/022:10:360
2779099706,47cyclictest28054-21fschecks_count19:10:230
2779099705,46cyclictest29926-21grep20:45:200
27790996948,11cyclictest0-21swapper/021:15:180
2779099693,47cyclictest5754-21grep21:05:340
2779099692,52cyclictest2734-21aten_rbpower_vo19:30:190
2779099692,52cyclictest2734-21aten_rbpower_vo19:30:190
2779099692,52cyclictest13408-21cat23:25:280
2779099692,46cyclictest29930-21uname21:40:250
2779099692,46cyclictest0-21swapper/019:20:040
319612680,2sleep20-21swapper/220:50:232
2779199685,10cyclictest20054-21ssh22:10:171
2779099686,45cyclictest8925-21/usr/sbin/munin23:20:240
2779099685,48cyclictest6120-21latency_hist19:40:040
2779099685,48cyclictest6120-21latency_hist19:40:040
2779299672,16cyclictest7750irq/56-dwc_otg20:10:172
2779299672,16cyclictest7750irq/56-dwc_otg20:10:172
2779299671,19cyclictest7950irq/56-dwc_otg_23:05:092
2779199673,12cyclictest7850irq/56-dwc_otg_19:40:351
2779199673,11cyclictest3044-21ssh22:30:171
2779099673,49cyclictest32370-21fschecks_count23:50:250
2779099669,44cyclictest0-21swapper/019:45:250
2779099663,48cyclictest0-21swapper/023:10:190
2779099663,45cyclictest0-21swapper/023:40:360
2779099662,52cyclictest24619-21grep20:30:210
274702662,14sleep00-21swapper/019:09:270
2779099652,50cyclictest0-21swapper/000:28:380
27790996512,22cyclictest0-21swapper/020:35:190
2779199643,5cyclictest7850irq/56-dwc_otg_00:05:281
27791996411,13cyclictest23102-21unixbench_singl00:20:331
2779099647,45cyclictest14946-21cut00:10:230
2779099647,45cyclictest14946-21cut00:10:230
2779099641,54cyclictest0-21swapper/019:55:100
2779199634,9cyclictest7750irq/56-dwc_otg23:45:181
2779199633,12cyclictest1146-21snmpd20:00:171
2779099636,45cyclictest23342-21/usr/sbin/munin22:55:350
2779199621,6cyclictest7850irq/56-dwc_otg_20:45:291
2779099629,43cyclictest4284-21taskset21:04:250
2779099629,43cyclictest4284-21taskset21:04:250
2779099626,44cyclictest20578-21cut23:35:190
2779099626,17cyclictest0-21swapper/020:10:040
2779099623,37cyclictest0-21swapper/000:15:340
2779099621,52cyclictest7334-21unixbench_multi19:40:350
2779099621,45cyclictest28257-21fschecks_count20:40:220
2779299612,15cyclictest7850irq/56-dwc_otg_21:00:172
2779299612,15cyclictest7850irq/56-dwc_otg_21:00:172
2779199617,5cyclictest7750irq/56-dwc_otg00:16:131
2779199614,4cyclictest7950irq/56-dwc_otg_19:10:341
27791996130,28cyclictest1120-21runrttasks23:15:351
27791996118,5cyclictest22216-21aten_rbpower_en22:55:201
2779199611,16cyclictest7950irq/56-dwc_otg_22:05:201
2779099617,44cyclictest0-21swapper/020:25:040
2779099611,44cyclictest0-21swapper/021:10:210
2779399602,14cyclictest0-21swapper/320:35:193
27791996015,8cyclictest1420-21ssh22:27:291
2779099608,35cyclictest19864-21latency_hist22:10:040
2779099605,43cyclictest0-21swapper/000:38:490
2779099604,48cyclictest0-21swapper/020:55:290
2779099603,46cyclictest0-21swapper/022:45:230
2779099602,43cyclictest0-21swapper/000:24:520
27790996021,20cyclictest12089-21latency_hist22:00:040
27790996021,20cyclictest12089-21latency_hist22:00:040
2779099601,45cyclictest1146-21snmpd23:19:520
2779199591,6cyclictest28926-21ssh00:30:041
2779199591,14cyclictest0-21swapper/123:20:581
2779099599,32cyclictest2222-21ssh21:45:470
2779099597,44cyclictest17507-21chrt21:25:030
2779199582,5cyclictest7061-21/usr/sbin/munin21:10:281
2779199582,4cyclictest4494-21sshd23:14:411
27791995817,5cyclictest18103-21ssh22:50:041
2779199581,5cyclictest7750irq/56-dwc_otg23:40:351
27791995813,11cyclictest7950irq/56-dwc_otg_22:20:171
2779099588,18cyclictest21831-21latency_hist22:55:050
2779099583,46cyclictest0-21swapper/023:45:310
2779099581,5cyclictest7624-21ntpq22:35:290
2779399572,31cyclictest25049-21cat23:40:293
2779299578,14cyclictest7850irq/56-dwc_otg_00:30:152
2779199571,23cyclictest18166-21timerwakeupswit20:10:331
2779199571,23cyclictest18166-21timerwakeupswit20:10:331
2779099578,41cyclictest0-21swapper/023:03:560
2779099577,39cyclictest0-21swapper/022:41:130
2779099576,38cyclictest3763-21runrttasks22:30:280
2779099572,44cyclictest0-21swapper/022:15:360
2779099571,40cyclictest0-21swapper/021:51:150
2779099571,39cyclictest0-21swapper/019:50:190
244892570,1sleep20-21swapper/220:30:192
44882560,2sleep30-21swapper/323:14:403
2779199565,4cyclictest7950irq/56-dwc_otg_23:10:041
2779199561,5cyclictest7850irq/56-dwc_otg_23:25:331
2779099561,47cyclictest17326-21ls20:10:200
2779099561,47cyclictest17326-21ls20:10:200
2779399556,16cyclictest0-21swapper/321:15:183
2779199557,4cyclictest0-21swapper/123:33:461
2779199556,4cyclictest0-21swapper/123:36:321
2779199554,4cyclictest7850irq/56-dwc_otg_00:30:341
2779199551,4cyclictest0-21swapper/123:01:151
2779099557,34cyclictest0-21swapper/022:21:470
2779099552,39cyclictest32542-21grep19:20:350
2779099551,41cyclictest0-21swapper/021:25:210
2779199545,4cyclictest0-21swapper/121:25:261
2779199541,9cyclictest12836-21ssh21:18:231
2779099542,41cyclictest0-21swapper/023:30:310
130542540,1sleep30-21swapper/322:42:443
2779199534,8cyclictest0-21swapper/119:20:181
2779199534,7cyclictest9202-21users19:45:361
2779199534,3cyclictest0-21swapper/100:04:551
2779199532,10cyclictest7850irq/56-dwc_otg_21:57:201
2779199532,10cyclictest7850irq/56-dwc_otg_21:57:201
2779199531,3cyclictest0-21swapper/121:35:301
2779099533,2cyclictest0-21swapper/020:50:180
2779199527,4cyclictest7750irq/56-dwc_otg21:40:211
2779199525,5cyclictest7750irq/56-dwc_otg21:45:291
2779199522,10cyclictest0-21swapper/100:37:221
2779099527,37cyclictest0-21swapper/019:25:170
2779099520,40cyclictest9691-21chrt00:02:520
2779199518,4cyclictest0-21swapper/100:13:071
2779199518,4cyclictest0-21swapper/100:13:071
2779099515,36cyclictest0-21swapper/020:25:230
2779299502,4cyclictest7850irq/56-dwc_otg_22:40:202
2779199501,5cyclictest7850irq/56-dwc_otg_20:40:041
2779299497,25cyclictest12352-21ssh22:00:102
2779199484,4cyclictest7750irq/56-dwc_otg22:00:061
2779199481,15cyclictest0-21swapper/121:34:531
2779099488,30cyclictest0-21swapper/020:01:410
2779399477,24cyclictest0-21swapper/320:55:223
2779399474,27cyclictest15723-21ssh21:21:253
2779199472,4cyclictest7750irq/56-dwc_otg23:55:281
2779199471,12cyclictest7750irq/56-dwc_otg20:40:211
2779399469,14cyclictest5125-21apt-get23:15:173
2779399466,25cyclictest0-21swapper/319:55:093
27793994619,12cyclictest7950irq/56-dwc_otg_19:10:283
2779299467,25cyclictest0-21swapper/221:50:202
2779399455,25cyclictest0-21swapper/321:42:093
2779399452,23cyclictest10022-21ssh23:20:333
2779399451,3cyclictest7950irq/56-dwc_otg_20:20:173
2779399451,26cyclictest25623-21rm21:35:213
2779299457,23cyclictest0-21swapper/221:30:182
2779299456,7cyclictest7950irq/56-dwc_otg_23:15:172
2779299455,4cyclictest7950irq/56-dwc_otg_22:35:192
2779199457,4cyclictest7750irq/56-dwc_otg20:08:321
2779199452,8cyclictest0-21swapper/120:25:181
2779199450,15cyclictest0-21swapper/122:37:431
2779399447,24cyclictest0-21swapper/323:35:173
2779299447,23cyclictest0-21swapper/219:10:192
2779199447,35cyclictest20531-21latency_hist20:20:041
2779199447,23cyclictest0-21swapper/120:50:191
2779199447,23cyclictest0-21swapper/119:30:181
2779199447,23cyclictest0-21swapper/119:30:181
2779399436,22cyclictest11104-21ssh00:05:183
2779399435,17cyclictest7950irq/56-dwc_otg_21:10:043
27793994324,9cyclictest7950irq/56-dwc_otg_23:03:433
27793994314,14cyclictest381ktimersoftd/320:15:173
27793994313,12cyclictest381ktimersoftd/321:29:383
2779299437,24cyclictest0-21swapper/220:20:182
2779199437,5cyclictest0-21swapper/119:15:221
2779199437,23cyclictest0-21swapper/121:00:191
2779199437,23cyclictest0-21swapper/121:00:191
177882430,6sleep22779299cyclictest21:25:092
2779399427,24cyclictest0-21swapper/320:40:183
27793994234,6cyclictest381ktimersoftd/300:00:173
27793994223,11cyclictest7950irq/56-dwc_otg_22:50:173
27793994218,9cyclictest7950irq/56-dwc_otg_00:30:263
27793994215,10cyclictest7950irq/56-dwc_otg_23:48:323
2779299427,23cyclictest0-21swapper/223:30:172
2779299427,22cyclictest0-21swapper/220:15:182
2779399416,19cyclictest5818-21sshd19:38:113
2779399416,19cyclictest5818-21sshd19:38:113
27793994113,15cyclictest7950irq/56-dwc_otg_22:13:503
2779399410,14cyclictest18000-21ssh22:50:003
2779299418,23cyclictest0-21swapper/220:25:192
2779299417,22cyclictest0-21swapper/219:55:092
2779299411,5cyclictest7750irq/56-dwc_otg21:20:172
27793994024,8cyclictest7950irq/56-dwc_otg_22:06:003
27793994020,8cyclictest7950irq/56-dwc_otg_20:26:373
2779399400,22cyclictest6780-21ssh23:59:453
2779399396,13cyclictest4049-21ssh22:30:333
2779399396,13cyclictest2260-21ssh23:53:223
2779399391,25cyclictest22835-21ls00:20:313
2779299397,24cyclictest0-21swapper/200:10:172
2779299397,24cyclictest0-21swapper/200:10:172
2779399386,13cyclictest23452-21users22:55:353
27793993819,4cyclictest13395-21ssh23:25:283
27793993810,12cyclictest32678-21taskset19:21:093
2779399380,11cyclictest1146-21snmpd21:57:073
2779399380,11cyclictest1146-21snmpd21:57:073
2779299388,4cyclictest7950irq/56-dwc_otg_00:05:162
2779299387,20cyclictest0-21swapper/200:20:172
27792993811,6cyclictest7850irq/56-dwc_otg_00:00:042
27791993814,7cyclictest140-21systemd-journal19:35:121
27791993814,7cyclictest140-21systemd-journal19:35:121
2779399378,16cyclictest0-21swapper/322:00:103
2779399375,17cyclictest7950irq/56-dwc_otg_20:01:483
2779399373,5cyclictest9175-21grep19:45:353
27793993713,13cyclictest7950irq/56-dwc_otg_21:10:243
27792993715,10cyclictest20747-21sh22:10:272
2779299371,25cyclictest10769-21timerandwakeup19:50:332
2779299371,22cyclictest13787-21ls21:20:042
2779199379,15cyclictest7950irq/56-dwc_otg_22:40:211
27791993722,4cyclictest15292-21uptime21:20:341
303402360,2sleep20-21swapper/219:15:282
2779399369,18cyclictest29879-21sh22:22:123
2779399367,25cyclictest0-21swapper/321:00:193
2779399367,25cyclictest0-21swapper/321:00:193
27793993619,8cyclictest7950irq/56-dwc_otg_22:16:163
27793993617,12cyclictest7950irq/56-dwc_otg_23:34:503
27793993610,14cyclictest19430-21grep00:15:343
2779399360,18cyclictest7750irq/56-dwc_otg20:10:163
2779399360,18cyclictest7750irq/56-dwc_otg20:10:163
2779299368,19cyclictest0-21swapper/221:35:182
2779299367,4cyclictest7850irq/56-dwc_otg_23:00:032
27792993619,5cyclictest32566-21sh21:44:232
27792993617,6cyclictest3213850wget23:50:222
2779299361,26cyclictest0-21swapper/222:22:062
2779299361,16cyclictest7850irq/56-dwc_otg_20:50:042
2779199367,23cyclictest0-21swapper/122:15:221
27793993514,5cyclictest381ktimersoftd/300:25:113
27793993513,9cyclictest7950irq/56-dwc_otg_19:50:153
27793993513,4cyclictest31603-21/usr/sbin/munin20:50:313
2779299352,5cyclictest7750irq/56-dwc_otg20:40:162
2779299351,20cyclictest3533-21latency_hist00:40:052
2779399346,4cyclictest6305-21ssh21:51:203
2779399342,7cyclictest7950irq/56-dwc_otg_20:50:053
27793993410,14cyclictest7950irq/56-dwc_otg_22:27:453
27793993410,11cyclictest16098-21taskset00:10:553
27793993410,11cyclictest16098-21taskset00:10:553
2779299342,6cyclictest7950irq/56-dwc_otg_20:05:232
27792993421,4cyclictest11637-21sh21:59:232
27792993421,4cyclictest11637-21sh21:59:232
27792993417,4cyclictest1685-21/usr/sbin/munin21:45:292
27792993410,3cyclictest0-21swapper/222:30:542
2779199341,7cyclictest7750irq/56-dwc_otg19:55:241
245922340,5sleep32779399cyclictest20:30:203
2779399335,12cyclictest2329-21taskset19:29:403
2779399333,15cyclictest0-21swapper/321:32:093
2779299334,20cyclictest23605-21ssh23:39:182
2779299333,18cyclictest7750irq/56-dwc_otg00:02:212
27792993317,8cyclictest7950irq/56-dwc_otg_23:10:082
27792993317,8cyclictest6410-21sh21:10:072
2779299331,5cyclictest0-21swapper/222:25:202
2779299331,3cyclictest7950irq/56-dwc_otg_22:15:202
27792993312,4cyclictest0-21swapper/223:00:232
2779199336,7cyclictest7850irq/56-dwc_otg_20:20:221
2779199332,14cyclictest7950irq/56-dwc_otg_22:50:211
27791993312,5cyclictest4663-21/usr/sbin/munin21:50:201
2779199331,13cyclictest7750irq/56-dwc_otg21:05:331
2779399325,12cyclictest7950irq/56-dwc_otg_00:36:253
2779399321,21cyclictest0-21swapper/319:40:203
2779299327,15cyclictest0-21swapper/219:45:212
2779299325,13cyclictest7850irq/56-dwc_otg_23:25:202
27792993223,6cyclictest4180-21idleruntime-cro19:35:042
27792993223,6cyclictest4180-21idleruntime-cro19:35:042
2779299321,22cyclictest1930-21timerandwakeup20:55:352
2779199328,4cyclictest7750irq/56-dwc_otg20:55:331
27793993110,4cyclictest31203-21latency_hist19:20:043
2779299313,14cyclictest7950irq/56-dwc_otg_22:50:322
2779299311,4cyclictest7850irq/56-dwc_otg_20:35:202
2779299311,20cyclictest18869-21missed_timers00:15:272
2779199313,5cyclictest7750irq/56-dwc_otg19:55:041
27791993117,10cyclictest0-21swapper/120:30:261
259582313,20sleep20-21swapper/219:05:082
2779399306,14cyclictest0-21swapper/320:05:193
27793993021,6cyclictest3735-21unixbench_multi19:30:323
27793993021,6cyclictest3735-21unixbench_multi19:30:323
2779299309,4cyclictest0-21swapper/223:20:322
2779299309,10cyclictest0-21swapper/200:26:092
2779299305,5cyclictest7850irq/56-dwc_otg_19:20:292
2779299302,7cyclictest24819-21if_eth023:40:272
2779299302,19cyclictest0-21swapper/221:09:362
27792993013,5cyclictest6433-21df_inode19:40:232
27792993011,5cyclictest28758-21latency23:45:252
2779299291,13cyclictest0-21swapper/222:05:172
2779299281,4cyclictest7850irq/56-dwc_otg_19:25:222
2779199281,17cyclictest0-21swapper/119:25:071
258342283,12sleep30-21swapper/319:05:053
2779299272,7cyclictest13535-21/usr/sbin/munin20:00:162
2779299252,8cyclictest5548-21unixbench_singl19:35:332
2779299252,8cyclictest5548-21unixbench_singl19:35:332
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional