You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-05 - 07:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri Sep 05, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2738622810,1sleep10-21swapper/119:05:281
197221030,3sleep123-21ksoftirqd/120:55:241
125722960,5sleep32867599cyclictest21:59:233
134332930,2sleep30-21swapper/322:42:103
2867299922,43cyclictest0-21swapper/021:40:210
2867299856,68cyclictest0-21swapper/023:15:220
18592850,2sleep10-21swapper/123:51:491
218742740,2sleep20-21swapper/220:20:232
218742740,2sleep20-21swapper/220:20:232
315442730,4sleep231519-21ssh23:05:362
2867299739,47cyclictest3767-21apt-get22:30:200
2867299733,44cyclictest0-21swapper/023:25:200
2867299726,50cyclictest22875-21head20:20:360
2867299726,50cyclictest22875-21head20:20:360
28672997230,10cyclictest13293-21ssh22:00:220
28672997230,10cyclictest13293-21ssh22:00:220
2867299722,39cyclictest0-21swapper/023:10:260
2867399716,47cyclictest8150irq/56-dwc_otg_00:25:281
2867399712,45cyclictest18527-21sh22:49:511
2867299714,64cyclictest19040-21ssh21:25:250
2867299714,37cyclictest0-21swapper/019:25:140
143712710,5sleep02867299cyclictest21:19:180
300132700,1sleep30-21swapper/322:20:533
300132700,1sleep30-21swapper/322:20:533
28672997019,11cyclictest0-21swapper/000:20:250
28672997019,11cyclictest0-21swapper/000:20:250
2867399694,45cyclictest8150irq/56-dwc_otg_21:25:391
2867299697,41cyclictest10986-21cut19:50:230
2867299694,52cyclictest0-21swapper/022:35:400
28672996919,16cyclictest3870-21apt-get23:55:150
2867399683,41cyclictest0-21swapper/121:45:201
2867299689,43cyclictest32096-21modprobe19:20:080
2867299689,36cyclictest0-21swapper/019:55:300
2867399672,47cyclictest12508-21ssh21:59:141
2867399671,43cyclictest27425-21ssh23:00:341
2867299677,43cyclictest9-21ksoftirqd/023:35:330
28672996711,43cyclictest28631-21ssh21:37:020
28672996711,43cyclictest28631-21ssh21:37:020
2867399662,37cyclictest8050irq/56-dwc_otg_19:55:351
2867299667,56cyclictest0-21swapper/019:45:230
2867299667,40cyclictest21574-21cat22:10:300
2867299666,50cyclictest405-21cat19:20:300
28672996617,27cyclictest26456-21unixbench_multi20:30:370
28672996614,29cyclictest19376-21aten_rbpower_po00:15:280
28672996611,51cyclictest8325-21latency00:00:310
2867399652,39cyclictest10450-21ssh00:03:321
2867399651,40cyclictest4087-21wc23:55:251
2867399651,40cyclictest21763-21ntp_states22:10:331
2867299659,34cyclictest6679-21head21:05:370
2867299655,45cyclictest4302-21ssh21:48:020
2867399642,42cyclictest8150irq/56-dwc_otg_19:25:391
2867399641,51cyclictest13265-21ssh00:06:161
2867399641,41cyclictest8150irq/56-dwc_otg_19:50:081
2867299645,42cyclictest27311-21wc23:00:320
28672996414,8cyclictest0-21swapper/021:55:210
28672996411,38cyclictest17033-21ssh22:46:440
2867399631,5cyclictest13168-21latency_hist23:25:101
2867399631,53cyclictest23075-21df22:55:261
2867399631,40cyclictest8884-21users22:35:411
2867399631,3cyclictest8150irq/56-dwc_otg_23:06:211
2867299634,41cyclictest19966-21aten_rbpower_en20:15:240
2867399622,45cyclictest0-21swapper/122:41:071
2867399621,48cyclictest31763-21ssh22:24:321
2867399621,48cyclictest31763-21ssh22:24:321
2867299629,35cyclictest61-21kswapd022:05:150
2867299621,49cyclictest797-21ssh23:08:480
2867399613,40cyclictest7950irq/56-dwc_otg00:20:311
2867399613,40cyclictest7950irq/56-dwc_otg00:20:311
2867399611,45cyclictest24089-21ssh23:39:271
2867299614,42cyclictest26527-21/usr/sbin/munin00:25:210
2867399604,45cyclictest8150irq/56-dwc_otg_23:33:551
2867399601,45cyclictest18600-21ssh00:14:291
2867399601,41cyclictest11760-21ssh23:22:431
2867399601,41cyclictest11138-21cut19:50:251
2867399601,38cyclictest3445-21apt22:30:081
2867299603,47cyclictest27947-21taskset23:44:300
2867299602,40cyclictest30291-21wc19:15:080
2867399593,42cyclictest8150irq/56-dwc_otg_23:25:311
2867399592,46cyclictest26810-21ssh23:42:111
2867399592,38cyclictest0-21swapper/122:50:331
2867399591,50cyclictest8435-21ssh21:53:411
2867399591,43cyclictest0-21swapper/121:35:341
2867399591,43cyclictest0-21swapper/121:35:341
2867399591,41cyclictest4347-21/usr/sbin/munin22:30:301
2867299598,43cyclictest0-21swapper/023:47:110
2867299598,27cyclictest0-21swapper/019:40:260
2867299598,26cyclictest2465850ssh22:56:450
2867299594,44cyclictest2491-21ssh00:36:180
2867599582,7cyclictest8050irq/56-dwc_otg_20:45:203
2867399582,42cyclictest25755-21ssh21:34:001
2867399581,42cyclictest3538-21sshd00:39:021
2867399581,39cyclictest3863-21wget21:00:241
2867299589,36cyclictest7119-21diskmemload22:16:050
2867299588,40cyclictest0-21swapper/000:31:070
2867299582,39cyclictest12018-21ssh22:40:300
2867299581,40cyclictest14808-21fschecks_time20:00:270
2867399571,44cyclictest0-21swapper/120:26:181
2867399571,42cyclictest0-21swapper/122:07:461
2867399571,42cyclictest0-21swapper/120:15:231
2867299579,44cyclictest1283-21sed20:50:390
2867299576,48cyclictest0-21swapper/023:50:240
2867299573,42cyclictest0-21swapper/000:11:430
2867299572,38cyclictest912-21runrttasks20:40:090
2867299572,24cyclictest0-21swapper/020:10:200
28672995711,34cyclictest14538-21ssh00:08:530
2867399568,35cyclictest0-21swapper/123:11:201
2867399562,39cyclictest0-21swapper/123:17:261
2867399561,43cyclictest21226-21ssh00:17:091
2867399561,40cyclictest26310-21rm22:16:071
2867399561,37cyclictest0-21swapper/120:01:081
2867399561,34cyclictest0-21swapper/121:22:011
2867299568,39cyclictest4855-21runrttasks19:31:200
2867299567,39cyclictest0-21swapper/021:10:250
2867299567,39cyclictest0-21swapper/021:10:250
2867299564,38cyclictest3478-21seq20:59:580
28672995625,16cyclictest15147-21aten_rbpower_cu21:20:240
2867299561,40cyclictest0-21swapper/022:26:370
2867399552,39cyclictest30661-21ssh00:30:331
2867399551,39cyclictest321-21/usr/sbin/munin19:20:291
2867299557,35cyclictest29720-21ps20:40:370
28672995530,3cyclictest0-21swapper/019:35:190
28672995530,3cyclictest0-21swapper/019:35:190
2867299552,37cyclictest5789-21sshd21:50:270
2867399544,32cyclictest15331-21ssh22:02:121
2867399544,32cyclictest15331-21ssh22:02:121
2867399542,43cyclictest30681-21ssh23:47:141
2867399542,39cyclictest0-21swapper/120:54:241
2867399541,41cyclictest9316-21ssh21:11:461
2867399541,41cyclictest9316-21ssh21:11:461
2867399541,37cyclictest29669-21ntp_states20:40:361
2867399541,35cyclictest0-21swapper/120:20:221
2867399541,35cyclictest0-21swapper/120:20:221
2867299543,48cyclictest0-21swapper/022:21:370
2867299543,48cyclictest0-21swapper/022:21:370
28672995426,4cyclictest8150irq/56-dwc_otg_20:30:080
2867299542,49cyclictest0-21swapper/022:54:570
2867299542,35cyclictest0-21swapper/021:31:320
2867399533,33cyclictest0-21swapper/119:31:221
2867399531,41cyclictest32625-21ssh21:42:271
2867399531,39cyclictest0-21swapper/121:06:031
2867399531,36cyclictest14652-21ssh21:19:541
2867299536,31cyclictest20075-21ssh23:33:560
2867299530,40cyclictest17793-21sshd20:09:280
2867399521,39cyclictest28487-21chrt20:37:281
2867299522,29cyclictest0-21swapper/023:20:360
2867299522,29cyclictest0-21swapper/023:20:360
2867499514,30cyclictest8050irq/56-dwc_otg_19:45:082
2867399511,38cyclictest0-21swapper/120:45:231
2867299511,42cyclictest0-21swapper/021:00:220
2867399507,40cyclictest28805-21aten_rbpower_vo19:10:201
2867399501,39cyclictest0-21swapper/119:36:491
2867399501,39cyclictest0-21swapper/119:36:491
2867399494,34cyclictest7950irq/56-dwc_otg20:05:321
2867399491,7cyclictest25382-21/usr/sbin/munin20:30:231
2867399481,5cyclictest7232-21apt-get19:40:241
2867399481,44cyclictest18404-21grep20:10:261
2867299483,32cyclictest0-21swapper/020:45:480
2867399471,42cyclictest31409-21sed19:15:351
63082460,3sleep3381ktimersoftd/323:57:513
2867599464,8cyclictest8150irq/56-dwc_otg_23:50:223
2867599459,6cyclictest37-21rcuc/321:25:223
2867599448,24cyclictest0-21swapper/319:10:193
282412442,14sleep00-21swapper/019:08:030
28675994318,5cyclictest1698-21munin-plugin-st20:55:083
2867499427,24cyclictest0-21swapper/223:55:142
2867499421,5cyclictest24762-21ssh23:40:202
277182420,2sleep30-21swapper/323:44:023
2867599418,13cyclictest8150irq/56-dwc_otg_00:20:093
28675994117,20cyclictest0-21swapper/319:44:283
2867499417,25cyclictest0-21swapper/220:15:242
189932410,2sleep20-21swapper/222:06:422
2867599406,22cyclictest7950irq/56-dwc_otg00:25:283
2867599401,32cyclictest0-21swapper/319:35:223
2867599401,32cyclictest0-21swapper/319:35:223
2867499407,25cyclictest0-21swapper/223:25:212
28674994013,9cyclictest0-21swapper/223:45:322
2867599396,24cyclictest0-21swapper/320:15:243
2867599396,21cyclictest0-21swapper/322:15:223
2867499397,25cyclictest0-21swapper/219:10:192
2867499397,25cyclictest0-21swapper/200:05:152
2867599381,26cyclictest14899-21ls21:20:093
28675993812,20cyclictest926-21snmpd23:10:243
2867499386,16cyclictest0-21swapper/221:40:232
2867499381,29cyclictest16639-21ls20:05:262
2867499381,20cyclictest8050irq/56-dwc_otg_22:26:252
2867599376,7cyclictest8050irq/56-dwc_otg_23:45:393
28675993718,5cyclictest1640-21cat23:10:093
2867599371,25cyclictest32117-21ls19:20:083
2867499376,3cyclictest7950irq/56-dwc_otg22:35:222
2867499373,4cyclictest7950irq/56-dwc_otg20:25:232
2867499371,4cyclictest8150irq/56-dwc_otg_21:50:242
2867499371,26cyclictest29755-21ls00:30:092
2867599367,24cyclictest0-21swapper/320:20:223
2867599367,24cyclictest0-21swapper/320:20:223
2867599367,18cyclictest0-21swapper/319:55:243
2867599361,8cyclictest12952-21cron23:25:093
2867599361,8cyclictest12952-21cron23:25:093
28675993618,5cyclictest21500-21sh23:35:303
28675993616,5cyclictest21343-21fschecks_time22:10:283
2867599361,18cyclictest0-21swapper/323:00:113
2867499364,3cyclictest0-21swapper/223:10:222
28674993620,5cyclictest20418-21sh22:50:512
28674993620,4cyclictest31-21ksoftirqd/219:55:272
2867599357,5cyclictest7950irq/56-dwc_otg22:55:093
2867599357,14cyclictest8150irq/56-dwc_otg_23:20:093
2867599357,12cyclictest8150irq/56-dwc_otg_21:05:233
28675993521,9cyclictest0-21swapper/322:35:223
2867599351,22cyclictest0-21swapper/319:25:143
2867599351,20cyclictest10638-21latency_hist19:50:093
2867499353,13cyclictest8150irq/56-dwc_otg_21:10:282
2867499353,13cyclictest8150irq/56-dwc_otg_21:10:282
2867499351,8cyclictest21263-21fschecks_count23:35:272
28674993516,4cyclictest24756-21/usr/sbin/munin22:15:292
28674993513,12cyclictest0-21swapper/220:54:352
2867499351,23cyclictest9308-21uniq19:45:272
2867599348,4cyclictest0-21swapper/300:14:233
2867599347,13cyclictest8150irq/56-dwc_otg_00:20:423
2867599347,13cyclictest8150irq/56-dwc_otg_00:20:423
28675993424,8cyclictest3884-21ssh22:30:243
2867499346,11cyclictest8150irq/56-dwc_otg_00:35:262
2867499344,14cyclictest8150irq/56-dwc_otg_21:20:082
2867499342,18cyclictest0-21swapper/220:35:272
2867499342,18cyclictest0-21swapper/200:00:282
2867499341,5cyclictest7278-21sed21:10:092
28674993413,10cyclictest5060-21sh22:30:382
2867499341,22cyclictest31024-21latency19:15:292
2867599339,7cyclictest8150irq/56-dwc_otg_22:05:093
2867599339,7cyclictest8150irq/56-dwc_otg_22:05:093
2867599339,7cyclictest15153-21sh00:10:083
2867599336,4cyclictest0-21swapper/300:35:513
2867599335,6cyclictest8050irq/56-dwc_otg_22:30:093
2867599335,18cyclictest5699-21ssh21:50:263
2867599334,4cyclictest8050irq/56-dwc_otg_22:45:303
2867599332,18cyclictest0-21swapper/321:25:003
2867599332,16cyclictest8150irq/56-dwc_otg_20:25:243
28675993311,12cyclictest0-21swapper/321:34:303
2867499338,14cyclictest9122-21apt23:20:092
2867499332,4cyclictest7950irq/56-dwc_otg19:20:232
2867499331,8cyclictest26295-21ssh23:00:082
2867499331,7cyclictest1024650ssh21:55:342
2867499331,15cyclictest0-21swapper/221:38:392
2867499331,15cyclictest0-21swapper/221:38:392
2867599329,4cyclictest2477-21load21:45:303
2867599328,12cyclictest0-21swapper/322:55:473
2867599327,12cyclictest0-21swapper/300:00:563
2867599321,23cyclictest573-21latency_hist00:35:093
2867599321,19cyclictest11226-21ls19:50:263
2867599321,14cyclictest7950irq/56-dwc_otg23:30:233
2867499326,5cyclictest8050irq/56-dwc_otg_19:30:092
2867499326,24cyclictest0-21swapper/219:30:252
2867499326,13cyclictest2251-21sh21:45:272
2867499326,12cyclictest8050irq/56-dwc_otg_22:00:242
2867499326,12cyclictest8050irq/56-dwc_otg_22:00:242
28674993216,9cyclictest16700-21sh21:21:142
2867499321,4cyclictest7109-21kworker/u8:421:30:212
2867499321,22cyclictest32075-21latency_hist22:25:102
2867499321,22cyclictest32075-21latency_hist22:25:102
2867499321,21cyclictest32665-21cpuspeed23:50:262
28674993212,12cyclictest12623-21unixbench_multi22:40:372
2867599319,4cyclictest0-21swapper/321:40:213
2867599317,3cyclictest0-21swapper/322:06:583
2867599317,11cyclictest8150irq/56-dwc_otg_21:10:293
2867599317,11cyclictest8150irq/56-dwc_otg_21:10:293
2867599316,14cyclictest8050irq/56-dwc_otg_20:05:363
2867599316,14cyclictest0-21swapper/319:30:253
2867499319,4cyclictest0-21swapper/200:13:162
2867499318,5cyclictest30255-21fschecks_time00:30:282
28674993121,8cyclictest8150irq/56-dwc_otg_22:45:262
2867499311,9cyclictest7950irq/56-dwc_otg20:58:192
28674993116,9cyclictest9418-21/usr/sbin/munin23:20:262
28674993116,9cyclictest9418-21/usr/sbin/munin23:20:262
2867499311,18cyclictest21943-21ls22:10:362
2867499311,18cyclictest0-21swapper/220:45:302
2867499311,15cyclictest8050irq/56-dwc_otg_19:35:352
2867499311,15cyclictest8050irq/56-dwc_otg_19:35:352
28674993110,8cyclictest28669-21cyclictest21:00:152
2867599306,5cyclictest8050irq/56-dwc_otg_21:35:223
2867599306,5cyclictest8050irq/56-dwc_otg_21:35:223
2867599301,16cyclictest32432-21aten_rbpower_vo19:20:243
28675993011,4cyclictest27576-21if_wlan020:35:283
2867499308,12cyclictest8050irq/56-dwc_otg_00:20:252
2867499308,12cyclictest8050irq/56-dwc_otg_00:20:252
2867499306,14cyclictest8150irq/56-dwc_otg_19:50:362
2867499302,5cyclictest7950irq/56-dwc_otg20:40:302
2867499302,14cyclictest0-21swapper/221:25:442
28674993015,5cyclictest27084-21latency_hist20:35:092
28674993013,9cyclictest159392chrt20:03:522
2867499301,23cyclictest0-21swapper/220:12:482
2867599296,14cyclictest0-21swapper/323:25:213
2867599295,5cyclictest25658-21fschecks_time20:30:263
2867599291,6cyclictest0-21swapper/321:00:153
2867499298,14cyclictest19267-21ssh00:15:232
2867499295,4cyclictest7950irq/56-dwc_otg23:30:252
2867499291,7cyclictest0-21swapper/223:04:132
2867599282,5cyclictest8050irq/56-dwc_otg_20:00:343
2867599282,19cyclictest0-21swapper/320:10:373
280682287,14sleep30-21swapper/319:05:513
268682286,12sleep20-21swapper/219:05:222
2867599278,9cyclictest8150irq/56-dwc_otg_20:40:313
28675992612,5cyclictest1925-21/usr/sbin/munin20:55:283
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional