You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-28 - 08:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Sat Mar 28, 2026 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
569321770,6sleep32671199cyclictest19:40:313
2670899962,73cyclictest12411-21/usr/sbin/munin20:00:240
2670899941,74cyclictest16082-21taskset23:17:110
2670899941,74cyclictest16082-21taskset23:17:110
26708999413,68cyclictest14190-21sh21:58:030
2670899921,68cyclictest31491-21grep19:20:390
2670899911,77cyclictest17423-21/usr/sbin/munin22:40:220
2670899902,71cyclictest0-21swapper/021:05:340
2670899891,77cyclictest744-21ssh22:20:280
2670899891,67cyclictest0-21swapper/020:35:250
2670899881,72cyclictest27142-21aten_rbpower_cu00:10:230
2670899871,68cyclictest23149-21ps23:25:340
2670899861,70cyclictest3408-21rm23:01:450
26708998526,47cyclictest22809-21latency_hist00:05:120
2670899851,72cyclictest5155-21rm21:47:090
2670899851,69cyclictest0-21swapper/000:15:540
18742850,2sleep20-21swapper/222:20:442
2670899841,69cyclictest0-21swapper/000:21:240
2670899841,62cyclictest6359-21ssh22:26:070
276732830,2sleep00-21swapper/023:30:400
2670899831,74cyclictest6343-21runrttasks23:45:140
2670899831,70cyclictest4558-21sshd23:42:240
2670899831,70cyclictest4558-21sshd23:42:240
2670899831,67cyclictest956-21ssh22:59:520
2670899831,67cyclictest956-21ssh22:59:520
2670899831,66cyclictest10899-21ssh22:31:480
2670899831,63cyclictest15472-21acpi00:35:210
2670899823,40cyclictest0-21swapper/023:55:210
2670899821,67cyclictest11601-21ssh21:16:320
261082820,9sleep1221ktimersoftd/122:50:311
2670899811,68cyclictest0-21swapper/021:30:380
2670899811,68cyclictest0-21swapper/000:29:320
2670899801,71cyclictest13509-21chrt23:53:380
2670899801,67cyclictest0-21swapper/022:09:130
2670899801,63cyclictest18308-21apt-get21:25:210
2670899791,69cyclictest29780-21ssh21:38:270
2670899791,68cyclictest0-21swapper/022:37:340
2670899791,58cyclictest17192-21rm22:00:510
2670899783,65cyclictest15727-21rm21:21:160
2670899781,65cyclictest0-21swapper/019:55:220
2670899781,64cyclictest1788-21wget21:00:250
244532780,2sleep10-21swapper/122:48:411
2670899771,67cyclictest20866-21rm23:23:100
2670899771,67cyclictest0-21swapper/022:17:380
2670899771,67cyclictest0-21swapper/000:32:330
2670899771,65cyclictest29136-21taskset22:54:390
2670899771,65cyclictest15936-21/usr/sbin/munin20:10:340
2670899771,63cyclictest26851-21aten_rbpower_vo19:10:240
2670899771,61cyclictest28218-21seq22:15:110
2670899771,61cyclictest28218-21seq22:15:110
2670899762,64cyclictest0-21swapper/023:14:210
2670899761,64cyclictest24641-21kworker/u8:021:51:320
2670899751,70cyclictest6258-21/usr/sbin/munin21:10:340
2670899751,62cyclictest5057-21diskmemload23:37:110
2670899751,58cyclictest21806-21grep22:45:280
2670899751,57cyclictest1970-21sh21:44:030
2670899751,55cyclictest29596-21sed19:15:380
2670899741,57cyclictest0-21swapper/000:07:560
2670899741,56cyclictest0-21swapper/020:19:320
2670899741,56cyclictest0-21swapper/020:19:320
173162740,4sleep22671099cyclictest00:36:362
63202730,4sleep17750irq/56-dwc_otg22:26:031
2670899731,64cyclictest10274-21sshd19:55:030
2670899731,60cyclictest0-21swapper/020:21:540
2670899731,57cyclictest0-21swapper/019:49:410
2670899721,60cyclictest7268-21ssh23:06:150
2670899721,60cyclictest0-21swapper/020:49:050
2670899721,60cyclictest0-21swapper/020:49:050
60292710,2sleep20-21swapper/223:05:292
2670899711,57cyclictest0-21swapper/020:51:520
2670899711,57cyclictest0-21swapper/020:51:520
2670899701,56cyclictest24566-21taskset20:33:020
2670899701,56cyclictest0-21swapper/019:39:070
2670899701,54cyclictest0-21swapper/019:25:360
2670899701,54cyclictest0-21swapper/019:25:360
26710996928,11cyclictest17985-21ssh23:59:162
2670899691,58cyclictest0-21swapper/020:41:070
26710996827,7cyclictest28238-21munin-run22:15:112
26710996827,7cyclictest28238-21munin-run22:15:112
2670899681,57cyclictest15344-21runrttasks20:05:480
2670899671,45cyclictest0-21swapper/019:44:420
26710996625,13cyclictest17615-21ssh23:19:592
26710996625,13cyclictest17615-21ssh23:19:592
26708996618,39cyclictest7950irq/56-dwc_otg_21:00:100
26710996528,11cyclictest28069-21rm23:31:112
26710996527,11cyclictest9793-21ssh21:52:502
26710996525,13cyclictest316-21ssh21:41:142
2670899651,56cyclictest22696-21chrt20:27:180
2670899651,51cyclictest0-21swapper/019:31:310
150702650,3sleep3381ktimersoftd/321:59:363
26710996426,13cyclictest28224-21ssh21:35:442
2670999648,10cyclictest7950irq/56-dwc_otg_23:00:281
2670999641,6cyclictest22953-21ssh21:30:291
26710996230,12cyclictest17756-21ssh21:24:482
26710996228,12cyclictest22627-21latency_hist00:05:112
26710996228,11cyclictest23416-21ssh22:09:152
26710996226,13cyclictest18823-21ssh22:03:412
26710996226,12cyclictest13202-21ssh21:19:182
26710996221,23cyclictest7950irq/56-dwc_otg_21:33:272
2670999621,8cyclictest16898-21sh22:40:111
2670999621,7cyclictest17722-21ssh23:20:111
2670999621,7cyclictest17722-21ssh23:20:111
201982620,2sleep220201-21missed_timers20:20:312
26710996126,10cyclictest25359-21/usr/sbin/munin20:35:292
2670999612,18cyclictest0-21swapper/122:12:121
2670999612,18cyclictest0-21swapper/122:12:121
26710996028,11cyclictest24819-21ssh22:49:212
26710996028,11cyclictest13912-21ssh00:32:572
26710996027,11cyclictest23811-21ssh23:25:592
26710996020,20cyclictest7950irq/56-dwc_otg_23:20:272
2670999607,7cyclictest7950irq/56-dwc_otg_22:40:361
2670999607,5cyclictest7950irq/56-dwc_otg_23:45:281
26710995928,19cyclictest10894-21ssh22:31:482
26710995928,11cyclictest4940-21ssh00:21:512
26710995927,14cyclictest9272-21rm00:27:022
26710995927,12cyclictest12563-21diskstats20:00:262
26710995925,12cyclictest20712-21ssh21:27:312
26710995923,22cyclictest8150-21ssh21:12:582
26710995921,10cyclictest4140-21ssh23:03:062
2670999598,5cyclictest7750irq/56-dwc_otg00:35:301
26710995827,16cyclictest32091-21ssh22:57:282
26710995827,16cyclictest32091-21ssh22:57:282
26710995827,13cyclictest25339-21ssh00:07:352
26710995826,15cyclictest10393-21ls23:10:322
26710995825,10cyclictest4794-21ssh23:42:502
26710995825,10cyclictest4794-21ssh23:42:502
26710995824,11cyclictest17187-21chrt20:11:102
26710995821,12cyclictest17182-21/usr/sbin/munin22:40:222
26710995821,11cyclictest7361-21ssh23:45:362
26710995817,12cyclictest1786-21ssh00:18:402
2670999589,5cyclictest0-21swapper/122:08:591
2670999583,14cyclictest0-21swapper/119:25:221
2670999583,14cyclictest0-21swapper/119:25:221
308652570,2sleep32671199cyclictest21:40:113
26710995725,11cyclictest10532-21/usr/sbin/munin19:55:222
26710995721,11cyclictest27087-21apt00:10:202
26710995628,8cyclictest31042-21ntp_states19:20:342
26710995624,12cyclictest29363-21/usr/sbin/munin19:15:362
2670999569,4cyclictest0-21swapper/100:17:151
26710995532,16cyclictest7979-21sshd22:29:022
26710995528,11cyclictest26821-21wget19:10:242
26710995426,8cyclictest31051-21ssh22:17:402
26710995426,12cyclictest2119-21if_wlan021:00:292
26710995423,16cyclictest7950irq/56-dwc_otg_20:30:242
26710995423,12cyclictest6786-21ssh21:50:012
26710995422,11cyclictest13711-21ssh23:54:012
2671199537,32cyclictest22613-21ssh21:30:253
26710995325,18cyclictest15346-21taskset20:05:502
26710995322,11cyclictest32751-21ssh23:37:112
26710995224,12cyclictest12776-21sed21:55:392
26710995218,13cyclictest6411-21chrt19:41:432
2670999525,13cyclictest7750irq/56-dwc_otg22:30:231
2670999524,3cyclictest0-21swapper/100:20:211
2670999522,20cyclictest0-21swapper/119:44:541
26711995114,13cyclictest31287-21sed19:20:383
26710995125,11cyclictest28804-21ssh22:54:032
26710995121,10cyclictest19051-21taskset20:16:572
26710995121,10cyclictest19051-21taskset20:16:572
2670999513,8cyclictest0-21swapper/120:00:221
26711995015,17cyclictest17717-21irqstats22:40:273
26710995024,12cyclictest4183-21ntpq21:05:342
26710995022,11cyclictest15689-21ssh22:37:582
2671199489,25cyclictest0-21swapper/322:45:243
26711994811,17cyclictest4393-21ls21:05:373
26710994838,7cyclictest1524-21apt-get19:30:232
2671099481,6cyclictest7950irq/56-dwc_otg_19:45:222
2671199477,27cyclictest6451-21chrt23:45:243
26711994711,15cyclictest22427-21ssh00:04:513
26710994723,13cyclictest614-21threads19:25:362
26710994723,13cyclictest614-21threads19:25:362
26709994720,10cyclictest7750irq/56-dwc_otg21:10:231
2671199465,4cyclictest7750irq/56-dwc_otg23:55:213
26711994625,8cyclictest23185-21apt-get20:30:223
26711994613,8cyclictest25329-21perl20:35:293
26711994613,11cyclictest18207-21ssh21:25:123
26711994610,11cyclictest11183-21latency_hist00:30:113
2671099467,27cyclictest0-21swapper/219:50:242
26710994621,14cyclictest0-21swapper/220:55:012
26710994621,14cyclictest0-21swapper/220:55:012
26710994620,12cyclictest908-21runrttasks20:46:462
26710994620,12cyclictest908-21runrttasks20:46:462
26710994619,12cyclictest7850irq/56-dwc_otg_21:00:112
2670999463,4cyclictest7950irq/56-dwc_otg_21:10:111
26710994522,11cyclictest28115-21taskset20:40:562
26710994521,13cyclictest3929-21cat19:35:322
2670999452,34cyclictest7950irq/56-dwc_otg_23:40:201
2670999452,34cyclictest7950irq/56-dwc_otg_23:40:201
262442452,35sleep00-21swapper/019:07:480
2671199447,23cyclictest0-21swapper/320:00:233
2671199445,23cyclictest7950irq/56-dwc_otg_00:35:283
2671199442,24cyclictest7850irq/56-dwc_otg_22:20:223
26711994414,18cyclictest11995-21rm22:33:493
26710994430,11cyclictest21584-21/usr/sbin/munin20:25:262
2671199437,22cyclictest0-21swapper/323:10:253
2671199431,25cyclictest7750irq/56-dwc_otg23:28:313
2671199427,24cyclictest0-21swapper/321:40:223
2671199424,24cyclictest0-21swapper/319:50:243
2671199421,25cyclictest0-21swapper/300:17:363
26711994211,5cyclictest5802-21cut21:10:293
2670999428,23cyclictest0-21swapper/119:10:231
2670999427,24cyclictest0-21swapper/119:30:251
2670999427,23cyclictest0-21swapper/120:55:231
2670999421,30cyclictest31539-21ssh23:35:371
266672420,2sleep10-21swapper/120:40:111
264762420,5sleep126469-21munin-run19:10:101
2671199419,12cyclictest26405-21ssh22:12:033
2671199419,12cyclictest26405-21ssh22:12:033
2671199417,23cyclictest0-21swapper/320:40:163
2671199417,22cyclictest7950irq/56-dwc_otg_23:00:213
26711994111,20cyclictest32597-21ssh23:36:553
2670999415,8cyclictest32411-21grep21:40:371
2671199406,21cyclictest0-21swapper/322:50:253
2670999403,6cyclictest0-21swapper/120:40:211
4612390,10sleep1221ktimersoftd/122:59:001
4612390,10sleep1221ktimersoftd/122:59:001
2671199398,12cyclictest15492-21ssh22:37:373
2671199395,23cyclictest0-21swapper/320:20:243
2671199395,11cyclictest11253-21ssh00:30:133
2671199391,28cyclictest2176-21ls21:00:303
26711993912,5cyclictest0-21swapper/323:40:223
26711993912,5cyclictest0-21swapper/323:40:223
2671199391,24cyclictest1542-21ls21:00:113
2670999397,23cyclictest0-21swapper/122:20:241
2670999395,10cyclictest7950irq/56-dwc_otg_21:20:381
2671199387,13cyclictest25347-21ssh00:07:363
2671199387,12cyclictest18882-21ssh22:03:473
26711993822,13cyclictest16739-21processes20:10:353
2670999387,4cyclictest7750irq/56-dwc_otg20:30:241
26709993812,6cyclictest23-21ksoftirqd/121:25:281
2671199379,12cyclictest7104-21ssh23:05:573
2671199377,19cyclictest28580-21ssh22:15:233
2671199377,16cyclictest0-21swapper/323:20:393
2671199376,11cyclictest29640-21ssh23:34:043
2671199372,10cyclictest0-21swapper/300:10:483
26711993718,10cyclictest7950irq/56-dwc_otg_21:24:533
26711993714,10cyclictest7067-21cat21:50:123
2671199371,22cyclictest0-21swapper/320:45:263
2671199371,22cyclictest0-21swapper/320:45:263
2670999372,3cyclictest7950irq/56-dwc_otg_19:35:241
26709993717,4cyclictest19470-21/usr/sbin/munin00:00:331
197702370,5sleep12670999cyclictest23:21:101
2671199369,11cyclictest26952-21diskstats19:10:263
2671199368,11cyclictest11665-21taskset21:16:393
2671199367,23cyclictest4786-21ssh22:25:243
2671199366,5cyclictest7750irq/56-dwc_otg19:55:343
2671199365,18cyclictest0-21swapper/300:21:333
2670999365,5cyclictest9824-21rm23:10:251
2670999364,15cyclictest7850irq/56-dwc_otg_20:05:251
26709993616,4cyclictest5057-21diskmemload00:05:311
2670999361,31cyclictest16957-21timerwakeupswit20:10:371
2671199357,14cyclictest7950irq/56-dwc_otg_23:51:183
2671199354,22cyclictest32092-21ssh22:57:293
2671199354,22cyclictest32092-21ssh22:57:293
2670999357,25cyclictest0-21swapper/123:25:221
2670999356,14cyclictest7950irq/56-dwc_otg_21:45:241
2670999354,6cyclictest0-21swapper/100:10:201
2670999353,29cyclictest28549-21ssh22:15:201
2670999353,19cyclictest7950irq/56-dwc_otg_23:05:331
2671199349,15cyclictest0-21swapper/322:05:253
2671199341,22cyclictest15705-21latency_hist20:10:113
26711993410,11cyclictest28530-21awk19:15:223
2670999349,5cyclictest7750irq/56-dwc_otg21:55:331
2670999346,14cyclictest7184-21fschecks_count19:45:271
2670999341,27cyclictest9233-21ls19:50:301
90582331,28sleep10-21swapper/121:51:331
2671199338,7cyclictest0-21swapper/321:52:433
2671199331,22cyclictest0-21swapper/320:18:453
2671199331,22cyclictest0-21swapper/320:18:453
2670999336,16cyclictest0-21swapper/100:25:161
2670999332,5cyclictest0-21swapper/121:35:241
2670999332,18cyclictest0-21swapper/123:30:251
2671199323,20cyclictest13930-21ssh23:15:243
2671199323,20cyclictest13930-21ssh23:15:243
2671199321,7cyclictest3392-21aten_rbpower_en19:35:253
2670999326,5cyclictest0-21swapper/122:00:321
2670999324,6cyclictest7750irq/56-dwc_otg23:50:291
2670999321,19cyclictest0-21swapper/123:55:431
2671199312,14cyclictest7850irq/56-dwc_otg_19:30:293
2671199311,4cyclictest0-21swapper/319:45:293
26711993112,7cyclictest21712-21fschecks_time20:25:293
2670999316,6cyclictest10825-21fschecks_time19:55:291
2670999315,5cyclictest7750irq/56-dwc_otg20:15:381
2670999315,5cyclictest7750irq/56-dwc_otg20:15:381
2670999313,5cyclictest7950irq/56-dwc_otg_20:45:291
2670999313,5cyclictest7950irq/56-dwc_otg_20:45:291
2670999313,5cyclictest12260-21/usr/sbin/munin00:30:361
2671199302,19cyclictest0-21swapper/319:25:273
2671199302,19cyclictest0-21swapper/319:25:273
26711993020,8cyclictest30515-21apt-get20:50:213
26711993020,8cyclictest30515-21apt-get20:50:213
26709993014,6cyclictest4962-1kworker/u9:021:15:251
2670999295,5cyclictest7750irq/56-dwc_otg19:15:301
2670999294,6cyclictest7850irq/56-dwc_otg_21:01:551
2670999292,5cyclictest7950irq/56-dwc_otg_20:25:251
2670999291,12cyclictest7750irq/56-dwc_otg19:20:371
2670999282,10cyclictest19659-21wget20:20:251
2670999281,13cyclictest7950irq/56-dwc_otg_20:50:281
2670999281,13cyclictest7950irq/56-dwc_otg_20:50:281
262982284,16sleep20-21swapper/219:08:302
248732273,15sleep30-21swapper/319:05:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional