You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-07 - 03:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Tue Apr 07, 2026 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
409799287,21cyclictest152491phc2sys23:25:251
410299250,24cyclictest29823-21diskmemload23:59:353
410299250,24cyclictest0-21swapper/300:00:363
4101992523,0cyclictest0-21swapper/221:48:402
4101992520,4cyclictest0-21swapper/221:26:342
410199250,24cyclictest0-21swapper/221:14:282
409599253,1cyclictest62550irq/127-eno122:20:000
4095992525,0cyclictest29823-21diskmemload00:16:390
409599250,24cyclictest0-21swapper/023:54:030
4102992422,1cyclictest0-21swapper/321:35:303
410299241,22cyclictest16588-21cron22:00:013
410299240,24cyclictest0-21swapper/321:53:003
410299240,23cyclictest0-21swapper/323:33:023
410299240,23cyclictest0-21swapper/322:39:383
410299240,1cyclictest29823-21diskmemload21:46:463
4101992423,1cyclictest29823-21diskmemload00:05:562
4101992423,1cyclictest0-21swapper/222:08:422
4101992423,0cyclictest0-21swapper/221:59:122
4101992422,1cyclictest29823-21diskmemload23:13:222
410199240,23cyclictest29823-21diskmemload23:06:192
410199240,23cyclictest29823-21diskmemload22:04:572
410199240,23cyclictest0-21swapper/222:28:282
410199240,23cyclictest0-21swapper/222:21:582
410199240,0cyclictest29823-21diskmemload00:14:072
4097992422,1cyclictest32077-21cron23:50:011
409799242,1cyclictest7406-21ssh23:58:471
409799240,23cyclictest31823-21sh22:47:201
409799240,0cyclictest0-21swapper/123:12:531
4095992423,1cyclictest29823-21diskmemload23:30:530
4095992423,1cyclictest29823-21diskmemload21:27:370
4095992423,1cyclictest17995-21sh23:36:400
4095992423,0cyclictest0-21swapper/023:28:500
4095992422,1cyclictest62550irq/127-eno122:48:590
4095992421,2cyclictest31960-21apt22:15:000
409599241,1cyclictest0-21swapper/020:05:000
409599240,24cyclictest0-21swapper/021:59:250
409599240,24cyclictest0-21swapper/021:13:050
409599240,23cyclictest14902-21perf22:30:000
409599240,0cyclictest0-21swapper/021:17:340
4102992323,0cyclictest0-21swapper/321:16:013
4102992323,0cyclictest0-21swapper/300:10:523
4102992321,2cyclictest9780-21rm21:22:553
4102992321,1cyclictest0-21swapper/322:59:033
4102992321,1cyclictest0-21swapper/321:30:223
410299231,21cyclictest15679-21apt-get23:03:573
410299230,2cyclictest29328-21sh23:48:223
410299230,23cyclictest0-21swapper/323:40:423
410299230,23cyclictest0-21swapper/322:28:283
410299230,23cyclictest0-21swapper/322:21:583
410299230,22cyclictest32096-21sh00:21:433
410299230,22cyclictest30452-21sh21:43:153
410299230,22cyclictest29823-21diskmemload22:50:533
410299230,22cyclictest0-21swapper/322:18:163
410299230,1cyclictest0-21swapper/300:15:093
410299230,0cyclictest0-21swapper/323:09:153
410299230,0cyclictest0-21swapper/322:47:503
4101992323,0cyclictest29823-21diskmemload23:04:442
4101992323,0cyclictest0-21swapper/222:36:142
4101992323,0cyclictest0-21swapper/221:16:012
4101992322,1cyclictest29823-21diskmemload23:39:332
4101992321,1cyclictest0-21swapper/222:11:022
4101992321,1cyclictest0-21swapper/221:44:342
4101992320,2cyclictest3474-21cron00:25:012
4101992320,2cyclictest19172-21rm21:31:402
4101992319,0cyclictest0-21swapper/200:26:572
410199230,23cyclictest29823-21diskmemload22:32:002
410199230,23cyclictest0-21swapper/223:56:552
410199230,23cyclictest0-21swapper/223:40:422
410199230,23cyclictest0-21swapper/222:18:162
410199230,22cyclictest7297-21sh21:51:012
410199230,1cyclictest29823-21diskmemload00:36:242
410199230,0cyclictest0-21swapper/223:51:092
410199230,0cyclictest0-21swapper/200:00:362
4097992323,0cyclictest0-21swapper/123:51:221
4097992323,0cyclictest0-21swapper/123:05:061
4097992323,0cyclictest0-21swapper/123:02:091
4097992321,1cyclictest32060-21ssh00:21:391
4097992321,1cyclictest0-21swapper/121:24:011
4097992321,1cyclictest0-21swapper/100:38:151
4097992320,2cyclictest12544-21latency_hist20:35:001
409799230,22cyclictest653-21ssh22:16:301
409799230,22cyclictest25253-21ssh00:14:191
409799230,22cyclictest23858-21sh23:42:451
409799230,22cyclictest12234-21ssh21:55:511
409799230,22cyclictest11283-21ssh00:01:391
409799230,22cyclictest0-21swapper/123:30:151
409799230,0cyclictest29823-21diskmemload22:38:511
409799230,0cyclictest0-21swapper/122:57:321
409799230,0cyclictest0-21swapper/121:46:431
409799230,0cyclictest0-21swapper/121:33:381
4095992323,0cyclictest0-21swapper/022:58:520
4095992323,0cyclictest0-21swapper/000:11:120
4095992322,1cyclictest29823-21diskmemload23:10:470
4095992322,1cyclictest0-21swapper/022:52:220
4095992322,1cyclictest0-21swapper/021:40:260
4095992322,1cyclictest0-21swapper/021:32:180
4095992321,1cyclictest8215-21sh00:29:220
4095992321,1cyclictest18541-21ssh00:39:450
4095992321,1cyclictest0-21swapper/023:47:010
4095992321,1cyclictest0-21swapper/000:05:010
409599230,23cyclictest29823-21diskmemload21:38:450
409599230,23cyclictest0-21swapper/021:52:210
409599230,22cyclictest5369-21ssh22:20:440
409599230,22cyclictest29823-21diskmemload22:30:090
409599230,22cyclictest21566-21ssh22:04:510
409599230,21cyclictest0-21swapper/022:06:310
409599230,0cyclictest0-21swapper/023:04:260
4102992222,0cyclictest0-21swapper/323:39:333
4102992222,0cyclictest0-21swapper/322:31:253
4102992222,0cyclictest0-21swapper/321:29:313
4102992222,0cyclictest0-21swapper/300:36:243
4102992220,1cyclictest0-21swapper/323:16:093
4102992220,1cyclictest0-21swapper/300:31:103
4102992219,2cyclictest6479-21awk19:15:003
410299220,22cyclictest0-21swapper/323:20:543
410299220,22cyclictest0-21swapper/323:11:153
410299220,22cyclictest0-21swapper/322:04:573
410299220,21cyclictest1674-21sh23:52:483
410299220,21cyclictest0-21swapper/323:25:383
410299220,21cyclictest0-21swapper/300:26:173
410299220,21cyclictest0-21swapper/300:05:503
410299220,1cyclictest7759-21perf20:25:013
410299220,1cyclictest28011-21apt-get21:08:583
4101992222,0cyclictest0-21swapper/223:30:582
4101992222,0cyclictest0-21swapper/221:35:302
4101992220,1cyclictest13702-21awk19:30:012
4101992219,2cyclictest8896-21cat19:20:002
410199220,21cyclictest8304-21apt-get20:28:572
410199220,21cyclictest29823-21diskmemload22:56:022
410199220,21cyclictest29823-21diskmemload00:15:192
410199220,21cyclictest24648-21grep21:00:012
410199220,21cyclictest0-21swapper/223:17:022
410199220,21cyclictest0-21swapper/222:50:562
410199220,21cyclictest0-21swapper/221:08:592
410199220,21cyclictest0-21swapper/219:48:562
410199220,1cyclictest9607-21ssh00:31:122
410199220,1cyclictest0-21swapper/221:20:122
410199220,1cyclictest0-21swapper/220:35:002
410199220,0cyclictest29823-21diskmemload22:47:502
410199220,0cyclictest0-21swapper/223:25:422
410199220,0cyclictest0-21swapper/223:20:542
4097992222,0cyclictest0-21swapper/122:05:331
4097992222,0cyclictest0-21swapper/121:39:191
4097992222,0cyclictest0-21swapper/100:09:011
4097992221,1cyclictest16876-21sh22:00:021
4097992220,2cyclictest8774-21/usr/sbin/munin22:24:061
4097992220,1cyclictest17322-21cron20:45:001
4097992220,1cyclictest17156-21ssh23:35:151
409799220,22cyclictest0-21swapper/122:33:151
409799220,22cyclictest0-21swapper/121:53:291
409799220,21cyclictest29823-21diskmemload22:26:341
409799220,21cyclictest13248-21ssh21:25:201
409799220,21cyclictest0-21swapper/100:16:121
409799220,1cyclictest2957-21rm21:15:071
409799220,1cyclictest27794-21ssh22:11:091
4095992222,0cyclictest0-21swapper/023:20:010
4095992222,0cyclictest0-21swapper/000:00:010
4095992221,1cyclictest29107-21sh23:15:100
4095992221,1cyclictest26104-21ssh22:41:190
4095992221,1cyclictest26104-21ssh22:41:190
4095992221,1cyclictest0-21swapper/021:45:070
409599220,22cyclictest0-21swapper/021:23:490
409599220,21cyclictest7038-21apt-get19:18:570
409599220,21cyclictest6467-21sh23:57:240
409599220,21cyclictest22550-21sh23:40:430
409599220,21cyclictest18490-21basename19:40:010
409599220,21cyclictest0-21swapper/019:33:560
409599220,1cyclictest5360-21awk20:20:010
409599220,1cyclictest19405-21ssh23:06:410
409599220,0cyclictest0-21swapper/022:36:590
409599220,0cyclictest0-21swapper/000:24:120
4102992120,1cyclictest13695-21perf19:30:013
4102992119,1cyclictest14910-21cron20:40:003
4102992119,1cyclictest0-21swapper/319:20:003
410299210,21cyclictest0-21swapper/322:40:453
410299210,21cyclictest0-21swapper/322:40:453
410299210,1cyclictest30306-21sh21:10:213
410299210,1cyclictest25748-21munin-run19:55:003
410299210,1cyclictest12533-21grep20:35:003
410299210,0cyclictest0-21swapper/322:11:323
410299210,0cyclictest0-21swapper/320:28:573
4101992119,1cyclictest6484-21cat19:15:002
4101992119,1cyclictest13102-21apt-get20:38:582
4101992118,2cyclictest313-21systemd-journal20:20:012
4101992117,4cyclictest0-21swapper/223:45:082
410199210,21cyclictest0-21swapper/222:40:452
410199210,21cyclictest0-21swapper/222:40:452
410199210,20cyclictest14247-21apt-get19:33:562
4097992121,0cyclictest29823-21diskmemload00:31:231
4097992121,0cyclictest0-21swapper/120:58:591
4097992120,1cyclictest26795-21ssh22:42:211
4097992120,1cyclictest26795-21ssh22:42:211
409799210,21cyclictest31627-21ssh21:12:231
409799210,0cyclictest0-21swapper/122:50:131
409799210,0cyclictest0-21swapper/100:25:121
4095992120,0cyclictest15517-21apt-get20:43:570
4095992119,1cyclictest20905-21munin-run19:45:000
409599210,0cyclictest25616-21apt-get21:03:570
409599210,0cyclictest0-21swapper/020:28:570
409599210,0cyclictest0-21swapper/019:53:570
409599210,0cyclictest0-21swapper/019:13:570
4102992018,1cyclictest5327-21grep20:20:013
410299200,0cyclictest31103-21apt-get20:08:573
410299200,0cyclictest0-21swapper/319:33:563
4101992020,0cyclictest0-21swapper/220:23:562
4101992018,1cyclictest28-21ksoftirqd/220:00:002
410199200,20cyclictest0-21swapper/220:08:572
4097992020,0cyclictest31103-21apt-get20:08:571
4097992020,0cyclictest0-21swapper/120:30:011
4097992019,0cyclictest0-21swapper/119:43:571
409799200,20cyclictest0-21swapper/120:15:011
4095992020,0cyclictest1076-21apt-get20:13:580
4095992020,0cyclictest0-21swapper/020:23:560
4102991919,0cyclictest0-21swapper/320:48:563
4102991919,0cyclictest0-21swapper/319:58:583
4102991919,0cyclictest0-21swapper/319:48:563
4102991918,0cyclictest0-21swapper/321:03:573
4101991919,0cyclictest19059-21apt-get19:43:572
4101991919,0cyclictest0-21swapper/221:03:572
4101991919,0cyclictest0-21swapper/220:48:562
4097991919,0cyclictest0-21swapper/119:38:571
4097991917,1cyclictest0-21swapper/119:30:011
4102991818,0cyclictest0-21swapper/320:13:593
4102991818,0cyclictest0-21swapper/319:43:573
4102991818,0cyclictest0-21swapper/319:23:573
4101991818,0cyclictest0-21swapper/220:13:592
4101991818,0cyclictest0-21swapper/219:23:572
4097991818,0cyclictest0-21swapper/121:03:571
4097991818,0cyclictest0-21swapper/120:03:571
4097991817,1cyclictest5909-21apt-get20:23:561
4095991818,0cyclictest0-21swapper/019:48:560
4095991817,1cyclictest11851-21apt-get19:28:570
4095991815,2cyclictest22143-21sleep20:55:000
4097991717,0cyclictest0-21swapper/120:18:571
4097991717,0cyclictest0-21swapper/119:18:571
4102991616,0cyclictest0-21swapper/320:53:593
4101991616,0cyclictest0-21swapper/220:53:592
4101991616,0cyclictest0-21swapper/219:53:572
410199160,15cyclictest17342-21ls20:45:002
4097991616,0cyclictest0-21swapper/120:38:581
4097991615,1cyclictest28125-21perf20:00:001
4097991614,1cyclictest6216-21seq19:14:061
4095991616,0cyclictest0-21swapper/020:33:570
4095991614,1cyclictest619-21polkitd20:50:000
4095991614,1cyclictest24657-21perf21:00:010
4095991614,1cyclictest14931-21dump-pmu-power20:40:000
4095991614,1cyclictest0-21swapper/019:59:070
410299150,14cyclictest0-21swapper/319:35:463
410199150,14cyclictest0-21swapper/219:39:062
4097991514,1cyclictest17911-21apt-get20:48:571
410299140,14cyclictest0-21swapper/320:40:053
410299140,14cyclictest0-21swapper/320:00:573
410199140,14cyclictest0-21swapper/220:00:522
4097991414,0cyclictest0-21swapper/119:45:331
409799140,14cyclictest0-21swapper/121:06:451
409799140,14cyclictest0-21swapper/120:51:061
409799140,14cyclictest0-21swapper/119:30:121
4095991414,0cyclictest0-21swapper/021:05:310
409599140,14cyclictest0-21swapper/020:05:360
409599140,14cyclictest0-21swapper/019:20:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional