You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 00:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot4s.osadl.org (updated Sat Nov 15, 2025 12:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5277662269190,26sleep60-21swapper/607:07:066
5278592234179,44sleep70-21swapper/707:08:267
5277552225162,20sleep40-21swapper/407:06:564
5279412221170,40sleep50-21swapper/507:09:385
5278372219180,26sleep30-21swapper/307:08:063
5278362216184,21sleep20-21swapper/207:08:052
5279712210181,18sleep10-21swapper/107:09:571
5276752206174,20sleep00-21swapper/007:05:460
52820899360,18cyclictest0-21swapper/711:28:057
52820899310,18cyclictest0-21swapper/709:38:167
52820899310,18cyclictest0-21swapper/709:38:157
52820899310,18cyclictest0-21swapper/707:59:247
52820399250,23cyclictest552122-21apt-get07:50:146
52819999250,24cyclictest568849-21perf08:20:005
52820899240,23cyclictest0-21swapper/710:55:017
528203992420,4cyclictest0-21swapper/607:40:006
52820399240,23cyclictest0-21swapper/607:24:596
528199992422,1cyclictest630396-21apt-get10:05:145
52819099240,24cyclictest557445-21sendmail-msp08:00:003
52818899240,22cyclictest0-21swapper/212:30:142
528208992321,1cyclictest0-21swapper/708:05:007
52820899230,22cyclictest647117-21munin-run10:35:007
52820899230,22cyclictest621817-21apt-get09:50:147
52820899230,22cyclictest534560-21basename07:19:597
52820899230,1cyclictest673143-21sendmail-msp11:20:017
528203992321,1cyclictest563554-21apt-get08:10:126
528199992321,1cyclictest0-21swapper/509:30:015
528199992321,1cyclictest0-21swapper/509:30:015
52819999230,22cyclictest652849-21munin-run10:45:005
52819999230,22cyclictest0-21swapper/512:40:015
52819999230,22cyclictest0-21swapper/511:45:135
52819999230,21cyclictest557848-21apt-get08:00:125
528194992322,1cyclictest0-21swapper/409:30:134
528194992322,1cyclictest0-21swapper/409:30:134
528194992321,1cyclictest0-21swapper/411:35:154
52819499230,23cyclictest696321-21basename11:59:594
52819499230,23cyclictest670263-21idleruntime-cro11:15:004
52819499230,1cyclictest2970800-21systemd-journal07:20:004
528190992322,1cyclictest0-21swapper/308:55:143
52819099230,22cyclictest0-21swapper/312:00:133
52819099230,22cyclictest0-21swapper/308:05:143
52819099230,1cyclictest0-21swapper/307:20:153
528185992323,0cyclictest0-21swapper/112:35:161
52818599230,22cyclictest0-21swapper/111:10:011
52818599230,22cyclictest0-21swapper/108:15:141
528183992322,1cyclictest0-21swapper/012:35:160
528183992321,1cyclictest0-21swapper/010:20:000
52818399230,22cyclictest603750-21idleruntime-cro09:20:010
52818399230,22cyclictest560702-21apt-get08:05:130
52818399230,1cyclictest598421-21apt-get09:10:130
528208992221,1cyclictest649994-21awk10:40:007
528208992221,1cyclictest0-21swapper/708:20:017
528208992221,0cyclictest0-21swapper/707:50:147
528208992220,1cyclictest661851-21apt-get11:00:157
528208992220,1cyclictest0-21swapper/709:20:137
52820899220,22cyclictest680536-21ntpq11:30:237
52820899220,22cyclictest621399-21perf09:50:007
52820899220,21cyclictest0-21swapper/712:15:137
52820899220,21cyclictest0-21swapper/709:30:007
52820899220,21cyclictest0-21swapper/709:30:007
52820899220,21cyclictest0-21swapper/708:30:137
52820899220,1cyclictest667816-21apt-get11:10:147
528203992222,0cyclictest0-21swapper/609:50:136
528203992220,1cyclictest551710-21awk07:49:596
52820399220,21cyclictest0-21swapper/611:30:176
52820399220,21cyclictest0-21swapper/610:55:146
52820399220,21cyclictest0-21swapper/610:15:176
528199992221,1cyclictest0-21swapper/511:10:015
528199992221,1cyclictest0-21swapper/508:30:145
528199992220,1cyclictest658974-21apt-get10:55:145
528199992220,1cyclictest0-21swapper/511:55:125
52819999220,21cyclictest685295-21apt-get11:40:135
52819999220,21cyclictest546419-21apt-get07:40:145
52819999220,21cyclictest0-21swapper/511:50:155
52819999220,21cyclictest0-21swapper/511:35:015
52819999220,21cyclictest0-21swapper/511:15:145
52819999220,21cyclictest0-21swapper/509:55:145
52819999220,21cyclictest0-21swapper/508:05:135
52819999220,1cyclictest638572-21basename10:20:005
52819999220,1cyclictest0-21swapper/508:50:005
528194992221,1cyclictest0-21swapper/410:50:134
528194992221,1cyclictest0-21swapper/408:05:144
528194992219,2cyclictest680553-21ntpq11:30:244
52819499220,22cyclictest0-21swapper/409:40:014
52819499220,22cyclictest0-21swapper/409:40:004
52819499220,21cyclictest621379-21sh09:50:004
52819499220,21cyclictest0-21swapper/412:00:134
52819499220,21cyclictest0-21swapper/411:00:164
52819499220,21cyclictest0-21swapper/409:15:004
52819499220,1cyclictest0-21swapper/409:20:024
528190992222,0cyclictest618962-21apt-get09:45:133
52819099220,22cyclictest0-21swapper/312:40:013
52819099220,21cyclictest530942-21munin-run07:15:003
52819099220,21cyclictest0-21swapper/312:15:143
52819099220,21cyclictest0-21swapper/310:05:143
52819099220,1cyclictest621817-21apt-get09:50:123
52819099220,1cyclictest566406-21apt-get08:15:133
52819099220,1cyclictest0-21swapper/311:45:143
52819099220,0cyclictest0-21swapper/312:05:113
528188992221,1cyclictest0-21swapper/208:40:142
528188992220,1cyclictest667176-21systemctl11:10:012
52818899220,22cyclictest609690-21apt-get09:30:022
52818899220,22cyclictest609690-21apt-get09:30:022
52818899220,22cyclictest0-21swapper/211:30:242
52818899220,22cyclictest0-21swapper/209:10:202
52818899220,22cyclictest0-21swapper/208:25:122
52818899220,21cyclictest0-21swapper/209:55:132
528185992221,1cyclictest0-21swapper/108:50:121
52818599220,22cyclictest652849-21munin-run10:45:011
52818599220,22cyclictest624236-21idleruntime-cro09:55:011
52818599220,21cyclictest557848-21apt-get08:00:111
52818599220,21cyclictest0-21swapper/110:20:151
52818599220,21cyclictest0-21swapper/108:10:131
528183992220,2cyclictest603703-21apt-get09:20:020
528183992220,1cyclictest0-21swapper/008:20:010
52818399220,21cyclictest624669-21apt-get09:55:140
52818399220,21cyclictest0-21swapper/009:00:130
52818399220,21cyclictest0-21swapper/008:50:020
52818399220,21cyclictest0-21swapper/008:10:120
52818399220,21cyclictest0-21swapper/007:39:590
52818399220,21cyclictest0-21swapper/007:15:000
528208992121,0cyclictest0-21swapper/710:20:157
528208992121,0cyclictest0-21swapper/708:36:447
528208992120,1cyclictest0-21swapper/712:10:007
528208992119,2cyclictest548850-21munin-run07:44:597
52820899210,0cyclictest0-21swapper/709:10:137
528203992121,0cyclictest0-21swapper/610:45:146
528203992121,0cyclictest0-21swapper/610:35:146
528203992120,1cyclictest554973-21apt-get07:55:146
528203992120,1cyclictest26550irq/139-nvme0q708:55:016
528203992120,1cyclictest0-21swapper/608:30:136
528203992119,1cyclictest693851-21apt-get11:55:126
528203992119,1cyclictest675990-21idleruntime-cro11:25:016
52820399210,0cyclictest0-21swapper/609:20:146
528199992121,0cyclictest0-21swapper/509:20:145
528199992121,0cyclictest0-21swapper/507:55:145
528199992119,2cyclictest0-21swapper/508:20:145
528199992119,1cyclictest0-21swapper/511:30:175
528199992119,1cyclictest0-21swapper/508:25:135
528199992119,1cyclictest0-21swapper/507:35:005
528199992119,1cyclictest0-21swapper/507:10:125
528199992117,3cyclictest1-21systemd12:10:005
52819999210,0cyclictest0-21swapper/509:45:125
528194992120,1cyclictest675978-21munin-run11:25:014
528194992120,1cyclictest568831-21apt-get08:20:004
52819499210,21cyclictest0-21swapper/410:15:014
52819499210,20cyclictest0-21swapper/411:15:134
52819499210,20cyclictest0-21swapper/410:55:144
52819499210,20cyclictest0-21swapper/408:50:124
52819499210,1cyclictest0-21swapper/409:10:004
528190992121,0cyclictest0-21swapper/308:50:013
528190992119,1cyclictest601277-21apt-get09:15:153
52819099210,21cyclictest0-21swapper/310:20:013
52819099210,21cyclictest0-21swapper/308:10:133
52819099210,0cyclictest0-21swapper/309:25:133
52819099210,0cyclictest0-21swapper/309:25:133
52819099210,0cyclictest0-21swapper/307:45:153
528188992120,1cyclictest557408-21munin-run08:00:002
52818899210,21cyclictest0-21swapper/209:25:132
52818899210,21cyclictest0-21swapper/209:25:122
52818899210,21cyclictest0-21swapper/208:50:122
52818899210,21cyclictest0-21swapper/207:15:132
52818899210,20cyclictest0-21swapper/210:35:142
52818899210,1cyclictest0-21swapper/207:35:122
52818899210,0cyclictest0-21swapper/212:25:122
52818899210,0cyclictest0-21swapper/212:00:132
52818899210,0cyclictest0-21swapper/211:40:132
528185992121,0cyclictest0-21swapper/111:20:111
528185992120,1cyclictest586328-21awk08:50:011
528185992120,1cyclictest0-21swapper/110:20:011
528185992120,0cyclictest0-21swapper/112:15:141
52818599210,21cyclictest0-21swapper/108:05:121
52818599210,1cyclictest685295-21apt-get11:40:131
52818599210,0cyclictest0-21swapper/109:10:131
528183992121,0cyclictest0-21swapper/012:15:130
528183992121,0cyclictest0-21swapper/010:40:150
528183992121,0cyclictest0-21swapper/008:50:010
528183992119,1cyclictest0-21swapper/008:00:110
528183992118,2cyclictest681739-21munin-run11:34:590
52818399210,20cyclictest0-21swapper/009:30:140
52818399210,20cyclictest0-21swapper/009:30:130
52818399210,20cyclictest0-21swapper/007:40:140
52818399210,0cyclictest0-21swapper/010:05:130
528208992020,0cyclictest0-21swapper/712:10:147
528208992020,0cyclictest0-21swapper/710:00:157
528208992020,0cyclictest0-21swapper/709:15:167
528208992020,0cyclictest0-21swapper/708:05:137
528208992020,0cyclictest0-21swapper/707:20:147
52820899200,0cyclictest0-21swapper/711:50:157
52820899200,0cyclictest0-21swapper/710:15:137
528203992020,0cyclictest0-21swapper/612:20:126
528203992020,0cyclictest0-21swapper/611:05:126
528203992020,0cyclictest0-21swapper/610:50:156
528203992020,0cyclictest0-21swapper/610:30:136
528203992020,0cyclictest0-21swapper/610:10:126
528203992020,0cyclictest0-21swapper/609:25:136
528203992020,0cyclictest0-21swapper/609:25:136
528203992020,0cyclictest0-21swapper/608:36:446
528203992020,0cyclictest0-21swapper/607:40:146
528203992020,0cyclictest0-21swapper/607:15:126
528203992018,1cyclictest0-21swapper/608:20:006
52820399200,1cyclictest702433-21apt-get12:10:146
52820399200,0cyclictest0-21swapper/607:10:116
528199992020,0cyclictest0-21swapper/507:50:145
528199992020,0cyclictest0-21swapper/507:29:595
528199992018,1cyclictest716301-21munin-run12:35:005
528199992018,1cyclictest0-21swapper/511:25:155
528194992020,0cyclictest0-21swapper/410:45:134
528194992020,0cyclictest0-21swapper/410:15:124
528194992020,0cyclictest0-21swapper/408:45:144
528194992020,0cyclictest0-21swapper/408:36:454
528194992020,0cyclictest0-21swapper/408:10:124
52819499200,0cyclictest0-21swapper/411:05:124
52819499200,0cyclictest0-21swapper/410:40:154
528190992020,0cyclictest0-21swapper/310:55:133
528190992020,0cyclictest0-21swapper/310:10:123
528190992020,0cyclictest0-21swapper/309:55:133
52819099200,20cyclictest649987-21systemctl10:40:013
528188992020,0cyclictest0-21swapper/211:25:132
528188992020,0cyclictest0-21swapper/210:30:132
528188992020,0cyclictest0-21swapper/209:45:132
528188992020,0cyclictest0-21swapper/208:35:002
528188992019,1cyclictest615686-21awk09:40:002
528188992019,1cyclictest615686-21awk09:40:002
528188992019,1cyclictest0-21swapper/210:15:132
52818899200,0cyclictest0-21swapper/211:10:022
52818899200,0cyclictest0-21swapper/209:40:162
52818899200,0cyclictest0-21swapper/207:40:152
528185992020,0cyclictest0-21swapper/110:36:451
528185992020,0cyclictest0-21swapper/107:30:131
528185992019,1cyclictest0-21swapper/107:50:141
528185992019,1cyclictest0-21swapper/107:25:131
528185992018,1cyclictest0-21swapper/111:45:121
52818599200,0cyclictest0-21swapper/107:39:591
528183992020,0cyclictest0-21swapper/010:36:440
528183992020,0cyclictest0-21swapper/009:50:140
528183992020,0cyclictest0-21swapper/007:25:130
528183992019,1cyclictest0-21swapper/009:45:120
528208991919,0cyclictest0-21swapper/710:10:127
528208991919,0cyclictest0-21swapper/709:30:027
528208991919,0cyclictest0-21swapper/709:30:017
528208991919,0cyclictest0-21swapper/708:40:147
528208991918,1cyclictest0-21swapper/707:39:597
528208991917,1cyclictest688145-21apt-get11:45:127
52820899190,1cyclictest596720-21strings09:05:197
528203991919,0cyclictest0-21swapper/608:50:016
528203991919,0cyclictest0-21swapper/608:25:126
528203991919,0cyclictest0-21swapper/608:05:136
528203991918,1cyclictest691003-21apt-get11:50:156
528203991915,3cyclictest684841-21latency_hist11:40:006
52820399190,18cyclictest600842-21latency_hist09:15:016
52820399190,18cyclictest0-21swapper/609:55:146
528199991918,1cyclictest0-21swapper/511:10:005
528199991918,1cyclictest0-21swapper/508:50:135
528199991918,1cyclictest0-21swapper/508:10:125
528194991919,0cyclictest0-21swapper/411:25:154
528194991919,0cyclictest0-21swapper/408:00:124
528194991918,1cyclictest0-21swapper/408:30:014
528190991919,0cyclictest0-21swapper/312:30:153
528190991919,0cyclictest0-21swapper/312:25:123
528190991919,0cyclictest0-21swapper/312:10:153
528190991919,0cyclictest0-21swapper/311:30:183
528190991919,0cyclictest0-21swapper/311:15:123
528190991919,0cyclictest0-21swapper/309:30:023
528190991919,0cyclictest0-21swapper/309:30:013
528190991919,0cyclictest0-21swapper/308:20:143
528190991917,1cyclictest592718-21apt-get09:00:143
528190991916,2cyclictest655707-21dump-pmu-power10:50:003
528188991919,0cyclictest0-21swapper/212:05:112
528188991919,0cyclictest0-21swapper/211:45:132
528188991919,0cyclictest0-21swapper/210:25:142
528188991919,0cyclictest0-21swapper/210:10:132
528188991919,0cyclictest0-21swapper/208:20:012
528188991918,1cyclictest0-21swapper/212:14:592
52818899190,19cyclictest0-21swapper/207:30:132
528185991919,0cyclictest0-21swapper/109:20:011
528185991919,0cyclictest0-21swapper/108:35:131
528185991919,0cyclictest0-21swapper/107:44:591
528185991918,1cyclictest681732-21apt-get11:35:021
528185991918,1cyclictest681732-21apt-get11:35:001
528185991918,0cyclictest0-21swapper/107:20:151
528185991917,1cyclictest0-21swapper/111:50:151
528185991917,1cyclictest0-21swapper/111:15:131
528185991917,1cyclictest0-21swapper/109:15:161
528183991919,0cyclictest0-21swapper/011:35:020
528183991919,0cyclictest0-21swapper/008:35:130
528183991919,0cyclictest0-21swapper/007:50:140
528183991919,0cyclictest0-21swapper/007:20:150
528183991917,1cyclictest653270-21apt-get10:45:140
528208991818,0cyclictest0-21swapper/709:40:157
528208991818,0cyclictest0-21swapper/708:50:027
528208991817,1cyclictest715097-21ntpq12:30:207
52820899180,18cyclictest0-21swapper/712:01:037
52820899180,18cyclictest0-21swapper/711:06:087
52820899180,18cyclictest0-21swapper/710:44:107
52820899180,18cyclictest0-21swapper/708:21:237
52820899180,18cyclictest0-21swapper/707:26:287
52820899180,0cyclictest0-21swapper/707:48:267
528203991818,0cyclictest0-21swapper/612:15:136
528203991818,0cyclictest0-21swapper/611:40:136
528203991818,0cyclictest0-21swapper/611:00:156
528203991818,0cyclictest0-21swapper/608:20:026
528203991817,1cyclictest699577-21apt-get12:05:126
528199991818,0cyclictest0-21swapper/509:15:165
528199991818,0cyclictest0-21swapper/508:55:145
528199991817,1cyclictest710599-21perf12:24:595
528199991816,1cyclictest0-21swapper/510:00:155
528199991816,1cyclictest0-21swapper/507:35:145
528194991818,0cyclictest0-21swapper/411:50:164
528194991818,0cyclictest0-21swapper/410:20:144
528194991818,0cyclictest0-21swapper/409:50:124
528194991818,0cyclictest0-21swapper/407:10:134
528194991817,1cyclictest649991-21awk10:40:014
528194991817,1cyclictest0-21swapper/410:00:004
528190991814,3cyclictest534513-21apt07:20:003
528188991818,0cyclictest0-21swapper/212:15:142
528188991818,0cyclictest0-21swapper/207:10:122
528188991817,1cyclictest629975-21munin-run10:05:002
52818899180,18cyclictest0-21swapper/208:45:152
52818899180,0cyclictest0-21swapper/210:20:142
528185991818,0cyclictest0-21swapper/109:55:131
528185991818,0cyclictest0-21swapper/109:45:131
528185991817,1cyclictest0-21swapper/111:10:021
528185991817,1cyclictest0-21swapper/110:05:131
528185991817,1cyclictest0-21swapper/109:05:141
528185991816,1cyclictest0-21swapper/110:30:131
528185991816,1cyclictest0-21swapper/109:30:021
528185991816,1cyclictest0-21swapper/109:30:021
528183991818,0cyclictest0-21swapper/012:05:120
528183991818,0cyclictest0-21swapper/011:50:150
528183991818,0cyclictest0-21swapper/011:45:120
528183991818,0cyclictest0-21swapper/011:40:130
528183991818,0cyclictest0-21swapper/011:15:130
528183991817,1cyclictest0-21swapper/007:55:130
52818399180,17cyclictest641265-21chrt10:23:180
528208991717,0cyclictest0-21swapper/708:45:157
528208991717,0cyclictest0-21swapper/708:25:137
528208991716,1cyclictest627100-21sendmail10:00:017
52820899170,17cyclictest0-21swapper/712:23:017
52820899170,0cyclictest0-21swapper/708:10:247
528203991717,0cyclictest0-21swapper/612:35:166
528203991717,0cyclictest0-21swapper/609:40:156
528203991715,1cyclictest713871-21apt-get12:30:166
528199991717,0cyclictest580287-21fwupd08:36:445
528199991716,1cyclictest0-21swapper/512:15:135
528199991715,1cyclictest615669-21munin-run09:40:015
528199991715,1cyclictest615669-21munin-run09:40:015
528194991717,0cyclictest0-21swapper/410:30:144
528194991717,0cyclictest0-21swapper/409:15:154
528194991717,0cyclictest0-21swapper/407:50:134
528194991717,0cyclictest0-21swapper/407:30:134
528194991716,1cyclictest716726-21apt-get12:35:174
52819499170,0cyclictest0-21swapper/407:40:154
52819499170,0cyclictest0-21swapper/407:35:144
528190991717,0cyclictest0-21swapper/311:55:133
528190991717,0cyclictest0-21swapper/310:50:143
528190991717,0cyclictest0-21swapper/310:20:143
528190991717,0cyclictest0-21swapper/308:45:143
528190991717,0cyclictest0-21swapper/307:35:123
528190991716,1cyclictest667816-21apt-get11:10:133
528188991717,0cyclictest0-21swapper/211:35:022
528188991717,0cyclictest0-21swapper/211:00:162
528188991717,0cyclictest0-21swapper/210:05:132
528188991717,0cyclictest0-21swapper/207:20:152
528188991716,1cyclictest537813-21apt-get07:25:122
528185991717,0cyclictest0-21swapper/112:05:111
528185991716,1cyclictest0-21swapper/111:55:131
528185991716,1cyclictest0-21swapper/110:55:131
528185991716,1cyclictest0-21swapper/110:45:141
528185991716,1cyclictest0-21swapper/107:55:131
528183991717,0cyclictest0-21swapper/011:55:130
528183991717,0cyclictest0-21swapper/011:10:020
528183991717,0cyclictest0-21swapper/010:30:130
528183991717,0cyclictest0-21swapper/009:05:130
528183991717,0cyclictest0-21swapper/008:30:130
528183991716,0cyclictest0-21swapper/011:05:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional