You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 00:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Thu Mar 12, 2026 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1364699533,34cyclictest30777-21ssh09:45:132
1363899522,7cyclictest131rcu_preempt07:35:121
13638995113,32cyclictest31701-21diskmemload09:45:131
13630994917,25cyclictest12-21ksoftirqd/007:50:130
1363899455,6cyclictest131rcu_preempt10:15:121
1363899455,6cyclictest131rcu_preempt10:15:111
1363099452,40cyclictest13924-21ssh11:05:130
1363099452,40cyclictest13924-21ssh11:05:120
1363099442,32cyclictest16631-21ssh09:55:120
1363899431,31cyclictest12359-21ssh11:40:111
1363899421,4cyclictest131rcu_preempt07:20:121
1363099427,27cyclictest681-21kworker/u8:011:55:140
1365599403,21cyclictest28366irq/58-eth009:35:123
1363899403,32cyclictest26765-21ssh11:30:111
1363899403,10cyclictest11467-21apt-get10:10:121
1363899403,10cyclictest11467-21apt-get10:10:111
1365599392,21cyclictest28366irq/58-eth009:45:123
13638993934,3cyclictest22916-21ssh11:10:121
1363899392,30cyclictest30438-21ssh11:50:111
13630993916,16cyclictest31701-21diskmemload12:20:130
1365599381,23cyclictest28366irq/58-eth010:50:123
1364699372,28cyclictest17938-21ssh11:41:222
1364699372,27cyclictest32161-21grep12:25:222
1363899374,10cyclictest1250-21kthreadcore07:30:191
1363899371,12cyclictest6850-21kthreadcore11:00:191
1363099373,21cyclictest15377-21ssh10:30:120
13655993618,10cyclictest28366irq/58-eth011:40:133
1364699362,17cyclictest8545-21kthreadcore11:55:202
1363899367,6cyclictest17681-21sed09:20:021
1363899362,11cyclictest19327-21grep10:50:161
1363899361,11cyclictest17864-21cut12:00:181
1364699352,20cyclictest10173-21ssh10:45:142
1363899359,5cyclictest23878-21grep12:20:291
1363899351,12cyclictest17976-21grep11:05:291
1363899351,12cyclictest17976-21grep11:05:291
13638993510,8cyclictest21794-21cat11:25:291
1363099352,27cyclictest11939-21cut11:20:210
1364699346,22cyclictest27-21ksoftirqd/212:21:012
1363899342,13cyclictest25764-21ssh10:00:121
1363899341,10cyclictest26622-21/usr/sbin/munin09:40:321
1365599338,23cyclictest28366irq/58-eth011:20:123
1363899332,12cyclictest30919-21timerwakeupswit09:25:281
1363899331,26cyclictest21365-21ssh11:45:111
1363899331,20cyclictest24442-21ssh10:35:131
1363899331,12cyclictest26734-21kthreadcore08:30:151
1363099333,22cyclictest32008-21ssh11:15:120
1365599327,18cyclictest0-21swapper/312:01:213
1364699322,23cyclictest0-21swapper/211:05:132
1364699322,23cyclictest0-21swapper/211:05:122
1363899323,18cyclictest8664-21ssh11:20:121
1363899323,10cyclictest20302-21tr08:55:221
1363899321,11cyclictest23021-21kthreadcore07:50:221
1363899321,10cyclictest15344-21/usr/sbin/munin10:30:261
1364699314,9cyclictest0-21swapper/208:15:132
1364699312,16cyclictest6348-21ssh10:25:122
1364699312,13cyclictest16742-21ssh11:24:222
1363899315,8cyclictest1789-21cat10:20:301
1363899311,11cyclictest20311-21ntp_states09:55:271
1363899311,10cyclictest13306-21tr10:45:221
1365599307,3cyclictest0-21swapper/311:02:583
1365599304,20cyclictest28366irq/58-eth009:30:123
1364699302,19cyclictest22346irq/40-dwc2_hso12:10:142
1364699302,12cyclictest27806-21ssh11:47:212
1364699301,3cyclictest131rcu_preempt09:30:122
1364699301,23cyclictest25115-21rm12:05:132
1364699301,23cyclictest25115-21rm12:05:122
13646993011,14cyclictest0-21swapper/207:55:132
1363899303,10cyclictest4554-21ssh11:00:011
1363899301,11cyclictest5374-21ntpq12:10:241
1363899301,11cyclictest28845-21ntp_states12:05:261
1363899301,11cyclictest28845-21ntp_states12:05:251
1363899301,11cyclictest25653-21memory09:00:211
1363899301,10cyclictest3577-21timerwakeupswit11:15:281
13630993021,2cyclictest0-21swapper/009:25:120
1363099301,23cyclictest1713-21ssh12:10:130
1363099301,22cyclictest25132-21ssh12:05:140
1363099301,22cyclictest25132-21ssh12:05:140
1365599294,2cyclictest0-21swapper/307:15:123
1364699292,20cyclictest0-21swapper/211:30:132
1364699291,22cyclictest8731-21ssh09:15:122
1364699291,19cyclictest22346irq/40-dwc2_hso10:55:112
1363899292,20cyclictest8035-21ssh12:30:201
1363899291,10cyclictest7555-21diskstats09:50:141
1363099292,21cyclictest3577-21sh11:51:210
1363099292,21cyclictest21671-21ssh12:01:210
1365599283,16cyclictest0-21swapper/312:06:223
1365599283,16cyclictest0-21swapper/312:06:213
1365599283,16cyclictest0-21swapper/311:50:203
1364699282,20cyclictest924-21snmpd07:20:132
1364699282,19cyclictest22346irq/40-dwc2_hso09:00:122
1364699282,16cyclictest22346irq/40-dwc2_hso09:20:112
13646992816,9cyclictest22628-21ssh11:25:312
1363899285,4cyclictest2288-21kworker/u8:210:25:051
1363899283,20cyclictest17851-21ssh09:20:131
1363899282,20cyclictest17982-21ssh12:18:221
1363899282,18cyclictest12620-21ssh09:35:111
1363899281,10cyclictest28183-21grep07:55:251
1363099281,20cyclictest28264-21rm10:55:130
1365599279,4cyclictest0-21swapper/312:26:333
1365599279,4cyclictest0-21swapper/311:58:123
1365599273,15cyclictest0-21swapper/312:14:203
1365599273,12cyclictest28366irq/58-eth011:45:113
1364699273,16cyclictest22346irq/40-dwc2_hso09:05:122
1364699273,16cyclictest22346irq/40-dwc2_hso09:05:112
1364699273,16cyclictest22346irq/40-dwc2_hso07:30:112
1364699273,10cyclictest0-21swapper/212:38:212
1364699272,19cyclictest10787-21ssh12:31:052
13646992711,11cyclictest261rcuc/209:55:122
1364699264,7cyclictest27-21ksoftirqd/209:42:132
1364699262,19cyclictest0-21swapper/208:45:142
1364699262,18cyclictest22346irq/40-dwc2_hso10:40:112
1364699262,11cyclictest27-21ksoftirqd/211:14:462
13646992613,8cyclictest261rcuc/209:10:132
1364699260,18cyclictest13871-21ls07:10:142
1363899261,8cyclictest17486-21ssh12:35:211
1365599251,3cyclictest0-21swapper/309:40:193
1365599251,20cyclictest0-21swapper/310:30:213
1364699253,16cyclictest22346irq/40-dwc2_hso08:30:102
1364699253,14cyclictest22346irq/40-dwc2_hso10:30:122
1364699253,13cyclictest22346irq/40-dwc2_hso08:50:132
1364699252,16cyclictest22346irq/40-dwc2_hso07:25:132
1364699252,16cyclictest22346irq/40-dwc2_hso07:25:122
1364699252,15cyclictest22346irq/40-dwc2_hso07:45:132
1364699251,14cyclictest24500-21ls10:35:142
1363899253,15cyclictest3543-21ssh09:30:121
13638992513,10cyclictest0-21swapper/108:05:121
1365599249,4cyclictest0-21swapper/310:27:153
1365599241,9cyclictest0-21swapper/312:25:013
1365599241,8cyclictest0-21swapper/309:25:163
1365599241,6cyclictest0-21swapper/311:15:163
1365599241,10cyclictest0-21swapper/311:25:213
1364699242,7cyclictest0-21swapper/211:38:202
1364699242,16cyclictest0-21swapper/212:15:202
13646992413,8cyclictest0-21swapper/211:15:582
1364699241,15cyclictest16088-21ssh12:00:112
1364699241,12cyclictest2452-21ssh10:05:132
13638992414,7cyclictest21852-21sshd07:15:211
13638992414,6cyclictest0-21swapper/110:40:131
1363099244,16cyclictest29634-21ssh10:20:130
1363099244,13cyclictest10152-21ssh10:45:130
1365599239,3cyclictest0-21swapper/310:20:323
1365599237,6cyclictest0-21swapper/311:09:193
1365599237,6cyclictest0-21swapper/311:09:183
1365599235,5cyclictest28366irq/58-eth010:40:263
1365599231,8cyclictest0-21swapper/310:15:213
1365599231,8cyclictest0-21swapper/310:15:203
1365599231,8cyclictest0-21swapper/309:50:193
1365599231,6cyclictest0-21swapper/308:30:013
1364699235,8cyclictest131rcu_preempt07:15:122
1364699234,16cyclictest22346irq/40-dwc2_hso08:00:112
1364699232,15cyclictest0-21swapper/207:50:042
13646992313,8cyclictest0-21swapper/210:15:142
13646992313,8cyclictest0-21swapper/210:15:132
13646992313,8cyclictest0-21swapper/209:25:132
13646992313,4cyclictest0-21swapper/208:20:152
1364699231,17cyclictest0-21swapper/208:10:122
1364699231,12cyclictest30240-21irqrtprio10:20:192
1363099238,9cyclictest0-21swapper/011:10:120
13630992310,8cyclictest12626-21ssh09:35:120
1365599228,11cyclictest0-21swapper/309:12:513
1365599227,3cyclictest0-21swapper/309:20:093
1365599226,3cyclictest28366irq/58-eth012:15:133
1365599223,13cyclictest0-21swapper/312:30:213
1365599221,8cyclictest0-21swapper/307:30:183
1365599221,7cyclictest0-21swapper/309:15:173
1365599221,6cyclictest0-21swapper/311:10:213
1365599221,6cyclictest0-21swapper/308:00:013
1365599221,10cyclictest0-21swapper/310:10:203
1365599221,10cyclictest0-21swapper/310:10:203
1365599220,12cyclictest28366irq/58-eth011:30:123
1364699224,10cyclictest3031-21grep08:40:022
1364699223,5cyclictest0-21swapper/207:35:122
1364699223,12cyclictest27-21ksoftirqd/209:38:322
1364699222,14cyclictest22346irq/40-dwc2_hso10:50:132
13646992217,3cyclictest27-21ksoftirqd/208:25:182
1364699221,6cyclictest4867-21rm11:00:122
13646992215,4cyclictest0-21swapper/210:11:482
13646992215,4cyclictest0-21swapper/210:11:472
13646992211,8cyclictest0-21swapper/210:02:462
1363899224,11cyclictest839-21munin-node08:25:141
1363099222,15cyclictest31701-21diskmemload11:43:210
1363099222,13cyclictest5437-21ssh12:30:140
13630992214,5cyclictest0-21swapper/008:35:270
1365599218,3cyclictest0-21swapper/309:57:253
1365599211,6cyclictest0-21swapper/312:35:193
1365599211,3cyclictest0-21swapper/311:35:143
1364699213,16cyclictest924-21snmpd08:40:122
13646992113,5cyclictest0-21swapper/209:50:132
1364699211,11cyclictest30404-21/usr/sbin/munin11:50:222
1363899212,15cyclictest32105-21ssh09:10:131
1363899211,18cyclictest12005-21ls11:40:021
1363899211,13cyclictest633-21Async12:11:531
1363899211,13cyclictest633-21Async12:11:531
1363899210,4cyclictest0-21swapper/112:26:251
1363099217,4cyclictest131rcu_preempt12:35:180
1363099211,17cyclictest6302-21find10:25:110
1363099211,15cyclictest29733-21ls08:00:160
1363099211,15cyclictest18342-21latency_hist07:15:020
1363099211,14cyclictest3277-21cut08:40:160
13630992110,7cyclictest1-21systemd07:15:120
1363099210,17cyclictest5117-21timerwakeupswit10:40:270
1365599209,7cyclictest28366irq/58-eth010:35:143
1364699202,10cyclictest12888-21kthreadcore07:40:192
1364699201,15cyclictest17705-21fschecks_time08:55:152
1363899204,11cyclictest11718-21grep08:15:161
1363899203,10cyclictest839-21munin-node07:25:141
1363899203,10cyclictest839-21munin-node07:25:141
1363099209,3cyclictest12-21ksoftirqd/010:07:140
1363099204,12cyclictest24397-21switchtime10:15:270
1363099204,12cyclictest24397-21switchtime10:15:270
1363099203,10cyclictest0-21swapper/011:47:220
1363099202,15cyclictest21181-21kthreadcore10:50:200
1363099202,14cyclictest12386-21sort09:35:010
1363099202,14cyclictest0-21swapper/010:35:120
1363099202,13cyclictest2778-21ssh12:27:210
1363099201,8cyclictest8188-21/usr/sbin/munin09:50:190
13630992013,3cyclictest19393-21latency_hist12:20:020
1365599197,7cyclictest28366irq/58-eth010:00:263
1365599192,11cyclictest0-21swapper/308:45:143
1365599192,10cyclictest0-21swapper/308:00:133
1365599191,12cyclictest0-21swapper/308:10:153
1364699191,9cyclictest2719-21kthreadcore08:05:192
1363899197,10cyclictest1-21systemd07:45:161
1363099199,3cyclictest131rcu_preempt09:40:280
1363099193,10cyclictest12-21ksoftirqd/008:20:120
13630991912,5cyclictest31701-21diskmemload11:35:140
1363099191,15cyclictest19084-21kthreadcore08:55:190
13630991911,5cyclictest1864-21missed_timers09:45:220
1363099191,14cyclictest17476-21idleruntime-cro09:20:020
1363099191,10cyclictest711-21rs:main0
1363099191,10cyclictest20294-21ls10:15:020
1363099191,10cyclictest20294-21ls10:15:010
13630991910,6cyclictest26195-21grep09:00:280
1365599188,3cyclictest0-21swapper/310:08:253
1365599182,10cyclictest0-21swapper/309:05:133
1365599182,10cyclictest0-21swapper/309:05:123
1365599182,10cyclictest0-21swapper/309:00:123
1365599182,10cyclictest0-21swapper/308:05:133
1365599182,10cyclictest0-21swapper/307:50:133
1365599181,10cyclictest0-21swapper/308:30:133
1364699181,13cyclictest13714-21latency_hist07:10:022
1363899183,12cyclictest1106-21kthreadcore08:35:211
1363899182,13cyclictest17985-21kthreadcore08:20:191
1363899182,12cyclictest7503-21irqcore11:55:191
1363899181,9cyclictest11512-21kthreadcore07:40:191
1363099188,8cyclictest28211-21apt-get07:25:130
1363099188,8cyclictest28211-21apt-get07:25:120
1363099187,3cyclictest131rcu_preempt10:00:240
1363099182,12cyclictest9932-21cron07:40:020
1363099182,12cyclictest19767-21sort11:25:200
1363099182,11cyclictest0-21swapper/007:55:120
1363099181,8cyclictest8299-21kthreadcore08:10:200
1363099181,14cyclictest27293-21fschecks_count09:05:150
1363099181,14cyclictest27293-21fschecks_count09:05:140
1363099181,14cyclictest21706-21/usr/sbin/munin09:20:270
1363099181,13cyclictest30484-21ntp_states11:30:260
1365599179,5cyclictest28366irq/58-eth010:45:183
1365599172,10cyclictest0-21swapper/308:15:133
1365599171,11cyclictest0-21swapper/307:10:113
13655991710,4cyclictest0-21swapper/310:58:133
1363899173,11cyclictest14127-21irqcore07:10:161
1363899172,11cyclictest15543-21kthreadcore08:50:201
13638991711,4cyclictest30134-21kernelversion08:00:181
1363899171,13cyclictest7024-21tail08:10:171
1363899171,13cyclictest6924-21grep08:40:271
1363899171,13cyclictest11403-21wc08:45:281
1363899171,12cyclictest2416-21/usr/sbin/munin10:05:171
1363099171,3cyclictest131rcu_preempt09:10:140
13630991711,4cyclictest0-21swapper/008:50:180
13630991711,4cyclictest0-21swapper/007:40:220
1363099171,13cyclictest7913-21apt-get08:45:130
1363099171,13cyclictest15388-21grep08:15:290
1365599162,10cyclictest0-21swapper/307:40:133
1363099168,5cyclictest24629-21kthreadcore07:20:180
1363099165,8cyclictest791-21wc07:30:150
1363099161,7cyclictest17613-21kthreadcore07:45:210
1363099161,13cyclictest5054-21sort11:00:130
1363099161,11cyclictest1994-21diskstats08:05:140
1365599152,10cyclictest0-21swapper/307:25:143
1365599152,10cyclictest0-21swapper/307:25:133
13655991510,3cyclictest0-21swapper/308:42:203
1363099152,10cyclictest13660-21latency_hist07:10:010
1363099151,8cyclictest0-21swapper/008:25:170
1365599147,4cyclictest0-21swapper/307:20:123
1365599141,11cyclictest0-21swapper/308:55:023
1363899141,10cyclictest13674-21latency_hist07:10:021
1365599136,5cyclictest0-21swapper/307:45:183
1365599127,3cyclictest0-21swapper/308:38:203
1365599121,9cyclictest0-21swapper/308:55:143
1365599112,5cyclictest0-21swapper/307:35:123
1365599111,7cyclictest0-21swapper/307:10:023
1365599111,6cyclictest0-21swapper/308:20:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional