You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-21 - 13:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Jun 21, 2026 00:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21345997423,11cyclictest131rcu_preempt23:25:121
2133699655,7cyclictest131rcu_preempt22:55:130
2133699613,43cyclictest14621-21ssh22:25:130
2134599595,44cyclictest377-21ssh22:35:131
2134599582,46cyclictest15741-21ssh21:50:161
21362995634,15cyclictest35966irq/58-eth022:40:143
21362995630,23cyclictest35966irq/58-eth000:25:143
2133699564,6cyclictest131rcu_preempt23:55:130
2134599534,40cyclictest11313-21ssh00:10:141
21362994926,17cyclictest35966irq/58-eth022:10:133
2133699483,38cyclictest19762-21ssh22:10:130
2133699472,38cyclictest25814-21ssh21:20:130
2134599451,36cyclictest23771-21ssh22:30:151
2134599441,32cyclictest2571-21ssh21:25:151
2133699442,34cyclictest29854-21ssh21:40:130
2136299433,26cyclictest35966irq/58-eth021:40:143
2136299431,24cyclictest35966irq/58-eth020:20:123
2136299431,24cyclictest35966irq/58-eth020:20:123
2135499434,24cyclictest27809-21ssh22:50:142
21336994225,8cyclictest536-21usb-storage23:00:110
2134599401,3cyclictest131rcu_preempt21:05:121
21336994022,11cyclictest0-21swapper/023:25:130
21354993923,3cyclictest7073-21diskmemload22:40:142
21345993911,19cyclictest21-21ksoftirqd/120:55:131
2133699391,27cyclictest10710-21ssh22:05:140
2136299382,22cyclictest35966irq/58-eth020:25:133
2136299382,21cyclictest35966irq/58-eth022:45:133
2134599385,2cyclictest21-21ksoftirqd/123:20:131
2134599383,30cyclictest18657-21ssh22:45:121
2136299371,21cyclictest35966irq/58-eth023:35:123
21354993518,5cyclictest1589-21ssh22:00:132
21354993518,5cyclictest1589-21ssh22:00:132
2134599355,19cyclictest24232-21ssh00:35:131
2133699353,26cyclictest28849-21sh22:15:120
2136299345,18cyclictest0-21swapper/321:45:133
2136299343,15cyclictest35966irq/58-eth000:00:123
21362993415,12cyclictest35966irq/58-eth023:45:133
2134599345,21cyclictest21-21ksoftirqd/120:15:131
2136299333,19cyclictest35966irq/58-eth022:55:153
2134599334,21cyclictest21-21ksoftirqd/120:40:131
2136299329,14cyclictest35966irq/58-eth022:15:133
2136299324,18cyclictest35966irq/58-eth019:20:113
2136299322,19cyclictest35966irq/58-eth021:20:133
2133699321,25cyclictest8579-21ssh23:15:150
2135499314,20cyclictest27346irq/40-dwc2_hso20:55:132
2135499311,22cyclictest0-21swapper/221:50:152
2133699311,22cyclictest16619-21sh21:15:130
2133699311,22cyclictest16619-21sh21:15:130
21336993112,10cyclictest12-21ksoftirqd/000:00:130
2135499302,20cyclictest27346irq/40-dwc2_hso20:05:112
2133699302,20cyclictest12-21ksoftirqd/023:35:120
2136299293,17cyclictest35966irq/58-eth019:25:123
2136299293,16cyclictest35966irq/58-eth023:00:123
2136299293,14cyclictest35966irq/58-eth023:30:113
2135499293,18cyclictest27346irq/40-dwc2_hso22:45:122
2136299281,15cyclictest35966irq/58-eth000:10:143
2135499281,15cyclictest11331-21rm00:10:162
2133699283,18cyclictest5599-21ssh22:20:140
2135499274,14cyclictest27346irq/40-dwc2_hso21:45:122
2135499273,19cyclictest0-21swapper/223:05:132
2135499272,17cyclictest19898-21ssh22:10:172
2135499272,16cyclictest27346irq/40-dwc2_hso20:40:132
2135499271,18cyclictest21503-21find19:10:122
2133699277,12cyclictest12-21ksoftirqd/020:10:120
21336992719,5cyclictest536-21usb-storage23:10:100
2136299266,17cyclictest0-21swapper/321:25:143
2136299263,16cyclictest0-21swapper/322:50:133
2134599261,19cyclictest2181-21ssh00:05:131
2133699269,9cyclictest21325-21cyclictest19:55:130
2133699262,18cyclictest7499-21ssh21:10:140
2133699261,19cyclictest663-21ssh00:20:280
2133699261,19cyclictest663-21ssh00:20:280
2136299252,14cyclictest35966irq/58-eth022:00:133
2136299252,14cyclictest35966irq/58-eth022:00:133
2135499251,8cyclictest732866ssh23:50:152
2134599257,5cyclictest131rcu_preempt23:15:151
2134599251,19cyclictest28477-21ssh21:55:251
2133699254,14cyclictest7246-21kworker/u8:023:20:120
2133699252,14cyclictest9514-21sh22:40:120
2135499245,12cyclictest25855-21kworker/u8:200:20:142
2135499245,12cyclictest25855-21kworker/u8:200:20:142
2135499241,7cyclictest93066snmpd20:29:592
2135499241,7cyclictest765166ssh22:38:172
2135499241,7cyclictest364066ssh23:10:282
2135499241,6cyclictest38966rm21:23:032
21354992412,9cyclictest7525-21ssh21:25:382
21354992410,5cyclictest2370-21fschecks_time00:05:162
21354992410,11cyclictest30780-21cron00:01:012
2134599241,17cyclictest20788-21sh21:35:141
2133699242,17cyclictest18677-21ssh22:45:130
2136299231,14cyclictest0-21swapper/322:05:153
2135499233,13cyclictest0-21swapper/219:55:252
2135499232,5cyclictest26726-21ssh23:25:122
2135499232,17cyclictest27346irq/40-dwc2_hso00:35:132
2135499232,12cyclictest4495-21ssh22:55:132
2135499231,9cyclictest11275-21ssh21:10:282
2135499231,6cyclictest1750066ssh22:07:512
21354992312,9cyclictest26242-21ssh00:16:452
21354992311,9cyclictest16585-21ssh23:19:152
2135499231,14cyclictest131rcu_preempt23:40:302
2135499231,14cyclictest131rcu_preempt23:40:302
21354992310,11cyclictest27-21ksoftirqd/223:30:262
2133699232,6cyclictest7306-21apt-get23:50:140
2136299229,8cyclictest35966irq/58-eth021:50:143
2136299222,9cyclictest0-21swapper/319:55:123
2136299221,13cyclictest0-21swapper/300:35:133
2135499227,5cyclictest17024-21ps21:15:192
2135499227,5cyclictest17024-21ps21:15:192
2135499225,9cyclictest3723-21fschecks_time19:25:162
2135499222,11cyclictest20787-21ssh21:35:142
21354992212,7cyclictest0-21swapper/223:45:132
2135499221,11cyclictest32445-21munin-plugin-st22:35:012
2134599221,2cyclictest131rcu_preempt21:40:131
2134599221,17cyclictest21728-21cut23:40:161
2134599221,17cyclictest21728-21cut23:40:161
2133699221,18cyclictest22957-21grep23:05:170
21336992210,3cyclictest131rcu_preempt21:25:310
21362992116,3cyclictest321rcuc/323:57:273
21362992114,5cyclictest35966irq/58-eth021:35:153
2136299211,11cyclictest0-21swapper/321:15:143
2136299211,11cyclictest0-21swapper/321:15:143
2135499218,11cyclictest0-21swapper/219:45:142
2135499215,13cyclictest27-21ksoftirqd/219:15:172
2135499212,9cyclictest20991-21irqrtprio20:50:182
2135499212,10cyclictest23668-21apt-get20:20:022
2135499211,9cyclictest20555-21ssh23:20:212
21354992116,3cyclictest27-21ksoftirqd/223:00:112
2135499211,15cyclictest16379-21grep23:35:282
2135499211,11cyclictest587966ssh00:25:052
2135499211,11cyclictest3445-21latency_hist19:25:022
2135499211,10cyclictest30854-21kthreadcore22:15:222
2135499211,10cyclictest14431-21tr21:30:212
21354992110,9cyclictest742-21ssh21:59:332
2134599218,9cyclictest31191-21grep23:45:171
2134599216,4cyclictest21-21ksoftirqd/119:15:121
2134599215,3cyclictest7073-21diskmemload21:11:331
2134599213,14cyclictest10729-21ssh22:05:151
2134599211,3cyclictest131rcu_preempt00:20:151
2134599211,3cyclictest131rcu_preempt00:20:151
21345992113,6cyclictest18256-21cron22:45:011
21345992112,7cyclictest21-21ksoftirqd/121:45:161
2134599211,13cyclictest21-21ksoftirqd/123:50:141
2134599210,15cyclictest28868-21ssh22:15:131
2133699211,15cyclictest27934-21ssh22:50:160
2133699211,14cyclictest513-21cut22:35:150
21362992015,3cyclictest321rcuc/323:53:223
21362992015,3cyclictest321rcuc/323:09:113
21362992014,4cyclictest321rcuc/300:32:333
21362992014,4cyclictest321rcuc/300:22:363
21362992014,4cyclictest321rcuc/300:22:353
21362992013,4cyclictest0-21swapper/322:25:143
21362992010,8cyclictest0-21swapper/323:41:243
21362992010,8cyclictest0-21swapper/323:41:243
2135499205,12cyclictest27-21ksoftirqd/222:20:202
2135499202,15cyclictest27700-21fschecks_time19:50:172
2135499202,15cyclictest27700-21fschecks_time19:50:162
2135499202,15cyclictest17678-21kthreadcore00:30:222
2135499202,10cyclictest1177-21ps21:00:262
2135499201,11cyclictest9429-21latency_hist20:05:022
2135499201,11cyclictest29958-21df21:40:142
2135499201,10cyclictest6208-21sed20:35:162
2135499201,10cyclictest19191-21cut19:40:192
2135499201,10cyclictest18960-21missed_timers20:45:232
2135499201,10cyclictest16625-21fschecks_time23:55:152
2135499201,10cyclictest15187-21ps22:25:182
2134599203,5cyclictest1255-21run-parts21:00:261
2134599202,7cyclictest131rcu_preempt00:25:141
2134599201,2cyclictest131rcu_preempt21:20:131
21345992011,6cyclictest28375-21irqrtprio22:50:191
2134599201,15cyclictest16632-21ssh21:15:141
2134599201,15cyclictest16632-21ssh21:15:141
2134599200,4cyclictest131rcu_preempt20:05:111
2133699209,8cyclictest11715-21sshd19:30:240
2133699202,12cyclictest11700-21ssh21:30:130
2133699201,17cyclictest11409-21ls00:10:150
2136299192,11cyclictest0-21swapper/320:45:143
2136299191,9cyclictest35966irq/58-eth020:05:123
21362991914,3cyclictest321rcuc/323:23:193
21362991913,4cyclictest321rcuc/322:35:123
21362991912,5cyclictest321rcuc/300:05:043
2136299191,13cyclictest0-21swapper/322:30:243
2136299191,12cyclictest0-21swapper/320:30:253
2136299191,10cyclictest35966irq/58-eth023:25:123
21362991910,6cyclictest35966irq/58-eth022:20:133
21362991910,4cyclictest0-21swapper/323:11:573
2135499191,9cyclictest23980-21/usr/sbin/munin20:20:172
2135499191,9cyclictest23980-21/usr/sbin/munin20:20:172
2135499191,8cyclictest13451-21irqcore19:35:182
2135499191,7cyclictest4348-21kthreadcore21:05:192
2135499191,15cyclictest8496-21fschecks_time19:30:162
2135499191,15cyclictest14409-21fschecks_count20:10:162
2135499191,10cyclictest1326-21fschecks_count20:30:132
2134599191,15cyclictest23998-21apt-get20:20:121
2134599191,15cyclictest23998-21apt-get20:20:121
2134599191,15cyclictest0-21swapper/122:00:141
2134599191,15cyclictest0-21swapper/122:00:141
2133699197,10cyclictest6534-21cat00:05:310
2133699193,11cyclictest0-21swapper/020:35:130
2133699191,8cyclictest24708-21threads21:35:280
21336991911,6cyclictest7102-21irqrtprio21:45:190
2133699191,15cyclictest23772-21missed_timers20:50:230
21336991910,6cyclictest23837-21cron00:35:020
2133699190,15cyclictest24854-21acpi21:55:120
2136299182,10cyclictest0-21swapper/321:05:123
2136299182,10cyclictest0-21swapper/320:35:133
21362991812,4cyclictest0-21swapper/323:16:463
21362991812,4cyclictest0-21swapper/321:55:193
2136299181,13cyclictest0-21swapper/320:55:163
2136299181,12cyclictest0-21swapper/300:15:293
2136299181,11cyclictest0-21swapper/320:40:133
2134599182,3cyclictest26552-21ssh22:12:481
2134599182,3cyclictest131rcu_preempt19:40:101
2134599181,7cyclictest32150-21cat19:55:011
2134599181,7cyclictest32150-21cat19:55:011
2134599181,3cyclictest131rcu_preempt22:27:511
2134599181,3cyclictest131rcu_preempt22:20:161
2134599181,15cyclictest25265-21ls19:10:281
2134599181,14cyclictest24585-21kthreadcore19:45:191
2134599181,14cyclictest19777-21ssh00:30:341
2134599181,12cyclictest5623-21cut23:30:201
2134599181,12cyclictest32082-21cut23:10:141
21345991810,6cyclictest31543-21gpgv23:10:021
2134599180,6cyclictest7073-21diskmemload00:00:201
2133699187,8cyclictest23869-21df_inode22:30:160
2133699186,9cyclictest25612-21ssh23:40:290
2133699186,9cyclictest25612-21ssh23:40:290
2133699185,10cyclictest20014-1kworker/u9:223:45:130
2133699183,12cyclictest30098-21grep19:15:280
2133699181,9cyclictest31447-21tr20:25:220
2133699181,9cyclictest10299-21basename22:05:010
2133699181,9cyclictest10299-21basename22:05:010
2133699181,6cyclictest6121-21df_inode00:25:150
2133699181,15cyclictest6032-21processes21:05:250
21336991810,4cyclictest0-21swapper/023:30:120
2133699180,15cyclictest23662-21latency_hist20:20:010
21362991712,3cyclictest321rcuc/319:50:133
21362991712,3cyclictest321rcuc/319:50:123
21362991712,3cyclictest0-21swapper/321:30:103
21362991711,4cyclictest0-21swapper/321:10:183
21362991711,4cyclictest0-21swapper/321:00:173
2136299171,12cyclictest0-21swapper/319:45:163
2136299171,10cyclictest0-21swapper/320:00:273
2134599176,2cyclictest131rcu_preempt22:57:571
2134599175,10cyclictest24987-21cron20:55:011
2134599171,9cyclictest32190-21kthreadcore19:20:181
2134599171,6cyclictest0-21swapper/120:10:141
21345991711,4cyclictest17768-21cat19:40:031
2134599171,13cyclictest20380-21ssh00:15:141
2134599171,12cyclictest17310-21ntp_states23:00:251
2134599171,12cyclictest16502-21awk23:55:141
21345991710,4cyclictest15886-21usage_Anybus21:30:301
2133699178,6cyclictest5901-21kthreadcore20:00:180
2133699178,6cyclictest2973-21kthreadcore20:30:180
2133699176,3cyclictest131rcu_preempt19:25:010
2133699173,9cyclictest0-21swapper/019:35:140
2133699173,8cyclictest0-21swapper/000:30:140
2133699172,10cyclictest0-21swapper/020:45:140
2133699171,13cyclictest31411-21kthreadcore21:00:190
2136299162,9cyclictest0-21swapper/319:15:133
2136299162,10cyclictest0-21swapper/320:50:143
21362991611,3cyclictest0-21swapper/320:10:233
2136299161,12cyclictest0-21swapper/319:30:233
21362991610,4cyclictest0-21swapper/319:35:273
2134599169,4cyclictest3604-21cut20:30:181
2134599168,5cyclictest2105-21kthreadcore19:55:211
2134599167,7cyclictest3539-21apt19:25:131
2134599161,3cyclictest12441-21apt-get23:35:121
2134599161,12cyclictest5159-21cut20:00:161
2134599161,12cyclictest31028-21cut20:25:211
21345991610,4cyclictest0-21swapper/120:45:281
2133699168,6cyclictest22812-21diskstats19:45:140
2133699165,7cyclictest20459-21rm00:15:140
2133699163,10cyclictest30951-21ntp_kernel_pll_19:50:250
2133699163,10cyclictest30951-21ntp_kernel_pll_19:50:250
2133699162,10cyclictest0-21swapper/019:25:140
2133699161,6cyclictest12461-21cut20:05:200
2133699161,12cyclictest15730-21ssh21:50:140
2133699161,11cyclictest25222-21apt-get20:55:120
2133699160,11cyclictest17941-21cpu19:40:120
2136299151,7cyclictest0-21swapper/319:40:103
21362991510,3cyclictest0-21swapper/320:15:273
21362991510,3cyclictest0-21swapper/319:10:273
2134599152,10cyclictest8981-21kthreadcore19:30:191
2134599152,10cyclictest5987-21apt-get20:35:111
2134599152,10cyclictest21388-21latency_hist19:10:011
2134599152,10cyclictest21388-21latency_hist19:10:011
2133699152,9cyclictest0-21swapper/019:10:140
2133699151,11cyclictest10810-21apt-get20:40:120
2133699151,10cyclictest25302-21kthreadcore20:20:190
2133699151,10cyclictest25302-21kthreadcore20:20:190
2133699151,10cyclictest21421-21grep19:10:020
2133699151,10cyclictest21421-21grep19:10:010
2135499131,9cyclictest21416-21latency_hist19:10:012
2135499131,9cyclictest21416-21latency_hist19:10:012
2136299102,6cyclictest321rcuc/319:10:013
2136299102,6cyclictest321rcuc/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional