You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-12 - 03:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Tue May 12, 2026 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1920621620,3sleep21838999cyclictest09:33:112
2052021440,2sleep119750-21/usr/sbin/munin08:16:201
742421020,2sleep20-21swapper/208:01:112
742421020,2sleep20-21swapper/208:01:112
52762850,2sleep20-21swapper/207:57:452
294232820,2sleep20-21swapper/209:46:082
294232820,2sleep20-21swapper/209:46:082
180732800,1sleep30-21swapper/308:52:133
22702790,3sleep12253-21ssh09:51:261
18387997623,35cyclictest11750-21/usr/sbin/munin08:06:130
1838799735,42cyclictest0-21swapper/007:16:110
143302720,6sleep11838899cyclictest07:30:221
29222690,2sleep30-21swapper/307:16:153
1838799693,34cyclictest6754-21runrttasks07:21:110
1838899686,9cyclictest18386-21cyclictest09:58:101
1838799683,15cyclictest0-21swapper/005:31:100
289602650,1sleep30-21swapper/307:08:263
18387996444,3cyclictest0-21swapper/006:56:110
72482630,2sleep20-21swapper/209:18:582
1838899638,6cyclictest7750irq/56-dwc_otg09:11:261
1838899638,6cyclictest7750irq/56-dwc_otg09:11:261
1838899636,10cyclictest7850irq/56-dwc_otg_07:15:571
1838899634,14cyclictest29997-21uname05:01:211
1838799634,46cyclictest61-21kswapd008:21:090
18387996327,13cyclictest0-21swapper/006:36:090
264752620,2sleep20-21swapper/208:23:462
18387996115,4cyclictest0-21swapper/009:06:100
1838799611,47cyclictest28859-21netstat08:26:190
1838899604,3cyclictest0-21swapper/105:00:581
18388996013,6cyclictest0-21swapper/106:56:261
1838899596,10cyclictest7850irq/56-dwc_otg_06:31:261
1838799591,37cyclictest0-21swapper/006:51:090
1838799591,37cyclictest0-21swapper/006:51:090
18387995843,5cyclictest7950irq/56-dwc_otg_07:01:110
1838799584,26cyclictest0-21swapper/007:26:100
18387995831,13cyclictest7750irq/56-dwc_otg05:06:240
1838799581,31cyclictest0-21swapper/008:01:080
1838799581,31cyclictest0-21swapper/008:01:080
66962570,12sleep00-21swapper/008:00:210
1838899577,5cyclictest0-21swapper/107:21:271
1838799572,25cyclictest0-21swapper/005:51:090
18387995711,27cyclictest13-21rcuc/009:41:250
1838899565,5cyclictest7750irq/56-dwc_otg08:06:241
1838799564,9cyclictest0-21swapper/006:11:100
1838799564,30cyclictest0-21swapper/006:06:090
18387995625,10cyclictest0-21swapper/006:41:090
18387995620,20cyclictest908-21runrttasks05:11:150
1838999555,18cyclictest7950irq/56-dwc_otg_09:11:112
1838999555,18cyclictest7950irq/56-dwc_otg_09:11:112
18387995536,6cyclictest7850irq/56-dwc_otg_08:15:580
18387995529,17cyclictest7950irq/56-dwc_otg_04:45:570
18387995515,19cyclictest17038-21/usr/sbin/munin09:31:110
177382550,2sleep30-21swapper/307:34:023
177382550,2sleep30-21swapper/307:34:023
1838899547,4cyclictest7850irq/56-dwc_otg_08:36:131
1838899547,15cyclictest0-21swapper/106:26:031
43792530,2sleep20-21swapper/207:17:192
1838899532,13cyclictest0-21swapper/109:36:291
1838899531,14cyclictest0-21swapper/108:05:591
1838899531,14cyclictest0-21swapper/108:05:591
1838799526,22cyclictest9-21ksoftirqd/009:01:190
1838799523,29cyclictest11-21rcu_preempt07:41:130
18387995230,5cyclictest7750irq/56-dwc_otg08:56:160
18387995222,14cyclictest13260-21grep08:46:250
18387995220,17cyclictest1448-21fschecks_count09:51:160
18387995216,21cyclictest0-21swapper/009:11:080
18387995216,21cyclictest0-21swapper/009:11:080
1838799519,21cyclictest12276-21timerandwakeup05:41:240
1838799517,21cyclictest0-21swapper/008:40:580
18387995130,5cyclictest7950irq/56-dwc_otg_07:11:120
18387995125,4cyclictest0-21swapper/006:29:270
18387995124,7cyclictest7750irq/56-dwc_otg06:46:110
18387995124,7cyclictest11898-21awk08:45:580
18387995124,7cyclictest11898-21awk08:45:580
1838799512,29cyclictest17056-21fschecks_time05:56:150
1838799511,7cyclictest24956-21unixbench_multi04:46:250
179242513,12sleep00-21swapper/004:28:250
1838799509,21cyclictest0-21swapper/008:51:130
1838799505,22cyclictest0-21swapper/004:36:190
18387995023,7cyclictest9206-21kernelversion09:21:170
18387995022,15cyclictest4531-21/usr/sbin/munin09:16:250
1838799501,28cyclictest0-21swapper/009:56:120
18387995012,6cyclictest16231-21unixbench_multi07:31:270
18387995012,6cyclictest16231-21unixbench_multi07:31:270
18387994923,3cyclictest0-21swapper/007:46:220
18387994921,15cyclictest21881-21/usr/sbin/munin09:36:220
18387994919,15cyclictest3986-21grep05:20:580
1838799482,30cyclictest0-21swapper/005:26:080
1838799481,26cyclictest0-21swapper/008:31:250
1839099471,6cyclictest7950irq/56-dwc_otg_07:11:083
1838999478,29cyclictest0-21swapper/206:26:032
1838799471,6cyclictest20049-21fschecks_count08:16:150
18387994716,22cyclictest9-21ksoftirqd/004:55:580
1838799471,24cyclictest0-21swapper/005:46:260
168262478,25sleep116929-21/usr/sbin/munin04:26:161
1838899467,26cyclictest0-21swapper/108:11:131
1838799469,10cyclictest7950irq/56-dwc_otg_06:31:120
1838799464,34cyclictest0-21swapper/005:21:180
18387994630,7cyclictest0-21swapper/009:46:250
18387994630,7cyclictest0-21swapper/009:46:250
18387994627,4cyclictest7750irq/56-dwc_otg09:26:170
1838799462,26cyclictest0-21swapper/007:09:310
18389994521,5cyclictest27539-21cut07:46:142
18389994520,7cyclictest14904-21latency_hist05:50:582
1838899456,19cyclictest7950irq/56-dwc_otg_08:26:031
1838899454,7cyclictest7750irq/56-dwc_otg09:46:111
1838899454,7cyclictest7750irq/56-dwc_otg09:46:111
1838799455,20cyclictest0-21swapper/007:36:170
1838799451,27cyclictest18642-21aten_rbpower_en06:01:110
1838899448,14cyclictest0-21swapper/107:01:131
1838799447,9cyclictest7850irq/56-dwc_otg_05:36:120
1838799447,9cyclictest7850irq/56-dwc_otg_05:36:120
1838799441,23cyclictest31615-21grep07:51:110
1838899437,25cyclictest0-21swapper/105:26:111
1838899437,22cyclictest0-21swapper/104:51:101
18388994317,6cyclictest29064-21fschecks_count09:06:161
18387994310,18cyclictest25113-21unixbench_multi06:16:250
255262420,4sleep31839099cyclictest09:01:233
1839099427,23cyclictest0-21swapper/304:51:103
18389994213,19cyclictest18429-21wget06:56:132
1838899427,22cyclictest0-21swapper/105:06:031
18388994219,5cyclictest30384-21gpgv07:10:581
18387994212,16cyclictest27469-21aten_rbpower_cu04:56:110
1838799421,19cyclictest0-21swapper/005:05:580
20312410,5sleep21838999cyclictest07:54:232
18390994118,5cyclictest5679-21latency_hist06:40:583
1838999413,24cyclictest0-21swapper/207:41:092
1838899417,22cyclictest0-21swapper/108:31:111
33842401,35sleep23394-21ssh09:53:092
1839099407,23cyclictest0-21swapper/306:26:033
18389994018,11cyclictest11292-21sh08:44:552
18389994018,11cyclictest11292-21sh08:44:552
18389994014,4cyclictest1335-21latency_hist06:35:582
1838899407,23cyclictest0-21swapper/106:51:101
1838899407,23cyclictest0-21swapper/106:51:101
1838899403,4cyclictest7750irq/56-dwc_otg09:21:581
1838799402,16cyclictest0-21swapper/004:35:580
314772390,1sleep10-21swapper/107:51:041
1839099396,24cyclictest25245-21ssh09:41:033
1838999398,23cyclictest0-21swapper/208:19:182
1838999392,32cyclictest12218-21ssh08:46:092
18389993918,6cyclictest14978-21/usr/sbin/munin06:51:222
18389993918,6cyclictest14978-21/usr/sbin/munin06:51:222
18389993914,5cyclictest17528-21grep08:51:272
1838899391,4cyclictest6413-21basename07:20:571
1838799391,17cyclictest26092-21sort06:21:150
1838799391,17cyclictest26092-21sort06:21:150
1839099381,4cyclictest7950irq/56-dwc_otg_05:31:093
1838999389,4cyclictest0-21swapper/208:36:142
18389993817,4cyclictest9284-21cron05:35:582
18389993812,13cyclictest7950irq/56-dwc_otg_04:31:242
1838899387,16cyclictest7950irq/56-dwc_otg_06:11:271
1838899385,22cyclictest1092-21ntpd06:36:091
1838899385,13cyclictest11906-21ssh08:45:581
1838899385,13cyclictest11906-21ssh08:45:581
1838899381,25cyclictest0-21swapper/105:31:111
137142380,5sleep31839099cyclictest09:26:243
1839099377,18cyclictest0-21swapper/307:41:093
18390993717,12cyclictest0-21swapper/309:31:083
1838999377,25cyclictest0-21swapper/209:41:042
1838999372,25cyclictest6851-21switchtime06:41:242
18389993717,4cyclictest24436-21if_wlan006:16:172
1838899376,6cyclictest19419-21cut07:36:141
1838899375,20cyclictest16935-21ssh09:31:101
18388993722,4cyclictest0-21swapper/108:56:241
1838899372,18cyclictest0-21swapper/109:41:161
166382376,18sleep216769-21/usr/sbin/munin04:26:132
65032360,4sleep36502-21sh07:59:593
1839099367,14cyclictest0-21swapper/304:31:113
18390993613,6cyclictest27904-21latency_hist08:25:583
1839099361,23cyclictest16524-21latency_hist05:55:583
1838999368,16cyclictest0-21swapper/205:21:102
1838999367,4cyclictest7850irq/56-dwc_otg_08:56:102
1838999367,25cyclictest0-21swapper/205:56:112
1838999362,18cyclictest0-21swapper/207:06:152
1838999360,27cyclictest0-21swapper/209:05:112
1838899361,28cyclictest17325-21switchtime08:51:251
65782351,2sleep30-21swapper/308:38:503
1839099358,23cyclictest0-21swapper/305:21:103
1839099355,21cyclictest6770-21ssh07:21:133
1839099353,3cyclictest7850irq/56-dwc_otg_08:56:093
18390993529,4cyclictest381ktimersoftd/308:11:123
18390993519,4cyclictest8145-21sh09:20:383
18390993517,4cyclictest908-21runrttasks08:29:183
18390993515,12cyclictest21442-21fschecks_time09:36:153
1839099351,26cyclictest14755-21ls05:50:573
1839099351,21cyclictest24955-21grep04:46:253
1838999359,18cyclictest0-21swapper/207:26:102
1838999359,14cyclictest0-21swapper/209:36:122
1838999353,5cyclictest29326-21diskmemload07:01:272
1838999352,26cyclictest0-21swapper/207:37:272
18389993514,5cyclictest31389-21fschecks_count05:06:152
1838899354,16cyclictest7850irq/56-dwc_otg_07:56:251
1838899351,25cyclictest7950irq/56-dwc_otg_09:01:091
1839099346,16cyclictest0-21swapper/308:41:103
1839099346,16cyclictest0-21swapper/308:41:103
1839099343,15cyclictest7950irq/56-dwc_otg_05:36:133
1839099343,15cyclictest7950irq/56-dwc_otg_05:36:133
1839099341,23cyclictest18601-21grep06:56:153
1838999348,5cyclictest7850irq/56-dwc_otg_06:25:592
1838999348,5cyclictest7850irq/56-dwc_otg_06:25:592
1838999348,11cyclictest10634-21latency_hist07:25:582
1838999347,24cyclictest0-21swapper/204:36:102
1838999347,15cyclictest0-21swapper/204:51:102
1838999346,11cyclictest7950irq/56-dwc_otg_04:46:252
1838999344,16cyclictest7950irq/56-dwc_otg_08:26:162
1838999342,4cyclictest0-21swapper/206:01:092
18389993416,5cyclictest30715-21sh09:07:392
1838899342,16cyclictest7950irq/56-dwc_otg_05:46:151
18388993412,12cyclictest0-21swapper/109:20:551
1838899341,20cyclictest16700-21ssh09:30:581
1838899341,17cyclictest0-21swapper/105:16:251
1838899341,16cyclictest16858-21wget05:56:121
133832340,1sleep30-21swapper/308:07:283
1839099337,16cyclictest0-21swapper/309:11:113
1839099337,16cyclictest0-21swapper/309:11:113
1839099332,7cyclictest7850irq/56-dwc_otg_09:51:183
18390993327,4cyclictest908-21runrttasks09:06:123
18390993323,7cyclictest5829-21ssh06:41:053
18390993318,5cyclictest8777-21sh10:00:283
1839099331,6cyclictest0-21swapper/307:26:143
1838999331,5cyclictest7850irq/56-dwc_otg_07:11:072
18389993313,5cyclictest908-21runrttasks04:56:212
18389993312,4cyclictest18802-21cron07:35:582
18389993312,4cyclictest18802-21cron07:35:582
18389993311,7cyclictest9963-21unixbench_singl09:21:262
1838999331,16cyclictest7950irq/56-dwc_otg_09:56:152
18389993311,4cyclictest2157-21kernelversion06:36:182
1838899337,16cyclictest0-21swapper/107:46:131
1838899336,7cyclictest7850irq/56-dwc_otg_06:21:261
1838899336,7cyclictest7850irq/56-dwc_otg_06:21:261
1838899335,5cyclictest0-21swapper/105:51:091
1838899335,13cyclictest7850irq/56-dwc_otg_07:31:181
1838899335,13cyclictest7850irq/56-dwc_otg_07:31:181
1838899335,12cyclictest7850irq/56-dwc_otg_06:46:271
1838899333,8cyclictest5715-21latency_hist05:25:581
1838899331,7cyclictest25068-21grep06:16:251
1839099326,5cyclictest7850irq/56-dwc_otg_05:20:573
1839099325,7cyclictest24598-21cat06:16:193
1839099324,5cyclictest7950irq/56-dwc_otg_06:25:593
18390993225,4cyclictest381ktimersoftd/306:46:273
1839099321,4cyclictest7950irq/56-dwc_otg_05:11:103
1838999327,4cyclictest0-21swapper/209:26:022
1838999327,12cyclictest787-21switchtime08:31:252
1838999325,4cyclictest7750irq/56-dwc_otg08:11:172
1838999323,15cyclictest7950irq/56-dwc_otg_05:26:182
18389993215,5cyclictest15302-21fschecks_time05:51:152
18389993214,4cyclictest11562-21aten_rbpower_en08:06:112
1838899327,15cyclictest0-21swapper/108:46:121
1838899321,22cyclictest24159-21cut04:46:151
18388993210,4cyclictest0-21swapper/107:41:131
1839099315,6cyclictest7750irq/56-dwc_otg08:31:243
18390993124,5cyclictest20570-21sh08:16:223
18390993113,9cyclictest27423-21apt-key06:25:583
18390993113,9cyclictest11572-21/usr/sbin/munin05:41:143
1839099311,19cyclictest1412-21cat06:35:583
1838999318,5cyclictest0-21swapper/206:11:182
1838999316,14cyclictest0-21swapper/206:06:112
18389993119,4cyclictest11346-21/usr/sbin/munin05:41:142
18389993111,5cyclictest10711-21missed_timers06:46:192
1838899311,6cyclictest7134-21unixbench_multi06:41:261
1839099309,4cyclictest0-21swapper/309:21:123
1839099306,13cyclictest7950irq/56-dwc_otg_05:26:163
1839099301,4cyclictest7950irq/56-dwc_otg_07:36:203
1839099301,18cyclictest21064-21switchtime04:36:223
1839099301,18cyclictest0-21swapper/308:48:413
1839099301,13cyclictest30330-21grep05:01:253
1838999308,2cyclictest0-21swapper/205:11:112
1838999306,14cyclictest0-21swapper/205:36:122
1838999306,14cyclictest0-21swapper/205:36:122
18389993013,8cyclictest23227-21sshd04:41:342
18389993011,5cyclictest29359-21cpu05:01:122
1838899301,7cyclictest10651-21grep05:36:271
1838899301,7cyclictest10651-21grep05:36:271
1838899301,24cyclictest761-21grep05:11:151
18388993012,4cyclictest25158-21unixbench_singl08:21:271
1838899301,20cyclictest11619-21uniq05:41:151
1838899301,16cyclictest0-21swapper/104:37:301
1838899301,14cyclictest7750irq/56-dwc_otg04:35:331
1839099299,5cyclictest31545-21taskset07:51:093
1839099297,5cyclictest0-21swapper/306:55:593
1839099295,5cyclictest23226-21chrt04:41:333
18390992914,4cyclictest31489-21fschecks_time05:06:163
1839099291,19cyclictest0-21swapper/307:49:063
1839099291,12cyclictest7950irq/56-dwc_otg_05:56:013
1839099287,12cyclictest0-21swapper/306:01:113
1839099286,15cyclictest16871-21ssh06:53:593
1839099285,12cyclictest39-21ksoftirqd/308:01:233
1839099285,12cyclictest39-21ksoftirqd/308:01:233
1839099281,18cyclictest0-21swapper/306:10:583
1839099281,13cyclictest0-21swapper/309:46:003
1839099281,13cyclictest0-21swapper/309:46:003
1838899287,8cyclictest0-21swapper/106:01:151
1838899286,4cyclictest7950irq/56-dwc_otg_04:41:151
165532283,15sleep30-21swapper/304:26:023
1839099275,9cyclictest7750irq/56-dwc_otg07:01:113
1839099275,10cyclictest22738-21ls06:11:193
1838899275,6cyclictest0-21swapper/106:06:141
1838999265,4cyclictest7850irq/56-dwc_otg_05:16:152
1839099251,17cyclictest0-21swapper/304:56:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional