You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-11-17 - 14:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Nov 17, 2026 00:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13265997453,12cyclictest33366irq/58-eth000:00:143
1323299666,48cyclictest25054-21ssh22:00:150
1325599542,34cyclictest17431-21ssh21:20:102
1323299544,35cyclictest5117-21ssh22:25:120
1324599534,38cyclictest21383-21ssh23:10:101
1323299535,35cyclictest8467-21ssh21:15:130
1323299524,36cyclictest32126-21ssh22:40:130
1323299524,36cyclictest32126-21ssh22:40:130
13245995116,29cyclictest0-21swapper/119:50:131
1324599504,40cyclictest32111-21ssh22:40:121
1324599504,40cyclictest32111-21ssh22:40:121
1323299501,9cyclictest131rcu_preempt21:40:130
1323299501,9cyclictest131rcu_preempt21:40:120
1323299492,7cyclictest131rcu_preempt22:05:120
1323299492,7cyclictest131rcu_preempt22:05:120
1323299483,8cyclictest131rcu_preempt20:05:130
1323299453,29cyclictest5310-21ssh23:55:130
1326599436,23cyclictest33366irq/58-eth019:30:133
1325599412,31cyclictest24319-21cut23:10:202
13255994118,16cyclictest0-21swapper/219:50:132
1323299401,31cyclictest26813-21ssh00:25:130
1323299401,31cyclictest26813-21ssh00:25:130
13265993926,7cyclictest33366irq/58-eth023:50:133
1325599392,29cyclictest10558-21ssh23:03:202
1325599391,31cyclictest8619-21ssh22:45:122
13245993926,9cyclictest28793-21ssh23:50:121
1325599382,28cyclictest27462-21cut23:30:202
1325599381,3cyclictest1960746ssh22:15:132
13245993827,7cyclictest0-21swapper/121:15:121
13232993818,8cyclictest12-21ksoftirqd/022:35:110
1325599361,29cyclictest1054-21snmpd23:35:132
1324599363,27cyclictest31889-21ssh21:10:151
13245993630,2cyclictest26525-21apt-get21:25:131
1325599350,25cyclictest30273-21ssh21:45:132
1324599359,9cyclictest9120-21ssh21:31:521
1323299357,19cyclictest628-21tcf-agent21:00:130
1326599342,21cyclictest33366irq/58-eth000:20:133
1326599342,15cyclictest33366irq/58-eth022:10:113
1325599341,28cyclictest22346irq/40-dwc2_hso21:10:132
13245993427,2cyclictest3285-21apt-get00:30:121
13245993426,5cyclictest5155-21apt-get19:35:121
13245993411,12cyclictest12839-21unixbench_multi00:15:281
1324599339,10cyclictest2244-21ssh22:22:011
1324599338,9cyclictest12459-21df_inode23:05:131
13245993310,10cyclictest30572-21ssh23:31:091
1326599323,16cyclictest33366irq/58-eth021:10:133
1326599322,16cyclictest33366irq/58-eth023:40:103
1324599328,8cyclictest6603-21ssh21:50:021
1324599328,8cyclictest20753-21ssh22:32:401
1324599327,9cyclictest25007-21sshd00:23:201
1324599327,9cyclictest23327-21ssh23:45:241
1324599327,8cyclictest22974-21ssh22:51:051
13245993221,8cyclictest23532-21find22:15:281
13245993212,10cyclictest4422-21cut23:00:181
13245993211,8cyclictest31472-21diskmemload21:55:281
1323299323,21cyclictest19578-21ssh22:15:110
1324599313,21cyclictest23270-21ssh00:05:131
13245993115,12cyclictest1597-21ssh22:05:131
13245993115,12cyclictest1597-21ssh22:05:131
13245993110,8cyclictest0-21swapper/123:27:481
1325599303,21cyclictest0-21swapper/223:23:092
1324599309,8cyclictest31472-21diskmemload21:38:051
1324599309,8cyclictest31472-21diskmemload21:38:051
1324599308,6cyclictest0-21swapper/123:40:261
1324599308,10cyclictest8572-21ssh22:45:101
1324599308,10cyclictest20109-21ssh00:01:401
13245993018,9cyclictest10533-21apt-get22:10:101
13245993017,9cyclictest10595-21irqstats19:40:211
1324599301,25cyclictest12316-21sh00:35:131
13245993011,7cyclictest28883-21kworker/u8:122:02:541
13245993010,8cyclictest31472-21diskmemload21:42:231
13245993010,8cyclictest31472-21diskmemload21:42:231
1323299304,22cyclictest17772-21ssh00:20:120
1323299302,20cyclictest3310-21ssh00:30:130
1323299300,23cyclictest0-21swapper/021:45:130
1325599293,17cyclictest22346irq/40-dwc2_hso20:25:142
1325599292,20cyclictest0-21swapper/220:50:112
1325599292,17cyclictest28585-21ssh22:20:132
1325599292,11cyclictest18644-21ssh00:37:202
1325599292,10cyclictest32449-21kthreadcore23:15:202
1324599297,8cyclictest28582-21rm22:36:161
13245992923,4cyclictest30468-21ssh23:15:121
13245992919,8cyclictest73-21kswapd023:55:161
13245992916,9cyclictest26693-21apt-get22:55:151
13245992915,10cyclictest0-21swapper/100:10:111
13232992920,5cyclictest14174-21sh22:30:130
13232992916,9cyclictest0-21swapper/019:20:120
1325599289,7cyclictest27-21ksoftirqd/223:57:082
1325599285,15cyclictest261rcuc/220:00:202
1325599285,15cyclictest261rcuc/220:00:202
1325599283,20cyclictest0-21swapper/222:53:022
1325599282,20cyclictest10554-21sh22:10:122
1325599282,19cyclictest0-21swapper/219:30:142
1325599281,19cyclictest868-21munin-node23:25:142
1324599287,7cyclictest0-21swapper/122:29:071
13245992818,7cyclictest0-21swapper/119:55:131
13245992814,10cyclictest16867-21ntpq19:10:251
1323299283,9cyclictest0-21swapper/023:09:200
1323299283,20cyclictest8789-21ssh00:15:120
1323299282,18cyclictest6113-21ssh23:00:200
13232992810,11cyclictest25973-1kworker/u9:023:35:130
1325599279,12cyclictest261rcuc/223:45:122
1325599277,4cyclictest27-21ksoftirqd/221:33:352
1325599277,11cyclictest131rcu_preempt00:01:322
1325599273,5cyclictest271ksoftirqd/200:30:192
1325599273,17cyclictest22346irq/40-dwc2_hso20:10:122
1325599271,22cyclictest31472-21diskmemload23:40:102
1325599271,18cyclictest13595-21cut19:10:162
1324599277,7cyclictest11034-21timerwakeupswit21:50:291
1324599276,7cyclictest1465-21sh00:28:151
1324599276,7cyclictest1465-21sh00:28:151
1324599276,7cyclictest0-21swapper/123:39:021
13245992715,9cyclictest2565-21users20:35:311
13245992715,8cyclictest14774-21apt-get19:45:121
13245992712,11cyclictest0-21swapper/119:25:131
1323299272,20cyclictest31472-21diskmemload00:38:230
1323299272,19cyclictest14169-21ssh23:23:090
13232992712,4cyclictest12-21ksoftirqd/022:55:150
1326599265,12cyclictest33566irq/59-eth021:55:123
1326599260,13cyclictest33366irq/58-eth021:20:103
1325599267,7cyclictest27-21ksoftirqd/200:11:462
1325599266,4cyclictest27-21ksoftirqd/200:26:552
1325599266,4cyclictest27-21ksoftirqd/200:26:552
1325599262,17cyclictest22346irq/40-dwc2_hso21:55:122
1325599261,13cyclictest31472-21diskmemload22:30:132
1324599266,7cyclictest6136-21latency_hist20:10:011
13245992618,3cyclictest0-21swapper/119:30:131
13245992615,7cyclictest25639-21apt-get20:30:051
13245992614,8cyclictest0-21swapper/120:55:121
13245992612,10cyclictest0-21swapper/120:40:121
1323299263,17cyclictest10556-21ssh22:10:120
13232992619,5cyclictest0-21swapper/021:35:140
13232992619,5cyclictest0-21swapper/021:35:140
1325599254,13cyclictest22346irq/40-dwc2_hso20:45:132
1325599254,13cyclictest22346irq/40-dwc2_hso20:35:142
1325599252,19cyclictest5348-21fschecks_time19:35:152
1325599252,18cyclictest600-21rngd22:55:202
1325599251,13cyclictest5123-21sh22:25:132
1325599251,12cyclictest1054-21snmpd23:50:242
13255992510,8cyclictest27-21ksoftirqd/221:18:532
13245992516,7cyclictest25484-21awk21:00:231
13245992514,8cyclictest7076-21ssh23:20:121
13245992513,9cyclictest22323-21kthreadcore20:25:201
13245992513,9cyclictest17374-21ssh21:20:071
13245992513,8cyclictest12201-21vmstat20:45:331
13245992512,9cyclictest19452-21ntp_states20:20:261
13245992511,10cyclictest0-21swapper/120:10:121
1323299255,11cyclictest13228-21cyclictest19:45:120
1323299252,20cyclictest28495-21ssh23:13:070
1323299252,17cyclictest28602-21rm22:20:150
1326599243,15cyclictest0-21swapper/321:50:143
1326599242,13cyclictest33366irq/58-eth021:45:113
1326599241,15cyclictest33366irq/58-eth022:15:133
1325599248,9cyclictest27-21ksoftirqd/200:20:132
1325599248,8cyclictest27-21ksoftirqd/222:40:412
1325599248,8cyclictest27-21ksoftirqd/222:40:412
1325599248,4cyclictest27-21ksoftirqd/221:35:082
1325599248,4cyclictest27-21ksoftirqd/221:35:082
1325599247,8cyclictest27-21ksoftirqd/221:54:142
1325599241,8cyclictest131rcu_preempt22:35:182
1325599241,19cyclictest14643-21kthreadcore23:05:192
13255992411,4cyclictest27-21ksoftirqd/221:25:122
13255992410,4cyclictest27-21ksoftirqd/200:17:042
1324599247,14cyclictest1347-21idleruntime-cro20:05:011
1324599247,14cyclictest1347-21idleruntime-cro20:05:011
1323299242,18cyclictest32246-21ssh00:10:120
13232992414,5cyclictest542-21usb-storage19:35:110
1326599235,10cyclictest33566irq/59-eth023:30:123
13265992313,7cyclictest0-21swapper/323:40:013
1325599237,13cyclictest23389-21kthreadcore21:40:202
1325599237,13cyclictest23389-21kthreadcore21:40:202
1325599235,9cyclictest27-21ksoftirqd/220:30:142
1325599232,4cyclictest271ksoftirqd/200:05:182
1323299231,8cyclictest21347-21kthreadcore19:50:200
1326599226,11cyclictest0-21swapper/322:53:013
1326599223,13cyclictest0-21swapper/323:07:203
13265992215,5cyclictest0-21swapper/323:15:133
13265992210,9cyclictest0-21swapper/322:40:523
13265992210,9cyclictest0-21swapper/322:40:523
1325599229,4cyclictest27-21ksoftirqd/222:05:512
1325599229,4cyclictest27-21ksoftirqd/222:05:512
1325599227,12cyclictest0-21swapper/219:25:252
1325599226,8cyclictest27-21ksoftirqd/219:15:152
1325599226,4cyclictest27-21ksoftirqd/221:00:302
1325599223,3cyclictest131rcu_preempt22:00:222
1325599222,15cyclictest14847-21cpuspeed19:45:132
1324599227,11cyclictest11375-21fschecks_time20:15:201
13245992214,4cyclictest23077-21apt-get19:20:121
13245992212,7cyclictest0-21swapper/120:50:111
13245992210,9cyclictest27161-21cat21:05:161
1326599216,11cyclictest0-21swapper/323:25:143
1326599213,11cyclictest33366irq/58-eth023:45:143
1326599212,13cyclictest33366irq/58-eth000:05:133
13265992115,4cyclictest0-21swapper/322:20:143
13265992115,4cyclictest0-21swapper/321:43:363
13265992115,4cyclictest0-21swapper/321:43:363
13265992114,4cyclictest0-21swapper/322:32:353
13265992113,6cyclictest0-21swapper/300:15:233
13265992113,5cyclictest0-21swapper/300:12:113
13265992111,7cyclictest0-21swapper/322:07:003
13265992111,7cyclictest0-21swapper/322:06:593
1326599211,16cyclictest0-21swapper/321:15:283
1325599213,4cyclictest27-21ksoftirqd/220:55:222
1325599212,16cyclictest20582-21cat20:25:012
1325599211,17cyclictest134-21systemd-journal21:10:012
13245992111,8cyclictest18463-21hwlatdetect19:15:171
1323299211,16cyclictest28948-21latency_hist20:00:020
1323299211,14cyclictest3072-21ssh21:10:260
13265992014,4cyclictest321rcuc/322:03:473
13265992014,4cyclictest0-21swapper/323:55:133
13265992014,4cyclictest0-21swapper/323:20:133
13265992014,4cyclictest0-21swapper/323:10:193
13265992014,4cyclictest0-21swapper/321:35:103
13265992014,4cyclictest0-21swapper/321:35:103
13265992014,4cyclictest0-21swapper/321:30:023
13265992014,4cyclictest0-21swapper/321:26:323
13265992014,4cyclictest0-21swapper/300:25:233
13265992014,4cyclictest0-21swapper/300:25:233
13265992013,5cyclictest0-21swapper/322:55:203
13265992013,5cyclictest0-21swapper/322:35:183
13265992011,7cyclictest0-21swapper/322:25:093
13265992011,6cyclictest33366irq/58-eth022:45:123
1325599202,9cyclictest13733-21kthreadcore20:15:252
13255992013,5cyclictest261rcuc/220:05:172
1325599201,11cyclictest27422-21kthreadcore19:55:232
1325599201,11cyclictest10106-21grep19:40:182
1325599201,10cyclictest3003-21sort20:40:162
1325599201,10cyclictest26241-21kthreadcore19:20:222
1324599206,11cyclictest13183-21latency_hist19:10:021
1323299207,3cyclictest131rcu_preempt00:05:180
1323299202,15cyclictest30451-21apt-get23:15:110
1323299202,14cyclictest32406-21netstat23:50:240
1323299202,13cyclictest16118-21df21:55:140
1323299201,7cyclictest16265-21sh23:25:150
13232992013,2cyclictest0-21swapper/021:30:120
1323299201,13cyclictest2708-21latency_hist21:30:010
13232992010,8cyclictest25659-21apt-get20:30:130
13232992010,8cyclictest13251-21cat22:45:290
1326599192,11cyclictest0-21swapper/320:30:143
13265991913,4cyclictest0-21swapper/323:00:123
13265991913,4cyclictest0-21swapper/321:10:023
13265991913,4cyclictest0-21swapper/320:35:313
13265991913,4cyclictest0-21swapper/320:10:263
13265991913,4cyclictest0-21swapper/319:35:183
13265991913,4cyclictest0-21swapper/319:15:313
13265991913,4cyclictest0-21swapper/300:35:153
13265991913,4cyclictest0-21swapper/300:30:143
1326599191,12cyclictest0-21swapper/319:50:023
1323299197,3cyclictest131rcu_preempt21:20:110
1323299193,9cyclictest0-21swapper/023:30:200
1323299193,8cyclictest18025-21kthreadcore20:55:190
1323299193,12cyclictest28335-21apt-get23:50:020
1323299193,10cyclictest10747-21ssh23:40:120
1323299191,9cyclictest9716-21missed_timers20:10:230
1323299191,8cyclictest10916-21find21:50:290
1326599182,10cyclictest0-21swapper/320:25:143
1326599182,10cyclictest0-21swapper/319:50:133
1326599182,10cyclictest0-21swapper/319:20:143
13265991812,4cyclictest0-21swapper/321:00:163
13265991812,4cyclictest0-21swapper/320:45:323
13265991812,4cyclictest0-21swapper/320:40:183
13265991812,4cyclictest0-21swapper/320:15:263
13265991812,4cyclictest0-21swapper/320:05:203
13265991812,4cyclictest0-21swapper/319:55:193
13265991812,4cyclictest0-21swapper/319:40:203
13265991812,4cyclictest0-21swapper/319:10:223
1326599181,12cyclictest321rcuc/320:55:123
1325599181,9cyclictest13279-21latency_hist19:10:022
1323299186,8cyclictest0-21swapper/019:40:160
1323299183,11cyclictest0-21swapper/020:15:170
1323299182,12cyclictest13797-21apt-get00:00:030
1323299181,8cyclictest8877-21kthreadcore20:45:220
1323299181,14cyclictest17583-21/usr/sbin/munin22:50:180
13265991712,3cyclictest0-21swapper/320:20:203
13265991711,4cyclictest0-21swapper/319:25:173
13265991711,3cyclictest0-21swapper/320:00:203
13265991711,3cyclictest0-21swapper/320:00:203
13265991710,3cyclictest321rcuc/320:50:113
1323299178,6cyclictest16211-21sed20:20:170
1323299173,6cyclictest0-21swapper/019:10:200
1323299173,11cyclictest31974-21kthreadcore20:35:210
1323299173,11cyclictest20256-21kthreadcore19:15:210
1323299172,5cyclictest12-21ksoftirqd/020:25:160
1323299171,5cyclictest32461-21/usr/sbin/munin19:30:010
1323299163,10cyclictest435-21hwlatdetect19:30:170
1323299162,11cyclictest29986-21kthreadcore20:00:190
1323299162,11cyclictest29986-21kthreadcore20:00:180
1323299162,10cyclictest13411-21kthreadcore20:50:170
1323299161,13cyclictest3879-21kthreadcore20:40:190
1323299161,11cyclictest31520-21latency_hist21:10:020
1323299151,10cyclictest13350-21latency_hist19:10:020
1326599101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional