You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-19 - 09:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Aug 19, 2026 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1325899601,39cyclictest6947-21ssh23:55:142
1325899601,39cyclictest6947-21ssh23:55:132
13258995739,12cyclictest14164-21sh00:35:132
1325899551,38cyclictest21290-21ssh23:45:132
1325899531,38cyclictest4837-21ssh23:00:162
1325899461,29cyclictest25008-21ssh00:05:162
13238994420,7cyclictest12-21ksoftirqd/020:35:130
1325899431,33cyclictest16012-21ssh00:00:142
1323899431,40cyclictest3517-21ssh21:30:130
13273994235,4cyclictest35966irq/58-eth022:40:123
1325899425,27cyclictest27-21ksoftirqd/222:35:142
1324799411,34cyclictest0-21swapper/119:45:141
13238994128,9cyclictest12-21ksoftirqd/000:30:130
1323899403,24cyclictest16456-21ssh21:55:130
13238993914,5cyclictest131rcu_preempt22:35:140
1325899371,26cyclictest11293-21ssh22:10:132
1324799375,29cyclictest28544-21ssh00:25:131
1324799375,28cyclictest29564-21ssh22:20:151
1323899375,3cyclictest131rcu_preempt21:20:150
1325899362,27cyclictest0-21swapper/222:55:132
1325899362,27cyclictest0-21swapper/222:55:122
13273993524,7cyclictest35966irq/58-eth022:50:153
1324799351,32cyclictest10607-21ssh00:15:141
13238993421,5cyclictest12-21ksoftirqd/023:25:110
1324799331,3cyclictest131rcu_preempt23:30:131
1323899331,28cyclictest19575-21ssh00:20:110
1323899331,26cyclictest29542-21sh22:20:150
13258993217,5cyclictest261rcuc/222:45:132
1327399314,17cyclictest35966irq/58-eth020:40:143
1325899311,14cyclictest26758-21ssh21:25:132
1323899312,24cyclictest10613-21ssh00:15:160
1327399303,16cyclictest35966irq/58-eth021:45:143
1327399302,19cyclictest35966irq/58-eth021:40:133
1327399302,19cyclictest35966irq/58-eth021:40:123
13273993017,8cyclictest35966irq/58-eth022:20:293
1327399301,18cyclictest35966irq/58-eth021:15:133
1327399294,20cyclictest35966irq/58-eth000:00:143
13273992914,9cyclictest35966irq/58-eth023:25:133
13273992911,12cyclictest35966irq/58-eth021:30:143
1325899293,19cyclictest27346irq/40-dwc2_hso20:20:132
1325899293,11cyclictest0-21swapper/220:45:052
1325899291,10cyclictest808-21ssh22:40:122
1324799291,24cyclictest8547-21ssh23:20:101
1323899292,21cyclictest30304-21ssh23:50:130
1323899292,21cyclictest30304-21ssh23:50:120
1325899282,17cyclictest27346irq/40-dwc2_hso23:35:132
1325899281,16cyclictest17640-21ssh23:25:132
1327399272,15cyclictest35966irq/58-eth023:50:133
1327399272,15cyclictest35966irq/58-eth023:50:123
1325899275,13cyclictest27346irq/40-dwc2_hso20:10:142
1325899270,18cyclictest30553-21uniq23:50:172
1325899270,18cyclictest30553-21uniq23:50:172
13247992713,10cyclictest0-21swapper/100:20:121
1323899275,20cyclictest20387-21ssh22:15:120
1327399264,14cyclictest0-21swapper/323:45:133
1325899262,17cyclictest21704-21rm21:40:132
1325899262,17cyclictest21704-21rm21:40:122
1325899262,15cyclictest12368-21ssh23:40:142
1325899262,14cyclictest8596-21ssh21:15:122
13247992613,5cyclictest21-21ksoftirqd/123:35:131
1327399252,18cyclictest35966irq/58-eth000:25:143
1327399252,18cyclictest0-21swapper/323:35:123
1327399251,15cyclictest35966irq/58-eth022:35:143
1325899256,9cyclictest2892-21cron23:35:012
1325899253,13cyclictest27346irq/40-dwc2_hso22:25:142
1324799253,18cyclictest20417-21ssh22:15:141
1327399244,17cyclictest35966irq/58-eth021:25:143
1327399243,18cyclictest35966irq/58-eth021:20:163
1327399242,5cyclictest0-21swapper/322:10:153
1327399242,15cyclictest35966irq/58-eth000:20:113
1327399241,18cyclictest0-21swapper/323:40:143
1327399241,17cyclictest0-21swapper/320:30:143
1325899248,11cyclictest26863-21latency_hist21:05:022
1325899242,11cyclictest0-21swapper/221:45:162
1325899241,15cyclictest17739-21ssh21:20:172
13258992411,4cyclictest25132-21sh22:00:022
1324799241,19cyclictest2159-21ssh22:05:131
1324799241,18cyclictest5146-21rm00:30:131
13238992416,6cyclictest13406-21ssh23:20:300
1327399233,13cyclictest0-21swapper/322:30:143
1327399233,13cyclictest0-21swapper/322:30:133
1327399233,10cyclictest0-21swapper/300:35:123
1327399232,15cyclictest35966irq/58-eth023:30:143
1325899233,14cyclictest27346irq/40-dwc2_hso00:10:162
1325899233,13cyclictest27346irq/40-dwc2_hso20:15:152
1325899232,6cyclictest0-21swapper/200:30:142
1325899232,6cyclictest0-21swapper/200:28:142
1325899232,14cyclictest27346irq/40-dwc2_hso21:35:142
13258992318,3cyclictest14983-21ssh00:15:322
13258992312,9cyclictest5643-21ssh22:24:512
1325899231,14cyclictest27346irq/40-dwc2_hso19:30:132
1324799234,11cyclictest0-21swapper/121:00:141
1324799232,15cyclictest3510-21diskmemload21:30:131
1323899234,15cyclictest21311-21ssh23:45:140
1323899232,16cyclictest19582-21apt-get19:50:120
13238992316,5cyclictest2178-21ssh22:05:140
1327399225,12cyclictest0-21swapper/321:10:143
1327399223,16cyclictest0-21swapper/323:05:143
1327399222,11cyclictest0-21swapper/322:15:143
13273992213,6cyclictest35966irq/58-eth023:20:133
1325899222,16cyclictest15215-21ssh22:30:132
1325899222,16cyclictest15215-21ssh22:30:132
1325899222,14cyclictest27346irq/40-dwc2_hso20:40:142
13258992211,9cyclictest9212-21ssh22:08:032
1325899221,17cyclictest20511-21kthreadcore19:50:192
13258992210,5cyclictest3811-21sort21:30:182
1324799226,11cyclictest21-21ksoftirqd/121:55:151
13247992211,3cyclictest131rcu_preempt19:40:001
1323899227,4cyclictest131rcu_preempt00:35:210
13238992217,3cyclictest16027-21ssh00:00:160
13238992213,6cyclictest26752-21ssh21:25:130
13238992212,7cyclictest167-21systemd-network20:30:150
13238992211,9cyclictest31140-21ssh22:55:220
13238992211,9cyclictest31140-21ssh22:55:210
1327399214,14cyclictest0-21swapper/322:25:143
1327399214,12cyclictest0-21swapper/321:55:153
1327399213,14cyclictest0-21swapper/322:00:153
1327399211,13cyclictest35966irq/58-eth000:15:163
1325899218,10cyclictest0-21swapper/221:15:022
1325899213,10cyclictest28143-21cron00:25:012
13258992115,3cyclictest26393-21ssh22:00:202
1325899211,16cyclictest0-21swapper/223:05:132
1325899211,11cyclictest30978-21kthreadcore19:25:202
1325899211,11cyclictest15760-21kthreadcore19:45:222
1325899211,10cyclictest31545-21munin-run23:15:022
1325899211,10cyclictest21898-21kthreadcore22:50:232
1325899211,10cyclictest12887-21kthreadcore19:40:242
1325899211,10cyclictest12887-21kthreadcore19:40:232
1324799215,4cyclictest0-21swapper/123:12:241
13247992113,6cyclictest31067-1kworker/u9:122:40:161
13247992112,6cyclictest3304-21latency_hist21:30:031
13247992112,6cyclictest19005-21ssh00:35:301
13247992112,6cyclictest1-21systemd22:26:281
13247992111,7cyclictest12484-21ssh23:40:141
1324799211,15cyclictest16241-21ssh21:35:271
1324799211,14cyclictest25535-21ssh22:00:151
13247992110,4cyclictest31524-21diskmemload22:40:011
1323899215,5cyclictest0-21swapper/020:55:130
13238992114,5cyclictest9949-21ssh21:15:200
1327399209,7cyclictest0-21swapper/300:05:183
1327399203,13cyclictest0-21swapper/323:00:153
1327399203,11cyclictest0-21swapper/319:25:123
13273992010,8cyclictest0-21swapper/323:11:493
13273992010,7cyclictest35966irq/58-eth000:10:153
1325899207,11cyclictest30798-21tail21:05:282
1325899204,12cyclictest27346irq/40-dwc2_hso19:20:142
1325899203,7cyclictest0-21swapper/223:15:132
1325899202,9cyclictest31505-21kthreadcore20:00:212
1325899202,9cyclictest15006-21kthreadcore20:50:222
1325899201,9cyclictest755-21kthreadcore20:35:242
1325899201,15cyclictest22965-21latency_hist19:20:022
1325899201,14cyclictest13394-21apt-get19:10:132
1325899201,11cyclictest28005-21kthreadcore20:30:212
1325899201,11cyclictest10996-21ntp_192.168.11421:50:242
1325899201,10cyclictest9672-21kthreadcore23:20:192
1325899201,10cyclictest5275-21grep19:35:162
1325899200,10cyclictest21166-21kthreadcore22:15:202
1324799201,12cyclictest31524-21diskmemload23:18:021
1324799201,10cyclictest31524-21diskmemload23:52:051
1324799201,10cyclictest31524-21diskmemload23:52:041
13247992010,7cyclictest26072-21ssh21:40:311
13247992010,7cyclictest26072-21ssh21:40:301
1323899201,5cyclictest0-21swapper/023:02:290
13238992013,5cyclictest18080-21ssh22:30:230
13238992013,5cyclictest18080-21ssh22:30:220
13238992012,6cyclictest3499-21ssh21:10:300
13238992010,7cyclictest2645-21ssh22:40:200
1327399192,12cyclictest0-21swapper/321:50:283
1327399192,10cyclictest0-21swapper/321:05:153
13273991910,6cyclictest0-21swapper/322:58:063
13273991910,6cyclictest0-21swapper/322:58:063
1325899192,9cyclictest3452-21kthreadcore20:05:262
1325899191,9cyclictest24544-21fschecks_time19:55:182
1325899191,9cyclictest20415-21kthreadcore20:55:222
1325899191,11cyclictest20786-21sshd20:25:132
1325899191,11cyclictest13224-21cyclictest19:10:022
1324799192,14cyclictest23037-21apt-get19:20:131
1324799191,9cyclictest0-21swapper/123:45:491
1324799191,9cyclictest0-21swapper/122:30:171
1324799191,9cyclictest0-21swapper/122:30:161
1324799191,4cyclictest0-21swapper/121:10:181
13247991913,4cyclictest0-21swapper/120:10:251
13247991912,5cyclictest19146-21ssh22:50:171
13247991911,5cyclictest3067-21kthreadcore00:10:211
13247991911,5cyclictest2032-21sshd22:57:151
13247991911,5cyclictest2032-21sshd22:57:141
1324799191,15cyclictest13847-21ssh23:05:141
1324799191,10cyclictest28077-21timerandwakeup19:55:291
1324799191,10cyclictest0-21swapper/123:29:341
13247991910,6cyclictest8426-21ssh23:00:261
1323899199,7cyclictest10189-21cron00:15:010
1323899197,9cyclictest17368-21ssh21:35:340
13238991913,4cyclictest28783-21fschecks_time00:25:180
13238991913,4cyclictest0-21swapper/000:05:180
13238991913,3cyclictest12-21ksoftirqd/023:15:120
13238991911,6cyclictest26887-21ssh22:00:220
13238991911,6cyclictest24904-21ssh23:10:230
1323899191,12cyclictest0-21swapper/019:25:060
1327399189,6cyclictest0-21swapper/321:39:523
1327399188,8cyclictest0-21swapper/323:16:193
1327399182,10cyclictest0-21swapper/320:50:153
1327399182,10cyclictest0-21swapper/320:35:153
1327399182,10cyclictest0-21swapper/320:20:143
1327399182,10cyclictest0-21swapper/320:15:153
1327399182,10cyclictest0-21swapper/320:10:143
1327399181,9cyclictest0-21swapper/320:45:123
13273991813,3cyclictest33-21ksoftirqd/319:15:263
13273991810,6cyclictest0-21swapper/300:32:233
1324799189,7cyclictest13761-21grep19:40:311
1324799189,7cyclictest13761-21grep19:40:311
1324799189,6cyclictest4885-21cron19:35:021
1324799189,6cyclictest30727-21timerwakeupswit21:05:291
1324799186,9cyclictest2652-21grep20:40:021
1324799186,10cyclictest7875-21tune2fs20:45:161
1324799183,12cyclictest1863-21/usr/sbin/munin21:45:251
1324799181,8cyclictest9614-21kthreadcore21:15:201
1324799181,7cyclictest28009-21sort19:25:111
1324799181,4cyclictest131rcu_preempt00:03:411
13247991812,3cyclictest20555-21munin-plugin-st20:25:021
1324799181,13cyclictest1096-21apt00:10:021
13247991810,6cyclictest13115-21cut20:50:181
13247991810,5cyclictest7314-21/usr/sbin/munin21:50:231
1324799180,3cyclictest131rcu_preempt22:10:141
1323899189,7cyclictest16829-21ssh23:05:210
1323899189,6cyclictest3399-21ssh23:35:140
1323899189,6cyclictest17136-21cron20:55:020
1323899184,11cyclictest20899-21awk23:45:010
1323899184,11cyclictest1540-21kthreadcore19:30:200
1323899183,12cyclictest9933-21ssh22:45:140
1323899181,14cyclictest10061-21ls22:25:290
13238991811,4cyclictest0-21swapper/021:50:130
1323899181,13cyclictest30767-21ssh21:45:140
13238991810,6cyclictest31506-21sshd21:06:160
1323899180,16cyclictest16697-21ls19:10:230
1327399172,10cyclictest0-21swapper/323:55:143
1327399172,10cyclictest0-21swapper/323:55:133
13273991712,3cyclictest33-21ksoftirqd/320:05:223
13273991712,3cyclictest33-21ksoftirqd/319:20:193
1327399171,13cyclictest0-21swapper/322:05:213
1327399171,11cyclictest0-21swapper/322:45:163
1324799178,6cyclictest3102-21ps20:40:181
1324799173,12cyclictest29512-21irqcore20:00:191
1324799172,2cyclictest131rcu_preempt21:22:461
1324799171,8cyclictest25549-21/usr/sbin/munin20:30:251
1324799171,2cyclictest131rcu_preempt23:56:531
1324799171,2cyclictest131rcu_preempt23:56:521
1324799171,13cyclictest10670-21ssh22:45:191
1324799171,13cyclictest0-21swapper/119:10:151
1323899179,6cyclictest26741-21sh21:05:010
1323899179,5cyclictest0-21swapper/020:40:240
1323899174,5cyclictest27072-21irqcore23:30:190
1323899174,10cyclictest20644-21kthreadcore19:15:220
1323899172,12cyclictest6941-21ssh23:55:140
1323899172,12cyclictest6941-21ssh23:55:130
1323899172,12cyclictest19019-21ssh22:50:140
1323899171,14cyclictest11289-21apt-get22:10:140
13238991711,3cyclictest31524-21diskmemload21:40:280
13238991711,3cyclictest31524-21diskmemload21:40:270
1323899171,13cyclictest20783-21apt-get20:25:120
13238991710,4cyclictest0-21swapper/019:20:250
1327399168,5cyclictest33-21ksoftirqd/320:55:303
1327399162,10cyclictest0-21swapper/321:00:153
1327399161,9cyclictest35966irq/58-eth020:00:153
1327399161,10cyclictest0-21swapper/319:55:153
1324799168,6cyclictest21997-21kthreadcore19:50:221
1324799168,6cyclictest20863-21grep20:55:281
1324799168,5cyclictest18397-21sed19:15:171
1324799162,8cyclictest0-21swapper/120:05:181
1323899163,11cyclictest10929-21cron20:15:010
1323899162,10cyclictest0-21swapper/020:05:190
1323899161,9cyclictest0-21swapper/019:40:160
1323899161,9cyclictest0-21swapper/019:40:150
1323899161,6cyclictest16118-21cut19:45:220
13238991611,3cyclictest5457-21grep19:35:190
13238991610,4cyclictest0-21swapper/020:00:180
1323899160,4cyclictest53646usb-storage20:45:100
1327399157,5cyclictest0-21swapper/319:30:133
1327399151,7cyclictest0-21swapper/319:40:023
1324799154,9cyclictest21135-21sed20:25:181
1324799151,10cyclictest11482-21irqcore20:15:181
1323899159,4cyclictest10178-1kworker/0:0H20:15:180
1323899158,5cyclictest26035-21kthreadcore19:55:220
1323899154,8cyclictest19086-21kthreadcore20:20:220
1327399149,3cyclictest33-21ksoftirqd/319:45:003
1327399149,3cyclictest33-21ksoftirqd/319:44:593
1327399149,3cyclictest33-21ksoftirqd/319:10:283
1323899141,10cyclictest13361-21latency_hist19:10:020
1327399134,6cyclictest0-21swapper/319:45:153
1327399133,6cyclictest0-21swapper/319:50:133
1327399133,4cyclictest0-21swapper/320:25:133
1324799125,5cyclictest13261-21grep19:10:011
1327399111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional