You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-26 - 10:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Wed Aug 26, 2026 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3130399270,26cyclictest24157-21diskmemload23:39:513
3130399270,22cyclictest24157-21diskmemload22:06:103
3130399270,1cyclictest5717-21expr21:50:403
3130399270,1cyclictest0-21swapper/322:40:013
3130399260,23cyclictest0-21swapper/300:21:283
3130399260,22cyclictest24157-21diskmemload21:22:483
31303992522,2cyclictest24157-21diskmemload23:30:593
31303992521,4cyclictest24157-21diskmemload21:10:483
3130399251,1cyclictest0-21swapper/322:23:053
3130399250,24cyclictest16126-21munin-run22:00:013
3130399250,24cyclictest0-21swapper/322:53:393
3130399250,24cyclictest0-21swapper/322:04:343
3130399250,21cyclictest0-21swapper/323:46:163
31299992523,1cyclictest24157-21diskmemload22:36:342
3129999250,1cyclictest24157-21diskmemload21:29:482
3129999250,1cyclictest0-21swapper/221:36:022
3129499250,24cyclictest4033-21apt-get19:20:391
3129099250,24cyclictest24231-21munin-run23:05:010
3129099250,24cyclictest24157-21diskmemload22:26:550
31303992422,2cyclictest0-21swapper/300:00:043
31303992422,1cyclictest32277-21grep00:10:003
31303992422,1cyclictest0-21swapper/300:35:423
31303992420,4cyclictest0-21swapper/319:50:403
31303992420,3cyclictest636-21apt-get00:10:413
31303992420,3cyclictest24157-21diskmemload23:55:443
31303992420,3cyclictest0-21swapper/323:53:503
31303992420,3cyclictest0-21swapper/322:55:263
31303992420,3cyclictest0-21swapper/300:32:473
3130399240,23cyclictest22088-21apt-get21:05:393
3130399240,23cyclictest18221-21sh23:27:513
3130399240,23cyclictest0-21swapper/323:15:393
3130399240,23cyclictest0-21swapper/321:44:413
3130399240,0cyclictest24157-21diskmemload21:15:183
3130399240,0cyclictest0-21swapper/322:46:173
3130399240,0cyclictest0-21swapper/321:30:403
31299992422,1cyclictest0-21swapper/223:15:402
31299992420,3cyclictest0-21swapper/222:24:152
31299992420,0cyclictest0-21swapper/200:07:222
3129999240,24cyclictest24157-21diskmemload23:42:252
3129999240,23cyclictest24157-21diskmemload21:47:572
3129999240,23cyclictest1294-21ssh22:44:562
3129999240,23cyclictest0-21swapper/223:28:092
3129999240,23cyclictest0-21swapper/223:09:212
3129999240,23cyclictest0-21swapper/222:08:402
31294992422,1cyclictest8350-21ssh21:52:161
3129499241,1cyclictest5716-21diskmemload00:15:021
3129499240,24cyclictest0-21swapper/122:53:211
3129499240,24cyclictest0-21swapper/121:49:381
3129499240,23cyclictest24157-21diskmemload23:51:141
3129499240,23cyclictest24157-21diskmemload23:20:081
3129499240,23cyclictest16174-21apt-get23:55:411
3129499240,1cyclictest24157-21diskmemload00:25:051
3129499240,0cyclictest0-21swapper/100:38:051
31290992422,1cyclictest29760-21ssh21:42:290
31290992421,2cyclictest16867-21ssh23:26:020
3129099240,24cyclictest0-21swapper/022:33:490
3129099240,24cyclictest0-21swapper/022:20:100
3129099240,23cyclictest0-21swapper/021:56:180
3129099240,1cyclictest24157-21diskmemload21:32:300
31303992323,0cyclictest0-21swapper/323:20:133
31303992323,0cyclictest0-21swapper/322:38:213
31303992323,0cyclictest0-21swapper/322:15:443
31303992320,2cyclictest0-21swapper/322:30:363
31303992319,3cyclictest24157-21diskmemload21:27:473
3130399230,22cyclictest6998-21irqstats00:15:443
3130399230,22cyclictest32683-21latency23:40:433
3130399230,22cyclictest0-21swapper/323:01:303
3130399230,0cyclictest24157-21diskmemload23:13:413
3130399230,0cyclictest24157-21diskmemload21:45:353
3130399230,0cyclictest0-21swapper/322:14:053
31299992323,0cyclictest5451-21apt-get21:50:402
31299992323,0cyclictest24157-21diskmemload22:13:502
31299992323,0cyclictest0-21swapper/223:47:542
31299992323,0cyclictest0-21swapper/221:13:172
31299992322,1cyclictest24157-21diskmemload22:58:332
31299992321,1cyclictest1050-21cron00:40:012
31299992320,2cyclictest1870-21sh22:15:442
3129999230,2cyclictest739-21ssh21:15:552
3129999230,23cyclictest24157-21diskmemload22:00:332
3129999230,22cyclictest18922-21diskmemload23:00:082
3129999230,22cyclictest18125-21cat19:50:012
3129999230,22cyclictest14416-21rm22:26:322
3129999230,21cyclictest151rcu_preempt22:32:002
3129999230,0cyclictest24157-21diskmemload00:25:332
3129999230,0cyclictest0-21swapper/200:22:362
31294992323,0cyclictest24157-21diskmemload00:24:091
31294992323,0cyclictest0-21swapper/123:03:321
31294992323,0cyclictest0-21swapper/119:15:391
31294992323,0cyclictest0-21swapper/100:32:451
31294992321,1cyclictest0-21swapper/122:47:021
31294992320,2cyclictest2731-21ssh23:43:041
31294992320,2cyclictest20497-21apt19:55:001
3129499230,23cyclictest0-21swapper/123:39:121
3129499230,23cyclictest0-21swapper/123:30:371
3129499230,23cyclictest0-21swapper/122:33:361
3129499230,22cyclictest32515-21idleruntime-cro20:20:001
3129499230,22cyclictest25070-21ssh00:02:401
3129499230,22cyclictest24157-21diskmemload21:33:351
3129499230,22cyclictest22680-21cstates22:05:421
3129499230,22cyclictest17432-21sh23:26:451
3129499230,22cyclictest12243-21ssh21:26:281
3129499230,0cyclictest24157-21diskmemload22:35:451
31290992323,0cyclictest24157-21diskmemload21:45:470
31290992323,0cyclictest0-21swapper/023:15:590
31290992323,0cyclictest0-21swapper/021:51:300
31290992323,0cyclictest0-21swapper/021:16:470
31290992323,0cyclictest0-21swapper/000:34:500
31290992322,1cyclictest21454-21sh00:00:110
31290992321,1cyclictest5885-21mailstats21:20:490
31290992321,1cyclictest1351-21ssh23:12:490
31290992320,2cyclictest30434-21kworker/0:200:00:000
3129099230,23cyclictest0-21swapper/022:40:300
3129099230,23cyclictest0-21swapper/000:11:360
3129099230,22cyclictest24361-21sh23:33:420
3129099230,22cyclictest24157-21diskmemload22:16:070
3129099230,22cyclictest23071-21sh21:36:010
3129099230,22cyclictest10699-21smartctl21:25:430
3129099230,22cyclictest0-21swapper/022:39:100
3129099230,1cyclictest0-21swapper/023:23:210
3129099230,0cyclictest0-21swapper/022:51:150
31303992220,1cyclictest0-21swapper/320:50:003
31303992219,3cyclictest24157-21diskmemload00:25:043
31303992219,3cyclictest0-21swapper/321:35:303
3130399220,21cyclictest24157-21diskmemload23:05:283
3130399220,21cyclictest0-21swapper/322:25:033
3130399220,21cyclictest0-21swapper/320:00:393
31299992222,0cyclictest24157-21diskmemload00:18:142
31299992222,0cyclictest24157-21diskmemload00:10:122
31299992222,0cyclictest0-21swapper/223:51:222
31299992220,1cyclictest0-21swapper/223:20:302
31299992220,1cyclictest0-21swapper/222:50:372
3129999220,22cyclictest0-21swapper/200:31:522
3129999220,21cyclictest28026-21apt-get20:10:402
3129999220,21cyclictest25629-21apt-get20:05:392
3129999220,21cyclictest24157-21diskmemload00:00:482
3129999220,21cyclictest1550-21apt-get19:15:402
3129999220,21cyclictest0-21swapper/223:35:312
3129999220,21cyclictest0-21swapper/222:45:072
3129999220,21cyclictest0-21swapper/221:55:432
3129999220,21cyclictest0-21swapper/221:40:512
3129999220,21cyclictest0-21swapper/221:20:432
3129999220,1cyclictest20820-21apt-get19:55:392
3129999220,1cyclictest15309-21sh21:30:182
3129999220,1cyclictest0-21swapper/223:57:112
3129999220,0cyclictest0-21swapper/223:30:262
31294992222,0cyclictest24157-21diskmemload22:40:011
31294992222,0cyclictest0-21swapper/123:46:581
31294992222,0cyclictest0-21swapper/123:11:351
31294992222,0cyclictest0-21swapper/121:57:301
31294992221,1cyclictest25170-21ssh21:10:281
31294992221,1cyclictest23235-21apt-get20:00:391
31294992220,2cyclictest20289-21ssh22:03:201
31294992220,1cyclictest29674-21sh00:06:251
31294992220,1cyclictest2899-21sh22:15:531
31294992220,1cyclictest24157-21diskmemload22:25:401
3129499220,21cyclictest12490-21apt-get20:45:391
3129499220,21cyclictest0-21swapper/119:50:011
3129499220,21cyclictest0-21swapper/119:35:401
3129499220,1cyclictest8645-21ssh22:21:001
3129499220,1cyclictest4152-21diskstats23:15:411
3129499220,1cyclictest24157-21diskmemload22:11:301
3129499220,1cyclictest22457-21/usr/sbin/munin21:35:471
31290992221,1cyclictest28026-21apt-get20:10:400
31290992221,1cyclictest0-21swapper/000:25:360
31290992220,1cyclictest2065-21apt-get22:45:390
31290992220,1cyclictest0-21swapper/023:45:020
3129099220,2cyclictest12194-21munin-plugin-st20:45:000
3129099220,21cyclictest24926-21apt-get23:05:390
3129099220,21cyclictest21792-21ls21:05:000
3129099220,21cyclictest1064-21perf00:40:010
3129099220,21cyclictest0-21swapper/023:50:350
3129099220,21cyclictest0-21swapper/022:55:130
3129099220,21cyclictest0-21swapper/022:10:160
3129099220,21cyclictest0-21swapper/020:15:400
3129099220,21cyclictest0-21swapper/000:15:400
3129099220,21cyclictest0-21swapper/000:05:030
3129099220,1cyclictest31676-21sh23:40:360
3129099220,1cyclictest26034-21ssh23:35:370
3129099220,1cyclictest22193-21ssh22:05:350
3129099220,0cyclictest0-21swapper/021:12:060
31303992120,0cyclictest0-21swapper/320:30:393
31303992119,1cyclictest2512-21sh20:25:013
3130399210,20cyclictest0-21swapper/321:00:403
3130399210,1cyclictest0-21swapper/319:40:393
31299992120,0cyclictest5282-21apt-get20:30:402
3129999210,20cyclictest31576-21apt-get19:10:402
3129999210,20cyclictest0-21swapper/220:50:412
3129999210,1cyclictest0-21swapper/223:10:212
31294992121,0cyclictest24157-21diskmemload22:55:111
31294992121,0cyclictest22903-21sendmail20:00:001
3129499210,1cyclictest4980-21idleruntime-cro20:30:011
3129499210,0cyclictest26832-21ssh23:05:571
3129499210,0cyclictest0-21swapper/121:20:131
31290992120,0cyclictest0-21swapper/019:35:400
3129099210,21cyclictest0-21swapper/019:25:380
3129099210,20cyclictest0-21swapper/020:50:410
3130399200,1cyclictest0-21swapper/320:10:453
3130399200,1cyclictest0-21swapper/319:10:413
3130399200,19cyclictest0-21swapper/319:45:393
31299992020,0cyclictest0-21swapper/220:25:392
31299992020,0cyclictest0-21swapper/220:20:402
31299992018,1cyclictest19398-21munin-run21:00:002
31299992018,1cyclictest0-21swapper/219:55:002
31294992020,0cyclictest0-21swapper/120:30:401
31294992020,0cyclictest0-21swapper/120:20:401
31294992019,1cyclictest7678-21apt-get20:35:401
31290992020,0cyclictest0-21swapper/020:45:390
31290992020,0cyclictest0-21swapper/020:00:390
31290992020,0cyclictest0-21swapper/019:30:390
31290992020,0cyclictest0-21swapper/019:15:390
31290992019,1cyclictest383-21apt-get20:20:400
31303991919,0cyclictest0-21swapper/319:25:383
31303991918,0cyclictest0-21swapper/319:55:393
31303991915,3cyclictest0-21swapper/320:20:003
3130399190,1cyclictest0-21swapper/320:40:473
3130399190,18cyclictest0-21swapper/319:20:003
3130399190,0cyclictest0-21swapper/319:30:393
31299991919,0cyclictest0-21swapper/220:00:392
31299991918,1cyclictest8822-21apt-get19:30:392
31294991919,0cyclictest10094-21apt-get20:40:401
31294991919,0cyclictest0-21swapper/120:10:401
31290991918,0cyclictest0-21swapper/019:45:390
31290991917,1cyclictest0-21swapper/019:40:400
31303991818,0cyclictest0-21swapper/319:35:413
31299991818,0cyclictest0-21swapper/220:35:402
31294991818,0cyclictest0-21swapper/121:00:401
31294991816,1cyclictest0-21swapper/121:05:391
31290991817,0cyclictest25629-21apt-get20:05:390
3130399170,1cyclictest0-21swapper/320:50:403
3130399170,16cyclictest4954-21cron20:30:013
3130399170,0cyclictest0-21swapper/320:35:393
31299991716,1cyclictest30420-21apt-get20:15:402
31290991716,1cyclictest0-21swapper/019:25:000
31290991714,2cyclictest3560-21if_enp1s0.820:25:420
3130399160,1cyclictest27777-21munin-plugin-st20:10:003
31299991616,0cyclictest0-21swapper/221:05:392
31299991616,0cyclictest0-21swapper/219:40:392
31294991614,1cyclictest27732-21munin-run20:10:001
31290991616,0cyclictest0-21swapper/020:30:390
3129099160,15cyclictest0-21swapper/020:55:510
3130399150,1cyclictest0-21swapper/320:55:533
3129999154,9cyclictest345-21systemd-udevd21:00:492
31299991515,0cyclictest0-21swapper/220:40:402
3129999150,1cyclictest14071-21cat20:45:522
31294991515,0cyclictest0-21swapper/119:45:011
31294991515,0cyclictest0-21swapper/119:25:381
3130399140,14cyclictest0-21swapper/319:20:053
3129999140,14cyclictest0-21swapper/219:35:382
3129999140,14cyclictest0-21swapper/219:20:162
3129999140,0cyclictest0-21swapper/219:25:472
3129499140,14cyclictest0-21swapper/120:55:061
3129499140,14cyclictest0-21swapper/119:30:061
3129499140,14cyclictest0-21swapper/119:11:391
31290991414,0cyclictest0-21swapper/019:10:510
3129099140,14cyclictest0-21swapper/021:05:150
3129099140,14cyclictest0-21swapper/020:35:220
3129099140,14cyclictest0-21swapper/019:55:530
3129099140,0cyclictest0-21swapper/019:50:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional