You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-28 - 04:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Sun Dec 28, 2025 00:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3066699262,0cyclictest0-21swapper/121:40:311
30666992523,1cyclictest27092-21perf20:10:001
3066699251,23cyclictest25293-21cron23:25:001
3066699250,1cyclictest25968-21sendmail22:20:011
30673992422,1cyclictest0-21swapper/200:30:302
3066699242,21cyclictest0-21swapper/121:20:321
3066699241,22cyclictest0-21swapper/121:00:301
3066499241,22cyclictest6739-21perf20:35:010
30674992321,1cyclictest12998-21apt-get19:40:303
3067499230,2cyclictest23948-21idleruntime-cro21:10:013
3067399230,22cyclictest12894-21awk23:00:012
30666992321,1cyclictest0-21swapper/123:40:311
30666992321,1cyclictest0-21swapper/122:55:361
3066699231,21cyclictest0-21swapper/121:55:311
3066699230,1cyclictest1600-21munin-run21:30:011
30664992321,1cyclictest4337-21dump-pmu-power20:30:010
3066499230,22cyclictest24713-21awk20:05:000
3066499230,22cyclictest22658-21perf00:25:000
3066499230,22cyclictest12652-21cron00:05:010
30674992221,1cyclictest32261-21apt-get20:20:313
30674992221,0cyclictest14639-21apt-get20:50:303
3067499220,2cyclictest28714-21cron21:20:013
3067499220,2cyclictest28714-21cron21:20:013
3067499220,21cyclictest30931-21apt-get19:10:283
3067499220,21cyclictest21489-21apt-get22:10:313
3067499220,1cyclictest6409-21plymouth21:40:003
3067499220,1cyclictest4007-21munin-run21:35:013
3067499220,1cyclictest19449-21apt-get21:00:293
3067499220,0cyclictest0-21swapper/320:05:303
30673992221,1cyclictest17033-21apt-get20:55:302
30673992220,1cyclictest23951-21perf21:10:002
30673992220,1cyclictest20504-21grep23:15:012
3067399220,21cyclictest29781-21apt-get20:15:332
3067399220,21cyclictest23205-21apt-get23:20:302
3067399220,21cyclictest1967-21munin-run20:25:012
3067399220,21cyclictest18813-21munin-run22:05:012
3067399220,21cyclictest0-21swapper/221:20:022
3067399220,1cyclictest3101-21grep19:20:002
3067399220,1cyclictest30388-21apt-get23:35:292
30666992221,1cyclictest0-21swapper/100:20:311
30666992221,0cyclictest0-21swapper/100:05:301
3066699220,2cyclictest24690-21ls20:05:001
3066699220,21cyclictest4348-21wc20:30:011
3066699220,21cyclictest27691-21idleruntime-cro23:30:001
3066699220,1cyclictest606-21dbus-daemon20:25:011
3066699220,1cyclictest31917-21latency_hist20:20:001
3066699220,1cyclictest21195-21awk22:10:011
30664992221,0cyclictest0-21swapper/000:25:310
30664992220,2cyclictest11208-21cat21:50:000
30664992220,1cyclictest15600-21apt-get23:05:300
3066499220,21cyclictest8044-21apt-get23:55:320
3066499220,21cyclictest26343-21sleep21:15:010
3066499220,21cyclictest25307-21grep23:25:000
3066499220,21cyclictest0-21swapper/023:00:310
3066499220,21cyclictest0-21swapper/022:20:300
3066499220,21cyclictest0-21swapper/019:55:310
3066499220,1cyclictest7885-21dump-pmu-power19:30:000
3066499220,1cyclictest313-21systemd-journal21:30:010
3066499220,1cyclictest17033-21apt-get20:55:290
30674992120,1cyclictest13891-21apt-get21:55:313
30674992120,1cyclictest12950-21apt-get00:05:303
3067499210,21cyclictest0-21swapper/323:00:313
3067499210,20cyclictest8344-21apt-get22:50:303
3067499210,20cyclictest0-21swapper/323:30:333
3067499210,0cyclictest0-21swapper/322:45:283
3067499210,0cyclictest0-21swapper/320:45:313
3067499210,0cyclictest0-21swapper/320:35:303
3067499210,0cyclictest0-21swapper/319:55:313
30673992120,1cyclictest17449-21apt-get00:15:022
30673992120,0cyclictest8185-21apt-get19:30:322
30673992120,0cyclictest13949-21apt-get20:50:032
30673992120,0cyclictest10556-21apt-get00:00:302
30673992119,1cyclictest29459-21cron20:15:002
30673992119,1cyclictest17715-21munin-run23:10:012
3067399210,20cyclictest28144-21apt-get00:35:322
3067399210,20cyclictest28144-21apt-get00:35:322
3067399210,20cyclictest0-21swapper/223:30:332
3067399210,20cyclictest0-21swapper/220:35:302
3067399210,20cyclictest0-21swapper/219:45:332
3067399210,1cyclictest3555-21apt-get22:40:312
3067399210,0cyclictest0-21swapper/223:25:312
30666992121,0cyclictest0-21swapper/123:35:291
30666992121,0cyclictest0-21swapper/122:35:311
30666992121,0cyclictest0-21swapper/119:30:321
30666992119,1cyclictest842-21awk22:35:001
3066699211,11cyclictest0-21swapper/123:55:411
3066699210,20cyclictest0-21swapper/122:25:311
3066699210,20cyclictest0-21swapper/120:50:291
3066699210,1cyclictest648-21perf19:15:001
3066699210,1cyclictest0-21swapper/121:05:331
3066699210,0cyclictest0-21swapper/122:20:301
3066699210,0cyclictest0-21swapper/119:55:311
30664992119,1cyclictest31921-21munin-run20:20:000
3066499210,21cyclictest0-21swapper/000:15:310
3066499210,20cyclictest0-21swapper/019:10:290
3066499210,1cyclictest31096-21apt-get22:30:290
3066499210,1cyclictest0-21swapper/022:00:020
3066499210,1cyclictest0-21swapper/021:00:290
3066499210,0cyclictest0-21swapper/022:45:290
3066499210,0cyclictest0-21swapper/020:20:310
3066499210,0cyclictest0-21swapper/000:05:300
30674992020,0cyclictest0-21swapper/323:50:323
30674992020,0cyclictest0-21swapper/323:40:313
30674992020,0cyclictest0-21swapper/321:25:293
30674992020,0cyclictest0-21swapper/319:35:303
30674992020,0cyclictest0-21swapper/319:30:323
30674992018,1cyclictest6706-21apt-get21:40:323
3067499200,20cyclictest0-21swapper/322:20:303
3067499200,0cyclictest0-21swapper/320:10:323
3067499200,0cyclictest0-21swapper/319:15:303
30673992020,0cyclictest0-21swapper/220:30:312
30673992020,0cyclictest0-21swapper/219:10:292
30673992020,0cyclictest0-21swapper/200:20:312
30673992019,0cyclictest0-21swapper/221:00:302
30673992019,0cyclictest0-21swapper/200:10:322
30673992018,1cyclictest313-21systemd-journal00:00:012
3067399200,20cyclictest0-21swapper/220:00:302
3067399200,20cyclictest0-21swapper/220:00:012
3067399200,1cyclictest1893-21apt-get21:30:312
3067399200,19cyclictest5650-21apt-get23:50:322
3067399200,0cyclictest0-21swapper/223:00:312
3066699203,16cyclictest7734-21idleruntime-cro23:55:011
30666992020,0cyclictest0-21swapper/122:10:311
30666992020,0cyclictest0-21swapper/120:35:301
30666992020,0cyclictest0-21swapper/119:35:301
30666992020,0cyclictest0-21swapper/100:15:021
30666992019,0cyclictest24241-21apt-get21:10:311
30666992019,0cyclictest0-21swapper/119:15:301
30666992018,1cyclictest25460-21perf00:30:011
30666992018,1cyclictest0-21swapper/122:45:001
30666992017,2cyclictest30110-21munin-run23:35:001
30666992017,2cyclictest0-21swapper/123:15:351
30666992017,2cyclictest0-21swapper/121:45:401
3066699200,20cyclictest0-21swapper/100:00:301
3066699200,0cyclictest0-21swapper/123:45:311
3066699200,0cyclictest0-21swapper/121:15:301
3066699200,0cyclictest0-21swapper/121:15:301
30664992020,0cyclictest0-21swapper/023:45:310
30664992020,0cyclictest0-21swapper/023:30:330
30664992020,0cyclictest0-21swapper/021:20:020
30664992020,0cyclictest0-21swapper/000:35:320
30664992020,0cyclictest0-21swapper/000:35:320
30664992019,1cyclictest13954-21perf20:50:010
30664992019,1cyclictest0-21swapper/023:10:020
30664992019,0cyclictest23882-21apt-get22:15:300
30664992018,1cyclictest56762sleep022:45:000
3066499200,0cyclictest0-21swapper/022:50:300
3066499200,0cyclictest0-21swapper/022:25:310
3066499200,0cyclictest0-21swapper/020:51:230
30674991918,0cyclictest1127-21apt-get22:35:313
30674991918,0cyclictest0-21swapper/321:45:333
3067499190,18cyclictest19100-21apt-get22:05:343
30673991919,0cyclictest9453-21apt-get20:40:312
30673991919,0cyclictest26633-21apt-get21:15:302
30673991919,0cyclictest26633-21apt-get21:15:302
30673991919,0cyclictest20785-21apt-get23:15:312
30673991919,0cyclictest0-21swapper/222:05:342
30673991918,1cyclictest0-21swapper/222:15:302
30673991918,0cyclictest0-21swapper/221:45:332
30673991916,2cyclictest2946-21cron23:45:012
30666991919,0cyclictest0-21swapper/123:10:021
30666991919,0cyclictest0-21swapper/123:00:311
30666991919,0cyclictest0-21swapper/100:10:321
30666991917,2cyclictest0-21swapper/119:50:331
30666991916,2cyclictest19163-21perf21:00:001
30666991916,2cyclictest0-21swapper/119:45:421
3066699190,0cyclictest0-21swapper/100:30:311
30664991919,0cyclictest0-21swapper/022:55:360
30664991919,0cyclictest0-21swapper/022:10:310
30674991818,0cyclictest23356-21apt-get00:25:313
30674991818,0cyclictest0-21swapper/323:55:313
30674991818,0cyclictest0-21swapper/320:25:303
30674991818,0cyclictest0-21swapper/300:30:303
30674991817,1cyclictest22596-21apt-get20:00:303
30674991816,1cyclictest18067-21apt-get00:15:023
3067499180,17cyclictest0-21swapper/321:10:313
30673991818,0cyclictest0-21swapper/219:20:302
30673991817,1cyclictest22674-21apt-get00:25:022
30673991817,0cyclictest0-21swapper/200:05:302
30673991816,1cyclictest10452-21turbostat22:55:012
30666991818,0cyclictest0-21swapper/123:05:301
30666991818,0cyclictest0-21swapper/120:40:311
30666991818,0cyclictest0-21swapper/120:10:321
30666991816,1cyclictest15112-21awk19:45:011
30664991818,0cyclictest0-21swapper/020:05:300
30664991817,1cyclictest2969-21perf23:45:010
30664991817,0cyclictest0-21swapper/022:05:370
30664991816,1cyclictest3052-21cron19:20:000
3066499180,17cyclictest5482-21dump-pmu-power19:25:000
30674991717,0cyclictest0-21swapper/323:25:003
30674991717,0cyclictest0-21swapper/322:40:303
30674991717,0cyclictest0-21swapper/322:15:303
30674991716,0cyclictest11495-21apt-get21:50:313
3067499170,16cyclictest19873-21apt19:55:013
3067399177,9cyclictest5949-21apt-get22:45:292
30673991717,0cyclictest3253-21apt-get23:45:312
30673991717,0cyclictest31849-21apt-get21:25:292
30673991717,0cyclictest0-21swapper/222:35:312
30673991717,0cyclictest0-21swapper/220:50:012
30673991716,1cyclictest13532-21munin-node22:30:342
3067399170,0cyclictest0-21swapper/221:45:002
30666991717,0cyclictest0-21swapper/121:50:311
30666991716,1cyclictest0-21swapper/122:00:301
30666991715,1cyclictest0-21swapper/120:35:011
30664991717,0cyclictest62550irq/127-eno121:07:020
30664991717,0cyclictest0-21swapper/019:30:320
30664991715,1cyclictest22877-21dump-pmu-power23:20:000
30674991616,0cyclictest0-21swapper/320:30:303
30674991614,1cyclictest0-21swapper/300:25:013
3067499160,6cyclictest0-21swapper/323:45:313
3067499160,3cyclictest0-21swapper/323:35:293
30666991616,0cyclictest0-21swapper/121:30:311
30666991616,0cyclictest0-21swapper/120:45:301
30666991615,1cyclictest4695-21grep19:20:371
30666991615,1cyclictest0-21swapper/100:35:341
30666991615,1cyclictest0-21swapper/100:35:341
30666991614,1cyclictest10457-21munin-run22:55:011
30664991616,0cyclictest0-21swapper/020:45:010
30664991616,0cyclictest0-21swapper/019:40:300
30664991616,0cyclictest0-21swapper/019:35:300
30664991614,1cyclictest27702-21latency_hist23:30:000
30664991614,1cyclictest0-21swapper/021:40:000
30664991614,1cyclictest0-21swapper/000:10:350
3066499160,7cyclictest27834-21cron00:35:010
3066499160,15cyclictest1-21systemd19:46:390
3067499151,7cyclictest313-21systemd-journal22:05:013
30674991515,0cyclictest28736-21apt-get21:20:023
30674991515,0cyclictest0-21swapper/320:15:333
3067499150,5cyclictest0-21swapper/320:55:293
30673991515,0cyclictest0-21swapper/221:10:312
30673991515,0cyclictest0-21swapper/220:05:302
30673991513,1cyclictest5787-21apt-get19:25:312
30673991513,1cyclictest28696-21apt-get22:25:312
30673991513,1cyclictest19898-21munin-run19:55:012
3067399150,1cyclictest16087-21apt-config22:00:012
30666991515,0cyclictest22-21ksoftirqd/119:26:101
30664991515,0cyclictest0-21swapper/022:35:310
3066499150,0cyclictest0-21swapper/023:40:010
30674991414,0cyclictest0-21swapper/323:15:283
30674991414,0cyclictest0-21swapper/323:10:133
30674991414,0cyclictest0-21swapper/322:25:513
30674991414,0cyclictest0-21swapper/300:35:323
30674991414,0cyclictest0-21swapper/300:35:323
30674991411,1cyclictest32198-21ntp_states22:30:343
3067499140,14cyclictest0-21swapper/323:25:053
3067499140,14cyclictest0-21swapper/323:05:043
3067499140,14cyclictest0-21swapper/320:40:313
3067499140,14cyclictest0-21swapper/319:25:133
3067499140,14cyclictest0-21swapper/300:10:033
3067499140,14cyclictest0-21swapper/300:01:243
3067499140,0cyclictest0-21swapper/319:46:093
3067499140,0cyclictest0-21swapper/319:20:123
30673991414,0cyclictest0-21swapper/219:40:502
3067399140,14cyclictest0-21swapper/222:20:252
3067399140,14cyclictest0-21swapper/221:51:592
3067399140,14cyclictest0-21swapper/219:35:142
3067399140,0cyclictest0-21swapper/222:10:162
3067399140,0cyclictest0-21swapper/221:35:052
3067399140,0cyclictest0-21swapper/220:26:052
30666991414,0cyclictest0-21swapper/121:35:111
3066699140,14cyclictest0-21swapper/122:45:031
30664991414,0cyclictest0-21swapper/021:55:210
30664991414,0cyclictest0-21swapper/020:10:180
30664991413,1cyclictest0-21swapper/021:40:310
3066499140,14cyclictest0-21swapper/019:50:200
3066499140,0cyclictest7036-21apt-get20:35:300
3066499140,0cyclictest0-21swapper/023:50:280
3066499140,0cyclictest0-21swapper/021:51:180
3066499140,0cyclictest0-21swapper/021:30:030
3066499140,0cyclictest0-21swapper/021:15:180
3066499140,0cyclictest0-21swapper/021:15:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional