You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 09:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Wed Aug 27, 2025 00:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16672992825,2cyclictest612-21gdbus23:05:003
1666799270,1cyclictest23821-21turbostat19:25:002
1666799270,1cyclictest0-21swapper/200:00:002
16657992422,1cyclictest0-21swapper/019:35:000
1665799240,23cyclictest15876-21turbostat20:15:010
1667299231,21cyclictest0-21swapper/320:15:003
1667299230,2cyclictest23369-21cron23:50:003
1667299230,22cyclictest18196-21apt-get21:29:363
1667299230,22cyclictest10375-21awk21:10:013
1667299230,22cyclictest10375-21awk21:10:013
1666799230,23cyclictest0-21swapper/223:00:012
1665799230,22cyclictest30729-21idleruntime-cro20:44:590
1665799230,22cyclictest21069-21cron20:25:010
16672992222,0cyclictest0-21swapper/321:44:353
16672992220,1cyclictest22753-21idleruntime-cro21:34:593
16672992220,1cyclictest10974-21apt-get21:14:373
1667299220,21cyclictest8914-21grep23:20:003
1667299220,21cyclictest6112-21apt-get00:24:373
1667299220,21cyclictest5451-21apt-get22:09:353
1667299220,21cyclictest2460-21basename22:00:003
1667299220,21cyclictest19303-21sleep22:35:003
1667299220,21cyclictest18314-21munin-run20:20:003
1667299220,21cyclictest0-21swapper/323:44:413
1667299220,21cyclictest0-21swapper/320:39:383
1667299220,1cyclictest15167-21awk21:20:003
16667992222,0cyclictest0-21swapper/223:24:362
16667992220,1cyclictest313-21systemd-journal19:45:012
16667992220,1cyclictest16493-21apt-get20:19:362
16667992220,1cyclictest15113-21apt-get22:29:372
16667992220,1cyclictest10705-21munin-run20:05:002
1666799220,21cyclictest0-21swapper/222:44:402
1666799220,21cyclictest0-21swapper/220:44:372
1666799220,1cyclictest6437-21apt-get19:59:362
1666799220,1cyclictest593-21dbus-daemon22:40:012
1666799220,1cyclictest24009-21apt-get23:54:372
1666799220,1cyclictest0-21swapper/223:04:422
16660992222,0cyclictest0-21swapper/122:09:341
16660992220,1cyclictest3437-21latency_hist19:49:591
16660992220,1cyclictest19304-21awk22:35:011
1666099220,21cyclictest26226-21perf19:30:011
1666099220,21cyclictest2328-21apt-get23:09:351
1666099220,21cyclictest1298-21apt-get20:54:351
1666099220,21cyclictest0-21swapper/122:59:371
1666099220,21cyclictest0-21swapper/121:50:011
1666099220,21cyclictest0-21swapper/121:44:371
1666099220,21cyclictest0-21swapper/120:20:021
1666099220,21cyclictest0-21swapper/119:14:391
1666099220,1cyclictest1-21systemd23:30:001
16657992222,0cyclictest32348-21apt-get23:04:410
16657992221,1cyclictest12720-21apt-get22:24:390
16657992219,2cyclictest30705-21munin-run00:04:590
16657992219,2cyclictest19982-21perf21:30:000
1665799220,22cyclictest0-21swapper/021:38:310
1665799220,21cyclictest25857-21apt-get21:44:350
1665799220,21cyclictest0-21swapper/023:19:360
1665799220,1cyclictest0-21swapper/023:40:010
16672992121,0cyclictest0-21swapper/300:09:423
16672992120,0cyclictest0-21swapper/322:44:383
16672992118,2cyclictest31309-21sh23:00:013
1667299210,21cyclictest0-21swapper/322:19:373
1667299210,20cyclictest0-21swapper/319:19:353
1667299210,0cyclictest0-21swapper/322:39:363
16667992120,0cyclictest19312-21apt-get20:24:352
16667992119,1cyclictest27656-21munin-run21:45:002
16667992119,1cyclictest21004-21perf23:45:002
1666799210,21cyclictest0-21swapper/220:59:412
1666799210,21cyclictest0-21swapper/219:34:342
1666799210,20cyclictest10904-21apt-get00:34:362
1666799210,1cyclictest7252-21munin-run22:10:002
1666799210,1cyclictest0-21swapper/222:24:382
1666799210,0cyclictest0-21swapper/200:02:112
16660992121,0cyclictest15790-21apt-get21:24:371
16660992121,0cyclictest0-21swapper/121:09:391
16660992121,0cyclictest0-21swapper/121:09:391
16660992121,0cyclictest0-21swapper/121:04:351
16660992121,0cyclictest0-21swapper/119:44:351
16660992120,1cyclictest13298-21apt-get00:39:361
16660992120,1cyclictest0-21swapper/123:59:371
16660992120,0cyclictest0-21swapper/122:29:381
16660992120,0cyclictest0-21swapper/119:24:371
16660992119,1cyclictest5531-21sendmail-msp21:00:001
1666099210,21cyclictest0-21swapper/120:04:351
1666099210,20cyclictest0-21swapper/121:19:351
1666099210,0cyclictest21711-21apt-get20:29:381
1666099210,0cyclictest0-21swapper/119:19:351
1666099210,0cyclictest0-21swapper/100:24:361
16657992121,0cyclictest31315-21apt-get00:09:410
16657992120,0cyclictest11304-21apt-get20:09:380
16657992119,1cyclictest7882-21cron21:05:000
16657992119,1cyclictest3400-21cron19:49:590
1665799210,21cyclictest0-21swapper/023:09:350
1665799210,20cyclictest29235-21apt-get19:39:360
1665799210,20cyclictest24736-21apt-get22:49:370
1665799210,0cyclictest0-21swapper/023:29:390
1665799210,0cyclictest0-21swapper/022:04:390
1665799210,0cyclictest0-21swapper/021:24:370
16672992020,0cyclictest9539-21apt-get23:24:363
16672992020,0cyclictest30650-21apt-get21:54:343
16672992020,0cyclictest0-21swapper/323:39:393
16672992020,0cyclictest0-21swapper/323:09:363
16672992020,0cyclictest0-21swapper/322:49:373
16672992020,0cyclictest0-21swapper/320:29:373
16672992020,0cyclictest0-21swapper/320:24:353
16667992020,0cyclictest27135-21apt-get22:54:402
16667992020,0cyclictest0-21swapper/222:04:402
16667992020,0cyclictest0-21swapper/221:49:372
16667992020,0cyclictest0-21swapper/220:09:372
16667992020,0cyclictest0-21swapper/200:39:362
16667992019,1cyclictest0-21swapper/222:14:392
16667992018,1cyclictest593-21dbus-daemon00:10:012
1666799200,20cyclictest0-21swapper/219:14:392
1666799200,0cyclictest0-21swapper/220:54:352
1666799200,0cyclictest0-21swapper/220:39:382
1666799200,0cyclictest0-21swapper/219:39:372
16660992020,0cyclictest0-21swapper/121:14:361
16660992020,0cyclictest0-21swapper/120:10:011
16660992020,0cyclictest0-21swapper/119:39:371
16660992019,1cyclictest26836-21apt-get19:34:351
16660992019,1cyclictest25240-21perf21:40:001
1666099200,20cyclictest0-21swapper/122:39:371
1666099200,19cyclictest0-21swapper/123:24:361
16657992020,0cyclictest19622-21apt-get19:19:360
16657992020,0cyclictest0-21swapper/022:14:390
16657992020,0cyclictest0-21swapper/021:14:370
16657992020,0cyclictest0-21swapper/019:54:360
16657992020,0cyclictest0-21swapper/000:39:370
16657992020,0cyclictest0-21swapper/000:19:370
16657992019,0cyclictest0-21swapper/022:59:370
16657992019,0cyclictest0-21swapper/020:05:010
1665799200,0cyclictest0-21swapper/022:19:380
1665799200,0cyclictest0-21swapper/021:54:340
1665799200,0cyclictest0-21swapper/021:34:360
1665799200,0cyclictest0-21swapper/020:49:350
16672991919,0cyclictest7853-21apt-get22:14:403
16672991919,0cyclictest28256-21apt-get21:49:373
16672991919,0cyclictest24437-21apt-get19:29:373
16672991919,0cyclictest1635-21apt-get19:49:363
16672991919,0cyclictest0-21swapper/320:34:383
16672991919,0cyclictest0-21swapper/319:39:363
16672991918,1cyclictest0-21swapper/321:24:363
1667299190,0cyclictest0-21swapper/319:24:363
16667991919,0cyclictest0-21swapper/223:09:362
16667991919,0cyclictest0-21swapper/221:14:362
16667991918,1cyclictest24736-21apt-get22:49:382
16667991915,3cyclictest0-21swapper/221:54:362
16667991914,4cyclictest0-21swapper/220:50:002
1666799190,1cyclictest3419-21perf19:50:002
16660991919,0cyclictest0-21swapper/122:14:391
16660991919,0cyclictest0-21swapper/121:54:351
16660991919,0cyclictest0-21swapper/120:44:361
16660991919,0cyclictest0-21swapper/120:39:381
16660991919,0cyclictest0-21swapper/100:04:591
16660991917,1cyclictest6512-21apt23:14:591
16657991919,0cyclictest0-21swapper/023:59:370
16657991919,0cyclictest0-21swapper/023:44:410
16657991919,0cyclictest0-21swapper/022:34:350
16657991919,0cyclictest0-21swapper/021:59:390
16657991919,0cyclictest0-21swapper/019:59:360
16657991919,0cyclictest0-21swapper/019:14:390
16657991918,1cyclictest23831-21perf19:25:010
16657991918,0cyclictest0-21swapper/019:44:350
1665799190,0cyclictest0-21swapper/000:34:370
16672991818,0cyclictest13298-21apt-get00:39:363
16672991818,0cyclictest0-21swapper/322:54:403
16672991818,0cyclictest0-21swapper/322:29:373
16667991818,0cyclictest0-21swapper/221:59:402
16667991818,0cyclictest0-21swapper/200:29:372
16667991818,0cyclictest0-21swapper/200:24:362
16667991818,0cyclictest0-21swapper/200:19:372
16667991817,1cyclictest4724-21apt-get23:14:362
16667991816,1cyclictest612-21gdbus23:30:002
1666799180,17cyclictest14393-21apt-get23:34:402
16660991818,0cyclictest0-21swapper/123:54:361
16660991818,0cyclictest0-21swapper/123:49:401
16660991818,0cyclictest0-21swapper/123:34:401
16660991818,0cyclictest0-21swapper/122:24:391
16660991818,0cyclictest0-21swapper/121:34:371
16660991818,0cyclictest0-21swapper/120:09:371
1666099180,17cyclictest8197-21cron19:59:591
16657991818,0cyclictest3715-21apt-get20:59:400
16657991818,0cyclictest0-21swapper/023:34:420
16657991818,0cyclictest0-21swapper/022:39:360
16657991818,0cyclictest0-21swapper/020:54:350
16657991818,0cyclictest0-21swapper/020:39:380
16657991818,0cyclictest0-21swapper/000:29:380
16657991818,0cyclictest0-21swapper/000:14:370
16657991816,1cyclictest26214-21dump-pmu-power19:30:000
16657991816,1cyclictest25905-21munin-plugin-st20:35:000
16672991717,0cyclictest13365-21hddtemp_smartct22:24:413
16672991717,0cyclictest0-21swapper/323:34:413
16672991717,0cyclictest0-21swapper/320:09:373
16672991717,0cyclictest0-21swapper/319:34:343
1667299170,0cyclictest17227-21apt-get19:14:403
16667991717,0cyclictest7121-21apt-get23:19:352
16667991717,0cyclictest16795-21apt-get23:39:392
16667991716,1cyclictest4041-21apt-get19:54:362
16667991715,1cyclictest21402-21perf19:20:002
16667991715,1cyclictest15187-21munin-run21:20:002
16660991717,0cyclictest0-21swapper/121:59:391
16660991717,0cyclictest0-21swapper/120:49:351
16660991717,0cyclictest0-21swapper/119:54:361
16660991717,0cyclictest0-21swapper/100:14:371
16660991715,1cyclictest5500-21dump-pmu-power00:20:001
16657991717,0cyclictest0-21swapper/023:24:370
16657991717,0cyclictest0-21swapper/023:14:370
16657991717,0cyclictest0-21swapper/021:19:360
16672991616,0cyclictest28918-21apt-get00:04:413
16672991616,0cyclictest0-21swapper/321:39:383
16672991616,0cyclictest0-21swapper/320:44:363
16672991616,0cyclictest0-21swapper/319:59:373
16672991616,0cyclictest0-21swapper/319:44:353
16672991616,0cyclictest0-21swapper/300:14:383
1667299160,15cyclictest11991-21apt-get23:29:373
16667991616,0cyclictest24113-21apt-get20:34:382
16667991616,0cyclictest19974-21apt-get21:30:012
16667991616,0cyclictest0-21swapper/221:04:362
16667991613,2cyclictest8530-21runrttasks21:09:402
16667991613,2cyclictest8530-21runrttasks21:09:392
1666799160,15cyclictest612-21gdbus20:30:002
1666799160,15cyclictest0-21swapper/200:13:102
16660991616,0cyclictest0-21swapper/123:39:401
16660991616,0cyclictest0-21swapper/122:54:391
16660991616,0cyclictest0-21swapper/100:34:361
1666099160,1cyclictest0-21swapper/121:25:221
16657991616,0cyclictest0-21swapper/020:19:370
16657991614,1cyclictest9681-21/usr/sbin/munin21:09:450
16657991614,1cyclictest9681-21/usr/sbin/munin21:09:450
1665799160,10cyclictest0-21swapper/000:24:370
1665799160,0cyclictest22340-21apt-get22:44:380
16672991515,0cyclictest6523-21perf23:15:003
16672991515,0cyclictest6125-21apt-get21:04:363
16672991514,1cyclictest26405-21apt-get23:59:373
16672991513,1cyclictest0-21swapper/300:29:433
1667299150,14cyclictest0-21swapper/323:50:283
1667299150,0cyclictest0-21swapper/320:04:353
16667991515,0cyclictest0-21swapper/223:49:392
16667991515,0cyclictest0-21swapper/220:10:012
16667991515,0cyclictest0-21swapper/219:29:362
16667991513,2cyclictest0-21swapper/222:19:462
16660991515,0cyclictest0-21swapper/123:44:421
16660991515,0cyclictest0-21swapper/123:19:351
16660991515,0cyclictest0-21swapper/122:44:391
16660991515,0cyclictest0-21swapper/122:04:391
16657991515,0cyclictest0-21swapper/022:09:340
16657991513,1cyclictest16914-21perf22:30:000
1667299140,14cyclictest0-21swapper/322:00:253
1667299140,14cyclictest0-21swapper/320:50:033
1667299140,14cyclictest0-21swapper/319:50:013
1667299140,14cyclictest0-21swapper/300:30:043
1667299140,14cyclictest0-21swapper/300:15:283
1667299140,0cyclictest0-21swapper/320:55:173
1667299140,0cyclictest0-21swapper/320:47:053
16667991414,0cyclictest0-21swapper/221:37:132
1666799140,14cyclictest0-21swapper/222:32:572
1666799140,0cyclictest0-21swapper/221:25:052
16660991414,0cyclictest0-21swapper/122:45:291
1666099140,14cyclictest0-21swapper/100:26:491
1666099140,0cyclictest0-21swapper/122:15:491
1666099140,0cyclictest0-21swapper/120:30:031
16657991414,0cyclictest0-21swapper/023:51:510
1665799140,14cyclictest0-21swapper/023:45:180
1665799140,14cyclictest0-21swapper/022:50:410
1665799140,0cyclictest0-21swapper/021:45:150
1665799140,0cyclictest0-21swapper/020:25:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional