You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 23:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Fri May 02, 2025 12:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
912992620,5cyclictest13117-21munin-run10:50:013
91099260,25cyclictest22930-21apt-get12:19:372
91299250,24cyclictest24997-21apt-get08:00:003
910992423,0cyclictest26483-21apt-get10:14:392
90599240,23cyclictest24239-21idleruntime-cro09:04:590
912992321,1cyclictest20315-21munin-plugin-st11:04:593
912992317,0cyclictest0-21swapper/310:04:403
91299230,22cyclictest8086-21cron07:25:003
91299230,22cyclictest17777-21cron07:45:013
91299230,22cyclictest10483-21apt-get09:44:423
910992321,1cyclictest23703-21apt-get10:09:422
91099230,2cyclictest17128-21awk12:04:592
909992322,1cyclictest2953-21apt-get11:39:401
909992321,1cyclictest27629-21apt-get09:14:401
909992320,2cyclictest24733-21grep12:20:001
909992320,2cyclictest17754-21apt-get12:05:021
90999230,22cyclictest10713-21turbostat10:45:001
90999230,1cyclictest0-21swapper/111:00:011
905992321,1cyclictest28263-21idleruntime-cro10:14:590
905992321,1cyclictest0-21swapper/007:19:390
905992320,2cyclictest12240-21sendmail08:40:000
905992319,3cyclictest0-21swapper/007:59:420
912992220,1cyclictest313-21systemd-journal07:19:593
912992220,1cyclictest29406-21perf09:15:013
912992220,1cyclictest0-21swapper/312:30:013
912992220,1cyclictest0-21swapper/309:39:383
912992219,2cyclictest12865-21cron07:35:013
91299220,21cyclictest313-21systemd-journal07:55:003
91299220,21cyclictest25703-21apt-get11:19:383
91299220,21cyclictest18500-21apt-get09:59:403
91299220,21cyclictest0-21swapper/312:00:003
91299220,21cyclictest0-21swapper/308:24:393
910992220,1cyclictest5904-21dump-pmu-power10:35:002
910992220,1cyclictest32333-21perf11:30:002
910992220,1cyclictest32333-21perf11:30:002
910992219,2cyclictest22694-21awk11:10:002
91099220,21cyclictest8114-21munin-run07:25:002
91099220,21cyclictest593-21dbus-daemon08:10:002
91099220,21cyclictest5038-21grep08:25:012
91099220,21cyclictest27508-21sendmail-msp11:20:002
91099220,21cyclictest22441-21apt-get09:04:362
91099220,21cyclictest17641-21apt-get08:54:412
91099220,21cyclictest12876-21awk07:35:012
909992221,0cyclictest28408-21apt-get08:09:411
909992220,1cyclictest15405-21munin-run07:40:001
909992220,1cyclictest1107-21turbostat10:24:591
90999220,21cyclictest8157-21apt-get11:49:401
90999220,21cyclictest26004-21apt-get08:04:351
90999220,21cyclictest20805-21apt-get07:54:361
90999220,21cyclictest1973-21wc12:40:001
90999220,21cyclictest16054-21apt-get09:54:391
90999220,21cyclictest0-21swapper/108:40:001
90999220,21cyclictest0-21swapper/107:59:411
90999220,1cyclictest20913-21apt-get10:04:401
90999220,1cyclictest20537-21apt-get12:14:381
90999220,1cyclictest20537-21apt-get12:14:381
90999220,1cyclictest13269-21apt-get09:49:391
90999220,0cyclictest0-21swapper/110:19:391
905992222,0cyclictest0-21swapper/007:14:380
905992221,1cyclictest20919-21apt-get11:09:400
905992220,1cyclictest5055-21munin-run08:25:010
905992219,2cyclictest15061-21munin-plugin-st09:50:000
90599220,2cyclictest22930-21apt-get12:19:360
90599220,21cyclictest313-21systemd-journal07:45:010
90599220,21cyclictest15497-21latency_hist10:55:010
90599220,21cyclictest0-21swapper/007:29:360
912992121,0cyclictest0-21swapper/308:19:373
912992119,1cyclictest3299-21awk07:14:593
91299210,21cyclictest0-21swapper/309:54:383
91299210,20cyclictest17861-21cron11:00:003
91299210,20cyclictest0-21swapper/310:54:403
91299210,20cyclictest0-21swapper/310:24:393
91299210,1cyclictest5367-21apt-get11:44:403
91299210,1cyclictest0-21swapper/310:10:013
91299210,0cyclictest0-21swapper/309:04:363
910992121,0cyclictest0-21swapper/208:19:372
910992120,1cyclictest12851-21apt-get08:44:382
910992120,0cyclictest0-21swapper/208:00:012
910992119,1cyclictest25234-21apt-get09:09:402
910992119,1cyclictest0-21swapper/211:59:592
910992119,1cyclictest0-21swapper/208:30:012
91099210,21cyclictest0-21swapper/207:59:402
91099210,20cyclictest0-21swapper/209:29:362
91099210,1cyclictest20031-21apt-get08:59:372
91099210,0cyclictest18417-21apt-get07:49:402
909992121,0cyclictest0-21swapper/111:19:391
909992120,1cyclictest15248-21apt-get08:49:401
909992120,0cyclictest12941-21apt-get11:59:381
90999210,21cyclictest0-21swapper/111:34:351
90999210,21cyclictest0-21swapper/107:24:391
90999210,20cyclictest0-21swapper/111:09:401
90999210,1cyclictest13720-21apt-get10:54:411
90999210,0cyclictest0-21swapper/111:44:391
90999210,0cyclictest0-21swapper/109:09:401
905992119,1cyclictest0-21swapper/009:59:410
905992118,2cyclictest8307-21idleruntime-cro10:40:000
90599210,20cyclictest2338-21latency_hist11:35:000
90599210,20cyclictest17914-21awk11:00:010
90599210,20cyclictest0-21swapper/009:29:370
90599210,1cyclictest32625-21sh08:15:000
90599210,1cyclictest31955-21turbostat.cron12:35:000
90599210,0cyclictest0-21swapper/011:24:390
90599210,0cyclictest0-21swapper/011:04:390
90599210,0cyclictest0-21swapper/010:49:400
90599210,0cyclictest0-21swapper/008:04:350
912992020,0cyclictest0-21swapper/311:54:403
912992020,0cyclictest0-21swapper/311:39:413
912992020,0cyclictest0-21swapper/310:39:393
912992020,0cyclictest0-21swapper/309:34:383
912992019,1cyclictest0-21swapper/312:39:403
912992019,0cyclictest0-21swapper/312:14:383
912992019,0cyclictest0-21swapper/312:14:373
912992019,0cyclictest0-21swapper/312:09:393
912992019,0cyclictest0-21swapper/307:49:413
91299200,0cyclictest0-21swapper/312:19:373
91299200,0cyclictest0-21swapper/311:14:373
910992020,0cyclictest0-21swapper/210:15:012
910992020,0cyclictest0-21swapper/210:04:392
910992020,0cyclictest0-21swapper/209:44:412
910992020,0cyclictest0-21swapper/209:39:372
910992020,0cyclictest0-21swapper/207:39:392
910992019,1cyclictest17129-21apt-get12:05:022
910992019,1cyclictest16020-21apt-get07:44:382
910992019,1cyclictest0-21swapper/209:54:382
910992019,1cyclictest0-21swapper/208:39:392
910992018,1cyclictest9933-21perf11:50:002
910992018,1cyclictest29409-21apt-get09:15:022
910992018,1cyclictest23310-21chrt11:14:382
910992018,1cyclictest20300-21perf11:05:002
909992020,0cyclictest0-21swapper/112:24:371
909992020,0cyclictest0-21swapper/110:14:391
909992020,0cyclictest0-21swapper/110:09:421
90999200,20cyclictest0-21swapper/108:34:401
90999200,19cyclictest0-21swapper/108:59:371
90999200,0cyclictest8066-21apt-get09:39:381
90999200,0cyclictest0-21swapper/112:29:391
90999200,0cyclictest0-21swapper/110:49:401
90999200,0cyclictest0-21swapper/110:29:401
905992020,0cyclictest0-21swapper/012:39:400
905992019,1cyclictest27133-21perf12:25:000
905992017,3cyclictest0-21swapper/008:34:400
905992017,2cyclictest32335-21idleruntime-cro11:29:590
905992017,2cyclictest32335-21idleruntime-cro11:29:590
90599201,18cyclictest7500-21apt-key11:45:010
90599200,19cyclictest23018-21apt-config10:05:000
90599200,19cyclictest0-21swapper/010:19:380
90599200,19cyclictest0-21swapper/010:05:010
90599200,19cyclictest0-21swapper/009:20:010
90599200,19cyclictest0-21swapper/008:44:380
90599200,19cyclictest0-21swapper/008:30:010
90599200,19cyclictest0-21swapper/007:54:380
90599200,0cyclictest0-21swapper/012:29:380
90599200,0cyclictest0-21swapper/012:14:380
90599200,0cyclictest0-21swapper/012:14:380
90599200,0cyclictest0-21swapper/012:04:380
90599200,0cyclictest0-21swapper/011:59:380
90599200,0cyclictest0-21swapper/009:09:390
912991919,0cyclictest0-21swapper/311:45:023
912991919,0cyclictest0-21swapper/311:34:353
912991919,0cyclictest0-21swapper/307:29:373
912991918,1cyclictest0-21swapper/308:09:403
910991919,0cyclictest0-21swapper/212:39:402
910991919,0cyclictest0-21swapper/212:29:382
910991919,0cyclictest0-21swapper/211:54:392
910991919,0cyclictest0-21swapper/211:25:002
910991919,0cyclictest0-21swapper/210:44:402
910991919,0cyclictest0-21swapper/209:45:022
910991919,0cyclictest0-21swapper/208:14:372
910991918,1cyclictest1500-21apt-get07:14:372
910991918,1cyclictest11324-21apt-get10:49:382
909991919,0cyclictest0-21swapper/111:24:391
909991919,0cyclictest0-21swapper/109:30:011
909991919,0cyclictest0-21swapper/109:15:011
909991919,0cyclictest0-21swapper/108:19:371
909991919,0cyclictest0-21swapper/107:34:371
909991918,1cyclictest428-21apt-get09:24:371
909991917,1cyclictest4662-21idleruntime-cro09:30:001
90999190,0cyclictest30135-21apt-get12:34:371
905991919,0cyclictest0-21swapper/011:45:020
905991919,0cyclictest0-21swapper/010:24:390
905991919,0cyclictest0-21swapper/009:44:420
905991919,0cyclictest0-21swapper/009:24:380
905991917,2cyclictest593-21dbus-daemon07:40:010
90599190,18cyclictest0-21swapper/011:40:010
90599190,18cyclictest0-21swapper/010:44:480
90599190,18cyclictest0-21swapper/010:34:420
90599190,18cyclictest0-21swapper/009:34:380
912991818,0cyclictest428-21apt-get09:24:383
912991818,0cyclictest0-21swapper/311:09:393
912991818,0cyclictest0-21swapper/308:59:373
912991818,0cyclictest0-21swapper/308:44:373
912991818,0cyclictest0-21swapper/308:29:383
912991818,0cyclictest0-21swapper/307:39:393
912991817,1cyclictest0-21swapper/309:05:023
912991816,1cyclictest29627-21dump-pmu-power11:25:003
912991816,1cyclictest10709-21perf10:45:003
910991818,0cyclictest31684-21apt-get10:24:392
910991818,0cyclictest0-21swapper/208:49:402
91099180,17cyclictest4734-21idleruntime-cro11:40:012
909991818,0cyclictest0-21swapper/111:14:381
90599182,15cyclictest21074-21ntp_states08:59:420
905991818,0cyclictest0-21swapper/011:14:380
90599180,18cyclictest0-21swapper/011:54:420
90599180,18cyclictest0-21swapper/008:19:370
90599180,17cyclictest0-21swapper/012:09:440
90599180,17cyclictest0-21swapper/008:54:420
90599180,17cyclictest0-21swapper/007:24:420
912991717,0cyclictest0-21swapper/310:14:393
912991717,0cyclictest0-21swapper/309:49:383
912991717,0cyclictest0-21swapper/308:14:383
91299170,1cyclictest0-21swapper/309:20:013
910991716,0cyclictest6518-21apt-get10:39:382
910991715,1cyclictest17911-21perf11:00:012
909991717,0cyclictest0-21swapper/111:04:391
909991717,0cyclictest0-21swapper/108:29:381
909991717,0cyclictest0-21swapper/107:44:371
909991717,0cyclictest0-21swapper/107:29:361
909991716,0cyclictest15355-21apt-get12:04:381
90999170,16cyclictest24255-21latency_hist09:05:001
905991717,0cyclictest0-21swapper/008:09:400
90599170,16cyclictest26705-21uname11:19:440
90599170,15cyclictest0-21swapper/007:49:480
912991616,0cyclictest0-21swapper/310:15:013
912991616,0cyclictest0-21swapper/308:54:403
912991616,0cyclictest0-21swapper/308:39:403
912991614,1cyclictest25063-21munin-run08:00:003
912991614,1cyclictest0-21swapper/312:04:393
910991616,0cyclictest9838-21perf08:35:002
910991616,0cyclictest0-21swapper/212:34:372
910991615,1cyclictest0-21swapper/210:54:412
910991614,1cyclictest0-21swapper/211:45:012
909991616,0cyclictest0-21swapper/109:44:411
909991616,0cyclictest0-21swapper/107:14:381
909991614,1cyclictest12321-21grep11:55:001
90599160,1cyclictest16854-21irqstats09:54:420
90599160,15cyclictest14938-21sshd08:46:030
90599160,15cyclictest0-21swapper/010:29:420
912991515,0cyclictest0-21swapper/311:29:393
912991515,0cyclictest0-21swapper/311:29:393
91299150,1cyclictest0-21swapper/310:34:453
91299150,14cyclictest16151-21memory08:49:433
910991515,0cyclictest5673-21apt-get09:34:372
909991515,0cyclictest0-21swapper/108:44:381
90999150,1cyclictest0-21swapper/111:29:441
90999150,1cyclictest0-21swapper/111:29:441
90599150,1cyclictest0-21swapper/009:36:050
90599150,15cyclictest0-21swapper/007:34:420
91299140,14cyclictest0-21swapper/308:30:193
91299140,0cyclictest0-21swapper/312:31:093
91299140,0cyclictest0-21swapper/312:20:243
91299140,0cyclictest0-21swapper/310:25:113
91299140,0cyclictest0-21swapper/309:29:133
910991414,0cyclictest0-21swapper/210:29:412
910991414,0cyclictest0-21swapper/207:15:452
91099140,14cyclictest0-21swapper/212:20:112
91099140,0cyclictest0-21swapper/212:10:452
91099140,0cyclictest0-21swapper/212:10:452
91099140,0cyclictest0-21swapper/211:30:242
91099140,0cyclictest0-21swapper/209:56:012
91099140,0cyclictest0-21swapper/209:23:012
91099140,0cyclictest0-21swapper/207:54:082
909991414,0cyclictest0-21swapper/107:15:261
90999140,14cyclictest0-21swapper/110:35:391
90999140,14cyclictest0-21swapper/109:55:081
90999140,14cyclictest0-21swapper/108:20:331
90999140,14cyclictest0-21swapper/107:48:461
90999140,0cyclictest0-21swapper/108:11:481
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional