You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 02:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Sun Jul 06, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11511992422,1cyclictest10221-21perf11:35:003
1151199241,22cyclictest10843-21munin-plugin-st08:15:003
1151199240,23cyclictest16317-21awk07:20:003
11511992321,1cyclictest27663-21latency_hist08:50:003
1151199230,22cyclictest9875-21apt12:40:003
1151199230,21cyclictest313-21systemd-journal10:25:003
1151199230,0cyclictest0-21swapper/307:20:123
11505992323,0cyclictest0-21swapper/107:20:131
1150599230,22cyclictest22115-21dump-pmu-power09:45:001
11501992321,1cyclictest0-21swapper/008:45:000
11501992321,1cyclictest0-21swapper/007:30:010
11511992220,1cyclictest31724-21perf10:05:003
11511992220,1cyclictest21100-21munin-plugin-st07:30:003
1151199220,21cyclictest30062-21munin-run08:55:003
1151199220,21cyclictest26365-21sendmail07:40:003
1151199220,21cyclictest1710-21munin-run10:10:003
1151199220,1cyclictest14509-21latency_hist09:29:593
11509992219,2cyclictest30840-21apt-get11:10:122
11509992219,2cyclictest18028-21cron08:30:002
1150999220,21cyclictest21758-21apt-get07:30:132
1150999220,1cyclictest593-21dbus-daemon07:15:002
1150999220,1cyclictest22691-21munin-run12:00:002
11505992221,0cyclictest13528-21apt-get08:20:101
11505992220,1cyclictest27485-21perf12:10:001
11505992220,1cyclictest12673-21apt-get11:40:001
1150599220,21cyclictest25526-21apt-get08:45:111
1150599220,1cyclictest28180-21wc11:04:591
1150599220,1cyclictest25790-21perf11:00:001
1150599220,1cyclictest0-21swapper/109:35:121
11501992222,0cyclictest0-21swapper/008:10:110
11501992221,1cyclictest0-21swapper/010:35:110
11501992220,1cyclictest18712-21munin-run07:25:000
11501992219,2cyclictest32290-21dump-pmu-power12:20:000
1150199220,21cyclictest20427-21idleruntime-cro08:35:000
1150199220,21cyclictest1165-21perf07:55:000
1150199220,21cyclictest0-21swapper/012:00:130
1150199220,21cyclictest0-21swapper/011:50:110
1150199220,21cyclictest0-21swapper/011:20:120
1150199220,21cyclictest0-21swapper/011:10:000
1150199220,21cyclictest0-21swapper/011:00:130
1150199220,21cyclictest0-21swapper/010:50:120
1150199220,21cyclictest0-21swapper/008:20:100
1150199220,21cyclictest0-21swapper/007:25:130
1150199220,1cyclictest32731-21apt-get09:00:110
11511992121,0cyclictest495-21apt-get12:20:133
11511992119,1cyclictest593-21dbus-daemon08:05:003
11511992118,2cyclictest17870-21sh11:50:003
1151199210,21cyclictest0-21swapper/312:15:133
1151199210,21cyclictest0-21swapper/308:30:123
1151199210,20cyclictest32449-21idleruntime-cro09:00:003
1151199210,20cyclictest0-21swapper/310:55:003
1151199210,20cyclictest0-21swapper/309:55:123
1151199210,20cyclictest0-21swapper/309:15:133
1151199210,0cyclictest0-21swapper/309:00:113
11509992120,1cyclictest14783-21apt-get09:30:112
11509992120,0cyclictest16572-21apt-get07:20:122
11509992119,1cyclictest8388-21perf08:10:002
11509992119,1cyclictest30139-21apt-get12:15:122
11509992119,1cyclictest24496-21sh09:50:002
1150999210,4cyclictest0-21swapper/208:15:212
1150999210,4cyclictest0-21swapper/208:15:212
1150999210,20cyclictest31985-21apt-get10:05:122
1150999210,20cyclictest27178-21apt-get09:55:122
1150999210,20cyclictest26675-21apt-get07:40:122
1150999210,20cyclictest22960-21apt-get12:00:122
1150999210,20cyclictest18967-21apt-get07:25:132
1150999210,20cyclictest0-21swapper/209:05:122
11505992121,0cyclictest29068-21apt-get07:45:121
11505992121,0cyclictest0-21swapper/108:10:111
11505992121,0cyclictest0-21swapper/107:50:121
11505992120,1cyclictest0-21swapper/110:20:121
11505992119,1cyclictest7231-21awk09:15:001
11505992119,1cyclictest22-21ksoftirqd/112:00:001
11505992119,1cyclictest0-21swapper/110:50:001
11505992119,1cyclictest0-21swapper/108:55:001
1150599210,20cyclictest0-21swapper/111:15:121
1150599210,20cyclictest0-21swapper/110:10:121
1150599210,1cyclictest2682-21perf12:25:011
1150599210,0cyclictest0-21swapper/111:20:221
1150599210,0cyclictest0-21swapper/111:10:131
1150599210,0cyclictest0-21swapper/110:50:121
11501992121,0cyclictest13890-21perf07:15:000
11501992121,0cyclictest0-21swapper/010:40:140
11501992121,0cyclictest0-21swapper/009:15:130
11501992121,0cyclictest0-21swapper/007:40:120
11501992120,1cyclictest0-21swapper/009:40:150
11501992120,0cyclictest0-21swapper/008:35:140
1150199210,1cyclictest0-21swapper/008:55:120
1150199210,0cyclictest0-21swapper/011:40:020
11511992020,0cyclictest5338-21apt-get12:30:143
11511992020,0cyclictest20720-21apt-get08:35:153
11511992020,0cyclictest0-21swapper/310:40:143
11511992020,0cyclictest0-21swapper/309:30:113
11511992020,0cyclictest0-21swapper/307:40:133
11511992019,1cyclictest14033-21apt-get10:35:123
11511992019,1cyclictest11637-21apt-get10:30:123
1151199200,20cyclictest0-21swapper/312:25:123
1151199200,20cyclictest0-21swapper/309:10:143
1151199200,19cyclictest12697-21sendmail-msp11:40:003
1151199200,0cyclictest0-21swapper/310:15:113
11509992020,0cyclictest5117-21apt-get09:10:142
11509992020,0cyclictest18321-21apt-get08:30:122
11509992020,0cyclictest0-21swapper/211:15:122
11509992020,0cyclictest0-21swapper/210:25:112
11509992020,0cyclictest0-21swapper/210:10:122
11509992020,0cyclictest0-21swapper/208:20:122
11509992019,0cyclictest29595-21apt-get10:00:132
11509992018,1cyclictest7734-21apt-get12:35:142
11509992018,1cyclictest12691-21grep11:40:012
11509992018,1cyclictest12673-21apt-get11:40:032
11509992018,1cyclictest0-21swapper/212:10:002
1150999200,20cyclictest0-21swapper/207:15:112
1150999200,0cyclictest5685-21apt-get11:25:122
1150999200,0cyclictest0-21swapper/211:50:112
1150999200,0cyclictest0-21swapper/210:35:122
11505992020,0cyclictest0-21swapper/107:35:111
11505992019,0cyclictest0-21swapper/111:45:161
11505992018,1cyclictest0-21swapper/108:45:001
1150599200,20cyclictest0-21swapper/110:30:121
1150599200,0cyclictest0-21swapper/112:35:131
1150599200,0cyclictest0-21swapper/110:05:121
1150599200,0cyclictest0-21swapper/109:20:121
11501992020,0cyclictest11103-21apt-get08:15:130
11501992020,0cyclictest11103-21apt-get08:15:130
11501992020,0cyclictest0-21swapper/011:55:130
11501992020,0cyclictest0-21swapper/011:35:150
11501992020,0cyclictest0-21swapper/011:15:120
11501992020,0cyclictest0-21swapper/010:55:090
11501992020,0cyclictest0-21swapper/010:15:100
11501992020,0cyclictest0-21swapper/009:35:130
11501992020,0cyclictest0-21swapper/009:25:120
11501992020,0cyclictest0-21swapper/009:05:130
11501992020,0cyclictest0-21swapper/008:25:120
11501992020,0cyclictest0-21swapper/007:45:120
11501992019,1cyclictest13-21ksoftirqd/008:01:380
11501992019,1cyclictest0-21swapper/011:10:130
11501992017,2cyclictest0-21swapper/012:35:130
1150199200,20cyclictest0-21swapper/010:20:110
1150199200,20cyclictest0-21swapper/008:10:000
1150199200,1cyclictest0-21swapper/009:30:110
1150199200,0cyclictest0-21swapper/012:05:140
1150199200,0cyclictest0-21swapper/010:00:130
1150199200,0cyclictest0-21swapper/008:50:120
11511991919,0cyclictest1971-21apt-get10:10:133
11511991919,0cyclictest0-21swapper/311:10:123
11511991919,0cyclictest0-21swapper/309:50:103
11511991919,0cyclictest0-21swapper/308:20:113
11511991919,0cyclictest0-21swapper/307:45:133
11511991919,0cyclictest0-21swapper/307:10:113
11511991918,0cyclictest2717-21apt-get09:05:123
11511991917,1cyclictest19725-21basename09:40:003
11511991917,1cyclictest0-21swapper/311:20:213
11511991916,2cyclictest7806-21ls11:30:003
1151199190,0cyclictest0-21swapper/311:40:123
11509991919,0cyclictest27919-21apt-get08:50:112
11509991919,0cyclictest27749-21apt-get12:10:152
11509991919,0cyclictest0-21swapper/210:45:112
11509991919,0cyclictest0-21swapper/209:40:152
11509991919,0cyclictest0-21swapper/209:35:142
11509991919,0cyclictest0-21swapper/208:40:092
11509991919,0cyclictest0-21swapper/207:50:132
11509991919,0cyclictest0-21swapper/207:35:102
11509991918,0cyclictest9928-21apt-get09:20:122
11509991917,2cyclictest28-21ksoftirqd/212:35:012
11509991917,1cyclictest447-21apt-get12:20:022
11509991917,1cyclictest28155-21sh11:04:592
1150999190,18cyclictest5961-21cron08:05:002
11505991919,0cyclictest0-21swapper/112:15:001
11505991919,0cyclictest0-21swapper/111:05:121
11505991919,0cyclictest0-21swapper/107:40:121
11505991919,0cyclictest0-21swapper/107:30:121
1150599190,1cyclictest0-21swapper/108:05:001
11501991919,0cyclictest0-21swapper/011:30:140
11501991919,0cyclictest0-21swapper/010:30:120
11501991917,1cyclictest13-21ksoftirqd/010:45:120
11501991916,3cyclictest13-21ksoftirqd/010:00:000
11511991818,0cyclictest31461-21apt-get07:50:133
11511991818,0cyclictest0-21swapper/312:10:153
11511991818,0cyclictest0-21swapper/310:25:113
11511991816,1cyclictest0-21swapper/310:50:013
1151199180,2cyclictest25258-21apt08:45:013
11509991816,1cyclictest1434-21apt-get07:55:112
11505991818,0cyclictest0-21swapper/112:00:121
11505991818,0cyclictest0-21swapper/109:45:121
11505991818,0cyclictest0-21swapper/109:05:121
11505991817,1cyclictest16920-21perf09:35:001
11505991817,0cyclictest0-21swapper/109:55:121
11505991816,1cyclictest593-21dbus-daemon08:39:591
11505991816,1cyclictest22-21ksoftirqd/107:30:001
11501991818,0cyclictest0-21swapper/010:25:110
11501991818,0cyclictest0-21swapper/009:45:120
11501991817,1cyclictest31985-21apt-get10:05:120
11501991817,1cyclictest0-21swapper/010:10:140
1150199180,18cyclictest0-21swapper/008:45:110
11511991717,0cyclictest19998-21apt-get09:40:153
11511991717,0cyclictest0-21swapper/311:50:113
11511991717,0cyclictest0-21swapper/308:05:113
11511991715,1cyclictest24500-21awk09:50:003
11509991717,0cyclictest2939-21apt-get12:25:122
11509991717,0cyclictest17889-21perf11:50:002
1150999171,15cyclictest578-21irqbalance08:37:102
11505991717,0cyclictest0-21swapper/111:30:141
11505991717,0cyclictest0-21swapper/110:25:121
11505991716,1cyclictest13354-21apt-get11:40:121
11505991715,1cyclictest593-21dbus-daemon11:30:001
11505991715,1cyclictest20461-21munin-run08:35:001
11505991715,1cyclictest16176-21latency_hist10:40:001
11505991715,1cyclictest0-21swapper/108:25:121
11501991716,1cyclictest0-21swapper/009:10:140
11511991615,0cyclictest0-21swapper/309:20:123
11509991616,0cyclictest8084-21apt-get11:30:152
11509991616,0cyclictest3242-21apt-get11:20:122
11509991616,0cyclictest24780-21apt-get09:50:092
11509991616,0cyclictest16451-21apt-get10:40:142
11509991616,0cyclictest0-21swapper/208:55:102
11509991616,0cyclictest0-21swapper/207:45:122
11509991614,1cyclictest8966-21munin-run10:25:002
1150999160,15cyclictest9630-21cron09:20:002
1150999160,15cyclictest593-21dbus-daemon08:15:002
11505991616,0cyclictest0-21swapper/111:50:111
11505991616,0cyclictest0-21swapper/107:55:111
11505991614,1cyclictest12799-21ntp_states07:10:161
11501991614,1cyclictest5075-21munin-run12:30:000
11511991515,0cyclictest805-21apt-get11:15:123
11511991512,2cyclictest25750-21sendmail11:00:003
1151199150,7cyclictest0-21swapper/311:55:163
1151199150,1cyclictest27387-21seq12:09:173
11509991515,0cyclictest12384-21apt-get09:25:122
11509991515,0cyclictest0-21swapper/208:45:112
11509991513,1cyclictest32731-21apt-get09:00:112
11509991512,2cyclictest30573-21idleruntime-cro11:10:002
11505991515,0cyclictest0-21swapper/112:30:141
1150199156,8cyclictest13-21ksoftirqd/012:20:160
11501991515,0cyclictest0-21swapper/007:55:110
1150199150,1cyclictest26939-21cat09:55:000
1150199150,1cyclictest0-21swapper/012:15:010
1150199150,15cyclictest0-21swapper/011:45:200
11511991414,0cyclictest0-21swapper/311:00:123
11511991414,0cyclictest0-21swapper/308:25:393
1151199140,14cyclictest0-21swapper/312:00:453
1151199140,14cyclictest0-21swapper/311:05:053
1151199140,14cyclictest0-21swapper/308:18:343
1151199140,14cyclictest0-21swapper/308:18:343
1151199140,14cyclictest0-21swapper/307:56:573
1151199140,0cyclictest0-21swapper/307:30:023
1150999140,14cyclictest0-21swapper/210:50:582
1150999140,14cyclictest0-21swapper/210:30:052
1150999140,0cyclictest0-21swapper/210:57:502
1150999140,0cyclictest0-21swapper/210:18:022
11505991414,0cyclictest0-21swapper/109:25:231
1150599140,5cyclictest0-21swapper/110:40:141
1150599140,14cyclictest0-21swapper/110:15:011
1150599140,14cyclictest0-21swapper/108:55:151
1150599140,0cyclictest0-21swapper/110:00:271
1150599140,0cyclictest0-21swapper/109:50:151
1150599140,0cyclictest0-21swapper/109:00:231
1150599140,0cyclictest0-21swapper/108:15:061
1150599140,0cyclictest0-21swapper/108:15:051
11501991414,0cyclictest0-21swapper/012:30:150
11501991414,0cyclictest0-21swapper/007:35:100
11501991413,0cyclictest0-21swapper/009:20:120
1150199140,14cyclictest0-21swapper/011:25:040
1150199140,0cyclictest0-21swapper/007:15:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional