You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-03 - 10:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu Sep 03, 2026 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
102632219165,29sleep10-21swapper/119:05:031
119252165102,27sleep30-21swapper/319:06:493
119522161109,30sleep00-21swapper/019:07:110
120932157102,29sleep20-21swapper/219:09:022
9862710,2sleep20-21swapper/221:10:232
130502650,3sleep313048-21ssh00:30:253
291462560,3sleep228-21ksoftirqd/200:19:202
298082530,2sleep00-21swapper/023:07:190
323382520,2sleep012-21ksoftirqd/021:33:560
52422510,2sleep00-21swapper/000:00:480
297012490,1sleep10-21swapper/122:43:321
92472460,2chrt0-21swapper/121:16:281
298852460,2sleep30-21swapper/323:31:063
222902450,1sleep30-21swapper/300:37:473
112072450,1sleep30-21swapper/300:30:013
38342430,1sleep30-21swapper/300:24:433
288852410,3sleep20-21swapper/223:55:122
1217124129,7sleep10-21swapper/119:10:011
83612350,5sleep122-21ksoftirqd/122:51:071
212302310,3chrt21217-21ssh23:25:163
104692300,1sleep20-21swapper/200:05:202
1244499282,10cyclictest9899-21ssh00:05:151
1244499281,25cyclictest0-21swapper/119:28:571
1244499281,25cyclictest0-21swapper/119:23:321
1244499281,25cyclictest0-21swapper/119:18:391
1244399281,25cyclictest0-21swapper/019:22:280
174562270,3chrt0-21swapper/122:10:211
174562270,3chrt0-21swapper/122:10:211
1244399278,12cyclictest31700-21diskmemload23:00:110
1244399273,16cyclictest7236-21ssh23:15:100
12443992724,2cyclictest0-21swapper/019:18:450
12443992723,2cyclictest0-21swapper/019:26:050
1244399271,24cyclictest0-21swapper/020:31:540
293112260,2chrt0-21swapper/022:19:310
12446992617,6cyclictest27646-21grep21:30:153
1244699261,2cyclictest141rcu_preempt00:00:003
1244699261,23cyclictest0-21swapper/319:30:483
12445992622,2cyclictest0-21swapper/219:18:542
12445992614,8cyclictest28245-21kernelversion23:30:182
1244599261,23cyclictest0-21swapper/219:26:182
1244599261,23cyclictest0-21swapper/219:21:032
1244399261,24cyclictest0-21swapper/020:37:310
49372250,1sleep00-21swapper/022:25:120
191452250,1sleep30-21swapper/323:47:313
12446992522,2cyclictest0-21swapper/319:18:583
1244699251,23cyclictest0-21swapper/320:33:573
1244699251,23cyclictest0-21swapper/319:20:393
1244699251,1cyclictest0-21swapper/319:29:063
12446992510,12cyclictest12874-21basename21:19:593
12445992513,7cyclictest0-21swapper/223:45:112
1244599250,23cyclictest0-21swapper/220:34:002
65892240,1sleep30-21swapper/323:38:283
65892240,1sleep30-21swapper/323:38:283
37262240,10sleep334-21ksoftirqd/321:36:003
1244699249,12cyclictest0-21swapper/321:15:003
12446992417,5cyclictest34-21ksoftirqd/321:50:013
1244599242,19cyclictest17930-21pmu-power22:10:262
1244599242,19cyclictest17930-21pmu-power22:10:252
1244599242,14cyclictest9926-21sh22:05:112
12445992412,9cyclictest6140-21cstates22:50:132
12444992414,7cyclictest9425-21cron00:05:001
146372230,2sleep00-21swapper/000:31:500
126142230,1sleep20-21swapper/219:10:122
12446992311,8cyclictest1916-21python323:10:283
1244699231,10cyclictest31700-21diskmemload22:15:103
12445992314,6cyclictest14532-21cron20:30:002
12445992312,9cyclictest31700-21diskmemload23:08:212
12445992310,10cyclictest15060-21cat23:45:002
1244399231,18cyclictest53050irq/142-enp3s0-22:30:120
63852220,1sleep10-21swapper/120:10:141
48812220,2sleep30-21swapper/322:00:363
154942220,19sleep315512-21rm00:09:233
12446992214,6cyclictest21997-21memory20:45:183
12446992211,7cyclictest52750irq/139-enp3s0-23:05:113
12445992213,7cyclictest3609-21smartctl22:00:162
12445992211,7cyclictest23057-21processes21:50:272
12444992212,7cyclictest278-21systemd-journal00:14:551
1244499221,18cyclictest560-21dbus-daemon00:35:241
12444992210,8cyclictest29643-21apt-get19:50:121
12443992215,5cyclictest278-21systemd-journal21:05:010
12443992214,6cyclictest32391-21ssh23:33:340
252912210,1sleep20-21swapper/223:04:322
249422210,4sleep024943-21expr00:15:250
1244699217,7cyclictest0-21swapper/321:59:433
1244699213,13cyclictest75-21kswapd023:40:113
12446992113,6cyclictest1-21systemd20:15:223
12446992112,6cyclictest278-21systemd-journal00:15:003
12446992111,6cyclictest34-21ksoftirqd/320:10:103
12445992113,6cyclictest28952-21diskstats21:55:142
12445992113,6cyclictest11729-21diskstats22:30:142
12445992112,7cyclictest560-21dbus-daemon21:00:012
12445992111,7cyclictest560-21dbus-daemon19:50:002
1244499218,11cyclictest5586-21date22:50:001
1244499218,10cyclictest8989-21forks23:40:141
12444992113,6cyclictest29702-21seq21:31:191
12444992113,6cyclictest27973-21fw_packets19:45:181
12444992113,6cyclictest16643-21python323:45:221
12444992113,5cyclictest52950irq/141-enp3s0-00:25:261
1244399219,10cyclictest2690-21latency_hist00:00:010
1244399218,9cyclictest12-21ksoftirqd/021:55:110
1244399212,13cyclictest21890-21ssh21:50:170
1244399212,13cyclictest1436-21chrt21:35:130
12443992113,6cyclictest4628-21seq23:12:490
12443992113,6cyclictest31096-21memory22:20:180
12443992113,6cyclictest17808-21seq21:46:290
12443992112,7cyclictest21963-21cat20:45:170
114972210,2sleep00-21swapper/022:54:150
5792200,2sleep1211rcuc/123:57:561
1244699207,4cyclictest141rcu_preempt22:20:193
1244699202,12cyclictest23211-21apt-get19:35:113
12446992014,4cyclictest34-21ksoftirqd/322:50:003
12446992012,6cyclictest9912-21apt-get22:05:103
12446992012,6cyclictest24269-21ntpq20:50:203
12446992011,6cyclictest22693-21cat21:50:233
1244699200,13cyclictest141rcu_preempt22:40:213
1244599209,8cyclictest30221-21apt-get00:20:102
12445992012,6cyclictest636-21runrttasks00:31:502
12445992012,6cyclictest31149-21cron22:45:002
12445992012,6cyclictest21676-21perl20:45:152
12445992012,6cyclictest17260-21seq22:58:162
12445992012,6cyclictest1498-21cat23:10:242
12445992012,5cyclictest21578-21cat21:50:012
1244599201,16cyclictest4685-21cut22:25:122
12445992010,7cyclictest20051-21awk20:40:212
1244499209,8cyclictest8747-21ssh21:40:161
1244499203,11cyclictest21241-21kworker/u8:021:10:121
1244499202,13cyclictest18077-21apt-get22:35:121
1244499202,12cyclictest636-21runrttasks22:25:111
12444992014,3cyclictest17666-21sh22:35:001
12444992013,4cyclictest8442-21gltestperf20:15:151
12444992012,6cyclictest8142-21cron23:40:001
12444992012,6cyclictest8142-21cron23:40:001
12444992012,5cyclictest21194-21grep23:25:161
12444992011,5cyclictest0-21swapper/123:20:131
12443992014,4cyclictest4836-21swap20:05:270
12443992013,5cyclictest30968-21irqstats00:20:160
12443992013,4cyclictest25998-21sh23:52:380
12443992012,6cyclictest8267-21df20:15:130
12443992012,6cyclictest26828-21seq23:29:460
12443992012,6cyclictest21065-21cat21:25:180
12443992011,7cyclictest25199-21cut19:40:000
1244399201,17cyclictest2007-21ssh22:46:310
12443992010,7cyclictest25937-21sed19:40:160
1244699199,7cyclictest29987-21cat19:50:153
1244699193,5cyclictest0-21swapper/321:40:493
1244699192,6cyclictest4001-21kworker/u8:123:15:503
12446991913,3cyclictest18040-21sendmail_mailst22:10:273
12446991913,3cyclictest18040-21sendmail_mailst22:10:273
12446991912,5cyclictest624-21gdbus19:44:593
12446991912,5cyclictest19390-21expr20:40:143
12446991911,6cyclictest560-21dbus-daemon00:05:003
12446991911,6cyclictest30568-21memory21:05:183
12446991910,6cyclictest13729-21ntpq22:55:203
1244599197,3cyclictest172772df20:35:152
1244599196,9cyclictest27673-21apt-get21:00:122
1244599192,11cyclictest23377-21apt-get20:50:122
1244599191,7cyclictest9639-21latency_hist00:05:012
12445991912,5cyclictest19079-21ntpq22:35:192
12445991911,6cyclictest23439-21awk22:15:092
12445991911,6cyclictest22243-21wc19:30:232
12445991911,6cyclictest19732-21sed00:35:252
12445991911,5cyclictest6452-21/usr/sbin/munin21:15:222
12445991911,5cyclictest22076-21ssh23:50:122
12445991911,5cyclictest14669-21swap21:20:252
12445991911,5cyclictest14669-21swap21:20:242
1244499195,4cyclictest31700-21diskmemload22:04:241
1244499193,3cyclictest22-21ksoftirqd/121:25:011
1244499192,12cyclictest21872-21apt-get21:50:171
1244499192,12cyclictest19430-21apt-get23:00:111
12444991917,1cyclictest0-21swapper/119:31:471
12444991913,4cyclictest17009-21sh21:45:401
12444991912,5cyclictest24942-21seq19:38:121
12444991912,5cyclictest22089-21seq23:50:131
12444991912,4cyclictest31700-21diskmemload23:30:171
12444991912,4cyclictest14130-21xargs22:55:241
12444991911,6cyclictest30584-21cat00:20:121
12444991911,6cyclictest13859-21fw_packets21:20:181
12444991911,6cyclictest13859-21fw_packets21:20:181
12444991911,5cyclictest31700-21diskmemload00:32:391
12444991911,5cyclictest28142-21grep21:00:161
1244499190,16cyclictest10479-21grep22:05:151
1244399198,7cyclictest53050irq/142-enp3s0-23:20:150
12443991915,3cyclictest15867-21ssh00:09:440
12443991913,4cyclictest32273-21cut21:10:130
12443991913,4cyclictest22610-21sh00:38:070
12443991913,4cyclictest22292-21sh22:14:150
12443991913,4cyclictest22292-21sh22:14:150
12443991913,4cyclictest18397-21sh23:46:460
12443991913,4cyclictest17867-21open_files00:10:220
12443991912,5cyclictest8066-21date21:15:260
12443991912,5cyclictest31998-21cat19:55:140
12443991912,5cyclictest25183-21expr22:40:150
12443991911,6cyclictest8997-21awk23:40:150
12443991911,6cyclictest14831-21ls21:20:260
12443991911,6cyclictest14831-21ls21:20:260
12443991911,6cyclictest13287-21cat19:10:190
12443991911,6cyclictest11243-21sensors_temp22:05:220
1244399191,15cyclictest25836-21df20:55:110
12443991910,7cyclictest9948-21users21:40:260
12443991910,7cyclictest2269-21tune2fs23:35:150
12443991910,7cyclictest2269-21tune2fs23:35:150
222472181,2sleep30-21swapper/322:37:533
21242181,15sleep20-21swapper/222:22:592
17922180,5sleep2271rcuc/220:00:132
1244699186,3cyclictest34-21ksoftirqd/319:45:173
1244699182,11cyclictest13320-21apt-get21:20:153
1244699182,11cyclictest13320-21apt-get21:20:153
12446991816,1cyclictest0-21swapper/320:37:313
12446991813,3cyclictest3716-21cpu20:05:173
12446991813,3cyclictest19399-21/usr/sbin/munin23:00:243
12446991813,3cyclictest0-21swapper/322:34:343
12446991811,5cyclictest2733-21ls20:00:243
12446991811,5cyclictest22323-21expr23:50:143
12446991811,4cyclictest14584-21gltestperf23:20:183
12446991810,6cyclictest4895-21expr22:25:123
12446991810,6cyclictest20488-21cat21:25:143
12446991810,5cyclictest52750irq/139-enp3s0-22:50:443
12446991810,5cyclictest23497-21acpi00:15:123
1244599189,6cyclictest7380-21grep23:15:112
1244599189,6cyclictest52850irq/140-enp3s0-23:25:122
12445991813,3cyclictest5932-21swap00:25:242
12445991812,4cyclictest7911-21/usr/sbin/munin20:15:172
12445991812,4cyclictest4457-21sh21:36:432
12445991812,4cyclictest31494-21sh21:33:072
12445991812,4cyclictest21770-21timerwakeupswit21:25:252
12445991812,4cyclictest19878-21sh23:24:342
12445991812,4cyclictest13250-21sh21:43:402
12445991811,5cyclictest460-21sensors22:45:232
12445991811,5cyclictest11402-21which20:20:222
1244599181,14cyclictest31700-21diskmemload00:13:032
12445991810,6cyclictest30152-21entropy21:05:152
1244499182,11cyclictest28579-21apt-get21:55:121
1244499182,11cyclictest1429-21apt-get21:35:131
1244499181,5cyclictest0-21swapper/123:15:151
12444991813,3cyclictest26539-21switchtime19:40:231
12444991813,3cyclictest26059-21sh23:05:041
12444991813,3cyclictest15559-21missed_timers20:30:181
12444991812,4cyclictest25269-21munin-plugin-st20:55:001
12444991812,3cyclictest11403-21mailstats20:20:211
12444991811,5cyclictest3824-21cat20:05:171
12444991811,5cyclictest31848-21cat19:55:131
12444991811,5cyclictest31484-21seq21:07:281
12444991811,5cyclictest24764-21tr22:15:201
12444991811,4cyclictest31700-21diskmemload23:10:031
1244499181,12cyclictest21238-21apt-get20:45:121
12444991810,6cyclictest20254-21sensors20:40:221
12444991810,6cyclictest17212-21cat20:35:141
12443991814,2cyclictest12632-21/usr/sbin/munin20:25:140
1244399181,3cyclictest30401-21memory19:50:190
12443991813,3cyclictest5432-21sh00:25:190
12443991813,3cyclictest5081-21sh22:00:480
12443991812,4cyclictest12536-21sh22:55:030
12443991811,5cyclictest27495-21acpi19:45:130
1244399181,13cyclictest18107-21taskset22:35:120
12443991810,6cyclictest29792-21basename21:05:110
1244699178,6cyclictest10512-21cat20:20:133
1244699172,11cyclictest31782-21apt-get19:55:133
12446991712,3cyclictest25488-21/usr/sbin/munin20:55:153
12446991710,5cyclictest34-21ksoftirqd/321:00:263
1244599179,6cyclictest26731-21cat19:40:242
1244599178,6cyclictest1423-21cut20:00:002
12445991712,3cyclictest7683-21sh20:15:012
12445991712,3cyclictest1668-21sh23:35:032
12445991712,3cyclictest1668-21sh23:35:032
12445991710,5cyclictest3863-21expr20:05:162
12445991710,5cyclictest30969-21cat19:50:242
12445991710,5cyclictest23638-21cut19:35:152
1244499179,6cyclictest30426-21cstates22:20:131
1244499178,6cyclictest12188-21cron20:25:001
1244499171,4cyclictest27405-21idleruntime-cro21:00:001
12444991711,3cyclictest31700-21diskmemload00:15:221
1244399179,6cyclictest22427-21cat19:30:240
12443991712,3cyclictest12197-21idleruntime-cro20:25:000
12443991711,4cyclictest19612sleep020:00:150
1244399171,12cyclictest23352-21/usr/sbin/munin20:50:270
12443991710,5cyclictest19276-21expr20:40:130
1244699162,10cyclictest12657-21apt-get20:25:113
12446991610,3cyclictest13018-21grep19:10:163
12444991611,3cyclictest1694-21cstates20:00:131
1244399168,5cyclictest6716-21ntpq20:10:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional