You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-27 - 11:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackaslot8s.osadl.org (updated Thu Aug 27, 2026 00:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28142230205,17sleep00-21swapper/019:09:280
3077021640,1sleep10-21swapper/122:04:411
2682214959,82sleep20-21swapper/219:07:452
25172142106,26sleep10-21swapper/119:05:371
25492139102,26sleep30-21swapper/319:06:023
216342630,3sleep0131rcuc/021:57:130
9372610,2chrt0-21swapper/022:05:320
142402610,2sleep30-21swapper/321:51:293
307232600,7sleep0131rcuc/022:52:220
43302580,2sleep00-21swapper/020:25:240
146332560,4sleep222181-21diskmemload22:40:172
250542540,4sleep00-21swapper/023:12:220
104832540,1sleep10-21swapper/100:14:341
261082530,1sleep30-21swapper/322:49:293
208822530,2sleep30-21swapper/323:33:393
71032510,2sleep012-21ksoftirqd/021:22:090
39452510,2sleep10-21swapper/123:45:131
223512490,2sleep30-21swapper/321:57:563
35122480,2sleep228-21ksoftirqd/219:10:142
18852450,2sleep10-21swapper/123:19:561
6692430,2chrt0-21swapper/321:41:193
152142430,1sleep10-21swapper/123:53:281
92012370,2chrt0-21swapper/122:12:071
3129993523,8cyclictest53050irq/142-enp3s0-23:35:102
285432320,2chrt0-21swapper/222:02:252
79672300,3sleep228-21ksoftirqd/220:35:172
281512280,1sleep30-21swapper/321:38:063
281512280,1sleep30-21swapper/321:38:053
313099271,24cyclictest0-21swapper/319:18:583
313099271,13cyclictest99082df20:40:133
312799271,25cyclictest0-21swapper/019:16:460
3130992619,5cyclictest6314-21ssh22:34:383
312999262,17cyclictest23768-21ssh21:35:112
312999262,17cyclictest23768-21ssh21:35:102
3129992617,6cyclictest23388-40systemctl00:00:002
3128992619,6cyclictest22-21ksoftirqd/121:00:011
312799261,1cyclictest0-21swapper/019:14:320
274292260,2sleep10-21swapper/100:02:231
312999256,9cyclictest141rcu_preempt00:20:002
312999252,18cyclictest24699-21kworker/u8:123:15:122
312999252,16cyclictest18152-21ssh21:55:112
312999250,23cyclictest0-21swapper/219:16:062
312899251,1cyclictest0-21swapper/119:16:521
3128992510,13cyclictest26186-21tr00:25:221
312899250,24cyclictest0-21swapper/119:14:551
312799253,15cyclictest640-21snmpd22:20:130
173292250,2sleep30-21swapper/323:06:193
142202250,1sleep20-21swapper/223:04:572
65912240,3sleep20-21swapper/222:34:552
3129992414,7cyclictest32162-21ntp_states20:15:192
312899246,5cyclictest22-21ksoftirqd/123:10:201
312899242,15cyclictest4778-21ssh21:45:121
312899241,16cyclictest19269-21ssh22:20:031
228322240,7sleep2271rcuc/221:10:122
313099230,22cyclictest0-21swapper/319:14:353
312999231,7cyclictest5909-21grep00:10:222
3129992315,4cyclictest30872-21sh23:40:232
3129992313,7cyclictest11604-21open_files23:50:212
312899233,14cyclictest5748-21ssh00:35:111
312899232,15cyclictest75-21kswapd022:50:131
312899231,7cyclictest14117-21/usr/sbin/munin22:40:121
312899231,6cyclictest0-21swapper/121:19:571
3128992313,7cyclictest12358-21ntp_offset00:15:191
3128992312,7cyclictest0-21swapper/123:25:121
3127992313,7cyclictest12910-21python319:30:240
50702220,2sleep00-21swapper/021:45:120
50352220,2sleep10-21swapper/123:21:211
3130992210,3cyclictest25350-21fschecks_time22:00:153
312999229,7cyclictest22181-21diskmemload21:15:132
312999223,10cyclictest22181-21diskmemload23:28:112
312999221,9cyclictest0-21swapper/200:05:242
3129992214,6cyclictest28571-21sed22:50:252
3129992211,8cyclictest278-21systemd-journal23:25:002
312899222,14cyclictest20368-21ssh22:45:091
312899221,5cyclictest15354-21apt-get23:30:031
3128992214,6cyclictest9958-21seq23:00:371
3128992214,6cyclictest16960-21ssh23:55:011
3128992213,7cyclictest2956-21awk22:55:251
3127992215,5cyclictest20677-21sh23:57:180
3127992213,6cyclictest278-21systemd-journal21:55:000
3127992211,9cyclictest1301-21apt22:55:100
157332220,2sleep10-21swapper/123:05:201
3129992114,4cyclictest22181-21diskmemload22:11:582
3129992113,6cyclictest22255-21sed19:55:002
3129992112,7cyclictest560-21dbus-daemon19:45:002
3129992112,5cyclictest53050irq/142-enp3s0-21:20:122
312899213,4cyclictest22181-21diskmemload21:43:051
312899211,5cyclictest0-21swapper/121:57:491
3128992115,4cyclictest22-21ksoftirqd/100:35:001
312899211,18cyclictest7177-21seq21:22:141
3127992113,6cyclictest9039-21ntpq23:00:190
3127992113,6cyclictest6909-21seq22:35:000
3127992113,6cyclictest11703-21sensors_temp21:25:230
3127992112,6cyclictest31026-21perl00:05:180
3127992112,6cyclictest23969-21users19:55:270
3127992111,7cyclictest24398-21sed21:35:150
3127992111,7cyclictest24398-21sed21:35:150
79372200,3sleep30-21swapper/319:20:133
319622200,0sleep2271rcuc/200:30:122
313099209,3cyclictest205172cstates19:50:133
3130992014,5cyclictest34-21ksoftirqd/323:30:003
3130992013,5cyclictest22372-21sed21:10:013
3130992013,5cyclictest18552-21ls23:55:233
3130992013,4cyclictest32612-21pmu-power00:30:193
3130992012,6cyclictest7359-21sed00:35:243
3130992012,6cyclictest29708-21cat23:15:293
3130992012,6cyclictest278-21systemd-journal00:25:003
3130992012,6cyclictest18501-21awk21:00:143
3130992012,6cyclictest16105-21seq23:54:233
3130992011,6cyclictest22810-21cat19:55:163
313099201,10cyclictest18655-21ssh21:30:293
312999209,3cyclictest28-21ksoftirqd/223:10:162
312999209,2cyclictest28-21ksoftirqd/219:20:202
312999201,4cyclictest4344-21cron21:45:002
3129992013,5cyclictest10732-21sensors20:40:212
3129992012,5cyclictest28-21ksoftirqd/222:20:042
3129992011,7cyclictest27305-21seq00:26:102
3129992011,7cyclictest15871-21awk23:05:212
3129992011,7cyclictest13337-21cron22:40:002
312899203,3cyclictest22181-21diskmemload22:07:211
312899202,16cyclictest7701-21sed22:35:161
312899201,7cyclictest13212-21chrt21:50:271
3128992014,4cyclictest24357-21sendmail19:59:591
3128992013,5cyclictest16144-21rm22:17:231
3128992012,6cyclictest7457-21find19:20:091
3128992011,7cyclictest770-21cat20:20:011
312899200,17cyclictest24918-21expr20:00:111
312799202,12cyclictest9053-21apt-get23:25:120
3127992012,6cyclictest408-21seq20:16:440
3127992012,6cyclictest29954-21smartctl21:15:160
3127992012,6cyclictest29296-21tune2fs20:10:150
3127992012,6cyclictest26212-21python300:25:240
3127992012,6cyclictest15398-21users19:35:250
3127992011,7cyclictest2690-21seq20:22:430
312799201,16cyclictest3872-21ssh23:45:130
62712191,15sleep00-21swapper/023:22:370
3130991914,3cyclictest34-21ksoftirqd/323:40:013
3130991913,4cyclictest7527-21sh00:11:343
3130991913,4cyclictest7005-21sh22:59:243
3130991913,4cyclictest12658-21ntp_offset19:30:223
3130991913,3cyclictest7780-21sh21:22:513
3130991912,5cyclictest7283-21ls22:35:133
3130991912,5cyclictest18754-21/usr/sbin/munin19:45:173
3130991911,6cyclictest4961-21cat21:45:133
3130991911,6cyclictest4043-21ntpq20:25:203
3130991911,6cyclictest30060-21grep00:05:013
3130991911,6cyclictest11584-21expr00:15:123
3130991911,5cyclictest21571-21cat23:10:113
313099191,12cyclictest5825-21runrttasks22:10:123
3130991910,7cyclictest24284-21awk21:10:233
312999196,9cyclictest53050irq/142-enp3s0-00:35:172
312999193,3cyclictest22181-21diskmemload21:31:012
312999192,3cyclictest22181-21diskmemload21:52:522
312999192,3cyclictest22181-21diskmemload00:00:382
3129991913,3cyclictest22181-21diskmemload22:55:082
3129991911,6cyclictest20099-21wc21:05:012
3129991911,6cyclictest16986-21awk23:30:212
3129991911,6cyclictest15920-21cut19:40:012
3129991911,6cyclictest12992-21tune2fs22:15:162
312999190,10cyclictest0-21swapper/223:45:292
312899197,4cyclictest52950irq/141-enp3s0-22:30:241
312899197,4cyclictest52950irq/141-enp3s0-19:50:251
312899197,4cyclictest52850irq/140-enp3s0-19:35:141
312899191,8cyclictest19805-21ssh21:31:251
3128991911,6cyclictest9577-21seq19:25:041
3128991911,6cyclictest8123-21sed20:35:181
312799199,7cyclictest560-21dbus-daemon21:10:000
312799198,7cyclictest9577-21apt-get20:40:110
312799197,3cyclictest141rcu_preempt22:00:230
3127991914,3cyclictest4067-21sh00:33:420
3127991914,3cyclictest24651-21sh23:35:290
3127991914,3cyclictest14021-21timerwakeupswit22:15:250
3127991914,3cyclictest11813-21sh22:14:460
3127991912,5cyclictest6581-21date00:35:180
3127991912,5cyclictest3640-21seq22:31:560
3127991912,5cyclictest14875-21ps20:50:220
3127991912,5cyclictest14875-21ps20:50:210
3127991911,6cyclictest9582-21seq19:25:090
3127991911,6cyclictest9227-21seq20:39:120
3127991911,6cyclictest560-21dbus-daemon23:35:010
3127991911,6cyclictest31792-21tr21:40:230
3127991911,6cyclictest278-21systemd-journal20:10:010
3127991911,6cyclictest25778-21python320:00:200
3127991911,6cyclictest22803-21expr21:10:110
3127991911,6cyclictest20555-21ls22:45:110
3127991911,6cyclictest18038-21sh00:20:120
3127991911,6cyclictest16248-21expr20:55:140
3127991911,6cyclictest12703-21cat00:15:230
3127991910,7cyclictest18556-21cat21:30:280
3127991910,6cyclictest6009-21cat20:30:170
150422190,2sleep334-21ksoftirqd/319:35:223
313099189,7cyclictest3947-21cpuspeed_turbos23:45:133
313099181,8cyclictest413-21python322:05:233
313099181,6cyclictest31156-21ssh23:40:243
3130991813,3cyclictest18629-21sh22:43:383
3130991812,4cyclictest34-21ksoftirqd/320:15:013
3130991812,4cyclictest30406-21sh00:05:113
3130991812,4cyclictest2883-21latency_hist20:25:003
3130991812,4cyclictest28003-21sh22:25:353
3130991812,4cyclictest28003-21sh22:25:343
3130991812,4cyclictest23808-21sh22:23:053
3130991812,4cyclictest2225-21sh23:20:033
3130991812,4cyclictest1423-21sh21:18:153
3130991811,5cyclictest9095-21seq20:37:273
3130991811,5cyclictest13908-21ls20:50:133
3130991811,5cyclictest13908-21ls20:50:123
3130991811,5cyclictest13272-21seq23:03:593
3130991811,4cyclictest22181-21diskmemload00:26:263
312999189,6cyclictest3939-21missed_timers20:25:182
312999186,3cyclictest141rcu_preempt22:05:112
312999182,3cyclictest22181-21diskmemload22:29:062
312999182,3cyclictest22181-21diskmemload22:29:052
3129991814,2cyclictest53050irq/142-enp3s0-21:46:242
3129991813,4cyclictest28-21ksoftirqd/220:00:012
3129991812,4cyclictest18202-21cpuspeed_turbos00:20:152
3129991811,4cyclictest20989-21sed21:05:182
3129991810,6cyclictest11013-21which21:25:172
312999180,5cyclictest0-21swapper/222:45:142
312899187,3cyclictest52950irq/141-enp3s0-20:45:141
312899187,3cyclictest22-21ksoftirqd/120:40:171
312899186,4cyclictest52950irq/141-enp3s0-20:25:131
312899184,3cyclictest141rcu_preempt23:40:011
312899182,4cyclictest22-21ksoftirqd/119:45:131
312899182,3cyclictest22181-21diskmemload00:20:431
312899182,11cyclictest991-21apt-get20:20:131
312899182,11cyclictest16051-21apt-get19:40:121
3128991811,5cyclictest26106-21basename22:25:111
3128991811,5cyclictest26106-21basename22:25:111
3128991811,5cyclictest20890-21cat21:05:181
3128991811,5cyclictest14182-21df20:50:141
3128991811,5cyclictest14182-21df20:50:131
3128991811,5cyclictest10949-21expr21:25:161
312799189,5cyclictest52850irq/140-enp3s0-22:25:120
312799189,5cyclictest52850irq/140-enp3s0-22:25:110
312799188,9cyclictest19003-21ssh23:08:010
312799182,11cyclictest18190-21apt-get19:45:120
3127991812,4cyclictest5253-21grep00:10:150
3127991812,4cyclictest2149-21sh23:43:450
3127991812,4cyclictest15930-21sh23:54:110
3127991811,5cyclictest20380-21ls19:50:120
3127991811,5cyclictest17631-21seq19:41:140
3127991810,6cyclictest14767-21ntp_kernel_pll_22:40:180
313099179,6cyclictest9912-21expr19:25:133
313099179,6cyclictest17204-21basename19:40:303
313099179,5cyclictest11455-21ntpq21:25:213
3130991712,3cyclictest29237-21sh22:50:533
3130991712,3cyclictest14648-21sh22:15:513
313099171,12cyclictest30793-21apt-get20:15:043
313099171,11cyclictest5279-21apt-get20:30:123
3130991710,5cyclictest25252-21cat20:00:153
3130991710,5cyclictest11906-21expr20:45:123
312999179,6cyclictest28-21ksoftirqd/219:30:222
312999178,7cyclictest15746-21dump-pmu-power20:55:002
312999178,7cyclictest15746-21dump-pmu-power20:54:592
3129991712,3cyclictest25536-21ntp_states20:00:182
3129991710,5cyclictest5358-21wc20:30:122
3129991710,5cyclictest29682-21sed20:10:192
312899179,6cyclictest560-21dbus-daemon20:35:001
312899179,6cyclictest27099-21diskstats20:05:151
312899179,6cyclictest18737-21basename21:00:161
312899179,5cyclictest31067-21grep00:05:191
312899179,4cyclictest23410-21sh21:10:161
312899177,7cyclictest30053-21sshd23:40:141
312899176,9cyclictest0-21swapper/121:36:541
312899176,9cyclictest0-21swapper/121:36:541
312799176,9cyclictest0-21swapper/023:15:060
3127991710,5cyclictest23771-21acpi00:00:110
3127991710,5cyclictest20055-21basename21:05:010
3127991710,5cyclictest12942-21cat20:45:220
313099169,5cyclictest27836-21gzip20:05:233
313099169,5cyclictest17399-21unixbench_multi20:55:263
312999169,5cyclictest12980-21sed20:45:232
312999168,6cyclictest9702-21grep19:25:132
312999168,6cyclictest17973-21sed21:00:012
312999168,6cyclictest1771-21cat20:20:202
3129991611,3cyclictest27305-21sed20:05:182
3129991610,4cyclictest19628-21grep19:45:252
3128991611,3cyclictest636-21runrttasks20:10:111
312799167,6cyclictest7616-21grep19:20:100
312899158,5cyclictest11928-21expr19:30:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional