You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-01 - 20:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat Aug 01, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
306752192170,15sleep00-21swapper/007:07:400
30575217396,25sleep20-21swapper/207:06:222
641721670,1sleep30-21swapper/308:40:233
3110621580,3sleep10-21swapper/107:10:011
30621215698,23sleep30-21swapper/307:06:593
305312149100,42sleep10-21swapper/107:05:471
1748621130,2sleep00-21swapper/012:14:010
226262770,3sleep122633-21latency09:35:191
310332720,2sleep10-21swapper/112:23:381
220312650,2chrt0-21swapper/008:00:280
88022610,1sleep30-21swapper/310:58:043
216642560,2sleep00-21swapper/009:11:340
194972520,1sleep20-21swapper/212:38:352
194972520,1sleep20-21swapper/212:38:352
124172510,1sleep312416-21ssh10:14:453
74502490,2sleep00-21swapper/010:10:250
291272480,2chrt0-21swapper/211:59:112
83142460,2sleep30-21swapper/309:48:173
153592440,1sleep10-21swapper/109:53:201
51552410,4sleep00-21swapper/011:41:150
194362410,3chrt0-21swapper/011:51:340
107852380,1sleep30-21swapper/309:50:133
69882340,2sleep30-21swapper/309:24:493
160162300,17sleep20-21swapper/209:53:582
130772300,3sleep00-21swapper/011:24:330
3112999291,24cyclictest23085-21df_inode10:45:150
7842270,9sleep012-21ksoftirqd/012:01:210
149542270,11sleep122-21ksoftirqd/109:00:191
4072260,3sleep1211rcuc/112:01:021
3113299263,9cyclictest18529-21ssh11:05:133
3113299263,9cyclictest18529-21ssh11:05:133
31129992618,6cyclictest9485-21ssh09:49:240
3112999261,7cyclictest28336-21df11:35:140
233022260,11sleep334-21ksoftirqd/309:35:253
81922250,2sleep30-21swapper/312:30:193
3112999259,11cyclictest640-21snmpd09:25:130
3112999250,24cyclictest0-21swapper/007:11:460
234452250,1sleep00-21swapper/012:18:030
97782240,1sleep30-21swapper/311:45:153
3113299243,8cyclictest32435-21apt-get11:15:113
3113299242,15cyclictest25822-21ssh09:15:133
3113299241,1cyclictest0-21swapper/307:11:173
31131992422,1cyclictest0-21swapper/207:11:202
31130992413,7cyclictest53050irq/142-enp3s0-10:20:141
31129992414,7cyclictest843-21awk10:05:260
31132992317,4cyclictest34-21ksoftirqd/308:50:003
31132992314,6cyclictest8991-21mailstats09:25:243
31132992313,7cyclictest26328-21diskstats12:20:143
31131992315,6cyclictest16136-21cron09:05:002
31130992313,7cyclictest10843-21sensors_temp07:35:241
173992230,2sleep10-21swapper/111:50:191
160482230,2sleep30-21swapper/310:16:343
3113299222,14cyclictest31575-21ssh10:05:133
31132992216,4cyclictest34-21ksoftirqd/312:00:003
31132992213,6cyclictest13073-21ntpq12:10:223
31131992213,6cyclictest343-21diskstats11:15:142
31131992213,6cyclictest278-21systemd-journal12:30:002
3113099228,5cyclictest5624-21expr12:05:161
31130992215,4cyclictest32261-21proc_pri08:25:221
31130992213,7cyclictest2637-21seq07:18:141
31130992212,7cyclictest2070-21df_inode10:30:141
31130992212,7cyclictest15371-21grep11:25:241
31130992210,9cyclictest18891-21sh11:05:151
31130992210,9cyclictest18891-21sh11:05:151
3112999224,10cyclictest28913-21ssh09:40:110
3112999222,8cyclictest0-21swapper/010:35:480
31129992215,5cyclictest12-21ksoftirqd/012:25:000
3112999220,19cyclictest17399-21seq09:55:040
262632220,2sleep30-21swapper/310:47:083
197532220,2sleep00-21swapper/011:05:230
197532220,2sleep00-21swapper/011:05:230
94932210,4chrt9489-21cron08:50:001
3113299211,16cyclictest18285-21diskmemload10:25:133
31132992111,6cyclictest0-21swapper/310:30:133
31131992113,6cyclictest6551-21diskstats10:10:172
31131992113,6cyclictest1454-21smartctl07:15:152
31131992112,6cyclictest8265-21irqstats08:45:172
31131992112,6cyclictest28965-21cut08:20:012
3113199211,17cyclictest11677-21munin-plugin-st07:40:002
3113199211,14cyclictest304552sleep212:00:122
31131992110,6cyclictest25832-21ssh09:15:132
31130992113,6cyclictest6555-21diskstats10:10:161
31130992113,6cyclictest2046-21gzip08:30:241
31130992112,6cyclictest27333-21cat10:25:141
31130992112,6cyclictest16835-21perl09:05:151
31130992111,6cyclictest53050irq/142-enp3s0-10:40:131
3112999214,10cyclictest24600-21ssh10:00:140
3112999211,7cyclictest141rcu_preempt09:20:150
3112999211,4cyclictest0-21swapper/009:16:410
31129992112,7cyclictest278-21systemd-journal08:20:000
3112999211,18cyclictest0-21swapper/010:27:270
31129992111,7cyclictest879-21cut08:30:000
31129992111,6cyclictest52750irq/139-enp3s0-09:35:130
26812210,1sleep00-21swapper/011:16:190
73402200,2sleep20-21swapper/209:25:002
70172200,2sleep30-21swapper/311:43:023
3113299208,9cyclictest0-21swapper/309:30:583
3113299208,9cyclictest0-21swapper/309:30:583
3113299206,12cyclictest401-21ssh12:01:023
3113299202,15cyclictest12248-21sort08:55:163
3113299202,13cyclictest11430-31updatedb.mlocat07:37:143
3113299202,12cyclictest8804-21ssh10:35:163
31132992013,5cyclictest11643-21munin-run08:55:003
31132992013,4cyclictest30209-21sh09:40:233
31132992011,6cyclictest2883-21grep08:35:003
31132992010,7cyclictest1443-21sed12:25:213
31131992014,4cyclictest27776-21sh11:35:022
31131992013,4cyclictest32252-21sh12:24:482
31131992012,6cyclictest19669-21date07:55:232
31131992012,6cyclictest19669-21date07:55:232
3113199201,17cyclictest21782-21ssh10:44:332
31131992011,6cyclictest7893-21expr09:25:152
31131992011,6cyclictest24151-21cat10:45:242
31131992010,6cyclictest52950irq/141-enp3s0-11:05:142
31131992010,6cyclictest52950irq/141-enp3s0-11:05:142
3113099201,5cyclictest0-21swapper/109:25:141
31130992014,4cyclictest9261-21sh09:49:111
31130992014,4cyclictest1374-21switchtime11:15:231
31130992014,3cyclictest24684-21cpuspeed_turbos10:00:151
31130992013,4cyclictest5405-21sh12:28:271
31130992013,4cyclictest25512-21sh11:55:431
31130992013,4cyclictest20941-21sh09:10:531
31130992013,4cyclictest17671-21sh12:36:511
31130992013,4cyclictest17671-21sh12:36:511
31130992013,4cyclictest17573-21sh11:04:451
31130992013,4cyclictest17573-21sh11:04:451
31130992012,6cyclictest9350-21seq07:34:031
31130992012,6cyclictest4060-21pluginstate08:35:211
31130992011,7cyclictest5835-21tune2fs07:25:151
31130992011,6cyclictest28635-21seq08:17:451
31130992011,6cyclictest13785-21idleruntime10:15:171
3113099201,14cyclictest4355-21ssh10:55:151
31129992015,3cyclictest7455-21latency_hist08:45:010
31129992013,5cyclictest18285-21diskmemload10:15:250
31129992013,5cyclictest13764-21cron09:00:000
31129992012,6cyclictest9925-21seq11:45:160
31129992012,6cyclictest8416-21ls12:30:200
31129992012,6cyclictest23561-21ntp_kernel_pll_08:05:210
31129992011,6cyclictest52750irq/139-enp3s0-10:20:140
31129992011,6cyclictest4749-21grep07:20:230
31129992010,7cyclictest25997-21cat08:10:250
3112999200,8cyclictest66050lldpd09:51:210
223522200,10sleep334-21ksoftirqd/309:58:203
150702200,9sleep012-21ksoftirqd/012:35:160
150702200,9sleep012-21ksoftirqd/012:35:160
3113299199,6cyclictest52850irq/140-enp3s0-11:20:123
3113299199,6cyclictest24763-21grep10:00:153
3113299197,3cyclictest13402fschecks_time07:15:143
3113299192,11cyclictest18736-21ssh09:10:123
31132991913,4cyclictest13839-21dump-pmu-power07:45:003
31132991913,3cyclictest30894-21sh11:13:573
31132991912,5cyclictest20961-21cron12:40:003
31132991912,5cyclictest20961-21cron12:40:003
31132991912,4cyclictest26806-21sh10:25:003
31132991911,6cyclictest7534-21cat07:30:123
31132991911,6cyclictest22246-21readlink11:30:223
31132991911,6cyclictest16223-21cut10:40:163
3113299191,14cyclictest4039-21latency_hist10:55:003
31132991910,7cyclictest560-21dbus-daemon07:55:003
31132991910,6cyclictest278-21systemd-journal11:29:553
3113199199,8cyclictest52750irq/139-enp3s0-09:10:112
3113199199,7cyclictest560-21dbus-daemon10:10:002
3113199199,7cyclictest17136-21cat09:05:182
3113199194,13cyclictest14204-21cut07:45:132
3113199192,12cyclictest7036-21ssh11:20:132
31131991914,3cyclictest22737-21cpuspeed_turbos08:05:142
31131991913,4cyclictest8741-21sh09:48:422
31131991913,4cyclictest26524-21sh09:38:122
31131991913,4cyclictest22770-21sh11:30:292
31131991913,4cyclictest19680-21idleruntime-cro10:20:002
31131991913,4cyclictest17401-21sh09:31:172
31131991913,4cyclictest17401-21sh09:31:172
31131991913,4cyclictest16876-21interrupts07:50:192
31131991913,4cyclictest14653-21sh12:11:192
31131991913,4cyclictest10580-21sh11:45:222
31131991912,5cyclictest21405-21ntp_kernel_err08:00:212
31131991912,5cyclictest19680-21cat12:15:182
31131991912,5cyclictest17828-21sh11:27:262
31131991912,4cyclictest18285-21diskmemload12:32:192
31131991911,6cyclictest4428-21python307:20:222
31131991911,6cyclictest21656-21ls10:20:272
31131991911,6cyclictest16873-21diskstats11:50:142
31131991911,6cyclictest12161-21ls08:55:152
31131991911,5cyclictest9002-21sed07:30:262
31131991911,5cyclictest26803-21mailstats11:10:252
31131991910,7cyclictest2192-21cat08:30:252
31131991910,6cyclictest5749-21diskstats07:25:142
3113099197,10cyclictest0-21swapper/110:48:321
3113099192,10cyclictest53050irq/142-enp3s0-11:30:121
31130991914,3cyclictest4980-21sh11:41:051
31130991914,3cyclictest25267-21gltestperf08:10:181
31130991913,4cyclictest23917-21sh12:18:301
31130991913,3cyclictest676-21sh10:51:591
31130991913,3cyclictest5611-21sh10:09:571
31130991913,3cyclictest14429-21sh12:11:051
31130991912,5cyclictest17815-21seq07:51:211
31130991912,4cyclictest636-21runrttasks11:47:241
31130991912,4cyclictest1847-21sh09:20:221
31130991911,6cyclictest11303-21seq08:50:511
31130991911,5cyclictest28911-21apt-get09:40:111
31130991911,5cyclictest16083-21mailstats09:30:231
31130991911,5cyclictest16083-21mailstats09:30:231
31130991910,6cyclictest29431-21diskstats08:20:141
31130991910,6cyclictest15482-21cat10:40:011
3112999194,8cyclictest15761-21ssh10:40:130
3112999192,12cyclictest5026-21apt-get12:05:130
3112999192,11cyclictest25529-21ssh11:10:150
31129991913,3cyclictest4299-21/usr/sbin/munin10:55:270
31129991912,5cyclictest25088-21unixbench-2d11:55:270
31129991912,5cyclictest10609-21netstat08:50:200
31129991911,6cyclictest16281-21date09:05:010
3112999191,13cyclictest18285-21diskmemload11:00:130
3112999191,13cyclictest18285-21diskmemload11:00:130
3113299189,7cyclictest16961-21smartctl09:05:163
3113299187,9cyclictest0-21swapper/311:35:453
3113299183,9cyclictest640-21snmpd07:20:113
3113299181,8cyclictest640-21snmpd11:53:143
3113299181,4cyclictest0-21swapper/312:17:363
31132991813,3cyclictest278-21systemd-journal07:50:003
31132991811,5cyclictest28068-21systemctl08:15:223
31132991811,4cyclictest13214-21sh11:00:363
31132991811,4cyclictest13214-21sh11:00:363
31132991810,6cyclictest31679-21tune2fs08:25:163
31132991810,6cyclictest20505-21sort08:00:133
3113199189,6cyclictest5583-21cat08:40:132
3113199187,9cyclictest0-21swapper/210:57:012
31131991813,3cyclictest9482-21sh12:08:022
31131991813,3cyclictest4575-21sh11:40:422
31131991813,3cyclictest21547-21sh09:57:342
31131991812,4cyclictest29763-21sh09:40:182
31131991812,4cyclictest27907-21sh10:25:192
31131991812,4cyclictest24454-21sh10:00:062
31131991812,4cyclictest12481-21sh10:37:212
31131991811,5cyclictest30379-21expr10:50:182
31131991811,5cyclictest30190-21which08:20:202
31131991811,5cyclictest2950-21ps10:30:222
31131991811,5cyclictest11693-21entropy11:00:152
31131991811,5cyclictest11693-21entropy11:00:152
3113199181,14cyclictest12494-21tail07:40:192
31131991810,6cyclictest829-21date08:30:002
31131991810,6cyclictest11695-21cut08:54:592
31131991810,5cyclictest4290-21switchtime08:35:232
3113099189,7cyclictest12019-21grep08:55:141
3113099186,10cyclictest29089-21runrttasks11:35:221
3113099182,11cyclictest20526-21apt-get08:00:151
31130991813,3cyclictest6875-21sh11:20:041
31130991813,3cyclictest30010-21sh11:13:071
31130991813,3cyclictest11711-21sh12:32:491
31130991811,5cyclictest6873-21seq08:41:321
31130991811,4cyclictest19712-21sh09:55:491
31130991810,6cyclictest25996-21expr09:15:141
31130991810,5cyclictest4004-21sed07:20:171
3112999189,7cyclictest29079-21cat08:20:100
3112999189,3cyclictest18285-21diskmemload11:30:240
3112999188,8cyclictest13947-21/usr/sbin/munin11:25:180
31129991813,3cyclictest12463-21gltestperf07:40:180
31129991812,4cyclictest937-21/usr/sbin/munin08:30:130
31129991812,4cyclictest21127-21sh09:34:480
31129991812,4cyclictest21127-21sh09:34:480
31129991811,5cyclictest8996-21awk07:30:260
31129991811,5cyclictest30147-21sh10:50:170
31129991810,5cyclictest1206-21cat07:15:120
3112999180,9cyclictest539650ssh10:32:160
3113299179,6cyclictest5923-21grep12:05:193
3113299178,6cyclictest278-21systemd-journal09:04:553
3113299178,6cyclictest20236-21cron08:00:003
3113299178,6cyclictest20236-21cron07:59:593
3113299175,8cyclictest22662-21apt-get08:05:133
3113299173,8cyclictest24801-21apt-get08:10:143
31132991711,3cyclictest6654-21xargs07:25:243
31131991710,5cyclictest25955-21sensors08:10:242
3113099179,6cyclictest13475-21seq07:40:291
31130991710,5cyclictest14337-21expr07:45:141
3112999178,7cyclictest10027-21diskstats07:35:160
3112999178,6cyclictest469-21/usr/sbin/munin12:25:190
3112999172,2cyclictest141rcu_preempt07:25:250
3112999172,11cyclictest14093-21apt-get07:45:130
3112999172,10cyclictest16398-21apt-get09:05:120
3112999171,11cyclictest7616-21apt-get08:45:120
31129991710,5cyclictest18573-21cat07:55:120
31129991710,5cyclictest18573-21cat07:55:120
3113299169,5cyclictest29407-21expr08:20:133
3113299167,6cyclictest3849-21sed08:35:183
3113099169,5cyclictest22781-21expr08:05:141
3113099169,5cyclictest18609-21cat07:55:121
3113099169,5cyclictest18609-21cat07:55:121
3112999168,5cyclictest3826-21grep08:35:180
31129991610,4cyclictest17560-21cat07:50:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional