You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-18 - 21:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Mon May 18, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23684217897,24sleep20-21swapper/207:05:382
23677216596,34sleep00-21swapper/007:05:320
239092140104,25sleep10-21swapper/107:08:341
221162139102,26sleep30-21swapper/307:05:033
1833221390,2sleep30-21swapper/312:04:063
1833221390,2sleep30-21swapper/312:04:063
245342820,2chrt0-21swapper/212:08:392
257002710,2sleep20-21swapper/210:09:002
136512640,1sleep00-21swapper/010:47:460
71412630,2sleep30-21swapper/309:55:123
64212620,3sleep10-21swapper/111:55:131
10082610,3sleep3331rcuc/308:45:143
8632590,2sleep20-21swapper/212:39:422
140772540,1sleep10-21swapper/111:12:111
237582520,2sleep30-21swapper/311:43:563
94682510,2sleep10-21swapper/111:33:021
218482510,2sleep00-21swapper/011:42:000
248472490,4sleep3331rcuc/309:20:013
55222480,2sleep30-21swapper/309:53:563
55222480,2sleep30-21swapper/309:53:563
229962480,1sleep10-21swapper/112:07:051
210112480,1chrt20082-21/usr/sbin/munin08:15:192
210112480,1chrt20082-21/usr/sbin/munin08:15:182
46342460,4sleep34633-21ssh11:29:553
105772460,1sleep00-21swapper/009:33:140
323002430,1sleep00-21swapper/011:25:230
160802430,2sleep30-21swapper/312:25:493
165182420,2sleep00-21swapper/011:38:230
40242330,9sleep334-21ksoftirqd/308:50:193
23822310,11sleep012-21ksoftirqd/012:15:310
89132280,2chrt0-21swapper/310:20:163
2449699283,17cyclictest6079-21chrt09:30:112
2449699274,16cyclictest25102-21ssh09:20:112
271662260,10sleep334-21ksoftirqd/311:45:363
2449499261,24cyclictest0-21swapper/107:12:351
2449399263,16cyclictest27177-21ssh10:10:040
2449399261,24cyclictest0-21swapper/007:12:070
24498992523,1cyclictest0-21swapper/307:11:553
24496992522,2cyclictest0-21swapper/207:12:052
191402250,2sleep00-21swapper/012:04:550
191402250,2sleep00-21swapper/012:04:550
24498992415,7cyclictest560-21dbus-daemon08:40:003
24498992410,3cyclictest26815-21apt-config10:10:013
24496992413,7cyclictest981-21sort09:50:162
24496992413,7cyclictest981-21sort09:50:162
2449499249,9cyclictest52950irq/141-enp3s0-10:40:121
2449499241,10cyclictest19852rm11:27:141
149362240,4sleep111252-21diskmemload12:25:191
261402230,2sleep10-21swapper/110:32:541
2449899232,6cyclictest8892ssh10:38:243
2449899232,6cyclictest8892ssh10:38:243
24498992313,7cyclictest20451-21cat08:15:143
24498992313,7cyclictest20451-21cat08:15:143
2449699232,15cyclictest32025-21ssh11:50:132
24496992316,5cyclictest3945-21awk10:40:202
24496992312,7cyclictest11680-21apt-get09:10:122
2449399233,14cyclictest27359-21ssh12:35:130
24493992312,7cyclictest22002-21grep08:20:000
24493992312,7cyclictest22002-21grep08:20:000
24498992213,6cyclictest10529-21awk09:05:243
24498992213,6cyclictest10529-21awk09:05:243
24498992211,8cyclictest5157-21apt-get11:30:123
24498992210,4cyclictest141rcu_preempt11:06:263
2449699222,17cyclictest22528-21cut10:30:162
2449699222,15cyclictest15026-21ssh10:25:122
2449699221,9cyclictest66050lldpd11:05:442
24496992213,5cyclictest0-21swapper/211:35:132
24496992210,9cyclictest278-21systemd-journal07:25:002
2449499227,3cyclictest140512sh11:35:531
24493992211,8cyclictest1-21systemd09:25:000
200362220,1sleep00-21swapper/010:05:000
149252220,1sleep30-21swapper/308:00:243
45142210,1sleep10-21swapper/110:16:371
2449899219,7cyclictest27653-21swap12:10:243
24498992114,5cyclictest26148-21ssh09:20:183
24498992113,6cyclictest16056-21seq11:37:553
24498992112,7cyclictest11626-21ntpq11:10:203
24498992112,6cyclictest6058-21awk08:55:223
24498992112,6cyclictest6006-21ntpq07:40:213
24498992112,6cyclictest4956-21seq12:18:083
24498992112,6cyclictest4769-21seq10:16:523
24498992112,6cyclictest28347-21ntp_kernel_pll_10:10:203
24498992111,7cyclictest27710-21df12:35:153
2449899210,18cyclictest26673-21date08:30:013
2449699216,5cyclictest28-21ksoftirqd/209:45:242
2449699212,13cyclictest30990-21apt-get07:25:132
2449699212,13cyclictest30990-21apt-get07:25:132
24496992112,6cyclictest10161-21wc07:50:202
24496992110,7cyclictest53050irq/142-enp3s0-12:25:122
24494992114,4cyclictest26042-21latency11:45:171
24494992113,6cyclictest24960-21diskstats08:25:141
24494992113,6cyclictest18770-21ls08:10:201
24494992113,6cyclictest11578-21seq09:34:141
24494992112,7cyclictest30144-21cat07:20:241
24494992112,6cyclictest14684-21awk08:00:211
24494992111,7cyclictest25641-21cron09:45:001
24494992111,6cyclictest52950irq/141-enp3s0-10:20:251
24494992110,9cyclictest560-21dbus-daemon10:00:001
2449399219,9cyclictest560-21dbus-daemon07:55:010
2449399214,7cyclictest0-21swapper/010:38:240
2449399214,7cyclictest0-21swapper/010:38:240
2449399212,8cyclictest21907-21/usr/sbin/munin10:30:220
2449399211,8cyclictest16302cpu10:15:130
24493992113,6cyclictest29264-21ssh11:47:430
24493992111,7cyclictest560-21dbus-daemon07:40:000
2449399211,17cyclictest23966-21apt-get11:20:110
2449399211,16cyclictest32308-21ssh09:25:150
2449399210,9cyclictest71442grep09:55:120
2449399210,5cyclictest0-21swapper/010:54:330
113942210,3chrt0-21swapper/209:10:002
113942210,3chrt0-21swapper/209:10:002
10422211,16sleep10-21swapper/109:50:181
10422211,16sleep10-21swapper/109:50:181
61342200,3sleep10-21swapper/108:55:231
299952200,3sleep122-21ksoftirqd/110:35:231
299952200,3sleep122-21ksoftirqd/110:35:231
263862200,3sleep00-21swapper/009:45:140
2449899203,13cyclictest29683-21ssh11:00:123
2449899202,12cyclictest26692-21apt-get07:15:133
2449899201,6cyclictest7640-21grep09:30:243
24498992014,4cyclictest34-21ksoftirqd/311:20:003
24498992014,4cyclictest22308-21sh10:30:143
24498992014,4cyclictest21934-21sendmail_mailtr12:30:233
24498992014,3cyclictest11608-21cpuspeed_turbos07:55:123
24498992013,4cyclictest11495-21sh09:10:023
24498992012,6cyclictest4060-21cut10:40:223
24498992012,6cyclictest24231-21seq12:08:213
24498992011,7cyclictest8145-21ntpq09:00:213
24498992011,7cyclictest11352-21ls07:55:013
2449699202,6cyclictest7006-21apt-get09:55:112
2449699202,14cyclictest23060-21ssh10:55:172
2449699202,12cyclictest2919-21apt-get08:50:102
2449699202,12cyclictest22483-21chrt08:20:132
24496992014,4cyclictest11814-21sh12:23:172
24496992013,5cyclictest6812-21cron12:20:012
24496992013,5cyclictest11453-21memory11:10:182
24496992012,6cyclictest12715-21cut12:00:002
24496992011,6cyclictest28075-21grep07:15:252
24496992011,6cyclictest27506-21sed08:30:192
24496992010,8cyclictest26909-21cron12:35:002
24496992010,8cyclictest13905-21cut09:35:222
24494992014,4cyclictest1245-21gltestperf08:45:161
24494992012,6cyclictest26711-21forks12:10:151
24494992012,6cyclictest18555-21which11:15:221
24494992012,6cyclictest15077-21awk10:25:121
24494992011,7cyclictest8415-21python307:45:231
24494992011,7cyclictest14544-21ls12:00:251
24494992011,7cyclictest14544-21ls12:00:251
2449499201,16cyclictest31949-21grep08:40:201
24494992010,7cyclictest5862-21load07:40:191
24494992010,7cyclictest29897-21grep08:35:231
24494992010,6cyclictest52950irq/141-enp3s0-12:30:121
24493992014,4cyclictest6638-21timerandwakeup11:30:250
24493992013,5cyclictest24038-21ssh12:32:060
24493992012,6cyclictest806-21tune2fs09:50:150
24493992012,6cyclictest806-21tune2fs09:50:150
24493992012,6cyclictest23545-21tune2fs10:55:200
24493992012,5cyclictest30434-21seq08:35:360
24493992012,4cyclictest11252-21diskmemload09:40:010
24493992010,7cyclictest278-21systemd-journal09:05:000
2449899197,9cyclictest0-21swapper/310:00:043
24498991914,3cyclictest19127-21sensors_temp08:10:243
24498991914,3cyclictest15755-21irqstats10:25:173
24498991913,4cyclictest16346-21sort10:50:123
24498991913,3cyclictest8194-21sh11:55:373
24498991912,5cyclictest34-21ksoftirqd/307:30:003
24498991912,5cyclictest34-21ksoftirqd/307:30:003
24498991912,5cyclictest1872-21smartctl07:30:233
24498991912,4cyclictest28662-21dump-pmu-power08:35:003
24498991912,4cyclictest23853-21sh11:20:043
24498991912,4cyclictest17983-21sh09:39:023
24498991912,4cyclictest14208-21sh10:48:203
24498991911,6cyclictest7556-21expr12:20:163
24498991911,6cyclictest31982-21cat11:50:113
24498991911,6cyclictest30945-21cat07:25:003
24498991911,6cyclictest19996-21expr09:40:163
24498991910,6cyclictest31051-21cat08:40:113
2449699198,8cyclictest5740-21sed11:30:172
2449699197,9cyclictest0-21swapper/211:00:162
2449699192,11cyclictest11252-21diskmemload12:00:122
2449699192,11cyclictest11252-21diskmemload12:00:122
2449699191,8cyclictest19929-21switchtime11:40:222
2449699191,6cyclictest11252-21diskmemload09:25:522
24496991913,3cyclictest11252-21diskmemload10:48:312
24496991912,4cyclictest18749-21sh09:15:142
24496991911,5cyclictest1229-21cut08:45:162
2449499197,10cyclictest10186-21rm12:21:381
2449499192,12cyclictest9359-21apt-get09:05:131
2449499192,12cyclictest9359-21apt-get09:05:131
2449499191,9cyclictest23635-21python308:20:231
24494991913,4cyclictest3627-21sh12:16:461
24494991913,4cyclictest28277-21sh09:21:521
24494991913,4cyclictest20011-21timerandwakeup11:40:221
24494991913,4cyclictest18950irq/143-ahci[0011:00:011
24494991912,5cyclictest30935-21seq12:37:181
24494991912,5cyclictest2553-21seq07:34:351
24494991912,5cyclictest23150-21seq09:18:291
24494991912,5cyclictest1340-21seq09:25:391
24494991911,6cyclictest9560-21seq10:45:091
24494991911,6cyclictest4460-21fw_packets11:05:181
24494991911,6cyclictest25057-21awk11:20:201
24494991911,6cyclictest12954-21which09:10:241
24494991911,6cyclictest10207-21cat07:50:191
24494991910,7cyclictest28267-21seq08:30:261
2449399196,9cyclictest14937-21netstat12:25:190
24493991914,3cyclictest4477-21timerwakeupswit10:40:250
24493991913,4cyclictest21638-21rm09:16:570
24493991913,4cyclictest17291-21grep08:05:250
24493991912,5cyclictest7031-21wc09:00:000
24493991912,5cyclictest21112-21/usr/sbin/munin10:05:170
24493991911,6cyclictest9092-21cut10:20:180
24493991910,7cyclictest560-21dbus-daemon08:30:000
161852190,4sleep0131rcuc/011:14:190
15752190,2sleep10-21swapper/111:03:041
323462180,2sleep00-21swapper/008:40:250
2449899189,6cyclictest7996-21cat07:45:183
2449899182,11cyclictest23035-21apt-get10:55:173
2449899182,11cyclictest22479-21apt-get08:20:133
2449899182,11cyclictest15834-21apt-get08:05:123
24498991813,3cyclictest28878-21sh09:46:313
24498991812,4cyclictest1686-21sh09:26:003
24498991810,6cyclictest4580-21find07:38:033
2449699189,6cyclictest15154-21grep08:00:252
2449699187,7cyclictest53050irq/142-enp3s0-11:45:122
2449699184,5cyclictest29028-21ssh10:35:142
2449699184,5cyclictest29028-21ssh10:35:142
2449699182,12cyclictest75-21kswapd010:00:112
2449699182,11cyclictest75-21kswapd010:20:122
2449699182,10cyclictest11252-21diskmemload12:10:132
24496991813,3cyclictest19311-21timerwakeupswit08:10:262
24496991813,3cyclictest1803-21ssh10:15:132
24496991812,4cyclictest19564-21tr09:40:132
24496991811,5cyclictest8417-21seq09:00:232
24496991811,5cyclictest631-21sed08:45:012
24496991811,5cyclictest28154-21cat10:10:182
24496991811,4cyclictest11252-21diskmemload11:25:072
24496991810,5cyclictest26106-21awk08:25:252
24496991810,5cyclictest24187-21expr11:20:132
2449499189,7cyclictest492-21cut07:30:001
2449499189,7cyclictest492-21cut07:30:001
2449499189,7cyclictest16694-21cut10:50:141
2449499189,6cyclictest3821-21awk08:50:171
2449499189,6cyclictest20294-21cat08:15:121
2449499189,6cyclictest20294-21cat08:15:121
2449499188,8cyclictest21102-21sed10:05:171
2449499188,7cyclictest32349-21cat11:50:131
2449499185,11cyclictest13567-21ssh10:00:031
24494991812,4cyclictest15242-21sh09:36:151
24494991812,3cyclictest12628-21sendmail_mailst07:55:221
24494991812,3cyclictest11252-21diskmemload10:10:011
24494991811,4cyclictest29466-21sh09:47:061
24494991810,6cyclictest7656-21perl09:00:151
24494991810,6cyclictest3608-21cut07:35:191
24494991810,6cyclictest17880-21date08:10:001
2449399188,8cyclictest32595-21gltestperf11:50:160
2449399186,9cyclictest0-21swapper/011:00:070
2449399186,8cyclictest5112-21apt-get07:40:120
2449399182,11cyclictest6285-21ssh11:55:120
2449399182,10cyclictest7302-21apt-get12:20:150
24493991813,3cyclictest28900-21cpu07:20:120
24493991811,5cyclictest16633-21sed10:25:250
24493991810,6cyclictest28154-21awk07:15:260
24493991810,6cyclictest23204-21sed08:20:180
24493991810,5cyclictest11252-21diskmemload11:05:140
2449699179,6cyclictest278-21systemd-journal08:57:212
2449699172,11cyclictest17288-21apt-get11:15:122
2449699171,6cyclictest17146-21missed_timers10:50:182
2449699171,3cyclictest141rcu_preempt07:45:122
24496991711,4cyclictest12728-21sed07:55:232
24496991710,5cyclictest30097-21sed08:35:252
24496991710,5cyclictest17187-21sed08:05:242
2449499170,15cyclictest28580-21munin-plugin-st07:20:001
2449399179,3cyclictest636-21runrttasks09:37:110
2449399178,6cyclictest27291-21needreboot12:10:200
2449399172,11cyclictest17661-21diskstats11:15:140
24493991712,3cyclictest15584-21idleruntime-cro08:05:000
24493991712,3cyclictest10830-21grep09:05:260
24493991712,3cyclictest10830-21grep09:05:260
24493991711,3cyclictest11252-21diskmemload09:12:540
24493991710,5cyclictest28653-21ls08:35:000
24493991710,5cyclictest2761-21cat08:50:000
24493991710,5cyclictest0-21swapper/008:10:260
2449699168,6cyclictest5509-21tune2fs07:40:162
2449699168,6cyclictest2590-21cron07:35:002
2449699167,7cyclictest4574-21anacron07:38:032
2449399169,5cyclictest8449-21cat07:45:230
2449399169,5cyclictest12999-21awk07:55:250
2449399167,7cyclictest31966-21grep07:25:220
2449399167,7cyclictest31966-21grep07:25:210
2449399162,10cyclictest75-21kswapd012:05:120
2449399161,11cyclictest66050lldpd07:30:120
2449399158,5cyclictest3568-21irqstats08:50:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional