You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-01 - 19:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed Jul 01, 2026 12:45:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
121472191169,15sleep30-21swapper/307:06:483
123222176101,24sleep20-21swapper/207:09:032
12368215298,25sleep10-21swapper/107:09:381
120472137101,25sleep00-21swapper/007:05:280
17002710,2sleep20-21swapper/209:55:312
61792660,8sleep334-21ksoftirqd/311:53:303
11242660,2sleep10-21swapper/109:55:221
280472600,6sleep127989-21ssh10:15:001
280472600,6sleep127989-21ssh10:15:001
253612570,2sleep20-21swapper/207:39:522
126222530,1sleep00-21swapper/010:04:040
167912510,2sleep10-21swapper/108:35:001
145522510,1sleep10-21swapper/110:27:531
74332490,2sleep00-21swapper/012:16:590
261872470,2sleep30-21swapper/310:13:263
261872470,2sleep30-21swapper/310:13:263
124092440,1sleep30-21swapper/310:49:373
116692420,1chrt11670-21/usr/sbin/munin09:40:200
12672993916,10cyclictest4335-21ssh09:35:173
1267099372,16cyclictest31488-21apt-get10:40:091
296472350,2chrt0-21swapper/310:38:423
207972340,3sleep334-21ksoftirqd/311:17:473
76302330,9sleep122-21ksoftirqd/111:54:531
155812300,1sleep00-21swapper/009:20:220
12672993011,7cyclictest34-21ksoftirqd/310:23:293
240522290,1sleep10-21swapper/108:50:171
218152290,2chrt0-21swapper/212:04:572
39702280,2sleep30-21swapper/312:15:093
1267099282,17cyclictest13278-21ssh10:50:121
203812270,2sleep00-21swapper/009:46:140
12672992713,9cyclictest53050irq/142-enp3s0-10:25:133
1267199277,14cyclictest24404-21ssh10:35:122
12672992614,8cyclictest53050irq/142-enp3s0-11:35:103
1267299261,23cyclictest0-21swapper/307:11:193
1267199261,24cyclictest0-21swapper/207:11:442
1267099261,12cyclictest191242sh12:25:191
1267299252,18cyclictest21628-21ssh09:25:123
12672992513,10cyclictest34-21ksoftirqd/312:25:003
1267099250,24cyclictest0-21swapper/107:11:401
1266999251,1cyclictest0-21swapper/007:11:410
57772240,2sleep10-21swapper/109:35:561
16252240,4sleep00-21swapper/008:00:000
143832240,10sleep334-21ksoftirqd/309:42:163
1267299244,10cyclictest34-21ksoftirqd/311:55:423
1267299242,15cyclictest17320-21taskset10:30:143
1267199241,13cyclictest6777-21irqstats08:10:182
1267099243,13cyclictest29378-21ssh12:10:111
1267099241,8cyclictest1773950ssh09:45:101
12670992418,4cyclictest31949-21mailstats11:25:211
12670992416,6cyclictest28105-21cat07:45:141
12669992417,5cyclictest9233-21ssh09:15:380
12669992413,8cyclictest2490-21python311:50:230
118882240,9sleep334-21ksoftirqd/311:11:043
268842230,2sleep10-21swapper/109:28:421
1267299239,2cyclictest141rcu_preempt12:30:133
1267299239,2cyclictest141rcu_preempt12:30:133
12671992312,8cyclictest17200-21cat08:35:162
12671992312,8cyclictest17200-21cat08:35:162
1267199231,12cyclictest32758-21irqstats07:55:192
1267099232,15cyclictest7282-21ssh09:15:131
1266999235,11cyclictest141rcu_preempt10:55:040
313262220,1sleep00-21swapper/012:10:370
251332220,1sleep00-21swapper/010:35:180
172862220,2sleep10-21swapper/110:06:471
1267299222,3cyclictest141rcu_preempt11:08:093
1267199224,10cyclictest28-21ksoftirqd/212:35:242
12671992218,2cyclictest271rcuc/209:15:232
12670992213,6cyclictest21419-21ntp_states10:55:201
12670992212,7cyclictest4321-21df_inode08:05:161
1266999223,3cyclictest141rcu_preempt12:35:240
12669992213,7cyclictest11711-21ls10:25:260
12669992210,3cyclictest20761-21cron10:10:000
12669992210,2cyclictest12-21ksoftirqd/010:53:220
175912210,2sleep10-21swapper/111:15:211
1267299216,5cyclictest1324-21kworker/u8:010:44:593
1267299211,5cyclictest0-21swapper/312:13:073
12672992113,6cyclictest14299-21ntp_kernel_pll_10:50:203
12672992113,5cyclictest636-21runrttasks09:22:423
12672992110,8cyclictest23828-21df_inode08:50:153
1267199211,7cyclictest0-21swapper/212:22:262
12671992112,7cyclictest20930-21sh11:17:542
1267199211,14cyclictest14086-21ssh10:05:112
1267199211,10cyclictest0-21swapper/212:29:522
12671992110,6cyclictest52950irq/141-enp3s0-11:30:112
1267099214,15cyclictest8334-21python310:00:231
1267099212,7cyclictest25107-21apt-get09:50:131
12670992113,6cyclictest15207-21awk07:15:151
12670992112,7cyclictest17581-21smartctl07:20:161
12670992111,7cyclictest13040-21fw_packets08:25:181
12669992114,5cyclictest12-21ksoftirqd/011:05:000
12669992113,4cyclictest31820-21diskmemload11:10:170
12669992111,6cyclictest6184-21apt-get10:45:120
201422200,2chrt0-21swapper/309:46:013
1267299209,2cyclictest34-21ksoftirqd/310:05:183
12672992016,2cyclictest341ksoftirqd/310:00:013
1267299201,5cyclictest0-21swapper/310:18:563
1267299201,5cyclictest0-21swapper/309:18:063
12672992012,5cyclictest28709-21/usr/sbin/munin09:30:373
1267199208,10cyclictest29530-21processes10:15:212
1267199201,9cyclictest4318-21grep10:20:242
1267199201,5cyclictest11540-21python310:25:252
1267199201,5cyclictest0-21swapper/210:14:292
1267199201,5cyclictest0-21swapper/210:14:292
12671992013,4cyclictest5943-21sh09:36:052
12671992012,6cyclictest23664-21python312:05:222
12671992012,6cyclictest10292-21date11:10:182
12671992012,5cyclictest17698-21cat10:30:152
12671992011,6cyclictest2264-21pmu-power11:50:222
1267199201,10cyclictest0-21swapper/209:26:402
12671992010,7cyclictest19342-21cat07:25:152
12670992013,5cyclictest21085-21seq08:44:541
12670992012,6cyclictest27658-21cat11:00:121
12670992012,6cyclictest24811-21ls07:35:251
12670992012,6cyclictest24097-21wc10:35:011
12670992012,6cyclictest17712-21seq12:01:061
12670992011,7cyclictest9391-21seq11:10:091
12670992011,7cyclictest30429-21tune2fs07:50:161
1267099201,17cyclictest29198-21ssh10:38:171
12670992010,7cyclictest17612-21memory08:35:191
12670992010,7cyclictest17612-21memory08:35:191
1266999209,8cyclictest21689-21df_inode08:45:140
1266999209,8cyclictest0-21swapper/010:20:000
1266999202,9cyclictest141rcu_preempt10:14:300
1266999202,9cyclictest141rcu_preempt10:14:290
1266999202,12cyclictest14720-21chrt08:30:140
12669992012,6cyclictest30948-21ls07:50:220
12669992012,6cyclictest27158-21smartctl11:45:170
12669992012,6cyclictest25372-21cron08:55:010
12669992012,6cyclictest1585-21uptime09:55:260
12669992011,7cyclictest9960-21seq09:39:480
12669992011,7cyclictest19551-21df08:40:150
12669992011,6cyclictest560-21dbus-daemon11:45:000
12669992010,6cyclictest52750irq/139-enp3s0-09:50:130
1267299199,6cyclictest53050irq/142-enp3s0-11:45:143
1267299198,7cyclictest53050irq/142-enp3s0-12:05:243
1267299198,2cyclictest34-21ksoftirqd/312:35:293
1267299197,2cyclictest34-21ksoftirqd/312:04:083
1267299193,9cyclictest20352-21ssh10:55:133
1267299192,11cyclictest30702-21ssh11:25:113
1267299191,8cyclictest19229-21ntp_kernel_pll_12:25:223
12672991915,2cyclictest341ksoftirqd/308:20:003
1267299191,4cyclictest0-21swapper/311:34:423
12672991913,4cyclictest31820-21diskmemload09:12:373
1267299191,2cyclictest141rcu_preempt10:00:243
12672991912,4cyclictest16750-21dump-pmu-power07:20:003
12672991911,6cyclictest30631-21sed09:05:183
1267299190,9cyclictest34-21ksoftirqd/309:50:153
1267199199,7cyclictest8598-21diskstats08:15:132
1267199196,4cyclictest141rcu_preempt08:50:002
1267199193,14cyclictest169832sleep209:21:092
1267199193,11cyclictest20341-21apt-get10:55:122
1267199192,13cyclictest4059-21ssh12:15:132
12671991914,3cyclictest30554-21proc_pri12:10:212
12671991913,4cyclictest27416-21sh11:22:202
12671991913,4cyclictest2484-21sh09:11:012
12671991913,3cyclictest2122-21gdbus11:05:012
12671991911,6cyclictest31552-21memory11:25:172
12671991911,6cyclictest31239-21ls09:05:242
12671991911,6cyclictest2761-21ntpq08:00:222
12671991911,6cyclictest26982-21vmstat12:30:282
12671991911,6cyclictest26982-21vmstat12:30:272
12671991911,6cyclictest24367-21seq09:49:552
12671991911,6cyclictest12355-21apt-get11:35:102
12671991910,7cyclictest19533-21cat08:40:152
1267099192,15cyclictest29712-21grep10:15:231
1267099191,8cyclictest31840-21ssh09:10:001
12670991915,2cyclictest8003-21cron08:15:001
12670991914,3cyclictest28591-21sh12:09:411
12670991912,5cyclictest21196-21sh11:40:251
12670991912,5cyclictest16021-21timerwakeupswit09:20:251
12670991912,4cyclictest4162-21open_inodes10:20:221
12670991911,6cyclictest9801-21seq08:15:411
12670991911,6cyclictest4467-21cut12:15:151
12670991911,6cyclictest2542-21cat11:05:131
12670991911,6cyclictest24148-21fw_packets11:20:151
12670991911,6cyclictest22692-21grep07:30:241
12670991911,6cyclictest11580-21sed09:40:191
12670991910,8cyclictest26393-21ssh11:45:061
12670991910,7cyclictest7267-21sh10:45:211
12670991910,6cyclictest9067-21cat11:55:191
12670991910,6cyclictest19935-21cat07:25:201
1266999199,7cyclictest13007-21tune2fs08:25:160
1266999197,10cyclictest653-21seq10:40:230
1266999193,8cyclictest141rcu_preempt12:08:380
1266999192,9cyclictest28765-21sh09:30:130
1266999191,8cyclictest26935-21ssh09:28:440
12669991914,3cyclictest8187-21acpi11:55:110
12669991914,3cyclictest19521-21sh11:16:350
12669991914,3cyclictest18114-21mailstats07:20:220
1266999191,2cyclictest141rcu_preempt08:10:000
12669991912,5cyclictest24041-21if_enp3s007:35:170
12669991912,5cyclictest19905-21users12:25:270
12669991911,6cyclictest671-21expr09:10:160
12669991911,6cyclictest560-21dbus-daemon12:24:590
12669991911,6cyclictest560-21dbus-daemon11:40:000
12669991911,6cyclictest31485-21ls11:25:170
12669991911,6cyclictest29364-21seq09:01:070
12669991911,6cyclictest16358-21awk07:15:250
12669991911,5cyclictest5309-21cat11:30:090
12669991910,7cyclictest25869-21expr08:55:160
12669991910,7cyclictest21617-21df07:30:130
1267299189,7cyclictest4050-21wc08:05:153
1267299187,6cyclictest53050irq/142-enp3s0-11:00:113
1267299187,2cyclictest34-21ksoftirqd/307:55:163
1267299187,2cyclictest34-21ksoftirqd/307:50:143
1267299183,3cyclictest141rcu_preempt08:05:003
12672991814,2cyclictest341ksoftirqd/309:05:003
1267299181,14cyclictest21058-21perf07:30:003
12672991810,6cyclictest25694-21ls08:55:153
12672991810,6cyclictest23402-21sed08:50:003
12672991810,6cyclictest17831-21sed07:20:193
12672991810,5cyclictest29154-21timerwakeupswit07:45:243
12672991810,5cyclictest22052-21cut07:30:173
1267199187,9cyclictest0-21swapper/211:46:592
1267199187,7cyclictest52950irq/141-enp3s0-11:00:122
1267199187,4cyclictest141rcu_preempt10:50:212
1267199185,9cyclictest27835-21apt-get07:45:122
1267199182,9cyclictest141rcu_preempt07:50:162
1267199182,11cyclictest19468-21apt-get11:40:122
1267199182,11cyclictest17018-21apt-get07:20:122
12671991813,3cyclictest8521-21sh10:45:592
12671991813,3cyclictest28845-21sh09:30:172
12671991813,3cyclictest25073-21/usr/sbin/munin09:50:132
12671991813,3cyclictest13120-21sh09:41:062
12671991812,4cyclictest21277-21acpi07:30:102
12671991811,5cyclictest8527-21expr11:55:132
12671991811,5cyclictest29100-21date09:00:242
12671991811,5cyclictest12927-21awk08:25:162
12671991811,5cyclictest11251-21date08:20:192
12671991810,6cyclictest7381-21entropy10:00:152
12671991810,6cyclictest557-21cat10:40:232
12671991810,6cyclictest4160-21cat08:05:152
12671991810,6cyclictest26469-21needreboot08:55:232
1267099189,7cyclictest27882-21ssh12:31:171
1267099189,7cyclictest27882-21ssh12:31:171
12670991814,2cyclictest444-21hddtemp_smartct12:35:161
12670991812,4cyclictest13430-21sh12:20:591
12670991811,5cyclictest11250-21date08:20:191
12670991810,6cyclictest682-21grep07:55:221
12670991810,5cyclictest278-21systemd-journal07:45:001
12670991810,5cyclictest13192-21latency11:35:171
1266999189,6cyclictest17890-21grep10:30:170
12669991813,3cyclictest4516-21timerandwakeup10:20:240
12669991813,3cyclictest23437-21sh11:20:030
12669991813,3cyclictest2341-21apt11:05:110
12669991811,5cyclictest15889-21basename12:00:160
12669991810,6cyclictest3708-21seq08:04:500
12669991810,6cyclictest27628-21date07:45:000
1267299178,6cyclictest53050irq/142-enp3s0-11:20:113
1267299178,6cyclictest278-21systemd-journal08:30:003
1267299172,11cyclictest23437-21apt-get07:35:123
1267299171,7cyclictest19821-21/usr/sbin/munin11:40:133
12672991712,3cyclictest10958-21cut08:20:163
12672991711,4cyclictest16597-21turbostat08:35:003
1267199177,8cyclictest0-21swapper/207:15:162
12671991710,5cyclictest23697-21expr08:50:142
1267099179,6cyclictest918-21cat09:10:181
1267099179,6cyclictest28251-21tune2fs09:00:151
1267099179,6cyclictest1977-21expr08:00:131
1267099172,11cyclictest28741-21apt-get09:30:131
12670991712,3cyclictest22739-21unixbench-2d08:45:261
1266999179,6cyclictest28397-21smartctl07:45:160
1266999178,7cyclictest624-21gdbus07:30:000
1266999178,6cyclictest6941-21cat08:10:190
1266999172,10cyclictest29933-21apt-get09:05:130
12669991711,4cyclictest25170-21sh12:30:100
12669991711,4cyclictest25170-21sh12:30:100
12669991710,5cyclictest18208-21cat08:35:250
12669991710,5cyclictest18208-21cat08:35:250
1267299168,6cyclictest26482-21sed07:40:193
12672991611,3cyclictest10745-21munin-node08:35:123
12672991611,3cyclictest10745-21munin-node08:35:123
1267299161,11cyclictest6127-21apt-get08:10:133
1267199169,5cyclictest25894-21ls07:40:132
1267099169,5cyclictest25758-21expr08:55:151
1267099161,6cyclictest5296-21/usr/sbin/munin11:30:171
1266999169,5cyclictest9929-21seq08:17:180
12669991610,4cyclictest11017-21idleruntime08:20:170
1267299158,4cyclictest0-21swapper/308:40:263
12671991510,3cyclictest16055-21timerwakeupswit08:30:262
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional