You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-14 - 10:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri Aug 14, 2026 00:45:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
302852144108,25sleep20-21swapper/219:07:052
30212214296,39sleep30-21swapper/319:06:113
286082141105,25sleep10-21swapper/119:05:071
302432138102,25sleep00-21swapper/019:06:320
240542910,9sleep334-21ksoftirqd/322:01:293
240542910,9sleep334-21ksoftirqd/322:01:283
13212840,1chrt0-21swapper/319:15:203
227802720,3chrt0-21swapper/121:36:431
125742680,7sleep3341ksoftirqd/322:17:143
287732540,1sleep10-21swapper/123:40:591
92682530,1sleep00-21swapper/023:26:500
30782995111,25cyclictest23946-21seq21:14:100
211802500,1sleep10-21swapper/100:00:141
89622480,2sleep012-21ksoftirqd/021:50:260
30782994713,20cyclictest28266-21sh22:53:170
183362450,1sleep20-21swapper/222:21:142
172392430,1sleep00-21swapper/021:05:240
52292410,2sleep30-21swapper/319:25:153
81472360,2chrt0-21swapper/323:50:163
30782993612,21cyclictest27304-21date00:05:000
74822350,1sleep30-21swapper/322:13:523
3078299350,7cyclictest29868-21rm00:05:580
30782993411,20cyclictest31426-21expr20:25:130
30782993411,20cyclictest31426-21expr20:25:130
30782993411,20cyclictest30204-21needreboot22:30:200
30782993411,20cyclictest13464-21seq20:56:330
14352340,2chrt0-21swapper/222:33:182
3078299337,6cyclictest12705-21sed21:30:000
3078299331,9cyclictest3053-21grep00:10:220
30782993310,20cyclictest16933-21sensors19:50:220
3078299329,5cyclictest2415-21seq19:19:510
30782993214,16cyclictest636-21runrttasks23:32:370
30782993214,16cyclictest636-21runrttasks23:32:360
30782993212,17cyclictest27157-21sh21:15:380
30782993212,17cyclictest12026-21sh00:17:080
30782993212,17cyclictest12026-21sh00:17:070
30782993211,18cyclictest7854-21sh22:38:000
3078299319,7cyclictest28738-21interrupts22:05:180
3078299318,5cyclictest15947-21grep21:55:270
3078299317,5cyclictest17871-21wc22:45:200
3078299317,5cyclictest17871-21wc22:45:190
30782993110,18cyclictest3850-21sh21:46:420
3078299307,5cyclictest6975-21basename19:30:110
3078299307,5cyclictest20549-21expr20:00:150
3078299302,7cyclictest18308-21apt-get23:10:120
3078299301,7cyclictest21796-21ssh21:35:430
3078299301,5cyclictest15146-21rm22:43:330
30782993013,15cyclictest28824-21latency_hist22:30:010
3078299301,12cyclictest3521-21fschecks_time00:35:160
30782993011,17cyclictest27149-21sh00:28:500
3078299301,10cyclictest29093-21df00:30:140
3078299301,10cyclictest23754-21chrt23:37:400
30782993010,17cyclictest30936-21sh21:43:080
238792300,12sleep228-21ksoftirqd/222:25:242
3078299299,3cyclictest32108-21cpuspeed_turbos21:20:140
3078299296,5cyclictest30684-21cat22:55:120
3078299292,9cyclictest0-21swapper/023:15:260
3078299291,9cyclictest6732-21cron19:30:000
3078299291,9cyclictest15745-21sed23:55:240
3078299291,9cyclictest15556-21wc19:50:010
3078299291,10cyclictest15891-21ssh00:20:150
30782992910,3cyclictest2886-21cstates19:20:130
30782992910,17cyclictest10091-21pmu-power19:35:220
30782992910,17cyclictest10091-21pmu-power19:35:210
30782992910,16cyclictest6281-21unixbench-2d23:00:240
30782992910,16cyclictest14510-21timerandwakeup21:30:270
303722290,4sleep334-21ksoftirqd/322:55:103
3078299289,3cyclictest19257-21timerwakeupswit19:55:240
3078299287,6cyclictest32512-21seq19:12:360
3078299287,3cyclictest3923-21unixbench-2d22:10:250
3078299284,6cyclictest278-21systemd-journal20:40:000
3078299281,8cyclictest27834-21python323:40:240
3078299281,8cyclictest21190-21ssh22:00:080
3078299281,8cyclictest21190-21ssh22:00:070
3078299281,7cyclictest9659-21/usr/sbin/munin22:15:160
3078299281,7cyclictest9090-21/usr/sbin/munin20:45:270
30782992812,14cyclictest18052-21diskmemload23:54:270
30782992812,14cyclictest17465-21unixbench_singl22:20:250
3078299276,3cyclictest25109-21tail20:10:160
3078299271,9cyclictest14945-21rm23:07:130
3078299271,8cyclictest7064-21apt23:50:010
3078299271,8cyclictest5170-21sshd23:24:250
3078299271,7cyclictest12577-21grep19:40:210
126502270,3chrt12608-21ssh23:05:223
3078399261,24cyclictest0-21swapper/119:11:291
3078299267,4cyclictest11229-21vmstat20:50:240
3078299267,3cyclictest24302-21sh20:09:590
3078299264,5cyclictest26920-21cpu20:15:130
3078299261,8cyclictest2298-21mailstats20:30:230
3078299261,7cyclictest29149-21tr20:20:120
296692260,1chrt0-21swapper/323:18:143
28532260,1chrt0-21swapper/121:21:581
3078299250,7cyclictest6526-21gzip20:40:220
3078299250,7cyclictest14104-21cut21:00:100
197732250,9sleep334-21ksoftirqd/321:10:253
95682240,9sleep122-21ksoftirqd/122:15:151
30785992422,1cyclictest0-21swapper/319:11:253
30784992422,1cyclictest0-21swapper/219:11:062
24412240,3sleep12443-21idleruntime-cro19:19:591
23322240,2chrt0-21swapper/323:21:343
3078599239,12cyclictest2041-21sh22:10:013
30785992317,4cyclictest34-21ksoftirqd/300:15:003
30785992312,7cyclictest23658-21grep00:25:263
30785992312,7cyclictest22506-21apt-get22:25:123
3078499230,20cyclictest7944-21uniq20:45:162
3272220,2sleep30-21swapper/322:32:153
3078599221,6cyclictest0-21swapper/323:33:223
3078599221,6cyclictest0-21swapper/323:33:223
30785992214,6cyclictest341ksoftirqd/319:40:013
30785992214,6cyclictest341ksoftirqd/319:40:003
30785992214,6cyclictest10449-21ssh21:51:313
3078499229,11cyclictest9322-21unixbench_multi21:50:292
3078499225,7cyclictest28-21ksoftirqd/221:20:002
3078499225,10cyclictest1037-21cstates21:45:152
30784992213,7cyclictest12412-21smartctl20:55:152
30784992213,6cyclictest14145-21apt-get23:55:102
3078399221,4cyclictest0-21swapper/122:01:381
3078399221,4cyclictest0-21swapper/122:01:371
30783992213,7cyclictest393-21netstat23:20:201
30783992213,6cyclictest18241-21cut22:45:241
30783992213,6cyclictest18241-21cut22:45:231
30783992212,7cyclictest8741-21sensors_temp20:45:251
30783992211,7cyclictest3008-21open_inodes00:10:211
44492210,1sleep20-21swapper/223:47:232
3078599211,18cyclictest5280-21ssh22:35:263
3078599210,2cyclictest141rcu_preempt21:19:593
3078499215,4cyclictest66050lldpd21:29:182
30784992114,4cyclictest18052-21diskmemload23:35:242
30784992113,6cyclictest30950-21cron20:25:002
30784992113,6cyclictest28212-21users23:40:262
30784992113,6cyclictest12359-21netstat19:40:192
30784992113,6cyclictest11154-21ls22:40:192
3078499211,18cyclictest19145-21ssh21:10:172
30784992111,7cyclictest5953-21grep20:40:162
3078399212,13cyclictest9137-21taskset19:35:121
3078399212,13cyclictest9137-21taskset19:35:121
30783992113,6cyclictest21239-21pluginstate20:00:231
30783992112,6cyclictest6012-21python319:25:231
284152210,1sleep10-21swapper/121:40:351
151802210,1sleep30-21swapper/300:20:023
100502210,3sleep3331rcuc/323:27:373
3078599203,12cyclictest640-21snmpd23:35:123
3078599201,8cyclictest7092-21latency_hist23:50:013
30785992014,4cyclictest34-21ksoftirqd/321:10:003
30785992013,5cyclictest52750irq/139-enp3s0-22:45:133
30785992013,5cyclictest52750irq/139-enp3s0-22:45:123
30785992013,5cyclictest278-21systemd-journal22:00:003
30785992012,5cyclictest26045-21grep20:10:263
30785992011,6cyclictest26686-21grep21:40:133
3078499208,3cyclictest110072ssh23:28:352
3078499206,5cyclictest28-21ksoftirqd/223:33:222
3078499206,5cyclictest28-21ksoftirqd/223:33:222
3078499202,13cyclictest640-21snmpd21:30:132
30784992015,3cyclictest24313-21dump-pmu-power20:10:002
30784992014,4cyclictest793-21python323:20:242
30784992014,4cyclictest28-21ksoftirqd/223:19:592
30784992013,4cyclictest26607-21sh22:04:032
30784992013,4cyclictest26607-21sh22:04:022
30784992012,6cyclictest3725-21cat00:35:182
30784992012,6cyclictest278-21systemd-journal21:45:002
30784992012,6cyclictest21268-21diskstats00:00:142
30784992012,6cyclictest17156-21ls19:50:242
30784992012,6cyclictest14525-21ntp_kernel_err19:45:202
30784992011,7cyclictest433-21cat21:20:192
30784992011,7cyclictest2096-21sed22:10:002
30784992011,6cyclictest24931-21tune2fs20:10:152
30784992010,6cyclictest52950irq/141-enp3s0-00:25:142
3078399203,12cyclictest18052-21diskmemload23:30:121
3078399203,12cyclictest18052-21diskmemload23:30:111
3078399202,14cyclictest18523-21ssh21:10:141
30783992016,2cyclictest18052-21diskmemload22:53:101
30783992014,4cyclictest4195-21hwlatdetect22:35:171
30783992013,4cyclictest53050irq/142-enp3s0-21:45:141
30783992012,6cyclictest16085-21seq23:08:241
30783992012,6cyclictest15677-21seq23:55:221
30783992011,6cyclictest53050irq/142-enp3s0-23:10:111
30783992011,6cyclictest31954-21timerwakeupswit22:55:251
3078399201,16cyclictest22-21ksoftirqd/122:40:121
30783992011,6cyclictest17910-21cat19:55:101
30783992011,6cyclictest12628-21seq21:53:441
30783992010,7cyclictest12187-21df_inode20:55:131
185522200,4sleep218052-21diskmemload22:45:262
185522200,4sleep218052-21diskmemload22:45:252
175402200,2sleep30-21swapper/323:57:073
3078599198,7cyclictest560-21dbus-daemon20:45:003
3078599198,7cyclictest52750irq/139-enp3s0-23:40:113
3078599197,9cyclictest560-21dbus-daemon20:55:003
3078599193,11cyclictest28749-21ssh00:30:133
3078599192,13cyclictest22419-21apt-get20:05:083
3078599192,13cyclictest20898-21ssh00:00:133
3078599192,12cyclictest6994-21apt-get19:30:133
30785991912,5cyclictest31964-21load20:25:183
30785991912,5cyclictest31964-21load20:25:173
3078599191,15cyclictest52750irq/139-enp3s0-21:20:263
30785991911,5cyclictest13363-21cat21:30:153
3078599191,12cyclictest15802-21apt-get22:20:133
30785991910,7cyclictest29440-21tune2fs20:20:143
30785991910,7cyclictest28553-21sed00:05:223
30785991910,6cyclictest7949-21grep21:25:273
30785991910,6cyclictest18727-21grep23:10:143
30785991910,6cyclictest11260-21grep22:40:203
3078499196,9cyclictest52950irq/141-enp3s0-21:37:332
3078499192,12cyclictest26907-21apt-get20:15:132
3078499191,8cyclictest18052-21diskmemload21:57:042
3078499191,5cyclictest18052-21diskmemload22:56:552
30784991913,3cyclictest8012-21sh22:38:102
3078499191,2cyclictest0-21swapper/223:01:502
30784991912,5cyclictest561-21seq20:28:572
30784991912,5cyclictest561-21seq20:28:562
30784991912,5cyclictest19072-21sensors19:55:222
30784991912,5cyclictest10290-21python319:35:232
30784991912,5cyclictest10290-21python319:35:222
30784991912,4cyclictest8799-21sh23:50:222
30784991912,2cyclictest22612sleep200:34:432
30784991911,6cyclictest9560-21cat00:15:202
30784991911,6cyclictest9560-21cat00:15:202
3078499191,15cyclictest18052-21diskmemload22:10:222
30784991910,7cyclictest8935-21cat19:35:002
30784991910,7cyclictest560-21dbus-daemon20:30:002
30784991910,6cyclictest14297-21expr21:00:112
3078399199,7cyclictest10287-21sed20:50:161
3078399192,12cyclictest19608-21ssh23:35:041
30783991913,4cyclictest14330-21sendmail_mailtr21:30:241
30783991912,5cyclictest636-21runrttasks22:13:291
30783991912,5cyclictest53050irq/142-enp3s0-23:15:131
30783991912,5cyclictest4115-21awk00:35:231
30783991912,5cyclictest3250-21seq20:35:031
30783991912,5cyclictest22959-21cat00:25:211
30783991912,5cyclictest17362-21seq21:56:441
30783991911,6cyclictest7644-21cat23:25:231
30783991911,6cyclictest30590-21wc22:30:241
30783991911,6cyclictest22504-21apt22:25:101
30783991911,6cyclictest17634-21seq00:20:591
30783991911,6cyclictest17441-21awk22:20:261
30783991911,5cyclictest1440-21idleruntime23:45:161
3078399191,10cyclictest53050irq/142-enp3s0-21:25:141
3078599189,6cyclictest3232-21cut20:35:013
3078599188,7cyclictest560-21dbus-daemon20:05:013
3078599182,3cyclictest141rcu_preempt23:00:133
30785991811,5cyclictest13894-21sed21:00:003
30785991810,6cyclictest23903-21ls22:50:143
30785991810,6cyclictest13995-21awk19:45:153
30785991810,5cyclictest14529-21sed21:00:133
3078499187,9cyclictest15953-21gltestperf00:20:162
3078499187,7cyclictest52950irq/141-enp3s0-00:10:102
3078499182,11cyclictest511-21apt-get19:15:142
3078499182,11cyclictest16158-21apt-get21:05:142
3078499181,7cyclictest18052-21diskmemload23:06:012
30784991813,3cyclictest8840-21sh22:15:022
30784991813,3cyclictest27530-21sh00:05:082
30784991812,4cyclictest28-21ksoftirqd/220:00:002
30784991811,5cyclictest3919-21cat20:35:172
30784991811,5cyclictest18601-21cat23:10:132
30784991811,5cyclictest10957-21ls20:50:232
3078399189,7cyclictest15031-21which21:00:201
3078399188,7cyclictest11567-21cat19:40:131
3078399181,6cyclictest30068-21python300:30:231
30783991812,4cyclictest7928-21sh23:01:551
30783991812,4cyclictest29319-21sh21:17:461
30783991812,4cyclictest13529-21sh00:18:371
30783991812,4cyclictest13529-21sh00:18:361
30783991811,5cyclictest8309-21sed23:50:181
30783991811,5cyclictest29731-21sh22:05:271
30783991810,6cyclictest7387-21date20:45:001
30783991810,6cyclictest32314-21run-parts20:25:221
30783991810,6cyclictest32314-21run-parts20:25:211
30783991810,6cyclictest29803-21sed20:20:171
30783991810,6cyclictest28077-21grep00:05:181
3078599178,6cyclictest1312-21cut21:45:183
3078599177,7cyclictest560-21dbus-daemon20:15:013
3078599173,8cyclictest7587-21apt-get20:45:143
3078599172,11cyclictest9953-21smartctl00:15:243
3078599172,11cyclictest9953-21smartctl00:15:233
30785991712,3cyclictest17892-21/usr/sbin/munin19:55:103
30785991711,4cyclictest34-21ksoftirqd/319:40:003
30785991710,5cyclictest20806-21ntpq21:35:223
30785991710,5cyclictest17263-21awk19:50:263
3078499177,8cyclictest0-21swapper/222:51:062
3078499171,11cyclictest2716-21apt-get19:20:112
3078399179,6cyclictest15332-21seq19:47:391
3078399177,7cyclictest278-21systemd-journal21:09:591
30783991712,3cyclictest1699-21hddtemp_smartct20:30:161
3078399171,14cyclictest8882-21munin-plugin-st19:35:001
30783991710,5cyclictest22693-21expr20:05:111
3078599168,6cyclictest4089-21ntpq00:35:233
3078599161,13cyclictest2691-21/usr/sbin/munin19:20:153
30785991610,4cyclictest34-21ksoftirqd/320:40:013
30784991610,4cyclictest5419-21hostname19:25:172
3078399169,5cyclictest25284-21sed20:10:171
3078399168,6cyclictest3210-21expr19:20:161
3078399159,4cyclictest28309-21unixbench-2d20:15:271
30783991510,3cyclictest15753-21/usr/sbin/munin19:50:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional