You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-20 - 23:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed May 20, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
46792222198,16sleep10-21swapper/107:07:581
46932181102,27sleep00-21swapper/007:08:100
44942175102,29sleep20-21swapper/207:05:352
1884221650,2sleep118844-21grep07:40:151
1964121630,2sleep3331rcuc/309:30:163
45102161109,28sleep30-21swapper/307:05:473
289842610,2sleep20-21swapper/208:05:012
33562600,1sleep00-21swapper/011:43:240
130692600,4sleep10-21swapper/112:15:091
299242480,2sleep30-21swapper/310:25:463
181542470,1sleep20-21swapper/211:05:312
257412440,1sleep00-21swapper/009:35:070
511099431,28cyclictest32362-21rm09:40:041
275992430,2sleep30-21swapper/311:36:543
511099418,18cyclictest0-21swapper/112:35:161
511099400,4cyclictest3057-21rm09:17:501
511099400,25cyclictest27398-21ssh10:49:021
104972390,1sleep30-21swapper/308:35:183
511099380,6cyclictest11150-21sh12:13:231
510999388,10cyclictest19336-21ssh11:30:240
510999378,7cyclictest15871-21fschecks_count10:40:140
510999378,7cyclictest15871-21fschecks_count10:40:130
510999378,11cyclictest2862-21sh10:05:310
230942370,3chrt0-21swapper/109:05:191
230942370,3chrt0-21swapper/109:05:181
511099362,2cyclictest0-21swapper/110:15:101
511099360,6cyclictest8466-21ssh09:45:251
511099360,3cyclictest27481-21/usr/sbin/munin09:35:251
510999368,12cyclictest7920-21awk09:45:220
510999367,12cyclictest29379-21/usr/sbin/munin08:05:130
510999364,19cyclictest0-21swapper/011:55:110
5109993610,9cyclictest5388-21ssh09:44:020
5109993610,11cyclictest22241-21rm11:09:400
510999357,12cyclictest16744-21cat08:50:180
5109993510,8cyclictest4956-21rm10:55:390
5109993510,8cyclictest16414-21taskset09:51:410
5109993510,8cyclictest15202-21rm11:51:410
5109993510,16cyclictest846-21ssh11:17:050
5109993510,15cyclictest6854-21rm12:34:530
511099349,22cyclictest21560-21grep09:55:231
511099348,22cyclictest20194-21cat12:20:131
511099340,5cyclictest32596-21ssh10:28:301
511099340,4cyclictest9277-21grep10:35:141
510999349,8cyclictest24263-21diskmemload11:20:300
510999349,15cyclictest26416-21ssh10:48:020
510999347,11cyclictest18153-21sh10:17:230
5109993410,8cyclictest24263-21diskmemload11:47:390
5109993410,8cyclictest24263-21diskmemload11:28:410
5109993410,8cyclictest24263-21diskmemload11:28:400
5109993410,13cyclictest2002-21ssh10:30:080
324772340,2chrt0-21swapper/209:40:102
511099338,21cyclictest1855-21tr12:30:201
5110993311,20cyclictest11866-21expr08:40:131
511099330,8cyclictest24109-21/usr/sbin/munin11:10:221
511099330,5cyclictest546-21cut12:05:181
511099330,5cyclictest12322-21ssh10:13:221
511099330,4cyclictest20687-21ssh11:55:291
510999339,7cyclictest24263-21diskmemload12:26:280
510999339,15cyclictest28810-21ssh12:01:590
510999337,8cyclictest24263-21diskmemload09:33:140
510999337,10cyclictest24263-21diskmemload09:23:160
510999337,10cyclictest1253-21ssh09:15:590
5109993310,8cyclictest1163-21ssh12:05:220
290592330,2chrt0-21swapper/209:13:062
110042330,3chrt0-21swapper/210:12:002
511099326,21cyclictest560-21dbus-daemon08:50:001
511099323,23cyclictest0-21swapper/111:25:121
511099323,23cyclictest0-21swapper/111:25:111
511099322,26cyclictest31116-21ssh11:40:121
511099321,6cyclictest24910-21fschecks_time11:35:151
5110993212,17cyclictest3741-21sh11:19:591
511099320,7cyclictest5982-21ssh11:20:301
511099320,6cyclictest28521-21sh12:25:461
511099320,6cyclictest24263-21diskmemload10:32:061
511099320,3cyclictest12673-21grep11:50:141
510999329,13cyclictest9913-21sh12:12:080
510999327,9cyclictest25706-21rm10:23:160
510999326,8cyclictest29814-21ssh10:01:430
510999326,6cyclictest30479-21users10:50:250
510999324,14cyclictest0-21swapper/011:36:160
5109993212,11cyclictest14992-21ssh12:15:320
5109993210,9cyclictest23272-21sh12:22:120
5109993210,7cyclictest27720-21ssh11:13:260
511099319,19cyclictest19279-21switchtime08:55:231
511099313,22cyclictest15936-21apt-get08:50:121
5110993111,17cyclictest636-21runrttasks08:25:191
5110993111,17cyclictest24263-21diskmemload11:46:561
511099310,7cyclictest4683-21ssh10:55:251
511099310,3cyclictest4680-21ssh10:07:211
511099310,3cyclictest27369-21sh10:24:581
510999319,8cyclictest11988-21ssh11:00:590
510999317,10cyclictest7750-21cpu10:10:110
510999316,9cyclictest29185-21awk10:25:220
510999316,15cyclictest0-21swapper/009:56:580
26062310,3sleep20-21swapper/210:30:142
511099301,4cyclictest28299-21sed08:00:241
5110993010,17cyclictest13333-21sh09:50:071
511099300,5cyclictest31707-21rm12:04:531
511099300,4cyclictest22115-21ssh11:33:061
510999307,7cyclictest14218-21tune2fs08:45:140
510999307,13cyclictest24263-21diskmemload09:25:590
510999307,11cyclictest8974-21seq07:17:310
510999305,13cyclictest0-21swapper/012:39:470
18102300,2sleep20-21swapper/211:41:502
511099299,17cyclictest25577-21missed_timers09:10:171
511099298,18cyclictest0-21swapper/107:35:001
511099291,7cyclictest16409-21needreboot10:40:191
511099291,7cyclictest16409-21needreboot10:40:181
511099291,6cyclictest6132-21sed07:10:241
5110992912,14cyclictest20188-21sh11:07:331
511099290,5cyclictest24263-21diskmemload10:54:541
511099290,4cyclictest7460-21ssh09:20:291
511099290,3cyclictest28049-21mailstats10:00:221
511099290,3cyclictest24705-21ssh09:34:301
510999299,6cyclictest10893-21ssh10:35:500
510999298,7cyclictest1824-21ntp_offset08:15:200
510999298,7cyclictest1824-21ntp_offset08:15:200
510999297,13cyclictest19622-21taskset08:55:530
510999297,10cyclictest11040-21seq08:35:430
510999296,11cyclictest24432-21seq07:54:190
5109992911,5cyclictest636-21runrttasks07:30:230
511099286,19cyclictest8219-21netstat08:30:191
511099286,19cyclictest32047-21needreboot08:10:211
511099281,7cyclictest9738-21ssh11:00:141
511099281,6cyclictest4456-21grep08:20:241
5110992810,15cyclictest26635-21munin-plugin-st08:00:001
511099280,3cyclictest16291-21ssh09:27:421
510999287,9cyclictest6890-21taskset07:13:160
510999286,9cyclictest26384-21seq07:56:550
510999286,10cyclictest6961-21seq08:28:170
510999286,10cyclictest13223-21seq07:26:540
510999285,14cyclictest26602-21ssh09:10:370
511099274,6cyclictest21063-21needreboot09:00:211
511099274,5cyclictest8423-21cat07:15:231
511099271,8cyclictest22755-21cpuspeed_turbos07:50:131
511099271,7cyclictest30482-21unixbench_multi08:05:251
511099271,7cyclictest14975-21awk07:30:251
511099271,7cyclictest11715-21tr07:25:131
511099271,6cyclictest9602-21cpuspeed_turbos08:35:111
510999276,9cyclictest9459-21seq07:20:120
510999275,10cyclictest27829-21ntp_states08:00:200
38242270,11chrt28-21ksoftirqd/208:20:162
511199263,16cyclictest6512-21ssh12:10:122
511099261,7cyclictest20858-21fschecks_count07:45:151
511099261,6cyclictest1082-21wc08:15:141
511099261,6cyclictest1082-21wc08:15:131
511099260,6cyclictest9439-21find07:20:111
510999267,6cyclictest4718-21seq08:22:040
260442260,9sleep228-21ksoftirqd/211:11:442
249522260,2sleep3331rcuc/307:55:143
511199253,16cyclictest26812-21ssh10:00:112
511199251,10cyclictest20681-21unixbench_singl09:30:262
510999257,6cyclictest9217-21seq08:34:480
510999256,6cyclictest20442-21seq07:45:030
511299249,10cyclictest9718-21apt-get11:00:153
511199243,15cyclictest13948-21apt-get10:15:042
5111992413,8cyclictest26347-21cron10:00:002
5111992412,8cyclictest15023-21sendmail10:40:002
5111992314,6cyclictest0-21swapper/209:35:032
510999239,12cyclictest31268-21/usr/sbin/munin08:10:140
5109992314,7cyclictest11760-21cut08:40:140
5109992313,8cyclictest24255-21/usr/sbin/munin09:10:010
5109992313,8cyclictest24255-21/usr/sbin/munin09:10:000
5109992313,7cyclictest18639-21expr07:40:140
5109992313,7cyclictest16967-21/usr/sbin/munin07:35:180
267562230,2sleep30-21swapper/310:00:113
173542230,2sleep20-21swapper/209:52:382
511299227,13cyclictest23637-21seq11:34:373
511299222,15cyclictest24665-21apt-get09:10:103
511299222,14cyclictest12235-21ssh11:50:113
5112992215,4cyclictest663-21/usr/sbin/munin12:30:223
5112992215,4cyclictest657-21sshd09:47:113
5112992213,7cyclictest7540-21cat08:30:133
511299221,19cyclictest29033-21ssh12:26:173
511199221,10cyclictest22123-21apt09:04:592
5111992210,10cyclictest28-21ksoftirqd/207:50:012
25592220,1sleep30-21swapper/311:42:363
141232220,2sleep30-21swapper/312:15:193
116022220,3sleep334-21ksoftirqd/309:24:423
511299217,4cyclictest141rcu_preempt12:35:133
511299211,8cyclictest0-21swapper/311:27:063
511299211,8cyclictest0-21swapper/311:27:053
5112992113,6cyclictest18054-21cron07:40:003
511299211,11cyclictest5489-21cut11:45:113
511199213,2cyclictest120032ssh11:01:002
511199213,13cyclictest26494-21chrt12:25:122
5111992115,4cyclictest28-21ksoftirqd/208:30:012
5111992113,6cyclictest27594-21sed10:25:002
5111992111,7cyclictest560-21dbus-daemon07:55:002
511199211,16cyclictest11617-21apt-get11:50:032
511199210,1cyclictest24263-21diskmemload12:05:402
5109992112,7cyclictest20950-21cat09:00:190
312222210,1sleep20-21swapper/210:27:062
511299201,7cyclictest22121-21idleruntime-cro09:04:593
5112992014,4cyclictest12328-21sh10:13:223
5112992013,5cyclictest15400-21ls10:15:173
5112992013,5cyclictest12620-21cut08:40:213
5112992012,6cyclictest5952-21cat07:10:213
5112992012,6cyclictest2992-21cat10:30:193
5112992012,6cyclictest29618-21smartctl08:05:163
5112992012,6cyclictest14993-21cat08:45:243
5112992011,7cyclictest5645-21tune2fs08:25:163
5112992011,6cyclictest24263-21diskmemload09:36:173
511199209,8cyclictest560-21dbus-daemon10:45:002
511199209,8cyclictest560-21dbus-daemon10:44:592
511199201,7cyclictest23026-21wc10:45:192
5111992013,4cyclictest636-21runrttasks10:57:002
5111992012,6cyclictest2164-21awk10:05:202
5111992012,6cyclictest14736-21open_inodes08:45:202
5111992012,5cyclictest24263-21diskmemload12:15:122
5111992011,6cyclictest1289-21sed12:30:152
511199201,14cyclictest75-21kswapd010:50:102
511299194,12cyclictest27642-21cat08:00:173
511299192,12cyclictest12305-21apt-get09:25:113
511299191,6cyclictest23640-21ssh12:22:333
5112991913,4cyclictest30303-21timerwakeupswit10:50:243
5112991913,4cyclictest26646-21sh11:12:213
5112991913,4cyclictest21994-21vmstat07:45:263
5112991913,4cyclictest2010-21sh12:05:523
5112991912,5cyclictest278-21systemd-journal07:35:013
5112991912,4cyclictest8403-21sh10:59:103
5112991912,4cyclictest4860-21sh09:43:303
5112991912,4cyclictest2478-21timerandwakeup10:05:223
5112991911,6cyclictest22669-21ls07:50:143
5112991911,6cyclictest20052-21cut09:00:013
5112991911,6cyclictest14870-21expr09:50:243
5112991911,6cyclictest10957-21users07:20:263
5112991911,6cyclictest0-21swapper/310:45:113
5112991910,7cyclictest18984-21cat11:55:123
511199192,3cyclictest141rcu_preempt07:55:132
511199192,12cyclictest7419-21apt-get12:35:122
511199192,12cyclictest13732-21apt-get07:30:122
511199192,11cyclictest640-21snmpd09:25:102
511199191,5cyclictest0-21swapper/209:47:482
5111991912,5cyclictest5986-21expr09:20:122
5111991912,5cyclictest5603-21cut07:10:172
5111991912,5cyclictest28-21ksoftirqd/211:50:012
5111991912,5cyclictest13691-21ls08:45:012
5111991912,5cyclictest13075-21awk07:25:262
5111991912,4cyclictest24263-21diskmemload12:20:182
5111991912,4cyclictest17821-21munin-run08:55:002
5111991911,6cyclictest25546-21ls11:35:222
511199191,11cyclictest31931-21gltestperf09:15:142
234212190,2chrt0-21swapper/309:57:003
511299189,6cyclictest8614-21timerwakeupswit07:15:253
511299189,6cyclictest278-21systemd-journal09:10:003
511299189,6cyclictest278-21systemd-journal09:09:593
511299189,6cyclictest15912-21acpi08:50:113
511299188,6cyclictest52750irq/139-enp3s0-10:35:103
511299187,9cyclictest0-21swapper/312:00:183
511299187,7cyclictest52750irq/139-enp3s0-11:20:133
511299182,11cyclictest75-21kswapd010:40:103
511299182,11cyclictest75-21kswapd010:40:093
511299182,11cyclictest6494-21apt-get12:10:133
5112991811,5cyclictest32481-21sed08:10:263
5112991811,5cyclictest3176-21expr08:20:133
5112991811,5cyclictest11828-21expr07:25:133
5112991811,4cyclictest16382-21sh11:05:123
511299181,11cyclictest21209-21taskset10:20:133
5112991810,6cyclictest31547-21sed11:15:243
511199183,10cyclictest17759-21apt-get11:30:112
511199182,11cyclictest908-21apt-get08:15:132
511199182,11cyclictest908-21apt-get08:15:122
511199182,11cyclictest22368-21apt-get09:05:132
511199182,11cyclictest22368-21apt-get09:05:122
5111991814,2cyclictest173172sleep207:35:242
5111991813,3cyclictest271rcuc/211:21:062
5111991813,3cyclictest19170-21needreboot07:40:182
5111991810,6cyclictest7380-21grep07:15:142
5112991712,3cyclictest1483-21hwlatdetect08:15:173
5112991712,3cyclictest1483-21hwlatdetect08:15:163
511299171,14cyclictest31640-21cstates09:15:123
5112991710,5cyclictest18601-21expr07:40:143
511199179,6cyclictest31385-21sed11:15:242
511199179,5cyclictest9391-21grep08:35:012
511199171,4cyclictest0-21swapper/212:01:192
5111991711,4cyclictest14069-21sh11:26:412
5111991711,4cyclictest14069-21sh11:26:412
5111991711,4cyclictest10233-21sed08:35:152
5111991711,3cyclictest24263-21diskmemload11:56:542
511199169,5cyclictest20116-21wc09:00:002
5111991611,3cyclictest32322-21mailstats08:10:252
511199159,4cyclictest10167-21latency07:20:172
511199158,5cyclictest18950irq/143-ahci[0008:05:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional