You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-14 - 05:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Jul 14, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
329640799961,12cyclictest150-21pr/ttyS223:00:020
329640799961,12cyclictest150-21pr/ttyS223:00:010
329640799884,38cyclictest0-21swapper/000:33:470
329641899718,16cyclictest309-21in:imuxsock20:25:002
3296415997061,7cyclictest23-21ksoftirqd/120:10:011
329641899697,22cyclictest3323975-21modprobe21:55:002
329642299682,61cyclictest3297905-21munin-run19:40:013
3296418996824,38cyclictest275-21dbus-daemon20:55:002
3296418996824,38cyclictest275-21dbus-daemon20:55:002
3296407996813,39cyclictest3374158-21ssh23:40:000
3296407996813,39cyclictest3374158-21ssh23:40:000
3296407996730,32cyclictest16850irq/134-800000022:51:370
3296418996629,34cyclictest30-21ksoftirqd/200:25:322
3296422996527,34cyclictest3375990-21sh23:43:373
329641899654,25cyclictest3355113-21sh23:00:012
329641899654,25cyclictest3355113-21sh23:00:002
329641899654,15cyclictest3467602-21systemd-journal22:10:292
3296418996519,41cyclictest309-21in:imuxsock21:30:012
3296418996510,49cyclictest0-21swapper/221:35:012
3296418996510,49cyclictest0-21swapper/221:35:002
3296407996514,47cyclictest3314759-21ssh21:35:170
3296422996424,34cyclictest275-21dbus-daemon20:30:013
329642299641,10cyclictest3340976-21cat22:30:033
329641899645,28cyclictest3321570-21grep21:50:002
329641899645,12cyclictest275-21dbus-daemon20:40:012
329641899643,15cyclictest3319179-21ssh21:45:002
3296407996423,10cyclictest3467602-21systemd-journal21:48:120
329640799641,45cyclictest3333508-21ssh22:15:000
329642299632,57cyclictest3381141-1kworker/u9:000:06:443
329642299631,8cyclictest3315031-21ssh21:36:013
329641899638,46cyclictest3467602-21systemd-journal21:19:592
329641899637,46cyclictest3296860-21modprobe19:20:012
329641899635,9cyclictest3364626-21modprobe23:20:002
329641899635,9cyclictest3364626-21modprobe23:19:592
329641899632,15cyclictest3347902-21cron22:45:002
3296418996320,37cyclictest3336002-21latency_hist22:20:022
3296418996320,37cyclictest3336002-21latency_hist22:20:012
3296407996325,10cyclictest3363647-21ssh23:17:510
3296407996325,10cyclictest3363647-21ssh23:17:500
329642299622,9cyclictest3467602-21systemd-journal21:51:213
329642299622,6cyclictest3336270-21ssh22:20:153
3296422996222,36cyclictest37-21ksoftirqd/321:43:133
329642299622,16cyclictest3298441-21munin-run19:50:023
329642299621,9cyclictest3304558-21ssh21:14:163
329642299621,8cyclictest3397603-21ssh00:28:483
329642299621,8cyclictest3367740-21ssh23:26:093
329642299621,8cyclictest3350916-21ssh22:51:043
329642299621,8cyclictest3308006-21ssh21:21:113
329642299621,7cyclictest3357859-21ssh23:05:243
329642299621,7cyclictest3310716-21ssh21:26:593
3296422996217,41cyclictest37-21ksoftirqd/322:47:153
329641899626,16cyclictest3369434-21ssh23:30:002
329641899626,12cyclictest3331319-21latency_hist22:10:022
329641899625,17cyclictest3299221-21unin-run20:05:012
329641899624,19cyclictest3298173-21latency_hist19:45:002
329641899623,22cyclictest3371813-21munin-run23:35:002
3296407996217,40cyclictest3392069-21ssh00:17:140
3296407996213,15cyclictest3302577-21diskmemload21:13:070
329642299613,17cyclictest3399572-21ssh00:32:533
329642299612,8cyclictest3320596-21ssh21:47:423
329642299612,6cyclictest3373956-21ssh23:39:343
329642299612,6cyclictest3373956-21ssh23:39:333
329642299612,20cyclictest3299999-21sendmail-msp20:20:003
329642299611,7cyclictest3385384-21ssh00:03:113
329642299611,7cyclictest3385384-21ssh00:03:103
329642299611,7cyclictest3380596-21ssh23:53:173
329642299611,7cyclictest3364949-21ssh23:20:093
329642299611,7cyclictest3354817-21ssh22:59:253
329642299611,7cyclictest3354817-21ssh22:59:243
329642299611,7cyclictest3344620-21ssh22:38:023
329642299611,7cyclictest3329949-21sshd22:07:143
329641899616,22cyclictest3467602-21systemd-journal00:25:002
329641899614,13cyclictest3383736-40mandb00:00:002
329641899614,13cyclictest3340755-21ssh22:30:012
3296418996126,31cyclictest3301255-21sshd20:41:132
3296418996123,33cyclictest3379262-21sh23:50:112
329641899612,17cyclictest3376571-21cron23:44:592
3296407996135,20cyclictest3296401-21cyclictest19:10:020
3296407996115,42cyclictest3387813-21ssh00:08:260
3296407996112,44cyclictest3348085-21latency_hist22:45:030
329642299602,7cyclictest3361492-21ssh23:13:193
329642299602,7cyclictest3327663-21ssh22:02:333
3296422996027,29cyclictest3300372-21cat20:25:013
3296422996023,33cyclictest3334236-21sshd22:16:143
3296422996023,33cyclictest3334236-21sshd22:16:143
329642299601,7cyclictest3402204-21ssh00:38:203
329642299601,7cyclictest3391538-21ssh00:16:003
329642299601,7cyclictest3346375-21ssh22:41:343
329642299601,7cyclictest3324487-21ssh21:55:483
329642299601,6cyclictest3369667-21ssh23:30:053
329642299601,6cyclictest3341635-21ssh22:31:313
329642299601,5cyclictest3394385-21ssh00:22:033
329642299601,5cyclictest3381773-21ssh23:55:353
329641899608,18cyclictest3390904-21modprobe00:15:012
329641899606,7cyclictest3326285-21modprobe21:59:592
329641899606,14cyclictest3298430-21munin-plugin-st19:50:002
329641899604,14cyclictest3400702-21latency_hist00:35:022
329641899603,17cyclictest3388497-21unin-run00:10:002
329641899603,15cyclictest3386132-21latency_hist00:05:032
329641899603,15cyclictest3386132-21latency_hist00:05:032
3296418996019,35cyclictest0-21swapper/222:40:012
329640799602,42cyclictest3343793-21ssh22:36:030
3296407996022,30cyclictest16850irq/134-800000021:51:350
3296407996018,38cyclictest3371791-21latency_hist23:35:010
3296407996016,40cyclictest16850irq/134-800000022:03:440
329642299594,48cyclictest3314582-21ssh21:35:033
329642299594,48cyclictest3314582-21ssh21:35:023
329642299592,5cyclictest3332575-21ssh22:12:513
329642299591,7cyclictest3377464-21ssh23:46:323
329642299591,5cyclictest3388744-21ssh00:10:063
329642299591,5cyclictest3363005-21ssh23:16:223
329642299591,5cyclictest3363005-21ssh23:16:223
329642299591,5cyclictest3356087-21ssh23:01:433
329641899597,13cyclictest3350280-21ssh22:50:002
329641899595,18cyclictest3297904-21modprobe19:40:002
329641899594,13cyclictest3467602-21systemd-journal19:35:002
329641899593,22cyclictest3374363-21sort23:40:032
329641899593,22cyclictest3374363-21sort23:40:032
329641899593,15cyclictest3378988-21munin-run23:50:012
329641899593,13cyclictest3359876-21modprobe23:09:592
3296418995919,33cyclictest0-21swapper/219:55:012
3296418995917,6cyclictest3338342-21cron22:25:002
3296418995914,10cyclictest309-21in:imuxsock23:03:132
3296407995919,35cyclictest364-21snmpd23:25:440
329641899585,17cyclictest3467602-21systemd-journal20:50:022
329641899585,14cyclictest3343077-21modprobe22:34:592
329641899585,11cyclictest3362256-21latency_hist23:15:022
329641899585,10cyclictest0-21swapper/220:10:022
329641899584,15cyclictest311-21rs:main2
3296418995832,22cyclictest3316813-21sendmail21:40:002
329641899583,12cyclictest3328724-21unin-run22:05:022
3296418995825,29cyclictest3302577-21diskmemload21:23:122
3296418995817,5cyclictest0-21swapper/221:12:492
3296407995823,4cyclictest3393626-21ssh00:20:150
3296407995822,32cyclictest16850irq/134-800000021:27:050
3296407995814,40cyclictest3302739-21grep21:10:020
329642299572,7cyclictest3298693-21munin-run19:55:013
329642299572,6cyclictest3300756-21munin-run20:35:003
3296422995718,35cyclictest37-21ksoftirqd/320:05:013
329642299571,4cyclictest3305092-21ssh21:15:043
329641899576,12cyclictest3297381-21modprobe19:30:002
329641899573,12cyclictest3298916-21cron20:00:002
329641899572,17cyclictest274-21cron20:20:002
3296415995719,14cyclictest3398403-21sh00:30:061
3296407995729,13cyclictest3338086-21ssh22:24:280
3296407995721,32cyclictest16850irq/134-800000023:42:560
3296407995720,3cyclictest3307244-21modprobe21:20:000
3296407995715,12cyclictest3309789-21cat21:25:020
329642299562,46cyclictest284-21systemd-logind20:00:003
329642299562,16cyclictest3301302-21munin-run20:45:023
329641899564,14cyclictest3296603-21munin-plugin-st19:15:002
329641899563,18cyclictest3401885-21ssh00:37:402
329641899561,18cyclictest3352667-21ssh22:55:012
329641599569,15cyclictest0-21swapper/100:23:381
3296415995613,14cyclictest3357111-21ssh23:04:071
3296407995621,31cyclictest338137850kworker/u8:323:55:020
3296407995620,31cyclictest16850irq/134-800000023:55:230
3296407995619,32cyclictest16850irq/134-800000021:40:370
3296407995618,33cyclictest0-21swapper/019:35:010
329641899554,10cyclictest3302578-21modprobe21:10:002
329641899553,16cyclictest284-21systemd-logind20:35:022
329641599551,10cyclictest275-21dbus-daemon20:50:011
3296407995528,23cyclictest0-21swapper/021:56:390
3296407995527,13cyclictest3366742-21ssh23:24:250
3296407995527,12cyclictest3334815-21ssh22:17:320
3296407995527,12cyclictest3334815-21ssh22:17:310
3296407995526,13cyclictest3388753-21ssh00:10:080
3296407995519,32cyclictest16850irq/134-800000022:26:300
3296407995515,36cyclictest3400841-21ssh00:35:100
3296407995514,11cyclictest3358589-21ssh23:07:020
329642299542,6cyclictest364-21snmpd20:51:013
329642299542,6cyclictest364-21snmpd20:51:013
329642299541,5cyclictest3301017-21munin-run20:40:013
329641899547,22cyclictest3467602-21systemd-journal20:15:002
3296407995417,33cyclictest3384863-21ssh00:02:060
3296407995417,33cyclictest3384863-21ssh00:02:050
3296407995411,18cyclictest3302541-21kworker/u8:121:32:520
3296407995411,18cyclictest3302541-21kworker/u8:121:32:510
329642299532,5cyclictest3302583-21munin-run21:10:013
329642299532,15cyclictest3299823-21cat20:15:023
329642299531,6cyclictest3302005-21sshd20:55:133
329642299531,14cyclictest3301616-21latency_hist20:50:023
3296418995318,4cyclictest3302577-21diskmemload00:18:212
329641599536,12cyclictest3302577-21diskmemload22:12:001
3296407995324,13cyclictest3378955-21ssh23:49:590
3296407995322,15cyclictest3356464-21ssh23:02:360
3296407995319,31cyclictest16850irq/134-800000023:10:210
3296407995318,5cyclictest16850irq/134-800000022:31:090
3296407995318,20cyclictest16850irq/134-800000022:46:390
3296407995317,6cyclictest16850irq/134-800000022:06:430
329642299522,4cyclictest3297114-21sshd19:21:453
329642299522,4cyclictest3297114-21sshd19:21:453
329641899525,16cyclictest1-21systemd20:30:002
329640799529,13cyclictest1-21systemd19:55:010
3296407995216,32cyclictest16850irq/134-800000000:26:160
329642299514,36cyclictest3296861-21modprobe19:20:003
329642299512,5cyclictest364-21snmpd19:44:573
329642299511,5cyclictest3297616-21sshd19:31:473
3296407995144,4cyclictest16950irq/224-800000020:30:000
329642299501,4cyclictest3302291-21sshd21:01:143
329641599507,33cyclictest3338494-21ssh22:25:011
3296415995023,21cyclictest3395945-21ssh00:25:041
329642299491,6cyclictest364-21snmpd19:29:543
3296418994940,6cyclictest3299222-21kworker/2:320:59:142
3296418994917,24cyclictest0-21swapper/221:05:022
329641599499,6cyclictest3307221-21systemctl21:20:001
3296415994925,20cyclictest3303921-21ssh21:12:471
3296415994921,23cyclictest3368725-21sh23:28:291
3296415994921,23cyclictest3334288-21ssh22:16:201
3296415994921,23cyclictest3334288-21ssh22:16:191
3296407994935,10cyclictest1-21systemd20:25:010
3296407994923,21cyclictest3301931-21ls20:55:030
3296407994923,21cyclictest3301931-21ls20:55:020
329642299482,22cyclictest3296609-21true19:15:013
329642299481,5cyclictest3299463-21systemd-run20:10:003
329641899484,14cyclictest3467602-21systemd-journal19:24:592
329641899484,14cyclictest3467602-21systemd-journal19:24:592
3296407994833,12cyclictest3296872-21systemctl19:20:000
3296407994825,17cyclictest275-21dbus-daemon19:25:010
3296407994825,17cyclictest275-21dbus-daemon19:25:000
3296415994732,11cyclictest3467602-21systemd-journal20:41:021
3296415994727,14cyclictest3345481-21munin-run22:40:001
3296415994724,18cyclictest3383797-21munin-run00:00:021
3296415994723,20cyclictest3374572-21ssh23:40:191
3296415994722,22cyclictest3366960-21sh23:24:511
3296415994721,23cyclictest3308838-21sh21:23:131
3296415994721,20cyclictest3386482-21ssh00:05:171
329640799479,3cyclictest3300744-21modprobe20:35:000
329642299462,39cyclictest3296489-21latency_hist19:10:013
3296415994625,18cyclictest3325432-21ssh21:57:561
3296415994625,17cyclictest3302777-21cat21:10:031
3296415994625,16cyclictest275-21dbus-daemon00:15:011
3296415994624,17cyclictest3323673-21ssh21:54:231
3296415994623,17cyclictest3346424-21ssh22:41:411
3296415994622,21cyclictest3319701-21sh21:45:391
3296415994622,19cyclictest3338859-21ssh22:25:471
329640799462,9cyclictest3299976-21cron20:20:000
3296415994530,12cyclictest3402891-21ssh00:39:561
3296415994528,14cyclictest3352676-21munin-run22:55:001
3296415994525,16cyclictest3330035-21ssh22:07:241
3296415994522,20cyclictest3313048-21ssh21:31:561
3296415994522,20cyclictest3313048-21ssh21:31:561
3296415994516,26cyclictest3327087-21sshd22:01:121
3296415994516,23cyclictest48-21kcompactd000:00:141
3296415994516,23cyclictest48-21kcompactd000:00:131
3296415994515,25cyclictest3341681-21sh22:31:461
3296407994523,16cyclictest3298971-21unin-run20:00:010
3296407994521,20cyclictest3298207-21latency_hist19:45:020
3296415994429,13cyclictest3364441-21ssh23:19:381
3296415994429,13cyclictest3364441-21ssh23:19:371
3296415994429,12cyclictest23-21ksoftirqd/100:17:441
3296415994429,11cyclictest3301803-21munin-run20:55:011
3296415994429,11cyclictest3301803-21munin-run20:55:001
3296415994427,12cyclictest275-21dbus-daemon20:15:011
3296415994423,18cyclictest3359696-21ssh23:09:311
3296415994423,16cyclictest3372610-21ssh23:36:261
3296415994423,16cyclictest3372610-21ssh23:36:261
3296415994422,18cyclictest3355315-21cat23:00:031
3296415994422,18cyclictest3355315-21cat23:00:031
3296415994415,25cyclictest3370952-21ssh23:33:001
3296407994437,4cyclictest16950irq/224-800000019:15:000
3296407994430,10cyclictest3299488-21sed20:10:010
3296407994425,14cyclictest1-21systemd20:45:000
3296407994423,17cyclictest3299207-21munin-plugin-st20:05:000
3296407994422,17cyclictest3299718-21/usr/sbin/munin20:15:010
3296407994420,18cyclictest3301020-21systemctl20:40:000
3296415994328,12cyclictest3310312-21ssh21:25:591
3296415994328,11cyclictest3297153-21systemd-run19:25:011
3296415994328,11cyclictest3297153-21systemd-run19:25:001
3296415994327,11cyclictest3300564-21cat20:30:021
3296415994326,13cyclictest3361886-21ssh23:14:091
3296415994325,13cyclictest3318083-21ssh21:42:261
3296415994323,15cyclictest3379207-21ssh23:50:021
3296415994322,16cyclictest3377006-21ssh23:45:261
3296415994322,16cyclictest3302368-21cat21:05:011
3296415994316,21cyclictest3299210-21munin-plugin-st20:05:011
3296407994337,4cyclictest16950irq/224-800000021:00:010
3296415994226,11cyclictest275-21dbus-daemon21:00:001
329641599422,5cyclictest275-21dbus-daemon19:15:011
3296415994224,13cyclictest3296849-21cron19:20:001
3296415994219,18cyclictest3348253-21ssh22:45:171
3296415994216,22cyclictest3301036-21sendmail-msp20:39:591
3296407994227,11cyclictest3467602-21systemd-journal19:30:010
3296407994224,12cyclictest3302331-21systemd-run21:05:010
329641599416,31cyclictest221rcuc/121:35:021
3296407994135,4cyclictest16950irq/224-800000020:46:120
3296415994020,16cyclictest3296436-21cut19:10:011
3296407994022,13cyclictest3298607-21grep19:50:030
3296415993927,9cyclictest3297620-21sshd19:32:121
3296415993926,10cyclictest3467602-21systemd-journal19:26:011
3296415993925,10cyclictest3299958-21cron20:20:011
3296415993923,12cyclictest275-21dbus-daemon20:25:001
3296415993917,17cyclictest3298668-21cron19:55:011
3296407993917,18cyclictest12-21ksoftirqd/019:35:460
3296415993822,11cyclictest3300857-21grep20:35:031
3296415993820,13cyclictest3298181-21systemd-run19:45:001
3296415993729,5cyclictest3298428-21munin-run19:50:011
3296415993722,12cyclictest3297867-21sshd19:36:471
3296415993721,10cyclictest1-21systemd20:00:001
329641899304,18cyclictest3296492-21latency_hist19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional