You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-25 - 09:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Thu Jun 25, 2026 00:43:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
151270499514,38cyclictest1549812-21ssh21:10:342
151270499514,38cyclictest1549812-21ssh21:10:342
1512702995137,11cyclictest1548710-21systemd-run21:10:010
1512702995137,11cyclictest1548710-21systemd-run21:10:010
1512702994933,8cyclictest1576575-21ssh21:55:010
1512703994839,6cyclictest1548683-21diskmemload22:35:301
1512703994838,6cyclictest1-21systemd21:40:011
1512702994828,15cyclictest12650irq/32-eth023:30:370
1512702994828,14cyclictest151ktimers/000:15:340
1512704994738,5cyclictest1553078-21mailstats21:15:442
1512703994638,5cyclictest1-21systemd23:40:011
1512703994635,4cyclictest1000-21mta-sts-daemon21:25:291
1512704994536,6cyclictest1657426-21if_eth000:05:252
1512704994536,5cyclictest3953742-21cups-browsed23:48:372
1512702994438,4cyclictest1678036-21munin-run00:40:010
1512703994333,6cyclictest420-21avahi-daemon23:20:321
1512702994335,5cyclictest0-21swapper/000:00:390
151270299433,33cyclictest1617708-21ssh23:00:320
151270299433,33cyclictest1593051-21ssh22:20:310
1512704994228,10cyclictest1605134-21latency22:40:252
151270399425,33cyclictest1633157-21ssh23:25:311
1512702994231,9cyclictest1648110-21sh23:50:250
1512704994133,5cyclictest1675029-21latency_hist00:35:022
1512703994134,4cyclictest1638259-21munin-plugin-st23:35:011
1512703994130,7cyclictest421-21dbus-daemon00:30:411
1512702994135,4cyclictest1554891-21munin-run21:20:010
1512702994135,4cyclictest1538202-21systemd-run20:35:010
151270299413,33cyclictest1562193-21ssh21:30:320
1512702994119,13cyclictest151ktimers/020:25:280
1512702994114,21cyclictest1568386-21ssh21:40:300
151270499403,30cyclictest421-21dbus-daemon19:55:292
1512704994022,11cyclictest1574606-21ssh21:50:342
1512703994033,4cyclictest1585850-21cron22:10:011
151270499394,27cyclictest1642370-21ssh23:40:302
1512704993931,5cyclictest1619827-21latency_hist23:05:022
1512703993931,5cyclictest1599457-21mailstats22:30:391
1512703993930,6cyclictest1514210-21systemd-run19:15:021
1512703993928,7cyclictest3953742-21cups-browsed22:49:211
1512703993921,3cyclictest171rcu_preempt23:59:351
1512702993933,4cyclictest1604328-21sendmail22:40:010
1512702993924,8cyclictest12650irq/32-eth000:10:320
151270499383,29cyclictest1667018-21ssh00:20:362
1512704993830,5cyclictest1662799-21cron00:15:012
1512704993829,6cyclictest420-21avahi-daemon22:04:372
1512704993829,5cyclictest1641369-21sendmail23:40:012
1512704993828,6cyclictest1607872-21aten_r2power_po22:45:202
1512704993820,10cyclictest0-21swapper/222:35:312
151270399387,23cyclictest1657783-21ssh00:05:321
151270399383,30cyclictest1623950-21ssh23:10:341
151270399383,27cyclictest1605308-21rm22:40:281
151270399383,25cyclictest1611554-21ssh22:50:331
151270399383,25cyclictest1611554-21ssh22:50:331
1512703993831,4cyclictest1003-21mta-sts-daemon19:40:371
1512703993831,3cyclictest1001-21mta-sts-daemon23:40:171
1512703993830,5cyclictest1551796-21ls21:15:001
1512703993830,5cyclictest1551796-21ls21:15:001
1512703993829,7cyclictest694-21cron23:10:011
1512703993829,6cyclictest485-21in:imuxsock23:05:011
1512703993825,10cyclictest1556155-21mailstats21:20:371
1512703993813,3cyclictest171rcu_preempt23:50:341
1512702993831,4cyclictest0-21swapper/021:45:330
1512702993828,4cyclictest1582776-21idleruntime-cro22:05:010
1512702993818,16cyclictest1666854-21ssh00:20:290
1512704993728,6cyclictest3953742-21cups-browsed21:05:212
1512704993728,6cyclictest3953742-21cups-browsed21:05:212
1512704993728,5cyclictest1599097-21needreboot22:30:282
1512704993728,3cyclictest0-21swapper/223:05:302
1512703993726,8cyclictest1589871-21needreboot22:15:301
1512703993726,7cyclictest3953742-21cups-browsed22:58:531
1512703993719,5cyclictest171rcu_preempt21:30:351
1512702993730,5cyclictest421-21dbus-daemon19:55:010
151270299372,29cyclictest1657949-21ssh00:05:380
151270299371,17cyclictest1559090-21ssh21:25:320
151270499363,26cyclictest1648661-21ssh23:50:382
1512704993628,5cyclictest1591989-21cron22:20:012
1512704993628,5cyclictest1565899-21ssh21:35:442
151270399363,28cyclictest1669976-21ssh00:25:361
1512703993628,5cyclictest1665856-21awk00:20:011
1512703993625,5cyclictest0-21swapper/100:20:341
1512703993624,6cyclictest0-21swapper/122:05:021
1512702993629,3cyclictest0-21swapper/023:10:310
1512702993628,5cyclictest1527713-21systemd-run20:00:010
1512702993627,6cyclictest0-21swapper/023:40:010
1512702993613,19cyclictest1586976-21ssh22:10:350
1512702993612,19cyclictest1642358-21ssh23:40:290
1512704993528,3cyclictest0-21swapper/222:20:202
1512704993527,5cyclictest1630655-21grep23:20:412
151270399353,27cyclictest1586922-21ssh22:10:331
1512703993530,2cyclictest0-21swapper/123:20:011
1512703993519,5cyclictest171rcu_preempt19:55:301
1512702993529,4cyclictest1564887-21ssh21:35:250
1512702993529,4cyclictest1532105-21head20:10:420
1512702993528,4cyclictest1001-21mta-sts-daemon23:49:030
151270299352,27cyclictest665-21snmpd20:50:380
1512702993519,11cyclictest151ktimers/023:55:350
1512704993428,4cyclictest1512701-21cyclictest20:00:432
1512704993428,4cyclictest1512701-21cyclictest20:00:432
1512704993427,4cyclictest1652880-21kworker/u8:5-rpciod00:00:012
1512704993426,5cyclictest1530698-21systemd-run20:10:012
1512704993426,4cyclictest1002-21mta-sts-daemon23:30:322
1512704993425,6cyclictest1000-21mta-sts-daemon22:52:502
1512704993425,6cyclictest1000-21mta-sts-daemon22:52:502
1512703993426,5cyclictest1517220-21systemd-run19:25:011
1512703993425,3cyclictest171rcu_preempt21:50:381
1512703993421,3cyclictest171rcu_preempt00:12:191
1512703993420,3cyclictest171rcu_preempt21:00:291
1512703993420,3cyclictest171rcu_preempt21:00:291
151270399341,25cyclictest665-21snmpd20:45:371
1512703993412,3cyclictest171rcu_preempt20:30:301
1512702993429,2cyclictest0-21swapper/019:40:010
1512702993428,4cyclictest1616656-21sendmail23:00:000
1512702993426,4cyclictest0-21swapper/022:05:290
1512702993424,7cyclictest1629616-21/usr/sbin/munin23:20:210
151270299342,14cyclictest1604346-21kworker/u8:0+rpciod22:50:310
151270299342,14cyclictest1604346-21kworker/u8:0+rpciod22:50:310
1512704993328,2cyclictest0-21swapper/220:45:272
1512704993326,5cyclictest1512701-21cyclictest22:59:222
151270399335,22cyclictest1577624-21ssh21:55:321
1512703993325,5cyclictest1526182-21munin-plugin-st19:55:011
1512703993325,5cyclictest1520131-21sed19:30:431
1512703993325,4cyclictest0-21swapper/121:45:471
1512703993324,4cyclictest0-21swapper/121:42:151
1512703993323,5cyclictest0-21swapper/121:18:111
1512703993318,3cyclictest171rcu_preempt22:20:301
1512702993327,3cyclictest1507362-21kworker/u8:2+rpciod19:35:010
151270499325,19cyclictest1670123-21ssh00:25:412
1512704993225,4cyclictest0-21swapper/220:20:012
1512704993224,5cyclictest1002-21mta-sts-daemon21:49:142
1512704993224,5cyclictest0-21swapper/223:20:012
151270499321,25cyclictest1663940-21ssh00:15:362
1512702993224,5cyclictest1549016-21/usr/sbin/munin21:10:370
1512702993224,5cyclictest1549016-21/usr/sbin/munin21:10:370
1512702993222,7cyclictest0-21swapper/023:05:320
1512702993221,9cyclictest1544630-21cpuspeed20:55:200
1512702993213,14cyclictest3953742-21cups-browsed20:05:340
1512704993126,2cyclictest0-21swapper/222:30:012
1512704993125,3cyclictest0-21swapper/221:55:362
1512704993125,3cyclictest0-21swapper/221:35:012
1512704993124,4cyclictest0-21swapper/200:00:302
1512704993123,5cyclictest1538700-21fschecks_time20:35:232
1512704993123,5cyclictest1520414-21/usr/sbin/munin19:35:312
1512704993122,7cyclictest1526180-21/usr/sbin/munin19:55:012
151270499312,23cyclictest1559110-21sshd21:25:332
1512704993120,8cyclictest1537626-21ntp_offset20:30:342
151270399314,23cyclictest1000-21mta-sts-daemon20:10:351
1512703993124,4cyclictest1512716-21kworker/u8:1-xprtiod19:45:111
1512703993123,4cyclictest0-21swapper/122:30:021
1512703993122,6cyclictest439-21systemd-logind21:00:011
1512703993114,3cyclictest171rcu_preempt19:25:291
1512703993113,9cyclictest265-21systemd-journal23:49:001
1512702993123,4cyclictest191rcuc/021:00:290
1512702993123,4cyclictest191rcuc/021:00:290
1512702993121,6cyclictest1577866-21mailstats21:55:390
1512702993116,9cyclictest0-21swapper/000:30:210
1512704993024,3cyclictest0-21swapper/223:25:412
1512704993024,3cyclictest0-21swapper/200:35:352
1512704993022,5cyclictest1585850-21cron22:10:012
151270499302,22cyclictest1556060-21sh21:20:352
1512704993021,6cyclictest1524603-21unixbench_singl19:45:432
151270399302,21cyclictest1001-21mta-sts-daemon20:40:331
1512703993012,3cyclictest171rcu_preempt00:35:411
151270299308,17cyclictest665-21snmpd20:15:300
1512702993024,4cyclictest1000-21mta-sts-daemon22:20:010
1512702993017,9cyclictest12650irq/32-eth023:25:290
151270299301,22cyclictest665-21snmpd20:20:310
1512704992920,7cyclictest1514995-21needreboot19:15:282
1512704992919,8cyclictest1518709-21sh19:30:012
1512704992919,6cyclictest1536401-21threads20:25:392
1512703992921,5cyclictest271ktimers/120:00:351
1512703992921,5cyclictest271ktimers/120:00:351
1512703992918,6cyclictest1536708-21systemd-run20:30:021
1512703992915,3cyclictest171rcu_preempt21:05:321
1512703992915,3cyclictest171rcu_preempt21:05:321
151270399291,22cyclictest1654749-21ssh00:00:381
1512702992922,3cyclictest0-21swapper/022:45:200
151270299292,21cyclictest1605388-21ssh22:40:310
1512702992921,6cyclictest3953742-21cups-browsed22:31:450
1512702992910,17cyclictest0-21swapper/022:25:430
1512704992820,5cyclictest3953742-21cups-browsed19:34:452
1512704992820,4cyclictest1532181-21cut20:15:002
151270399283,15cyclictest428-21mta-sts-daemon19:35:331
1512703992822,3cyclictest0-21swapper/120:40:011
1512703992821,5cyclictest1529478-21aten_r2power_en20:05:191
1512703992812,3cyclictest171rcu_preempt20:20:321
151270299282,21cyclictest1555944-21ssh21:20:300
1512702992821,4cyclictest1002-21mta-sts-daemon00:25:520
151270499272,18cyclictest1586863-21ssh22:10:312
1512704992718,6cyclictest665-21snmpd23:10:342
151270499271,20cyclictest1568464-21ssh21:40:352
1512703992720,4cyclictest1544102-21unixbench_singl20:50:451
151270399271,5cyclictest0-21swapper/120:15:341
1512703992710,3cyclictest171rcu_preempt19:15:321
1512702992721,4cyclictest1523571-21aten_r2power_vo19:45:210
1512702992720,5cyclictest1626707-21latency23:15:260
1512702992620,4cyclictest1515684-21cron19:20:010
1512702992620,4cyclictest1003-21mta-sts-daemon20:36:340
151270499255,14cyclictest665-21snmpd20:55:292
1512704992518,4cyclictest1541180-21cron20:45:012
1512704992517,4cyclictest0-21swapper/219:40:372
151270299251,14cyclictest151ktimers/019:10:300
1512704992416,5cyclictest3953742-21cups-browsed21:00:352
1512704992416,5cyclictest3953742-21cups-browsed21:00:352
1512704992416,5cyclictest3953742-21cups-browsed19:14:452
1512702992418,4cyclictest1541183-21latency_hist20:45:020
1512702992415,6cyclictest1529190-21munin-plugin-st20:05:010
1512702992415,6cyclictest1529190-21munin-plugin-st20:05:010
151270499235,12cyclictest4053189-21munin-node19:20:282
1512704992318,2cyclictest0-21swapper/220:25:022
1512702992318,3cyclictest1003-21mta-sts-daemon19:40:380
151270299231,17cyclictest665-21snmpd20:45:370
151270299231,16cyclictest1516591-21ntpq19:20:290
1512702992217,3cyclictest627-21lldpd19:25:290
1512704992113,5cyclictest1543953-21timerwakeupswit20:50:442
1512703992113,3cyclictest171rcu_preempt19:10:031
151270599202,10cyclictest0-21swapper/323:25:303
1512704992014,3cyclictest0-21swapper/219:10:022
151270299191,15cyclictest420-21avahi-daemon19:10:010
151270599182,9cyclictest0-21swapper/300:20:353
151270599182,8cyclictest0-21swapper/321:30:323
151270599174,9cyclictest0-21swapper/323:20:333
151270599172,9cyclictest0-21swapper/323:35:353
151270599162,8cyclictest0-21swapper/322:15:383
151270599162,8cyclictest0-21swapper/300:05:353
151270599162,7cyclictest0-21swapper/321:50:293
151270599162,6cyclictest0-21swapper/322:30:333
151270599161,8cyclictest0-21swapper/322:45:313
151270599161,8cyclictest0-21swapper/322:10:303
151270599153,9cyclictest0-21swapper/319:10:323
151270599152,8cyclictest0-21swapper/321:15:363
151270599152,7cyclictest0-21swapper/323:00:333
151270599152,7cyclictest0-21swapper/322:55:293
151270599151,8cyclictest0-21swapper/321:20:323
151270599151,8cyclictest0-21swapper/300:25:373
151270599151,7cyclictest0-21swapper/321:25:283
151270599151,6cyclictest0-21swapper/320:25:273
151270599150,7cyclictest0-21swapper/300:35:313
151270599143,7cyclictest0-21swapper/300:15:303
151270599143,6cyclictest0-21swapper/323:30:333
151270599142,7cyclictest0-21swapper/322:20:313
151270599142,6cyclictest0-21swapper/321:05:343
151270599142,6cyclictest0-21swapper/321:05:343
151270599141,8cyclictest0-21swapper/322:25:353
151270599141,7cyclictest0-21swapper/323:10:313
151270599141,7cyclictest0-21swapper/321:35:333
151270599141,7cyclictest0-21swapper/320:45:373
151270599141,7cyclictest0-21swapper/319:30:343
151270599141,6cyclictest0-21swapper/322:05:243
151270599141,6cyclictest0-21swapper/320:00:333
151270599141,6cyclictest0-21swapper/320:00:333
151270599141,4cyclictest0-21swapper/320:10:293
151270599132,7cyclictest0-21swapper/323:40:253
151270599132,6cyclictest0-21swapper/323:55:183
151270599132,5cyclictest0-21swapper/319:55:323
151270599132,5cyclictest0-21swapper/300:00:203
151270599131,7cyclictest0-21swapper/322:40:233
151270599131,7cyclictest0-21swapper/322:00:343
151270599131,7cyclictest0-21swapper/321:45:213
151270599131,6cyclictest0-21swapper/320:40:303
151270599131,5cyclictest0-21swapper/319:45:323
151270599122,6cyclictest0-21swapper/322:50:373
151270599122,6cyclictest0-21swapper/322:50:373
151270599122,6cyclictest0-21swapper/321:55:223
151270599122,6cyclictest0-21swapper/321:10:253
151270599122,6cyclictest0-21swapper/321:10:253
151270599122,6cyclictest0-21swapper/300:30:243
151270599121,7cyclictest0-21swapper/323:45:363
151270599121,7cyclictest0-21swapper/321:40:213
151270599121,7cyclictest0-21swapper/319:15:293
151270599121,7cyclictest0-21swapper/300:10:303
151270599121,6cyclictest0-21swapper/323:50:213
151270599121,6cyclictest0-21swapper/323:05:203
151270599121,6cyclictest0-21swapper/322:40:033
151270599121,6cyclictest0-21swapper/320:20:313
151270599121,6cyclictest0-21swapper/320:05:343
151270599121,6cyclictest0-21swapper/319:50:383
151270599121,6cyclictest0-21swapper/319:25:283
151270599113,5cyclictest0-21swapper/320:50:313
151270599112,6cyclictest0-21swapper/320:30:313
151270599112,5cyclictest0-21swapper/320:55:293
151270599112,5cyclictest0-21swapper/319:35:253
151270599111,6cyclictest0-21swapper/323:15:213
151270599111,6cyclictest0-21swapper/321:00:293
151270599111,6cyclictest0-21swapper/321:00:293
151270599111,5cyclictest0-21swapper/319:20:273
151270599104,5cyclictest0-21swapper/320:35:363
151270599101,6cyclictest0-21swapper/320:15:303
15127059991,5cyclictest0-21swapper/319:40:233
15127059971,5cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional