You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 05:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Thu Dec 04, 2025 00:39:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1319518996353,7cyclictest3675390-21cups-browsed21:34:401
1319517996347,11cyclictest1419265-21cron22:35:010
1319518996249,8cyclictest1-21systemd21:55:021
1319519995948,7cyclictest3675390-21cups-browsed22:09:232
1319519995948,7cyclictest3675390-21cups-browsed21:49:212
131951899573,28cyclictest1392282-21ssh21:59:211
1319518995628,25cyclictest1336147-21systemd-run20:05:011
1319518995327,3cyclictest0-21swapper/100:00:011
1319518995324,4cyclictest0-21swapper/122:53:351
131951899512,20cyclictest1460439-21ssh23:29:241
1319518995021,4cyclictest0-21swapper/123:07:471
1319517995022,20cyclictest12450irq/32-eth020:04:240
131951899495,34cyclictest1410972-21ssh22:24:221
1319518994924,4cyclictest0-21swapper/121:40:011
1319518994837,6cyclictest1399103-21aten_r2power_fr22:09:121
1319518994739,5cyclictest1457403-21ssh23:25:011
1319518994738,6cyclictest1352644-21systemd-run21:00:011
1319519994638,3cyclictest0-21swapper/223:49:202
1319519994620,9cyclictest1394402-21ssh22:02:192
1319519994618,10cyclictest1451047-21sh23:17:202
1319519994616,9cyclictest1322036-21ntpq19:19:242
1319519994616,9cyclictest1322036-21ntpq19:19:242
131951899466,15cyclictest1425994-21ssh22:44:201
131951899466,15cyclictest1425994-21ssh22:44:201
1319518994639,4cyclictest1438361-21latency_hist23:00:021
131951799466,14cyclictest14-21ksoftirqd/021:14:210
1319519994536,6cyclictest1484069-40logrotate00:00:012
1319517994531,11cyclictest1483829-21unixbench_singl23:59:350
1319519994417,8cyclictest1427811-21ssh22:46:292
1319517994434,6cyclictest1-21systemd22:20:020
1319517994428,13cyclictest1457467-21latency_hist23:25:020
1319519994335,5cyclictest1498918-21latency_hist00:20:022
1319519994319,9cyclictest1408934-21ssh22:21:252
1319519994319,8cyclictest1341384-21memory20:24:172
1319519994316,17cyclictest0-21swapper/219:39:192
1319519994315,10cyclictest1401581-21ssh22:11:432
1319518994335,5cyclictest1404351-21latency_hist22:15:021
1319518994334,6cyclictest1-21systemd22:05:021
1319517994333,7cyclictest422-21dbus-daemon00:09:240
1319519994216,14cyclictest1489678-21ssh00:08:182
1319519994215,9cyclictest1358720-21ntpq21:14:202
1319518994234,5cyclictest1495081-21unixbench_multi00:14:321
1319518994231,6cyclictest0-21swapper/123:54:151
1319518994228,10cyclictest1362121-21kernelversion21:19:161
1319517994235,4cyclictest1365714-21cut21:24:130
1319517994232,5cyclictest12550irq/33-eth021:29:260
131951799422,34cyclictest1403545-21ssh22:14:260
1319519994131,4cyclictest1320487-21ntpq19:14:232
1319519994118,7cyclictest1462326-21sh23:31:522
1319519994117,8cyclictest1392134-21ntpq21:59:162
1319519994117,8cyclictest1331825-21latency_hist19:50:032
1319519994117,7cyclictest1476497-21cron23:50:032
1319519994117,7cyclictest1355106-21ntp_states21:09:242
1319519994116,10cyclictest1336341-21cut20:05:022
1319519994115,10cyclictest1439037-21sh23:01:042
1319518994132,6cyclictest1333136-21systemd-run19:55:011
1319517994135,4cyclictest1480306-21systemd-run23:55:020
1319519994032,5cyclictest1460491-21proc_pri23:29:252
1319519994032,5cyclictest1327154-21latency_hist19:35:022
1319519994031,5cyclictest1345115-21munin-run20:35:012
1319519994027,7cyclictest1388471-21ntpq21:54:202
1319519994026,8cyclictest1508666-21ntpq00:34:242
1319519994024,12cyclictest1435503-21ssh22:56:372
1319519994016,9cyclictest1444256-21sh23:09:042
1319519994015,18cyclictest1377323-21ssh21:39:212
1319519994013,9cyclictest1343065-21ntpq20:29:192
1319519994013,8cyclictest1381383-21sed21:44:302
1319518994030,4cyclictest0-21swapper/123:19:211
1319517994023,14cyclictest1399737-21sh22:09:230
1319519993930,6cyclictest422-21dbus-daemon21:35:012
1319519993927,7cyclictest1332432-21needreboot19:54:152
1319519993926,9cyclictest1350497-21ntp_kernel_pll_20:54:182
1319519993925,10cyclictest1352931-21aten_r2power_en21:04:082
1319519993924,11cyclictest1364350-21ssh21:21:542
1319519993923,12cyclictest1492702-21ssh00:11:412
1319519993915,9cyclictest1412240-21sh22:25:302
1319519993915,7cyclictest1432815-21ssh22:53:592
1319519993914,8cyclictest1501517-21ntpq00:24:202
1319519993914,8cyclictest1418486-21ntpq22:34:252
1319519993913,9cyclictest1347816-21switchtime20:44:292
131951899394,30cyclictest1471810-21ssh23:44:221
1319518993932,4cyclictest1324136-21latency_hist19:25:021
1319518993931,5cyclictest1-21systemd21:10:011
1319518993930,5cyclictest1490330-21cpuspeed00:09:121
1319518993930,5cyclictest0-21swapper/123:14:291
1319518993930,3cyclictest0-21swapper/123:49:181
1319518993928,6cyclictest0-21swapper/122:29:211
131951899392,33cyclictest1422108-21ssh22:39:191
131951899392,33cyclictest1422108-21ssh22:39:191
131951899392,29cyclictest1418474-21sh22:34:241
1319517993918,13cyclictest12450irq/32-eth022:24:240
1319517993917,13cyclictest12450irq/32-eth023:44:270
1319519993827,7cyclictest1456616-21ntpq23:24:252
1319519993826,8cyclictest1329005-21aten_r2power_po19:44:082
1319519993825,9cyclictest1424110-21sh22:41:482
1319519993825,9cyclictest1424110-21sh22:41:482
1319519993825,10cyclictest1470024-21ssh23:41:542
1319519993824,11cyclictest1323559-21ntpq19:24:242
1319519993823,11cyclictest1509698-21ssh00:35:052
1319519993814,7cyclictest1355612-21diskmemload22:36:562
1319519993814,7cyclictest1355612-21diskmemload22:36:562
1319519993813,9cyclictest1337707-21cut20:10:022
1319519993813,8cyclictest1346105-21ntpq20:39:242
1319519993813,7cyclictest1349019-21ntpq20:49:172
131951999381,31cyclictest1369868-21ssh21:29:222
1319518993832,3cyclictest0-21swapper/100:35:021
1319518993830,5cyclictest1322549-21munin-run19:20:011
1319518993830,5cyclictest1322549-21munin-run19:20:011
1319518993830,4cyclictest0-21swapper/122:20:011
1319518993829,6cyclictest3675390-21cups-browsed23:34:261
1319518993827,8cyclictest1486939-21needreboot00:04:191
1319517993831,5cyclictest1422686-21unixbench_multi22:39:310
1319517993831,5cyclictest1422686-21unixbench_multi22:39:310
1319517993829,6cyclictest1392766-21unixbench_multi21:59:290
1319517993827,8cyclictest1468897-21sendmail23:40:010
1319519993729,5cyclictest1468869-21cron23:40:012
1319519993725,9cyclictest1405033-21ssh22:16:062
1319519993725,8cyclictest1352039-21ntpq20:59:212
1319519993723,11cyclictest1447941-21ssh23:13:542
1319519993722,9cyclictest1484540-21ssh00:00:262
1319519993722,11cyclictest1504024-21ssh00:28:292
1319519993722,11cyclictest1334265-21swap19:59:282
1319519993717,5cyclictest1324333-21sshd19:26:332
1319518993727,7cyclictest1512800-21sendmail00:40:011
1319517993727,3cyclictest171rcu_preempt19:49:190
1319517993714,18cyclictest12450irq/32-eth021:54:210
1319517993713,3cyclictest171rcu_preempt19:39:200
1319519993624,9cyclictest1360139-21ssh21:15:432
1319519993623,8cyclictest1337865-21/usr/sbin/munin20:14:072
1319519993623,8cyclictest1337865-21/usr/sbin/munin20:14:072
1319518993628,4cyclictest0-21swapper/123:40:011
1319518993628,4cyclictest0-21swapper/100:19:301
131951899362,27cyclictest1369778-21ssh21:29:181
1319517993623,10cyclictest1487916-21cron00:05:010
1319517993620,13cyclictest1342123-21cron20:25:000
1319517993618,3cyclictest171rcu_preempt21:19:190
1319519993525,6cyclictest1339367-21munin-node20:19:062
1319519993525,6cyclictest1339367-21munin-node20:19:062
1319518993527,5cyclictest1346660-21latency_hist20:40:021
1319517993528,3cyclictest0-21swapper/000:35:020
1319517993519,6cyclictest1433609-21ntp_kernel_pll_22:54:200
131951799351,5cyclictest171rcu_preempt00:24:200
1319518993428,3cyclictest0-21swapper/123:05:011
1319518993425,6cyclictest422-21dbus-daemon19:25:391
131951899342,25cyclictest1358771-21ssh21:14:211
131951799343,26cyclictest657-21snmpd21:09:200
1319517993428,4cyclictest1333056-21unixbench_singl19:54:300
1319517993426,5cyclictest3675390-21cups-browsed23:07:590
1319519993323,7cyclictest1319532-21munin-run19:10:012
131951899333,25cyclictest69-21kswapd000:29:251
1319518993325,5cyclictest1337615-21cron20:10:011
1319518993325,5cyclictest1331519-21unixbench_multi19:49:291
1319518993325,4cyclictest0-21swapper/122:49:241
1319517993326,5cyclictest1319516-21cyclictest00:13:370
1319517993325,5cyclictest1384975-21sendmail_mailqu21:49:280
1319517993324,6cyclictest0-21swapper/020:10:010
1319517993322,8cyclictest1505593-21unixbench_multi00:29:330
131951899325,21cyclictest437-21mta-sts-daemon20:34:181
1319518993225,5cyclictest0-21swapper/121:24:311
1319518993224,5cyclictest957-21mta-sts-daemon21:40:341
1319518993224,5cyclictest1502394-21ssh00:25:021
1319518993222,7cyclictest1349628-21systemd-run20:50:011
1319518993222,4cyclictest0-21swapper/121:48:591
1319518993221,4cyclictest959-21mta-sts-daemon20:44:241
1319517993227,2cyclictest0-21swapper/020:55:010
131951799322,25cyclictest1381039-21ssh21:44:200
1319517993221,4cyclictest1321600-21fschecks_time19:19:130
1319517993221,4cyclictest1321600-21fschecks_time19:19:130
1319518993122,6cyclictest1342200-21latency_hist20:25:021
1319517993125,4cyclictest1319516-21cyclictest19:31:280
1319517993125,3cyclictest69-21kswapd021:39:240
131951799312,25cyclictest1396903-21grep22:05:020
1319517993121,7cyclictest1414457-21latency22:29:160
1319517993120,8cyclictest0-21swapper/023:14:120
1319517993117,5cyclictest171rcu_preempt20:34:200
1319517993115,4cyclictest1321004-21grep19:15:010
1319517993112,17cyclictest151ktimers/023:33:280
1319517993110,15cyclictest657-21snmpd23:19:440
1319518993022,5cyclictest1352588-21kworker/u8:9-xprtiod21:04:131
1319518993022,5cyclictest1340566-21grep20:19:281
1319518993022,5cyclictest1340566-21grep20:19:281
1319518993021,6cyclictest1334599-21cron20:00:011
1319517993023,4cyclictest453-21systemd-logind19:25:500
131951799302,22cyclictest1429760-21ssh22:49:170
1319517993019,8cyclictest0-21swapper/019:45:010
1319517992924,3cyclictest955-21mta-sts-daemon20:49:160
1319517992921,6cyclictest1498470-21timerandwakeup00:19:330
1319517992918,9cyclictest1351656-21fschecks_time20:59:130
1319517992916,9cyclictest12450irq/32-eth023:49:360
1319518992822,3cyclictest0-21swapper/119:40:011
1319518992816,8cyclictest1327268-21latency_hist19:35:031
131951799283,19cyclictest1426175-21ps22:44:250
131951799283,19cyclictest1426175-21ps22:44:250
1319517992822,4cyclictest1512535-21grep00:39:340
1319517992822,4cyclictest1334603-21cron20:00:010
131951799273,3cyclictest171rcu_preempt20:19:210
131951799273,3cyclictest171rcu_preempt20:19:210
1319517992722,3cyclictest0-21swapper/020:40:010
131951799272,20cyclictest1460403-21ssh23:29:220
1319517992719,6cyclictest956-21mta-sts-daemon21:32:340
131951799271,7cyclictest0-21swapper/020:14:170
131951799271,7cyclictest0-21swapper/020:14:170
1319517992716,7cyclictest12450irq/32-eth022:59:260
1319517992714,8cyclictest12450irq/32-eth023:04:240
131951799271,21cyclictest1323197-21awk19:24:130
131951899266,13cyclictest959-21mta-sts-daemon20:54:191
1319518992617,6cyclictest1330080-21unixbench_singl19:44:291
131951799260,19cyclictest657-21snmpd21:04:180
131951799254,16cyclictest437-21mta-sts-daemon20:29:180
131951899244,13cyclictest437-21mta-sts-daemon20:14:241
131951899244,13cyclictest437-21mta-sts-daemon20:14:241
1319518992412,9cyclictest955-21mta-sts-daemon19:14:191
1319518992312,8cyclictest1343509-21unixbench_multi20:29:281
131951799225,6cyclictest14-21ksoftirqd/020:44:270
131952099185,6cyclictest0-21swapper/300:14:233
131952099182,9cyclictest0-21swapper/300:29:253
1319518991813,2cyclictest0-21swapper/119:10:021
1319517991810,3cyclictest171rcu_preempt19:10:010
131952099173,7cyclictest0-21swapper/323:24:203
131952099162,8cyclictest0-21swapper/322:44:163
131952099162,8cyclictest0-21swapper/322:44:163
131952099161,6cyclictest0-21swapper/321:10:023
131952099152,8cyclictest0-21swapper/323:34:193
131952099151,8cyclictest0-21swapper/300:24:213
131952099151,6cyclictest0-21swapper/323:54:233
131952099151,6cyclictest0-21swapper/322:04:203
131952099147,5cyclictest0-21swapper/321:34:253
131952099146,5cyclictest0-21swapper/323:40:013
131952099145,6cyclictest0-21swapper/323:59:243
131952099145,5cyclictest0-21swapper/323:19:223
131952099142,7cyclictest0-21swapper/322:39:183
131952099142,7cyclictest0-21swapper/322:39:183
131952099142,6cyclictest0-21swapper/322:09:203
131952099142,6cyclictest0-21swapper/320:44:213
131952099142,5cyclictest0-21swapper/300:34:183
131952099141,8cyclictest0-21swapper/321:19:263
131952099141,7cyclictest0-21swapper/322:49:243
131952099141,7cyclictest0-21swapper/322:19:213
131952099141,7cyclictest0-21swapper/321:49:203
131952099141,6cyclictest0-21swapper/321:39:183
131952099141,6cyclictest0-21swapper/320:14:173
131952099141,6cyclictest0-21swapper/320:14:173
131952099141,6cyclictest0-21swapper/320:04:223
131952099136,4cyclictest0-21swapper/320:20:013
131952099136,4cyclictest0-21swapper/320:20:013
131952099136,3cyclictest0-21swapper/322:15:033
131952099134,6cyclictest0-21swapper/323:14:193
131952099132,6cyclictest0-21swapper/323:49:133
131952099132,5cyclictest0-21swapper/320:29:183
131952099131,7cyclictest0-21swapper/321:59:173
131952099131,7cyclictest0-21swapper/321:24:233
131952099131,7cyclictest0-21swapper/300:40:013
131952099131,6cyclictest0-21swapper/321:29:183
131952099131,6cyclictest0-21swapper/320:54:223
131952099131,6cyclictest0-21swapper/320:34:163
131952099131,5cyclictest0-21swapper/322:24:223
131952099131,5cyclictest0-21swapper/320:39:173
131952099131,5cyclictest0-21swapper/319:54:203
131952099131,5cyclictest0-21swapper/319:34:183
131952099126,4cyclictest0-21swapper/322:29:103
131952099125,4cyclictest0-21swapper/300:07:343
131952099122,6cyclictest0-21swapper/323:44:123
131952099122,6cyclictest0-21swapper/323:29:073
131952099122,5cyclictest0-21swapper/321:50:343
131952099121,7cyclictest0-21swapper/322:59:233
131952099121,7cyclictest0-21swapper/322:34:223
131952099121,7cyclictest0-21swapper/321:14:173
131952099121,6cyclictest0-21swapper/321:00:013
131952099121,6cyclictest0-21swapper/319:49:173
131952099121,6cyclictest0-21swapper/300:19:223
131952099121,6cyclictest0-21swapper/300:04:113
131952099121,5cyclictest0-21swapper/319:24:203
131952099116,3cyclictest0-21swapper/323:04:113
131952099116,3cyclictest0-21swapper/320:24:243
131952099116,3cyclictest0-21swapper/319:14:133
131952099111,6cyclictest0-21swapper/321:44:233
131952099111,6cyclictest0-21swapper/319:19:233
131952099111,6cyclictest0-21swapper/319:19:233
131952099111,5cyclictest0-21swapper/320:09:193
131952099105,3cyclictest0-21swapper/322:52:113
131952099102,5cyclictest0-21swapper/323:09:163
131952099101,5cyclictest0-21swapper/321:04:183
131952099101,5cyclictest0-21swapper/319:59:183
131952099101,5cyclictest0-21swapper/319:10:023
13195209994,3cyclictest0-21swapper/319:39:233
13195209993,4cyclictest0-21swapper/320:49:153
13195209991,5cyclictest0-21swapper/319:44:083
13195209981,5cyclictest0-21swapper/319:25:383
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional