You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 00:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Sun May 04, 2025 12:41:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1356289996549,13cyclictest1419316-40mandb09:51:500
1356291996136,22cyclictest1-21systemd10:50:022
135629199602,28cyclictest1511571-21ssh12:07:222
1356291995733,20cyclictest0-21swapper/210:00:012
135629199569,6cyclictest171rcu_preempt11:42:262
1356289995542,10cyclictest1359271-21munin-plugin-st07:20:010
1356290995445,5cyclictest1392195-21idleruntime-cro09:10:011
135629199533,26cyclictest1477743-21ssh11:22:212
1356289995333,18cyclictest477-21rs:main0
1356291995243,5cyclictest0-21swapper/209:51:522
1356291995229,2cyclictest0-21swapper/209:05:012
1356291995229,2cyclictest0-21swapper/209:05:012
1356291995125,5cyclictest1379257-21fschecks_count08:27:152
135629199502,20cyclictest1500227-21rm11:52:212
1356291994938,7cyclictest1-21systemd11:30:012
1356290994937,5cyclictest1406205-21kworker/u8:0-rpciod09:51:491
1356290994838,4cyclictest0-21swapper/112:27:241
1356291994635,6cyclictest0-21swapper/209:25:012
1356291994635,6cyclictest0-21swapper/209:25:012
1356290994638,5cyclictest1505845-21systemd-run12:00:011
1356290994529,6cyclictest57-21kcompactd009:32:271
135628999458,28cyclictest1527654-21ssh12:28:280
135628999453,24cyclictest1461448-21ntpq10:57:200
135628999453,24cyclictest1461448-21ntpq10:57:200
135628999451,42cyclictest1431960-21ssh10:11:060
135628999450,29cyclictest1424611-21sh09:59:020
1356291994435,6cyclictest1363777-21systemd-run07:35:012
1356291994434,6cyclictest1374254-21systemd-run08:10:012
1356291994434,6cyclictest1374254-21systemd-run08:10:012
1356290994416,4cyclictest171rcu_preempt09:42:251
1356289994424,18cyclictest1408781-21munin-plugin-st09:35:010
135628999441,25cyclictest0-21swapper/010:54:420
135628999440,34cyclictest1512435-21ssh12:07:590
135628999440,34cyclictest1478751-21ssh11:23:220
1356291994333,6cyclictest1459759-21cron10:55:012
1356291994333,6cyclictest1444020-21cron10:30:012
1356291994326,9cyclictest1392185-21diskmemload12:22:302
1356290994335,5cyclictest1390719-21systemd-run09:05:011
1356290994335,5cyclictest1390719-21systemd-run09:05:011
1356290994334,5cyclictest1530956-21head12:32:351
1356289994335,5cyclictest1-21systemd10:35:010
135628999433,36cyclictest1442567-21ntpq10:27:250
135628999433,22cyclictest1500314-21ntpq11:52:240
135628999431,39cyclictest1449018-21ntpq10:37:240
135628999431,39cyclictest1411256-21ssh09:37:460
135628999431,26cyclictest1523711-21sh12:23:030
1356291994234,5cyclictest1463076-21sendmail11:00:012
1356291994234,5cyclictest1463076-21sendmail11:00:012
1356290994235,4cyclictest1438408-21kworker/u8:0-rpciod10:22:201
135629099422,33cyclictest1500340-21ssh11:52:261
1356289994236,3cyclictest0-21swapper/011:40:010
1356289994224,15cyclictest0-21swapper/010:20:010
135628999421,26cyclictest1374871-21irqstats08:12:200
135628999420,32cyclictest1392781-21ssh09:10:440
1356291994133,5cyclictest1411092-21sed09:37:372
1356291994132,6cyclictest1397063-21sh09:17:182
1356291994132,6cyclictest1397063-21sh09:17:182
1356291994131,6cyclictest1434549-21systemd-run10:15:022
1356290994133,5cyclictest1533972-21needreboot12:37:191
1356290994133,5cyclictest1466374-21systemd-run11:05:011
135628999417,30cyclictest1361713-21ntpq07:27:250
135628999417,30cyclictest1361713-21ntpq07:27:250
1356289994124,15cyclictest1392183-21latency_hist09:10:020
135628999412,35cyclictest1379590-21ntpq08:27:210
135628999412,21cyclictest1404024-21ntpq09:27:220
135628999411,37cyclictest1369112-21ntpq07:52:220
135628999411,37cyclictest1369112-21ntpq07:52:220
135628999411,36cyclictest1515433-21ntpq12:12:250
135628999411,33cyclictest1507791-21ssh12:02:230
135628999411,24cyclictest1505153-21sh11:58:410
135628999411,23cyclictest1467399-21aten_r2power_cu11:07:130
135629199403,28cyclictest1439256-21ssh10:22:262
1356291994032,5cyclictest1415331-21cron09:45:012
1356290994034,3cyclictest0-21swapper/109:40:011
1356290994032,5cyclictest1377101-21unixbench_multi08:17:401
1356290994032,5cyclictest1-21systemd09:20:021
1356290994032,5cyclictest1-21systemd09:20:021
1356290994020,7cyclictest1404101-21ntpq09:27:251
135628999402,35cyclictest1492793-21ntpq11:42:250
135628999402,34cyclictest1474424-21ntpq11:17:240
135628999402,34cyclictest1382575-21ntpq08:37:200
135628999402,23cyclictest1463932-21sh11:01:450
135628999401,36cyclictest1438159-21ssh10:20:420
135628999401,36cyclictest1384141-21ntpq08:42:230
135628999401,35cyclictest1414676-21ssh09:43:030
135628999401,35cyclictest1387094-21ntpq08:52:220
135628999401,34cyclictest1370602-21ntpq07:57:210
135628999401,34cyclictest1360185-21ntpq07:22:240
135628999401,27cyclictest1396213-21sh09:15:330
135628999401,27cyclictest1396213-21sh09:15:330
135628999400,37cyclictest1528875-21ssh12:30:380
135628999400,22cyclictest1400490-21needreboot09:22:210
135628999400,22cyclictest1400490-21needreboot09:22:210
1356291993931,5cyclictest1470899-21fschecks_time11:12:152
1356291993931,5cyclictest1470899-21fschecks_time11:12:152
1356290993930,6cyclictest1431482-21latency_hist10:10:031
1356290993913,4cyclictest171rcu_preempt12:02:251
135628999396,30cyclictest1416283-21ssh09:46:570
135628999392,33cyclictest1372119-21ntpq08:02:220
135628999392,32cyclictest1519071-21ntpq12:17:240
135628999391,35cyclictest1429056-21ssh10:06:400
135628999391,35cyclictest1367707-21ntpq07:47:280
135628999391,34cyclictest1357455-21switchtime07:12:330
135628999391,33cyclictest1376579-21ntpq08:17:270
135628999390,37cyclictest1532725-21ssh12:35:510
135628999390,37cyclictest1482122-21ssh11:27:360
135628999390,36cyclictest1472535-21ssh11:14:030
135628999390,36cyclictest1472535-21ssh11:14:030
1356291993830,5cyclictest1437676-21systemctl10:20:012
1356291993829,5cyclictest1520877-21basename12:20:012
1356291993829,5cyclictest0-21swapper/208:40:012
1356291993826,9cyclictest1515846-21timerwakeupswit12:12:342
1356291993826,9cyclictest1431341-21sh10:10:022
1356291993826,9cyclictest1381066-21ntp_kernel_err08:32:212
1356290993828,6cyclictest1455205-21mailstats10:47:311
135628999382,32cyclictest1373620-21ntpq08:07:210
135628999382,32cyclictest1373620-21ntpq08:07:210
135628999381,34cyclictest1453027-21ssh10:44:310
135628999381,34cyclictest1364717-21ntpq07:37:270
135628999381,33cyclictest1390075-21ntpq09:02:210
135628999381,33cyclictest1390075-21ntpq09:02:210
135628999381,32cyclictest1388569-21ntpq08:57:200
135628999381,32cyclictest1388569-21ntpq08:57:200
135628999381,23cyclictest1495239-21ssh11:46:080
135628999380,36cyclictest1483600-21ssh11:30:220
135628999380,36cyclictest1425978-21ssh10:02:040
135628999380,34cyclictest1363143-21ntpq07:32:230
1356291993731,3cyclictest0-21swapper/212:37:212
1356291993728,6cyclictest1525922-21aten_r2power_en12:27:122
1356291993727,7cyclictest1532152-21latency_hist12:35:022
1356291993727,6cyclictest1386349-21latency_hist08:50:022
1356290993729,5cyclictest1459889-21latency_hist10:55:031
1356290993728,6cyclictest1475018-21unixbench_singl11:17:341
1356290993727,7cyclictest1389182-21grep09:00:011
1356290993727,7cyclictest1389182-21grep09:00:011
1356290993719,5cyclictest1361014-21/usr/sbin/munin07:27:321
1356290993719,5cyclictest1361014-21/usr/sbin/munin07:27:321
135628999371,31cyclictest1366235-21ntpq07:42:480
135628999371,30cyclictest1455051-21ntpq10:47:280
1356291993629,4cyclictest0-21swapper/212:03:042
1356291993628,5cyclictest1449224-21ssh10:37:312
1356291993628,3cyclictest0-21swapper/210:35:012
135629199362,27cyclictest1404129-21ssh09:27:262
1356290993630,3cyclictest0-21swapper/110:20:011
1356290993629,4cyclictest1400928-21switchtime09:22:341
1356290993629,4cyclictest1400928-21switchtime09:22:341
1356290993629,4cyclictest0-21swapper/111:00:021
1356290993629,4cyclictest0-21swapper/111:00:021
1356290993628,5cyclictest1472655-21systemd-run11:15:011
1356290993628,5cyclictest1472655-21systemd-run11:15:011
1356290993627,6cyclictest453-21systemd-logind10:45:011
1356290993625,8cyclictest1492517-21latency11:42:171
1356290993625,3cyclictest171rcu_preempt11:32:221
1356290993612,8cyclictest28-21ksoftirqd/112:17:311
135628999361,32cyclictest1385605-21ntpq08:47:230
135628999361,32cyclictest1378219-21ntpq08:22:440
1356291993527,5cyclictest1475856-21munin-run11:20:012
1356291993527,4cyclictest0-21swapper/211:37:202
1356291993526,5cyclictest1467712-21uniq11:07:172
1356291993524,5cyclictest0-21swapper/211:02:242
135629199352,26cyclictest1407367-21ssh09:32:232
1356290993528,4cyclictest1383211-21awk08:40:011
1356290993527,6cyclictest422-21dbus-daemon08:45:011
1356290993521,7cyclictest1426695-21sh10:02:231
1356290993518,6cyclictest1395191-21sh09:13:311
1356290993518,3cyclictest171rcu_preempt08:12:241
1356290993517,3cyclictest171rcu_preempt07:22:281
1356289993515,17cyclictest0-21swapper/007:10:020
135629199342,24cyclictest1485226-21ssh11:32:232
135629199341,30cyclictest1451834-21ssh10:42:512
135629199341,25cyclictest1504008-21ssh11:57:232
1356290993429,2cyclictest0-21swapper/110:40:011
1356290993427,4cyclictest959-21mta-sts-daemon09:45:141
1356290993426,3cyclictest171rcu_preempt12:11:551
1356290993423,8cyclictest1372530-21timerwakeupswit08:02:351
135629099342,26cyclictest1445829-21ssh10:32:241
1356291993327,3cyclictest0-21swapper/207:20:012
1356290993325,5cyclictest1359277-21awk07:20:011
1356290993317,7cyclictest171rcu_preempt12:07:251
1356290993310,4cyclictest0-21swapper/107:52:251
1356290993310,4cyclictest0-21swapper/107:52:251
135629199325,18cyclictest1394207-21ssh09:12:242
1356291993225,4cyclictest1371227-21munin-run08:00:012
1356291993222,7cyclictest0-21swapper/210:02:142
135629199322,24cyclictest1417196-21ssh09:47:282
1356291993220,5cyclictest0-21swapper/211:47:242
1356290993221,8cyclictest1423360-21missed_timers09:57:251
1356290993216,6cyclictest918-21sendmail-mta12:22:351
135629099313,9cyclictest1442564-21ssh10:27:261
1356290993123,6cyclictest1468248-21ssh11:07:301
1356290993123,5cyclictest1385243-21fschecks_time08:47:151
1356290993123,5cyclictest0-21swapper/111:30:021
1356290993118,3cyclictest171rcu_preempt07:47:231
1356290993116,3cyclictest171rcu_preempt11:22:261
1356291993022,5cyclictest1384710-21cut08:45:022
1356290993022,5cyclictest1490763-21sendmail11:40:011
1356290993019,3cyclictest171rcu_preempt08:00:021
1356290993017,6cyclictest171rcu_preempt07:32:221
1356290993017,3cyclictest171rcu_preempt10:15:021
135629099301,4cyclictest0-21swapper/107:10:011
1356291992919,5cyclictest1372970-21kworker/u8:2-xprtiod09:00:012
1356291992919,5cyclictest1372970-21kworker/u8:2-xprtiod09:00:012
1356291992918,7cyclictest266-21systemd-journal09:10:002
1356290992923,3cyclictest0-21swapper/108:22:331
1356290992921,5cyclictest1387707-21systemd-run08:55:011
1356290992920,6cyclictest1380235-21true08:30:011
1356290992915,3cyclictest171rcu_preempt08:07:201
1356290992915,3cyclictest171rcu_preempt08:07:201
1356290992911,3cyclictest171rcu_preempt07:42:501
135629199284,13cyclictest1360776-21cron07:25:012
1356291992818,7cyclictest491-21gdbus07:52:312
1356291992818,7cyclictest491-21gdbus07:52:312
1356291992818,5cyclictest1377217-21idleruntime-cro08:20:012
1356290992813,3cyclictest171rcu_preempt07:12:261
1356290992812,3cyclictest171rcu_preempt11:50:021
1356290992810,3cyclictest171rcu_preempt08:32:311
1356291992718,6cyclictest1-21systemd07:15:012
1356291992717,6cyclictest1-21systemd08:25:012
1356291992711,11cyclictest351ktimers/207:42:492
1356290992717,2cyclictest171rcu_preempt07:37:241
1356291992620,3cyclictest0-21swapper/208:13:042
1356291992620,3cyclictest0-21swapper/208:02:362
1356291992620,3cyclictest0-21swapper/207:10:022
1356291992617,5cyclictest955-21mta-sts-daemon07:47:252
1356291992615,8cyclictest1365126-21tail07:37:342
1356291992317,4cyclictest0-21swapper/208:52:232
1356291992213,6cyclictest1361809-21grep07:27:292
1356291992213,6cyclictest1361809-21grep07:27:292
135629299173,7cyclictest0-21swapper/311:22:233
135629299172,8cyclictest0-21swapper/310:27:203
135629299171,9cyclictest0-21swapper/311:57:233
135629299162,9cyclictest0-21swapper/309:51:503
135629299161,5cyclictest0-21swapper/308:07:203
135629299161,5cyclictest0-21swapper/308:07:203
135629299153,4cyclictest0-21swapper/307:37:233
135629299152,8cyclictest0-21swapper/310:52:243
135629299152,7cyclictest0-21swapper/311:27:213
135629299151,7cyclictest0-21swapper/312:02:253
135629299151,6cyclictest0-21swapper/307:47:283
135629299143,6cyclictest0-21swapper/311:17:223
135629299143,5cyclictest0-21swapper/310:02:253
135629299143,5cyclictest0-21swapper/309:22:233
135629299143,5cyclictest0-21swapper/309:22:233
135629299143,5cyclictest0-21swapper/309:12:293
135629299143,4cyclictest0-21swapper/308:02:263
135629299143,4cyclictest0-21swapper/307:52:273
135629299143,4cyclictest0-21swapper/307:52:273
135629299141,8cyclictest0-21swapper/312:22:323
135629299141,8cyclictest0-21swapper/311:42:213
135629299141,8cyclictest0-21swapper/311:12:223
135629299141,8cyclictest0-21swapper/311:12:223
135629299141,8cyclictest0-21swapper/311:02:293
135629299141,7cyclictest0-21swapper/312:37:213
135629299141,7cyclictest0-21swapper/311:52:203
135629299141,7cyclictest0-21swapper/310:12:203
135629299141,7cyclictest0-21swapper/309:17:143
135629299141,7cyclictest0-21swapper/309:17:143
135629299141,6cyclictest0-21swapper/310:37:233
135629299141,6cyclictest0-21swapper/308:27:233
135629299137,4cyclictest0-21swapper/310:45:013
135629299137,4cyclictest0-21swapper/310:22:333
135629299136,5cyclictest0-21swapper/312:30:043
135629299136,4cyclictest0-21swapper/309:37:193
135629299135,5cyclictest0-21swapper/308:57:323
135629299135,5cyclictest0-21swapper/308:57:323
135629299132,6cyclictest0-21swapper/311:07:253
135629299132,6cyclictest0-21swapper/310:47:113
135629299132,6cyclictest0-21swapper/308:15:013
135629299131,7cyclictest0-21swapper/312:07:203
135629299131,7cyclictest0-21swapper/309:47:223
135629299131,7cyclictest0-21swapper/309:42:263
135629299131,6cyclictest0-21swapper/311:47:203
135629299131,6cyclictest0-21swapper/309:02:213
135629299131,6cyclictest0-21swapper/309:02:213
135629299131,5cyclictest0-21swapper/308:47:213
135629299131,5cyclictest0-21swapper/308:32:203
135629299131,5cyclictest0-21swapper/307:12:223
135629299126,4cyclictest0-21swapper/312:17:183
135629299126,4cyclictest0-21swapper/310:57:123
135629299126,4cyclictest0-21swapper/310:57:123
135629299126,4cyclictest0-21swapper/310:07:333
135629299126,4cyclictest0-21swapper/309:57:253
135629299126,4cyclictest0-21swapper/309:10:023
135629299122,5cyclictest0-21swapper/308:42:303
135629299121,6cyclictest0-21swapper/312:12:133
135629299121,6cyclictest0-21swapper/309:32:243
135629299121,6cyclictest0-21swapper/309:27:203
135629299121,5cyclictest0-21swapper/308:37:193
135629299121,5cyclictest0-21swapper/307:22:233
135629299115,4cyclictest0-21swapper/308:00:013
135629299112,6cyclictest0-21swapper/312:27:213
135629299112,5cyclictest0-21swapper/310:17:203
135629299111,6cyclictest0-21swapper/311:32:213
135629299111,6cyclictest0-21swapper/308:17:323
135629299111,5cyclictest0-21swapper/310:30:043
135629299111,5cyclictest0-21swapper/307:27:213
135629299111,5cyclictest0-21swapper/307:27:213
135629299106,3cyclictest0-21swapper/308:22:403
135629299105,3cyclictest0-21swapper/307:17:153
135629299102,5cyclictest0-21swapper/311:37:153
135629299101,5cyclictest0-21swapper/308:55:013
13562929993,4cyclictest0-21swapper/307:32:123
13562929992,4cyclictest0-21swapper/307:10:013
13562929991,3cyclictest0-21swapper/307:42:383
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional