You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-03 - 02:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Mon Nov 03, 2025 00:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
159203822250,3sleep3586-21kworker/3:2-events23:16:283
154519121520,2sleep3471rcuc/322:26:363
16464142830,5chrt0-21swapper/200:10:002
1346748996847,18cyclictest1581900-21/usr/sbin/munin23:06:451
16074672660,6chrt0-21swapper/223:31:322
15261442650,6sleep3451irq_work/322:06:413
1346748996355,5cyclictest1556823-21latency_hist22:40:011
1346749995748,7cyclictest1616293-21cp23:40:002
15350742550,1sleep01535080-21cut22:16:340
1346748995444,7cyclictest1346581-21latency_hist19:10:011
1346748995232,18cyclictest282-21systemd-journal21:35:001
1346745995243,6cyclictest1626348-21latency_hist23:50:010
1346745994635,9cyclictest1421835-21ntpq20:21:390
1346748994536,7cyclictest1331646-21kworker/u8:1+flush-8:019:26:431
1346748994326,14cyclictest1537106-21latency_hist22:20:011
1346748994233,6cyclictest1402953-21latency_hist20:05:011
1346745994232,7cyclictest1472317-21apt-get21:15:000
1346745994230,8cyclictest282-21systemd-journal19:50:010
1346748994131,8cyclictest282-21systemd-journal22:47:501
1346745994130,8cyclictest1441590-21ntpq20:41:360
1346748994019,18cyclictest1456402-21ntpq20:56:351
1346749993934,4cyclictest1427830-21kworker/u8:3+flush-8:021:51:372
1346748993932,4cyclictest1393415-21apt-get19:56:271
1346748993922,14cyclictest1471574-21ntpq21:11:381
1346745993932,5cyclictest1557187-21apt-get22:41:240
1346745993932,5cyclictest1557187-21apt-get22:41:240
1346745993930,6cyclictest1331646-21kworker/u8:1+flush-8:019:16:280
1346745993929,7cyclictest1620547-21ntpq23:41:340
1346748993828,7cyclictest1351613-21latency_hist19:15:011
1346749993731,4cyclictest1331646-21kworker/u8:1+flush-8:020:36:432
1346745993730,5cyclictest1408230-21apt-get20:11:220
1346745993729,5cyclictest1596396-21latency_hist23:20:010
1346749993631,3cyclictest1512125-21kworker/u8:2+flush-8:022:01:362
1346749993628,6cyclictest1596383-21cron23:20:002
1346748993630,4cyclictest1497611-21apt-get21:41:221
1346748993628,6cyclictest1596770-21apt-get23:21:251
1346748993626,8cyclictest1436721-21ntpq20:36:391
1346748993626,7cyclictest1676275-21latency_hist00:40:011
1346745993628,6cyclictest1491442-21ntpq21:31:360
1346745993626,8cyclictest1367600-21ntpq19:26:390
1346745993623,10cyclictest282-21systemd-journal00:00:000
1346748993527,6cyclictest1606639-21apt-get23:31:241
1346748993526,6cyclictest1407880-21latency_hist20:10:011
1346748993525,8cyclictest1605457-21ntpq23:26:401
1346749993426,6cyclictest1636601-21cat00:00:012
1346748993418,12cyclictest0-21swapper/100:25:031
1346745993428,4cyclictest1482434-21latency_hist21:25:010
1346749993325,5cyclictest1576654-21awk23:00:002
134674899334,5cyclictest0-21swapper/123:51:241
134674899332,6cyclictest0-21swapper/119:51:241
1346748993319,12cyclictest0-21swapper/123:40:011
1346748993316,15cyclictest1517663-21diskstats22:01:271
1346745993329,2cyclictest1446469-21needreboot20:46:370
1346745993324,7cyclictest1581089-21kworker/u8:3+flush-8:023:11:560
1346745993324,6cyclictest1502198-21latency_hist21:45:010
1346749993226,4cyclictest1294598-21kworker/u8:0+flush-8:019:56:372
1346749993223,7cyclictest1462460-21awk21:05:002
134674899324,4cyclictest0-21swapper/120:46:261
134674899324,4cyclictest0-21swapper/120:41:241
134674899324,4cyclictest0-21swapper/120:21:251
134674899323,7cyclictest0-21swapper/100:21:241
134674899323,5cyclictest0-21swapper/122:56:261
134674899323,5cyclictest0-21swapper/120:11:231
1346748993224,6cyclictest1631656-21apt-get23:56:241
1346748993221,9cyclictest0-21swapper/121:01:251
1346748993217,12cyclictest1656290-21latency_hist00:20:011
1346748993216,14cyclictest0-21swapper/121:16:411
1346748993215,15cyclictest0-21swapper/119:41:301
1346748993214,16cyclictest0-21swapper/120:31:511
1346748993213,16cyclictest0-21swapper/122:06:431
134674899320,30cyclictest1650961-21unixbench_multi00:11:481
1346745993226,4cyclictest1661237-21gpgv00:25:000
1346745993226,4cyclictest1606286-21latency_hist23:30:010
1346749993120,8cyclictest1388132-21awk19:50:002
134674899313,6cyclictest0-21swapper/122:41:251
134674899313,6cyclictest0-21swapper/122:41:251
134674899313,4cyclictest0-21swapper/123:11:261
1346748993127,2cyclictest1544546-21kthreadcore22:26:351
134674899312,4cyclictest0-21swapper/100:06:231
1346748993121,7cyclictest1452320-21latency_hist20:55:001
1346748993121,7cyclictest1452320-21latency_hist20:55:001
1346748993119,11cyclictest0-21swapper/121:06:431
1346748993119,10cyclictest832-21mta-sts-daemon22:11:341
1346748993119,10cyclictest0-21swapper/121:46:311
1346748993119,10cyclictest0-21swapper/119:46:311
1346748993118,11cyclictest282-21systemd-journal22:24:511
1346745993127,2cyclictest1452318-21cron20:55:000
1346745993127,2cyclictest1452318-21cron20:55:000
1346745993126,3cyclictest1666428-21systemd-run00:30:000
1346745993126,2cyclictest1532233-21latency_hist22:15:010
1346745993121,8cyclictest1526094-21ntpq22:06:400
1346749993025,4cyclictest1501419-21kworker/u8:1+events_unbound21:41:422
1346749993024,4cyclictest1341679-21kworker/u8:2+events_unbound19:35:022
1346748993026,3cyclictest831-21mta-sts-daemon19:40:001
1346748993022,6cyclictest1353872-21apt-get19:16:231
1346748993021,7cyclictest1547299-21/usr/sbin/munin22:31:411
1346748993020,8cyclictest282-21systemd-journal21:56:511
1346748993018,10cyclictest0-21swapper/121:36:331
1346748993015,13cyclictest0-21swapper/121:26:421
1346748993015,13cyclictest0-21swapper/100:31:301
1346748993014,14cyclictest0-21swapper/123:16:431
1346748993014,14cyclictest0-21swapper/121:21:291
1346748993012,16cyclictest1358831-21apt-get19:21:241
1346748993012,16cyclictest1358831-21apt-get19:21:241
1346748993010,18cyclictest1416406-21cut20:16:331
1346745993026,2cyclictest1431670-21needreboot20:31:370
1346745993025,3cyclictest1642536-21irqrtprio00:06:300
1346745993021,6cyclictest1551882-21latency_hist22:35:010
1346745993020,8cyclictest1546106-21ntpq22:26:380
16496732290,2sleep3481ktimers/300:11:383
134674899293,5cyclictest0-21swapper/122:51:241
134674899292,12cyclictest1423021-21apt-get20:26:251
1346748992915,12cyclictest0-21swapper/121:55:001
1346748992912,15cyclictest0-21swapper/123:01:271
1346748992911,16cyclictest1637146-21grep00:01:311
1346748992911,15cyclictest0-21swapper/123:45:391
1346745992926,2cyclictest1398036-21systemd-run20:00:010
1346745992925,2cyclictest1373328-21systemd-run19:35:010
1346745992922,5cyclictest1506765-21grep21:46:440
1346745992914,12cyclictest1294598-21kworker/u8:0+flush-8:021:26:430
1346749992823,3cyclictest1427830-21kworker/u8:3+events_unbound20:55:022
1346749992821,5cyclictest1368722-21apt-get19:31:232
1346748992812,14cyclictest0-21swapper/123:41:331
1346745992824,2cyclictest1656277-21cron00:20:000
1346745992823,3cyclictest1671313-21apt00:35:000
1346745992823,3cyclictest1460027-21kthreadcore21:01:320
1346745992823,2cyclictest0-21swapper/023:05:000
1346745992822,4cyclictest1572958-21kthreadcore22:56:340
1346749992723,2cyclictest1671787-21aten2_r5power_v00:36:292
1346749992721,4cyclictest1341679-21kworker/u8:2+events_unbound19:25:012
1346749992720,5cyclictest1477456-21awk21:20:002
1346748992710,15cyclictest0-21swapper/119:31:341
1346745992720,5cyclictest1341679-21kworker/u8:2+flush-8:019:36:400
1346745992719,6cyclictest1512773-21fschecks_time21:56:280
1346749992623,2cyclictest1451951-21unixbench_multi20:51:432
1346749992623,2cyclictest1451951-21unixbench_multi20:51:432
1346749992622,2cyclictest1403356-21aten2_r5power_e20:06:252
1346749992621,4cyclictest1651294-21cron00:15:002
134674599262,16cyclictest1462821-21taskset21:06:270
1346745992621,4cyclictest1331646-21kworker/u8:1+flush-8:020:36:280
1346745992621,3cyclictest1511781-21unixbench_singl21:51:420
1346745992620,4cyclictest1341679-21kworker/u8:2+flush-8:019:11:270
1346745992618,6cyclictest1388127-21latency_hist19:50:010
1346749992522,2cyclictest1442066-21unixbench_multi20:41:432
1346749992520,3cyclictest1417343-21unixbench_multi20:16:422
134674999250,23cyclictest1467368-21turbostat.cron21:10:002
134674599252,16cyclictest584-21snmpd20:06:240
1346749992421,2cyclictest1546610-21unixbench_multi22:26:462
1346749992420,2cyclictest1630981-21unixbench_singl23:51:432
1346749992418,4cyclictest1591181-21unixbench_singl23:11:462
134674599242,21cyclictest1566742-21kworker/0:2+events00:36:410
1346745992419,4cyclictest1556463-21unixbench_multi22:36:430
1346745992419,3cyclictest1477501-21systemd-run21:20:010
1346745992418,4cyclictest1615833-21grep23:36:430
1346745992418,4cyclictest1341679-21kworker/u8:2+flush-8:020:16:420
1346745992418,3cyclictest0-21swapper/022:46:290
134674599241,22cyclictest1566742-21kworker/0:2+events23:51:570
15312752230,5sleep239-21ksoftirqd/222:11:422
1346749992319,2cyclictest1620490-21needreboot23:41:332
1346749992319,2cyclictest1581255-21unixbench_singl23:01:442
134674599235,16cyclictest1566742-21kworker/0:2+events23:31:250
1346745992314,2cyclictest421rcuog/221:36:520
134674599231,21cyclictest1566742-21kworker/0:2+events00:01:450
1346749992219,2cyclictest1537763-21fschecks_count22:21:272
1346749992219,2cyclictest1512810-21fschecks_time21:56:282
1346749992217,4cyclictest1581089-21kworker/u8:3+events_unbound23:47:022
1346745992218,2cyclictest1581955-21aten2_r5power_c23:06:280
1346745992217,3cyclictest1341679-21kworker/u8:2+flush-8:019:41:310
1346745992216,4cyclictest1597496-21grep23:21:330
1346745992216,4cyclictest1522043-21latency_hist22:05:010
134674599220,21cyclictest1437513-21kworker/0:1+events20:56:290
1346749992118,2cyclictest1605964-21unixbench_singl23:26:462
1346749992116,4cyclictest1512125-21kworker/u8:2+events_unbound22:41:402
1346749992116,4cyclictest1512125-21kworker/u8:2+events_unbound22:41:402
1346749992116,4cyclictest1331646-21kworker/u8:1+flush-8:019:21:292
1346749992116,4cyclictest1331646-21kworker/u8:1+flush-8:019:21:292
1346749992115,4cyclictest1447043-21unixbench_singl20:46:462
134674599213,17cyclictest1341695-21kworker/0:0+events19:21:250
134674599213,17cyclictest1341695-21kworker/0:0+events19:21:250
1346745992116,3cyclictest0-21swapper/020:05:010
1346745992116,3cyclictest0-21swapper/000:11:480
1346745992115,4cyclictest1331646-21kworker/u8:1+events_unbound20:26:330
13465062212,13sleep00-21swapper/019:09:140
157293222011,5sleep30-21swapper/322:56:353
134674999202,12cyclictest1378633-21apt-get19:41:252
1346749992016,2cyclictest0-21swapper/221:15:012
1346749992011,7cyclictest1503336-21sed21:46:312
1346749992011,7cyclictest1424918-21tr20:26:322
134674999198,8cyclictest1346997-21apt-get19:11:232
134674999192,12cyclictest1353872-21apt-get19:16:242
1346749991916,2cyclictest1666124-21unixbench_singl00:26:452
1346749991915,3cyclictest1341679-21kworker/u8:2+events_unbound20:10:102
1346749991915,2cyclictest1535282-21kthreadcore22:16:342
1346749991915,2cyclictest0-21swapper/222:50:002
1346749991915,2cyclictest0-21swapper/221:26:362
1346749991915,2cyclictest0-21swapper/200:25:002
1346749991914,3cyclictest1556491-21unixbench_singl22:36:432
1346749991914,3cyclictest0-21swapper/219:55:002
1346749991912,5cyclictest1432256-21grep20:31:452
1346745991914,3cyclictest1537715-21sh22:21:260
13419202193,9sleep30-21swapper/319:05:313
13418892194,9sleep20-21swapper/219:05:062
1346749991814,2cyclictest1512125-21kworker/u8:2-writeback22:55:012
1346749991813,3cyclictest0-21swapper/220:01:352
14500092170,2sleep3481ktimers/320:51:333
14500092170,2sleep3481ktimers/320:51:333
13687252172,9sleep30-21swapper/319:31:243
134674999173,9cyclictest1581925-21apt-get23:06:282
1346749991713,2cyclictest0-21swapper/223:21:332
1346749991713,2cyclictest0-21swapper/221:36:442
1346749991713,2cyclictest0-21swapper/200:31:392
1346749991711,4cyclictest0-21swapper/200:01:362
134674999171,12cyclictest1487704-21taskset21:31:252
134674599172,11cyclictest1567075-21apt-get22:51:230
14926562162,9sleep30-21swapper/321:36:253
134674999168,6cyclictest1419006-21cut20:21:332
134674999163,8cyclictest1522398-21apt-get22:06:282
1346749991612,2cyclictest0-21swapper/222:30:082
1346749991612,2cyclictest0-21swapper/221:22:462
134674999159,4cyclictest0-21swapper/200:16:322
16377572145,5sleep30-21swapper/300:01:363
15071132143,6sleep30-21swapper/321:50:013
15978432132,7sleep30-21swapper/323:21:353
15568572132,6sleep30-21swapper/322:40:013
15568572132,6sleep30-21swapper/322:40:013
14572912132,6sleep30-21swapper/321:00:013
14226932132,6sleep30-21swapper/320:25:013
13980032132,6sleep30-21swapper/320:00:013
16758012122,6sleep30-21swapper/300:36:473
16663922120,6sleep30-21swapper/300:30:013
16603982123,6sleep30-21swapper/300:21:383
16560372122,6sleep30-21swapper/300:17:013
16161622120,8sleep30-21swapper/323:40:013
16063062122,6sleep30-21swapper/323:30:013
15872522122,6sleep30-21swapper/323:11:313
15664412122,6sleep30-21swapper/322:46:433
15524792122,6sleep30-21swapper/322:36:283
15488262122,6sleep30-21swapper/322:31:383
15414942122,6sleep30-21swapper/322:21:433
14827322124,5sleep30-21swapper/321:26:013
14777122122,6sleep30-21swapper/321:20:113
14332912122,6sleep30-21swapper/320:36:293
14172332122,6sleep30-21swapper/320:16:433
14082342122,7sleep30-21swapper/320:11:253
14073752122,6sleep30-21swapper/320:06:433
13793442122,6sleep30-21swapper/319:41:313
13641652122,6sleep30-21swapper/319:26:303
16666632113,5sleep30-21swapper/300:30:293
16422012112,5sleep30-21swapper/300:06:293
16313462112,5sleep30-21swapper/323:55:013
16269462112,6sleep30-21swapper/323:51:283
16220162112,5sleep30-21swapper/323:46:313
16208252112,5sleep30-21swapper/323:41:413
16017422112,5sleep30-21swapper/323:26:283
15824022112,5sleep30-21swapper/323:06:343
15772542112,5sleep30-21swapper/323:01:283
15669642112,5sleep30-21swapper/322:50:173
15363792112,5sleep30-21swapper/322:16:423
15278292112,5sleep30-21swapper/322:11:323
15125732112,5sleep30-21swapper/321:56:273
14975832112,5sleep30-21swapper/321:41:223
14765942112,5sleep30-21swapper/321:16:383
14677322112,5sleep30-21swapper/321:11:243
14627512112,6sleep30-21swapper/321:05:553
14529282112,5sleep30-21swapper/320:56:283
14431902112,5sleep30-21swapper/320:46:323
14377762112,5sleep30-21swapper/320:40:503
14325312112,6sleep30-21swapper/320:34:493
14220512112,5sleep30-21swapper/320:21:443
13937232112,5sleep30-21swapper/319:56:323
13884492112,5sleep30-21swapper/319:51:223
13835422112,5sleep30-21swapper/319:46:253
13736282112,5sleep30-21swapper/319:36:223
13587652112,6sleep30-21swapper/319:20:533
13587652112,6sleep30-21swapper/319:20:533
13525932112,5sleep30-21swapper/319:15:033
13515802112,5sleep30-21swapper/319:14:513
15173422102,5sleep30-21swapper/322:00:483
15073532102,5sleep30-21swapper/321:50:053
14875752102,5sleep30-21swapper/321:30:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional