You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-02 - 12:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Wed Jul 02, 2025 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2895499260,25cyclictest0-21swapper/020:49:500
2896099240,1cyclictest22425-21apt-get00:30:022
2895699240,23cyclictest30919-21lockfile-create20:20:001
2895699240,23cyclictest0-21swapper/119:54:591
28954992420,4cyclictest2833-21sensors23:50:010
2895499240,0cyclictest8829-21apt-get20:44:500
2896299230,22cyclictest31320-21cron19:15:003
2896299230,22cyclictest30240-21cron21:25:003
2896299230,22cyclictest18902-21idleruntime-cro19:55:003
2896299230,1cyclictest313-21systemd-journal21:05:003
2896099230,2cyclictest5794-21munin-run20:35:002
2896099230,22cyclictest22420-21cron00:30:002
2896099230,0cyclictest0-21swapper/221:54:472
28956992321,1cyclictest20321-21apt-config00:25:001
2895699230,22cyclictest14720-21latency_hist00:15:001
2895499230,2cyclictest26106-21cron20:10:000
2895499230,2cyclictest22912-21wc23:25:000
28962992220,1cyclictest6180-21perf22:50:013
28962992220,1cyclictest32644-21ls21:30:003
2896299221,20cyclictest11972-21apt-get19:44:493
2896299220,3cyclictest7576-21sendmail-msp00:00:003
2896299220,21cyclictest29753-21apt-get22:34:553
2896299220,21cyclictest28369-21idleruntime-cro23:35:003
2896299220,21cyclictest18293-21apt-get00:24:503
2896299220,21cyclictest15877-21apt-get00:19:503
2896299220,21cyclictest15877-21apt-get00:19:503
2896299220,21cyclictest0-21swapper/300:29:533
2896299220,1cyclictest0-21swapper/320:10:003
28960992221,1cyclictest0-21swapper/222:44:522
28960992220,1cyclictest14954-21grep00:15:012
28960992220,1cyclictest13781-21dump-pmu-power23:04:592
2896099220,21cyclictest29565-21apt-get19:14:482
2896099220,21cyclictest21749-21apt-get23:24:542
2896099220,21cyclictest0-21swapper/223:55:002
2896099220,21cyclictest0-21swapper/222:09:492
2896099220,21cyclictest0-21swapper/221:10:012
2896099220,21cyclictest0-21swapper/200:19:512
2896099220,21cyclictest0-21swapper/200:19:502
28956992220,1cyclictest4763-21apt-get19:29:491
28956992220,1cyclictest24224-21cron22:20:011
28956992220,1cyclictest22961-21cron21:10:001
28956992220,1cyclictest20601-21munin-run21:05:001
2895699220,21cyclictest13749-21munin-run19:45:001
2895699220,21cyclictest12870-21apt-get21:59:491
2895699220,21cyclictest0-21swapper/123:54:531
2895699220,21cyclictest0-21swapper/123:49:531
2895699220,21cyclictest0-21swapper/122:44:521
2895699220,21cyclictest0-21swapper/100:29:531
28954992220,2cyclictest0-21swapper/019:14:480
28954992220,1cyclictest8908-21sh19:35:000
28954992220,1cyclictest28588-21ls20:15:000
28954992220,1cyclictest0-21swapper/020:35:000
28954992220,1cyclictest0-21swapper/000:30:020
2895499220,21cyclictest863-21apt-get21:34:510
2895499220,21cyclictest27285-21apt-get22:29:500
2895499220,21cyclictest17155-21apt-get19:54:480
2895499220,21cyclictest10475-21apt-get21:54:470
2895499220,21cyclictest0-21swapper/022:04:480
2895499220,1cyclictest25384-21wc21:15:000
2895499220,1cyclictest22475-21apt-get22:19:490
2895499220,1cyclictest14639-21latency_hist22:00:000
28962992121,0cyclictest23870-21apt-get00:34:483
28962992120,0cyclictest11221-21apt-get20:49:493
28962992120,0cyclictest0-21swapper/323:24:533
28962992119,1cyclictest14706-21cron00:15:003
2896299210,21cyclictest0-21swapper/322:04:483
2896299210,20cyclictest0-21swapper/323:14:503
2896299210,1cyclictest0-21swapper/323:20:003
2896299210,0cyclictest0-21swapper/320:39:483
28960992120,1cyclictest30887-21apt-get21:29:492
28960992120,0cyclictest6192-21apt-get23:59:522
28960992120,0cyclictest0-21swapper/222:54:512
28960992120,0cyclictest0-21swapper/219:39:482
28960992118,2cyclictest23665-21cron20:05:002
2896099210,20cyclictest28483-21apt-get21:24:492
2896099210,20cyclictest0-21swapper/223:00:002
2896099210,20cyclictest0-21swapper/222:34:562
2896099210,20cyclictest0-21swapper/221:19:492
2896099210,20cyclictest0-21swapper/221:14:482
2896099210,20cyclictest0-21swapper/220:59:492
2896099210,0cyclictest0-21swapper/223:19:512
2896099210,0cyclictest0-21swapper/221:45:002
28956992121,0cyclictest0-21swapper/120:49:491
28956992121,0cyclictest0-21swapper/120:29:501
28956992121,0cyclictest0-21swapper/120:14:501
28956992120,1cyclictest8082-21apt-get21:49:491
28956992120,1cyclictest6938-21apt-get22:54:511
28956992120,1cyclictest0-21swapper/123:44:531
28956992120,0cyclictest0-21swapper/100:39:491
28956992119,1cyclictest1972-21apt-get19:24:501
28956992119,1cyclictest16931-21apt-get23:14:511
2895699210,20cyclictest0-21swapper/122:59:551
2895699210,20cyclictest0-21swapper/120:34:481
2895699210,20cyclictest0-21swapper/100:19:501
2895699210,20cyclictest0-21swapper/100:19:501
2895699210,1cyclictest24138-21apt-get23:29:491
2895699210,0cyclictest19554-21apt-get19:59:501
2895699210,0cyclictest0-21swapper/123:04:501
2895699210,0cyclictest0-21swapper/121:54:471
2895699210,0cyclictest0-21swapper/119:19:511
28954992119,1cyclictest593-21dbus-daemon20:05:000
28954992119,1cyclictest26266-21apt-get00:39:500
28954992119,1cyclictest20078-21apt-get22:14:500
2895499210,0cyclictest0-21swapper/019:24:500
28962992020,0cyclictest0-21swapper/323:44:533
28962992020,0cyclictest0-21swapper/323:29:493
28962992018,1cyclictest3549-21grep22:45:013
28962992018,1cyclictest30917-21munin-run20:20:003
2896299200,20cyclictest8828-21apt20:44:503
2896299200,20cyclictest0-21swapper/322:55:023
2896299200,20cyclictest0-21swapper/322:14:493
2896299200,20cyclictest0-21swapper/321:39:503
2896299200,20cyclictest0-21swapper/320:50:013
2896299200,0cyclictest4013-21apt-get20:34:493
2896299200,0cyclictest0-21swapper/323:49:523
2896299200,0cyclictest0-21swapper/323:09:493
2896299200,0cyclictest0-21swapper/319:39:483
2896299200,0cyclictest0-21swapper/300:10:003
28960992020,0cyclictest8695-21apt-get00:04:522
28960992020,0cyclictest0-21swapper/222:14:502
28960992020,0cyclictest0-21swapper/221:49:502
28960992020,0cyclictest0-21swapper/221:04:492
28960992020,0cyclictest0-21swapper/220:54:562
28960992020,0cyclictest0-21swapper/220:39:482
28960992020,0cyclictest0-21swapper/219:49:492
28960992020,0cyclictest0-21swapper/219:44:512
28960992020,0cyclictest0-21swapper/219:25:012
28960992018,1cyclictest0-21swapper/220:10:002
2896099200,20cyclictest0-21swapper/221:34:512
2896099200,20cyclictest0-21swapper/220:14:492
2896099200,19cyclictest15285-21apt-get22:04:482
2896099200,0cyclictest0-21swapper/222:19:492
28956992020,0cyclictest0-21swapper/123:34:491
28956992020,0cyclictest0-21swapper/122:14:501
28956992020,0cyclictest0-21swapper/119:39:491
28956992018,1cyclictest18173-21basename21:00:011
28956992018,1cyclictest0-21swapper/121:44:501
2895699200,20cyclictest0-21swapper/119:49:491
2895699200,19cyclictest0-21swapper/122:49:511
2895699200,0cyclictest0-21swapper/120:44:501
2895699200,0cyclictest0-21swapper/100:34:481
28954992020,0cyclictest14535-21apt-get23:09:500
28954992020,0cyclictest0-21swapper/021:24:490
28954992020,0cyclictest0-21swapper/021:19:490
28954992020,0cyclictest0-21swapper/021:09:470
28954992020,0cyclictest0-21swapper/021:04:490
28954992020,0cyclictest0-21swapper/019:44:500
28954992020,0cyclictest0-21swapper/000:04:520
28954992018,1cyclictest593-21dbus-daemon19:40:000
2895499200,0cyclictest0-21swapper/023:55:010
2895499200,0cyclictest0-21swapper/000:25:020
28962991919,0cyclictest3799-21apt-get23:54:523
28962991919,0cyclictest0-21swapper/322:29:503
28962991919,0cyclictest0-21swapper/321:34:513
28962991919,0cyclictest0-21swapper/319:19:513
28962991919,0cyclictest0-21swapper/300:04:523
28962991917,1cyclictest972-21perf20:25:003
28962991917,1cyclictest25369-21sh21:15:003
2896299190,18cyclictest7160-21apt-get19:34:483
28960991919,0cyclictest0-21swapper/223:39:502
28960991919,0cyclictest0-21swapper/223:29:492
28960991919,0cyclictest0-21swapper/220:24:502
28960991919,0cyclictest0-21swapper/219:59:502
28956991919,0cyclictest0-21swapper/121:19:491
28956991919,0cyclictest0-21swapper/100:04:531
28956991917,1cyclictest8915-21ls19:35:011
28956991917,1cyclictest7746-21/usr/sbin/munin00:00:001
28956991917,1cyclictest593-21dbus-daemon19:15:001
28956991917,1cyclictest31632-21apt-get20:24:491
2895699190,1cyclictest26140-21munin-run20:10:001
2895499194,14cyclictest6192-21apt-get23:59:520
28954991919,0cyclictest8489-21apt-get22:55:020
28954991919,0cyclictest0-21swapper/023:44:530
28954991919,0cyclictest0-21swapper/023:04:510
28954991919,0cyclictest0-21swapper/021:49:490
28954991919,0cyclictest0-21swapper/020:54:520
28954991919,0cyclictest0-21swapper/019:49:500
28954991918,0cyclictest0-21swapper/000:09:480
2895499190,1cyclictest0-21swapper/020:19:500
2895499190,18cyclictest0-21swapper/022:45:000
28962991818,0cyclictest0-21swapper/321:54:473
28962991818,0cyclictest0-21swapper/321:09:473
28962991818,0cyclictest0-21swapper/319:24:503
28962991818,0cyclictest0-21swapper/300:39:503
2896299180,17cyclictest1219-21cron22:40:003
28960991818,0cyclictest0-21swapper/223:14:502
28960991818,0cyclictest0-21swapper/200:09:472
28960991816,1cyclictest593-21dbus-daemon23:45:002
2896099180,17cyclictest0-21swapper/222:30:002
28956991818,0cyclictest21970-21apt-get20:04:501
28956991818,0cyclictest0-21swapper/122:34:551
28956991818,0cyclictest0-21swapper/121:34:511
28956991816,1cyclictest0-21swapper/122:30:001
2895699180,17cyclictest16321-21cat23:10:001
28954991816,1cyclictest593-21dbus-daemon00:25:000
28962991717,0cyclictest0-21swapper/323:04:503
28962991717,0cyclictest0-21swapper/322:19:493
28962991717,0cyclictest0-21swapper/321:19:503
28962991717,0cyclictest0-21swapper/320:59:493
28962991715,1cyclictest3722-21apt-get19:25:013
28960991717,0cyclictest0-21swapper/223:34:492
28960991717,0cyclictest0-21swapper/219:34:482
28960991715,1cyclictest26650-21latency_hist22:25:002
2896099170,16cyclictest1389-21apt-get23:49:522
2896099170,16cyclictest12975-21apt-get20:50:002
28956991717,0cyclictest6408-21apt-get20:39:481
28956991717,0cyclictest3269-21apt-get21:39:511
28956991717,0cyclictest0-21swapper/123:19:511
28956991717,0cyclictest0-21swapper/122:04:471
28956991716,1cyclictest0-21swapper/122:09:491
2895699171,1cyclictest1237-21munin-run22:40:001
2895699170,16cyclictest25452-21sed21:15:001
28954991717,0cyclictest0-21swapper/022:09:500
28954991717,0cyclictest0-21swapper/021:39:500
28954991717,0cyclictest0-21swapper/019:59:500
28954991714,2cyclictest8194-21munin-run20:40:000
28954991713,3cyclictest0-21swapper/023:39:550
28962991615,1cyclictest5688-21apt-get21:44:503
28962991614,1cyclictest26758-21apt-get20:14:503
28962991614,1cyclictest17681-21apt-get22:09:493
2896299160,8cyclictest0-21swapper/323:39:513
2896299160,15cyclictest9858-21munin-run21:50:003
2896299160,15cyclictest15409-21ntp_states19:49:553
2896299160,15cyclictest0-21swapper/322:00:003
28960991616,0cyclictest0-21swapper/220:44:502
28960991616,0cyclictest0-21swapper/200:39:492
28960991615,0cyclictest0-21swapper/200:24:502
28960991613,2cyclictest4272-21taskset21:39:542
2896099161,14cyclictest14535-21apt-get23:09:502
2896099160,1cyclictest1544-21users19:20:012
2896099160,15cyclictest0-21swapper/220:20:002
28956991616,0cyclictest11085-21apt-get00:09:481
28956991616,0cyclictest0-21swapper/120:54:511
28954991616,0cyclictest0-21swapper/020:59:490
28954991616,0cyclictest0-21swapper/020:24:490
28954991616,0cyclictest0-21swapper/000:14:530
28954991614,1cyclictest31965-21apt-get19:19:520
28954991613,2cyclictest0-21swapper/021:29:500
2895499160,1cyclictest0-21swapper/023:19:560
2895499160,15cyclictest5807-21ntp_states19:29:540
2895499160,15cyclictest0-21swapper/022:49:570
2895499160,0cyclictest0-21swapper/023:28:240
28962991515,0cyclictest21970-21apt-get20:04:503
28962991515,0cyclictest0-21swapper/322:54:513
28962991515,0cyclictest0-21swapper/320:29:503
28962991513,1cyclictest21282-21munin-run20:00:003
28960991514,1cyclictest3014-21/usr/sbin/munin20:29:582
28960991513,1cyclictest18917-21turbostat.cron19:54:592
28956991515,0cyclictest0-21swapper/123:39:501
28954991515,0cyclictest0-21swapper/022:24:480
28954991514,1cyclictest0-21swapper/022:52:110
2895499150,15cyclictest0-21swapper/023:30:130
28962991414,0cyclictest0-21swapper/322:20:083
28960991414,0cyclictest0-21swapper/222:45:252
2896099140,14cyclictest0-21swapper/221:55:142
2896099140,14cyclictest0-21swapper/219:17:272
2896099140,0cyclictest0-21swapper/222:36:322
28956991414,0cyclictest0-21swapper/123:20:171
2895699140,0cyclictest0-21swapper/121:20:381
28954991414,0cyclictest0-21swapper/000:15:200
28954991414,0cyclictest0-21swapper/000:15:200
2895499140,14cyclictest0-21swapper/020:25:270
2895499140,0cyclictest0-21swapper/022:30:560
2895499140,0cyclictest0-21swapper/021:40:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional