You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 04:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Sat Nov 29, 2025 00:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3247399260,25cyclictest1-21systemd00:10:001
3247799251,22cyclictest313-21systemd-journal19:25:002
32477992421,2cyclictest10256-21sh22:50:002
3247399240,23cyclictest10742-21apt-get22:52:461
32480992321,1cyclictest0-21swapper/323:50:003
3248099230,22cyclictest0-21swapper/321:37:453
3248099230,22cyclictest0-21swapper/321:37:453
3247799230,2cyclictest25129-21idleruntime-cro23:20:002
3247799230,22cyclictest12818-21perf19:35:002
32469992320,2cyclictest2401-21sh19:15:000
3246999230,2cyclictest11926-21cron23:59:590
32480992220,1cyclictest7648-21perf19:24:593
32480992220,1cyclictest22198-21apt-get21:02:473
32480992220,1cyclictest0-21swapper/321:22:453
3248099220,22cyclictest0-21swapper/322:32:453
3248099220,21cyclictest14525-21cron20:45:003
3248099220,21cyclictest0-21swapper/320:12:423
3248099220,21cyclictest0-21swapper/320:07:433
3248099220,1cyclictest30286-21munin-run22:25:003
3248099220,1cyclictest29934-21turbostat23:30:013
3247799221,20cyclictest15259-21awk19:40:002
3247799220,2cyclictest3072-21awk22:35:002
3247799220,21cyclictest29665-21cron20:09:592
3247799220,21cyclictest2829-21apt-get23:42:472
3247799220,21cyclictest25433-21cron22:14:592
3247799220,21cyclictest16265-21apt-get21:57:442
3247799220,21cyclictest13866-21apt-get21:52:442
3247799220,21cyclictest0-21swapper/220:35:002
3247799220,21cyclictest0-21swapper/219:57:432
3247799220,1cyclictest31279-21munin-run00:40:002
3247799220,1cyclictest27826-21cron22:19:592
3247399220,21cyclictest5953-21apt-get22:42:461
3247399220,21cyclictest4253-21apt-get21:32:441
3247399220,21cyclictest27764-21apt-get20:07:431
3247399220,21cyclictest0-21swapper/120:55:001
32469992222,0cyclictest0-21swapper/023:02:440
3246999220,21cyclictest10890-21apt-get19:32:440
3246999220,21cyclictest0-21swapper/022:32:450
3246999220,21cyclictest0-21swapper/019:47:420
3246999220,1cyclictest12527-21apt-get00:02:450
32480992120,0cyclictest0-21swapper/300:37:453
32480992119,1cyclictest4763-21grep23:44:593
32480992119,1cyclictest0-21swapper/300:00:003
3248099210,20cyclictest0-21swapper/323:12:453
3248099210,20cyclictest0-21swapper/322:47:453
3248099210,20cyclictest0-21swapper/320:02:443
3248099210,20cyclictest0-21swapper/319:30:003
3248099210,20cyclictest0-21swapper/300:07:463
32477992121,0cyclictest0-21swapper/223:27:462
32477992121,0cyclictest0-21swapper/220:37:442
32477992120,1cyclictest20352-21apt-get23:12:452
32477992120,1cyclictest0-21swapper/200:27:442
32477992120,0cyclictest0-21swapper/200:32:462
32477992117,3cyclictest16920-21perf20:50:002
3247799210,20cyclictest32343-21sleep23:34:592
3247799210,0cyclictest22137-21apt-get00:22:462
3247799210,0cyclictest0-21swapper/222:37:442
3247799210,0cyclictest0-21swapper/200:02:452
32473992121,0cyclictest30289-21dpkg22:25:001
32473992121,0cyclictest0-21swapper/123:42:471
32473992121,0cyclictest0-21swapper/100:17:441
32473992120,1cyclictest25617-21apt-get23:22:451
32473992120,1cyclictest24532-21apt-get00:27:451
3247399211,0cyclictest29323-21apt-get00:37:451
3247399210,21cyclictest15560-21apt-get23:02:441
3247399210,21cyclictest0-21swapper/120:47:461
3247399210,20cyclictest593-21dbus-daemon22:10:001
3247399210,20cyclictest12141-21munin-run00:00:001
3247399210,20cyclictest10166-21apt-get20:37:441
3247399210,20cyclictest0-21swapper/121:17:461
3247399210,0cyclictest25369-21apt-get20:02:441
3247399210,0cyclictest0-21swapper/122:12:451
32469992121,0cyclictest0-21swapper/022:17:450
32469992121,0cyclictest0-21swapper/021:02:470
32469992119,1cyclictest26469-21munin-run00:30:000
32469992119,1cyclictest23528-21apt-get22:12:450
32469992119,1cyclictest0-21swapper/020:20:010
3246999210,21cyclictest61550irq/126-eno123:17:450
3246999210,21cyclictest0-21swapper/022:22:460
3246999210,21cyclictest0-21swapper/019:27:440
3246999210,20cyclictest0-21swapper/023:52:460
3246999210,20cyclictest0-21swapper/023:37:450
3246999210,20cyclictest0-21swapper/022:47:440
3246999210,20cyclictest0-21swapper/021:37:450
3246999210,20cyclictest0-21swapper/021:37:450
3246999210,20cyclictest0-21swapper/020:02:440
3246999210,20cyclictest0-21swapper/000:37:450
3246999210,1cyclictest5487-21ls22:40:010
3246999210,0cyclictest0-21swapper/023:32:460
3246999210,0cyclictest0-21swapper/022:57:440
3246999210,0cyclictest0-21swapper/022:52:450
32480992020,0cyclictest0-21swapper/323:22:453
32480992020,0cyclictest0-21swapper/322:17:443
32480992020,0cyclictest0-21swapper/321:32:433
32480992020,0cyclictest0-21swapper/321:17:463
32480992019,1cyclictest22317-21apt-get23:15:023
32480992019,0cyclictest0-21swapper/321:47:453
32480992019,0cyclictest0-21swapper/319:12:443
32480992018,1cyclictest26457-21grep00:30:003
32480992018,1cyclictest0-21swapper/300:15:013
3248099200,0cyclictest0-21swapper/321:57:443
32477992020,0cyclictest8096-21apt-get19:27:442
32477992020,0cyclictest32569-21apt-get20:17:452
32477992020,0cyclictest30157-21apt-get20:12:412
32477992020,0cyclictest0-21swapper/221:37:442
32477992020,0cyclictest0-21swapper/221:37:442
32477992020,0cyclictest0-21swapper/221:12:442
32477992020,0cyclictest0-21swapper/220:22:452
32477992019,1cyclictest20500-21apt-get19:52:452
3247799200,20cyclictest0-21swapper/200:17:462
3247799200,19cyclictest0-21swapper/219:50:002
3247799200,0cyclictest2885-21apt-get19:17:452
3247799200,0cyclictest15711-21apt-get19:42:462
3247799200,0cyclictest0-21swapper/223:22:442
3247799200,0cyclictest0-21swapper/221:02:172
32473992020,0cyclictest6653-21apt-get21:37:451
32473992020,0cyclictest6653-21apt-get21:37:451
32473992020,0cyclictest2974-21apt-get20:22:461
32473992020,0cyclictest0-21swapper/123:47:441
32473992020,0cyclictest0-21swapper/122:47:451
32473992020,0cyclictest0-21swapper/122:32:451
32473992020,0cyclictest0-21swapper/119:32:441
32473992019,1cyclictest14529-21latency_hist20:45:001
3247399200,0cyclictest0-21swapper/119:57:421
3247399200,0cyclictest0-21swapper/100:32:461
32469992020,0cyclictest0-21swapper/023:47:440
32469992020,0cyclictest0-21swapper/021:22:450
32469992020,0cyclictest0-21swapper/021:12:440
32469992019,0cyclictest9071-21apt-get21:42:440
32469992019,0cyclictest2091-21apt-get20:20:020
3246999200,0cyclictest0-21swapper/020:12:420
32480991919,0cyclictest0-21swapper/323:02:453
32480991919,0cyclictest0-21swapper/321:52:443
32480991919,0cyclictest0-21swapper/320:27:433
32480991918,1cyclictest352-21apt-get23:37:453
32480991918,1cyclictest34-21ksoftirqd/319:40:003
32480991918,1cyclictest0-21swapper/320:37:443
32480991918,0cyclictest0-21swapper/300:17:463
32480991917,2cyclictest12813-21apt19:35:003
32477991919,0cyclictest0-21swapper/223:07:462
32477991919,0cyclictest0-21swapper/222:27:442
32477991919,0cyclictest0-21swapper/221:07:462
32477991918,1cyclictest0-21swapper/200:12:462
3247799190,18cyclictest5370-21apt-get20:27:442
32473991919,0cyclictest24648-21apt-get21:07:451
32473991919,0cyclictest0-21swapper/121:57:441
32473991919,0cyclictest0-21swapper/119:47:431
32473991919,0cyclictest0-21swapper/119:12:441
32473991917,1cyclictest7642-21awk19:25:001
32473991917,1cyclictest641-21basename22:30:001
3247399190,2cyclictest14666-21irqstats21:52:481
3247399190,0cyclictest0-21swapper/123:17:441
32469991919,0cyclictest0-21swapper/021:57:440
32469991919,0cyclictest0-21swapper/021:17:460
32469991919,0cyclictest0-21swapper/020:32:430
32469991919,0cyclictest0-21swapper/019:42:450
32469991918,1cyclictest0-21swapper/020:27:430
32469991917,1cyclictest24909-21perf20:00:000
32469991917,1cyclictest16848-21turbostat00:09:590
32469991916,2cyclictest20604-21turbostat22:05:000
32480991818,0cyclictest0-21swapper/321:12:443
32480991818,0cyclictest0-21swapper/320:32:423
32480991818,0cyclictest0-21swapper/319:57:433
32480991818,0cyclictest0-21swapper/319:52:443
3248099180,17cyclictest0-21swapper/319:42:463
32477991818,0cyclictest0-21swapper/222:42:462
32477991818,0cyclictest0-21swapper/222:02:452
32477991818,0cyclictest0-21swapper/220:02:442
32477991817,0cyclictest19779-21apt-get20:57:452
32477991816,1cyclictest31366-21turbostat.cron21:19:592
32477991816,1cyclictest11922-21cron00:00:002
32473991818,0cyclictest10036-21perf19:30:011
32473991818,0cyclictest0-21swapper/123:12:451
32473991818,0cyclictest0-21swapper/121:22:451
32473991818,0cyclictest0-21swapper/120:12:421
32473991818,0cyclictest0-21swapper/119:37:451
32473991817,1cyclictest25925-21apt-get22:17:441
32473991817,0cyclictest0-21swapper/121:02:461
32469991818,0cyclictest0-21swapper/023:07:460
32469991818,0cyclictest0-21swapper/000:22:460
32480991717,0cyclictest26927-21apt-get00:32:453
32480991717,0cyclictest0-21swapper/322:57:443
32480991717,0cyclictest0-21swapper/321:42:443
32480991717,0cyclictest0-21swapper/320:47:453
32480991717,0cyclictest0-21swapper/320:22:453
32480991716,1cyclictest5486-21sendmail-msp22:40:013
32480991716,1cyclictest0-21swapper/320:52:473
32477991717,0cyclictest0-21swapper/222:57:442
32477991717,0cyclictest0-21swapper/219:12:442
32477991715,1cyclictest3793-21perf21:30:002
32473991717,0cyclictest0-21swapper/121:12:441
32473991717,0cyclictest0-21swapper/119:52:451
32473991716,1cyclictest0-21swapper/120:27:441
32473991716,0cyclictest0-21swapper/121:47:451
32473991715,1cyclictest19253-21apt00:15:011
32469991717,0cyclictest4830-21apt-get19:20:020
32469991717,0cyclictest11469-21apt-get21:47:440
32469991717,0cyclictest0-21swapper/020:42:450
32469991717,0cyclictest0-21swapper/020:37:430
32469991717,0cyclictest0-21swapper/000:12:460
32469991716,0cyclictest0-21swapper/020:57:450
32469991715,1cyclictest19687-21/usr/sbin/munin00:17:480
32469991715,1cyclictest141rcuc/022:45:010
3246999170,0cyclictest13290-21apt-get19:37:450
32480991616,0cyclictest0-21swapper/322:02:453
32480991616,0cyclictest0-21swapper/300:02:453
32480991614,1cyclictest3788-21awk21:30:003
32477991616,0cyclictest0-21swapper/222:22:462
32477991616,0cyclictest0-21swapper/221:42:442
32477991614,1cyclictest31861-21apt-get21:22:442
32473991614,1cyclictest8671-21ntp_states23:52:521
32473991614,1cyclictest19898-21perf23:09:591
3247399160,15cyclictest7962-21tr20:32:441
32469991616,0cyclictest0-21swapper/022:07:460
32469991616,0cyclictest0-21swapper/021:32:440
32469991616,0cyclictest0-21swapper/020:52:450
32469991614,1cyclictest26233-21tail23:22:480
32480991515,0cyclictest0-21swapper/322:42:463
3248099150,1cyclictest8524-21memory23:52:503
3248099150,14cyclictest0-21swapper/322:27:493
32477991515,0cyclictest14927-21apt-get00:07:462
32477991515,0cyclictest0-21swapper/223:52:462
3247799150,14cyclictest0-21swapper/223:37:462
3247799150,14cyclictest0-21swapper/220:55:002
32473991515,0cyclictest0-21swapper/119:42:461
32473991514,1cyclictest28014-21apt-get23:27:471
32473991514,1cyclictest13141-21apt-get22:57:441
3247399150,14cyclictest0-21swapper/122:35:371
32469991515,0cyclictest13866-21apt-get21:52:440
32469991515,0cyclictest0-21swapper/023:27:470
32469991515,0cyclictest0-21swapper/022:25:020
32469991515,0cyclictest0-21swapper/021:07:460
32469991514,0cyclictest0-21swapper/021:27:460
32480991414,0cyclictest0-21swapper/323:32:463
32480991414,0cyclictest0-21swapper/321:07:463
32480991414,0cyclictest0-21swapper/319:47:403
32480991414,0cyclictest0-21swapper/300:21:413
32480991413,0cyclictest0-21swapper/322:15:003
3248099140,14cyclictest0-21swapper/323:06:453
3248099140,14cyclictest0-21swapper/320:55:403
3248099140,14cyclictest0-21swapper/319:17:553
3248099140,0cyclictest0-21swapper/322:51:403
3248099140,0cyclictest0-21swapper/322:05:123
3247799140,14cyclictest0-21swapper/223:00:162
3247799140,14cyclictest0-21swapper/222:51:332
3247799140,14cyclictest0-21swapper/221:46:052
3247799140,14cyclictest0-21swapper/221:30:192
3247799140,14cyclictest0-21swapper/220:41:182
32473991414,0cyclictest0-21swapper/122:01:591
32473991414,0cyclictest0-21swapper/121:42:441
32473991414,0cyclictest0-21swapper/120:57:161
32473991414,0cyclictest0-21swapper/120:15:251
32473991414,0cyclictest0-21swapper/100:01:571
3247399140,14cyclictest0-21swapper/123:36:051
3247399140,14cyclictest0-21swapper/100:21:231
3247399140,0cyclictest0-21swapper/123:31:431
3247399140,0cyclictest0-21swapper/121:27:061
3246999140,14cyclictest0-21swapper/023:40:370
3246999140,14cyclictest0-21swapper/020:47:540
3246999140,0cyclictest0-21swapper/023:10:020
3246999140,0cyclictest0-21swapper/019:51:140
32480991313,0cyclictest0-21swapper/320:15:043
3247799130,8cyclictest313-21systemd-journal23:46:012
32477991212,0cyclictest0-21swapper/222:07:472
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional