You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 08:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Fri Nov 28, 2025 00:44:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
912399240,23cyclictest4918-21tr22:20:003
912399240,23cyclictest0-21swapper/323:57:413
912099240,23cyclictest16309-21turbostat.cron19:25:000
912399230,2cyclictest9812-21basename23:34:593
912399230,1cyclictest12586-21cron23:39:593
9122992321,1cyclictest0-21swapper/220:34:592
9122992321,1cyclictest0-21swapper/219:55:012
9121992323,0cyclictest0-21swapper/121:52:401
912199230,22cyclictest6430-21apt-get20:12:381
912199230,22cyclictest313-21systemd-journal23:55:001
9120992321,1cyclictest4842-21latency_hist21:15:000
9123992221,1cyclictest0-21swapper/323:02:413
9123992221,1cyclictest0-21swapper/323:02:413
912399220,21cyclictest0-21swapper/321:32:443
912399220,21cyclictest0-21swapper/320:12:383
912399220,21cyclictest0-21swapper/319:17:383
912399220,21cyclictest0-21swapper/300:37:383
912399220,21cyclictest0-21swapper/300:02:393
912399220,1cyclictest10139-21apt-get21:27:393
9122992220,1cyclictest9686-21munin-run21:24:592
9122992220,1cyclictest9686-21munin-run21:24:592
9122992220,1cyclictest14552-21perf22:40:002
912299220,21cyclictest612-21gdbus00:30:002
912299220,21cyclictest3582-21awk20:05:002
912299220,21cyclictest30603-21apt-get22:07:422
912299220,21cyclictest27176-21basename00:10:002
912299220,21cyclictest25083-21apt-get23:02:402
912299220,21cyclictest25083-21apt-get23:02:402
912299220,21cyclictest19760-21munin-run22:50:002
912299220,21cyclictest0-21swapper/221:45:002
912299220,21cyclictest0-21swapper/219:32:412
912299220,21cyclictest0-21swapper/219:17:392
912299220,1cyclictest23996-21basename19:39:592
912299220,1cyclictest15332-21apt-get21:37:422
9121992220,1cyclictest22538-21latency_hist21:50:001
912199220,21cyclictest0-21swapper/122:30:011
912199220,21cyclictest0-21swapper/120:47:401
912199220,1cyclictest27643-21turbostat20:55:001
9120992220,1cyclictest27030-21awk23:05:000
9120992220,1cyclictest27030-21awk23:05:000
9120992220,1cyclictest13084-21apt-get23:42:400
9120992220,1cyclictest0-21swapper/022:22:420
9120992219,2cyclictest30064-21sendmail-msp20:59:590
912099220,21cyclictest5318-21apt-get21:17:400
912099220,21cyclictest4920-21sh22:20:000
912099220,21cyclictest24944-21munin-run21:55:000
912099220,21cyclictest0-21swapper/020:42:400
912099220,21cyclictest0-21swapper/019:42:400
912099220,1cyclictest541-21apt-get22:12:390
912099220,1cyclictest2462-21munin-run21:10:000
912099220,1cyclictest20279-21apt-get23:57:400
912099220,1cyclictest13908-21ls19:19:590
9123992121,0cyclictest0-21swapper/323:07:413
9123992120,1cyclictest0-21swapper/323:27:403
9123992120,1cyclictest0-21swapper/322:42:403
9123992120,1cyclictest0-21swapper/300:17:413
9123992120,0cyclictest8828-21apt-get20:17:373
912399210,20cyclictest12123-21cron22:35:003
912399210,20cyclictest0-21swapper/320:22:423
912399210,0cyclictest16773-21apt-get19:27:383
9122992121,0cyclictest11260-21apt-get20:22:422
9122992121,0cyclictest0-21swapper/223:47:402
9122992120,1cyclictest2990-21apt-get22:17:392
9122992120,1cyclictest24620-21grep23:00:002
9122992120,1cyclictest10207-21apt-get22:32:392
9122992120,0cyclictest0-21swapper/221:12:402
912299210,20cyclictest28205-21apt-get22:02:402
912299210,20cyclictest22546-21munin-run21:50:002
912299210,20cyclictest0-21swapper/220:17:382
912299210,20cyclictest0-21swapper/219:12:402
912299210,20cyclictest0-21swapper/200:17:402
912299210,20cyclictest0-21swapper/200:02:392
912299210,1cyclictest7277-21apt-get00:37:392
912299210,1cyclictest2682-21apt-get23:22:392
912299210,0cyclictest0-21swapper/223:42:412
912299210,0cyclictest0-21swapper/220:47:392
9121992121,0cyclictest0-21swapper/123:35:011
9121992121,0cyclictest0-21swapper/119:27:371
9121992121,0cyclictest0-21swapper/100:22:421
9121992120,1cyclictest32462-21turbostat21:05:001
912199210,0cyclictest4038-21apt-get20:07:401
9120992120,1cyclictest19671-21apt-get21:44:590
9120992120,0cyclictest0-21swapper/000:37:390
9120992119,1cyclictest7001-21sh23:30:000
912099210,20cyclictest7736-21apt-get21:22:420
912099210,20cyclictest7736-21apt-get21:22:420
912099210,20cyclictest0-21swapper/023:35:010
912099210,20cyclictest0-21swapper/021:32:440
912099210,0cyclictest19571-21apt-get19:32:410
912099210,0cyclictest15420-21apt-get22:42:400
912099210,0cyclictest0-21swapper/023:17:410
912099210,0cyclictest0-21swapper/000:17:410
9123992020,0cyclictest0-21swapper/323:22:403
9123992020,0cyclictest0-21swapper/322:52:393
9123992020,0cyclictest0-21swapper/322:37:413
9123992020,0cyclictest0-21swapper/322:22:423
9123992020,0cyclictest0-21swapper/321:17:403
9123992020,0cyclictest0-21swapper/300:32:413
9123992019,0cyclictest22995-21apt-get21:52:403
9123992019,0cyclictest0-21swapper/323:52:413
9123992019,0cyclictest0-21swapper/321:07:393
9123992019,0cyclictest0-21swapper/319:12:403
9123992018,2cyclictest27336-21apt-get22:00:023
912399200,0cyclictest0-21swapper/321:57:433
912399200,0cyclictest0-21swapper/320:02:383
9122992020,0cyclictest26835-21apt-get19:47:392
9122992020,0cyclictest14379-21apt-get19:22:412
9122992020,0cyclictest0-21swapper/222:40:012
9122992020,0cyclictest0-21swapper/221:52:402
9122992020,0cyclictest0-21swapper/220:42:402
9122992020,0cyclictest0-21swapper/200:22:422
9122992019,1cyclictest5318-21apt-get21:17:402
9122992018,1cyclictest10428-21apt-get23:35:012
912299200,0cyclictest25396-21apt-get21:57:432
912299200,0cyclictest0-21swapper/221:02:402
9121992020,0cyclictest0-21swapper/122:00:011
9121992020,0cyclictest0-21swapper/120:27:391
9121992020,0cyclictest0-21swapper/119:22:411
9121992020,0cyclictest0-21swapper/100:02:381
9121992020,0cyclictest0-21swapper/100:00:001
9121992019,1cyclictest8828-21apt-get20:17:371
9121992019,1cyclictest768-21runrttasks22:40:011
9121992019,0cyclictest14471-21apt-get21:35:021
9121992018,1cyclictest32507-21cron22:09:591
912199200,20cyclictest0-21swapper/120:35:001
912199200,20cyclictest0-21swapper/119:47:401
912199200,1cyclictest0-21swapper/121:57:421
912199200,0cyclictest20870-21apt-get20:42:401
912199200,0cyclictest0-21swapper/122:52:391
9120992020,0cyclictest0-21swapper/020:32:380
9120992020,0cyclictest0-21swapper/019:52:420
9120992019,1cyclictest0-21swapper/021:45:010
9120992019,0cyclictest0-21swapper/020:02:380
9120992019,0cyclictest0-21swapper/000:27:400
9120992018,1cyclictest32544-21perf22:10:000
9120992018,1cyclictest27336-21apt-get22:00:000
9123991919,0cyclictest7803-21apt-get22:27:403
9123991919,0cyclictest32502-21apt-get00:22:423
9123991919,0cyclictest0-21swapper/323:42:403
9123991919,0cyclictest0-21swapper/320:37:403
9123991919,0cyclictest0-21swapper/319:57:393
9123991919,0cyclictest0-21swapper/319:52:413
9123991919,0cyclictest0-21swapper/300:27:403
9123991919,0cyclictest0-21swapper/300:12:413
9123991918,1cyclictest0-21swapper/319:42:403
9123991918,0cyclictest0-21swapper/321:45:013
9122991919,0cyclictest0-21swapper/221:27:402
9122991919,0cyclictest0-21swapper/220:57:412
9122991919,0cyclictest0-21swapper/219:57:392
9122991918,1cyclictest24439-21apt-get19:42:392
9122991917,1cyclictest0-21swapper/222:55:002
9122991916,2cyclictest7340-21turbostat.cron22:25:002
9121991919,0cyclictest27480-21apt-get23:07:411
9121991919,0cyclictest12541-21apt-get21:32:451
9121991919,0cyclictest0-21swapper/122:47:401
9121991919,0cyclictest0-21swapper/121:07:381
9121991919,0cyclictest0-21swapper/100:32:421
9121991918,0cyclictest31629-21apt-get19:57:391
912199190,0cyclictest12611-21apt-get22:37:431
9120991919,0cyclictest0-21swapper/023:12:400
9120991919,0cyclictest0-21swapper/020:17:380
9120991918,1cyclictest0-21swapper/022:02:410
9120991917,1cyclictest20213-21apt-get22:52:390
9120991917,1cyclictest0-21swapper/000:22:420
9123991818,0cyclictest0-21swapper/321:37:423
9123991818,0cyclictest0-21swapper/321:02:403
9123991818,0cyclictest0-21swapper/320:47:383
9123991818,0cyclictest0-21swapper/319:37:403
9123991818,0cyclictest0-21swapper/319:22:413
9123991817,1cyclictest0-21swapper/319:47:383
9123991816,1cyclictest0-21swapper/320:10:003
9123991815,2cyclictest313-21systemd-journal23:50:003
9122991818,0cyclictest25717-21apt-get20:52:412
9122991818,0cyclictest0-21swapper/223:30:012
9122991818,0cyclictest0-21swapper/220:12:382
9122991818,0cyclictest0-21swapper/220:07:402
912299180,17cyclictest313-21systemd-journal20:30:002
9121991818,0cyclictest29235-21apt-get19:52:421
9121991818,0cyclictest22022-21apt-get19:37:401
9121991818,0cyclictest0-21swapper/122:17:401
9121991818,0cyclictest0-21swapper/120:22:411
9121991818,0cyclictest0-21swapper/100:27:401
9121991818,0cyclictest0-21swapper/100:12:411
9121991816,1cyclictest26383-21perf19:45:001
9120991818,0cyclictest0-21swapper/023:47:400
9120991818,0cyclictest0-21swapper/000:15:000
9120991816,1cyclictest19805-21basename23:54:590
9120991816,1cyclictest0-21swapper/023:22:400
9120991816,1cyclictest0-21swapper/020:22:410
9120991813,5cyclictest0-21swapper/020:27:390
912099180,18cyclictest0-21swapper/020:36:070
912099180,17cyclictest0-21swapper/022:57:400
9123991717,0cyclictest0-21swapper/321:22:413
9123991717,0cyclictest0-21swapper/321:22:413
9123991717,0cyclictest0-21swapper/320:42:393
9123991717,0cyclictest0-21swapper/300:07:393
9122991717,0cyclictest0-21swapper/223:07:412
9122991717,0cyclictest0-21swapper/219:27:382
9122991716,1cyclictest541-21apt-get22:12:392
9121991717,0cyclictest17751-21apt-get21:42:401
9121991717,0cyclictest0-21swapper/122:57:411
9121991717,0cyclictest0-21swapper/121:17:391
9121991717,0cyclictest0-21swapper/119:30:021
9121991716,1cyclictest9809-21turbostat23:35:001
9121991716,0cyclictest0-21swapper/121:22:411
9121991716,0cyclictest0-21swapper/121:22:411
9121991715,1cyclictest30079-21apt-get00:17:411
9120991717,0cyclictest0-21swapper/020:52:410
9120991717,0cyclictest0-21swapper/020:12:370
9120991717,0cyclictest0-21swapper/019:37:400
9120991717,0cyclictest0-21swapper/000:35:000
9120991717,0cyclictest0-21swapper/000:07:390
9120991716,0cyclictest0-21swapper/019:47:390
9120991715,1cyclictest5964-21sh20:10:000
9123991616,0cyclictest28115-21apt-get20:57:403
9123991616,0cyclictest0-21swapper/319:30:013
9123991615,0cyclictest0-21swapper/323:17:413
912399160,10cyclictest0-21swapper/322:47:403
9122991616,0cyclictest4886-21apt-get00:32:422
9122991616,0cyclictest0-21swapper/222:27:392
9121991616,0cyclictest0-21swapper/123:27:411
9121991616,0cyclictest0-21swapper/123:17:411
9121991616,0cyclictest0-21swapper/100:37:391
912199160,15cyclictest9984-21diskstats19:12:411
9120991616,0cyclictest0-21swapper/021:37:400
9120991613,2cyclictest14574-21awk22:40:000
912099160,15cyclictest17295-21grep19:27:400
912099160,15cyclictest0-21swapper/019:12:410
912099160,15cyclictest0-21swapper/000:02:470
9123991515,0cyclictest0-21swapper/322:12:393
912399150,1cyclictest0-21swapper/323:12:393
912399150,1cyclictest0-21swapper/322:57:443
9122991515,0cyclictest31811-21apt-get23:15:012
9122991515,0cyclictest17879-21apt-get23:52:412
9122991514,1cyclictest18451-21apt-get20:37:402
9121991515,0cyclictest25234-21apt-get00:07:371
9121991515,0cyclictest0-21swapper/121:27:401
9121991515,0cyclictest0-21swapper/120:37:411
9121991513,1cyclictest7335-21apt22:25:011
9120991515,0cyclictest0-21swapper/023:30:010
9120991515,0cyclictest0-21swapper/022:47:400
9120991515,0cyclictest0-21swapper/021:27:400
9120991515,0cyclictest0-21swapper/021:02:390
9120991512,2cyclictest24121-21irqstats20:47:430
912399140,14cyclictest0-21swapper/322:07:593
912399140,14cyclictest0-21swapper/321:10:553
912399140,14cyclictest0-21swapper/320:54:423
912399140,0cyclictest0-21swapper/320:30:133
9122991414,0cyclictest0-21swapper/223:11:382
912299140,14cyclictest0-21swapper/223:55:042
912299140,14cyclictest0-21swapper/221:32:452
912299140,14cyclictest0-21swapper/221:07:472
912299140,14cyclictest0-21swapper/200:10:252
9121991414,0cyclictest0-21swapper/123:42:221
9121991414,0cyclictest0-21swapper/123:10:271
9121991414,0cyclictest0-21swapper/120:00:101
9121991413,0cyclictest0-21swapper/122:12:391
912199140,0cyclictest0-21swapper/123:48:341
912199140,0cyclictest0-21swapper/123:20:491
912199140,0cyclictest0-21swapper/123:00:311
912199140,0cyclictest0-21swapper/123:00:301
912199140,0cyclictest0-21swapper/121:10:051
912199140,0cyclictest0-21swapper/120:56:301
912199140,0cyclictest0-21swapper/119:16:171
912099140,14cyclictest0-21swapper/022:25:470
912099140,0cyclictest0-21swapper/023:06:500
912099140,0cyclictest0-21swapper/019:55:500
9123991313,0cyclictest0-21swapper/320:25:083
9122991313,0cyclictest0-21swapper/223:25:092
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional