You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 02:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Mon May 12, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1467499240,23cyclictest12268-21grep12:15:002
1467199240,23cyclictest21829-21cron07:25:001
14679992323,0cyclictest0-21swapper/311:45:303
14679992321,1cyclictest0-21swapper/311:40:003
14679992321,1cyclictest0-21swapper/309:10:293
14674992321,1cyclictest593-21dbus-daemon08:40:002
1467499230,22cyclictest24289-21munin-plugin-st12:40:002
1467499230,22cyclictest16980-21sendmail11:00:002
14671992323,0cyclictest18587-21apt-get11:10:261
1467199230,22cyclictest4625-21apt12:00:011
1467199230,22cyclictest24276-21perf12:40:001
1467199230,22cyclictest19172-21dump-pmu-power08:25:001
14669992321,1cyclictest0-21swapper/008:25:290
1466999230,22cyclictest25324-21apt-get11:35:300
1466999230,22cyclictest22785-21cron09:40:000
1467999220,21cyclictest0-21swapper/308:50:293
1467999220,1cyclictest22159-21apt-get12:35:293
14674992222,0cyclictest0-21swapper/210:30:262
14674992222,0cyclictest0-21swapper/209:35:312
14674992220,1cyclictest3083-21apt-get10:05:282
14674992220,1cyclictest27450-21perf11:40:002
14674992220,1cyclictest25202-21awk09:45:002
14674992220,1cyclictest18308-21perf11:10:002
1467499220,21cyclictest19150-21cron08:25:002
1467499220,21cyclictest0-21swapper/212:25:272
1467499220,1cyclictest2513-21apt-get11:55:312
1467499220,1cyclictest14981-21apt-get12:20:322
14671992220,1cyclictest313-21systemd-journal08:50:001
1467199220,21cyclictest7716-21apt-get12:05:301
1467199220,21cyclictest26667-21apt-get08:40:291
1467199220,21cyclictest20693-21apt-get09:35:301
1467199220,21cyclictest16733-21cron08:20:001
1467199220,21cyclictest12260-21perf12:15:001
1467199220,21cyclictest0-21swapper/110:05:291
1467199220,1cyclictest19470-21basename07:20:011
14669992222,0cyclictest0-21swapper/007:35:300
1466999220,21cyclictest4625-21apt-get12:00:010
1466999220,21cyclictest0-21swapper/008:10:320
14679992121,0cyclictest0-21swapper/308:55:313
14679992120,0cyclictest0-21swapper/309:45:293
14679992119,1cyclictest29169-21idleruntime-cro07:40:003
14679992118,2cyclictest23957-21latency_hist08:35:003
1467999210,20cyclictest0-21swapper/307:25:283
1467999210,0cyclictest0-21swapper/311:10:263
1467999210,0cyclictest0-21swapper/309:25:303
14674992120,0cyclictest1929-21apt-get07:50:022
14674992120,0cyclictest0-21swapper/207:40:302
1467499210,20cyclictest18299-21apt-get09:30:312
1467499210,20cyclictest0-21swapper/211:40:322
1467499210,1cyclictest12260-21apt-get08:10:322
1467499210,0cyclictest0-21swapper/211:10:262
14671992121,0cyclictest13507-21apt-get09:20:311
14671992120,1cyclictest31453-21apt-get08:50:291
14671992119,1cyclictest16998-21tr11:00:001
1467199210,21cyclictest0-21swapper/108:00:271
1467199210,20cyclictest0-21swapper/111:45:301
1467199210,20cyclictest0-21swapper/107:45:291
1467199210,20cyclictest0-21swapper/107:45:281
1467199210,1cyclictest21997-21perf11:25:001
14669992121,0cyclictest14651-21apt-get08:15:290
14669992120,1cyclictest0-21swapper/010:50:260
14669992119,1cyclictest3529-21sendmail09:00:000
14669992119,1cyclictest0-21swapper/011:45:000
14669992118,2cyclictest7569-21cron10:15:000
1466999210,21cyclictest0-21swapper/012:10:320
1466999210,21cyclictest0-21swapper/011:10:260
1466999210,20cyclictest17367-21apt-get12:25:280
1466999210,20cyclictest0-21swapper/008:30:310
1466999210,0cyclictest0-21swapper/009:00:290
14679992020,0cyclictest29063-21apt-get08:45:313
14679992020,0cyclictest0-21swapper/310:10:303
14679992020,0cyclictest0-21swapper/309:00:283
14679992020,0cyclictest0-21swapper/308:40:303
14679992020,0cyclictest0-21swapper/308:10:313
14679992020,0cyclictest0-21swapper/308:05:303
14679992020,0cyclictest0-21swapper/308:00:273
14679992019,1cyclictest15621-21perf09:25:003
1467999200,0cyclictest5006-21apt-get07:55:303
1467999200,0cyclictest0-21swapper/308:25:303
14674992020,0cyclictest0-21swapper/212:00:312
14674992020,0cyclictest0-21swapper/209:25:292
14674992020,0cyclictest0-21swapper/209:15:292
14674992020,0cyclictest0-21swapper/207:10:282
14674992019,0cyclictest22139-21apt-get07:25:292
14674992018,1cyclictest0-21swapper/212:35:002
1467499200,14cyclictest0-21swapper/211:30:262
1467499200,0cyclictest0-21swapper/210:00:282
1467499200,0cyclictest0-21swapper/209:45:292
1467499200,0cyclictest0-21swapper/209:05:292
14671992020,0cyclictest0-21swapper/110:20:001
14671992020,0cyclictest0-21swapper/109:50:321
14671992020,0cyclictest0-21swapper/108:10:321
14671992020,0cyclictest0-21swapper/107:50:011
14671992017,2cyclictest26377-21sendmail-msp08:40:001
1467199200,20cyclictest0-21swapper/111:40:331
1467199200,0cyclictest25499-21apt-get09:45:281
1467199200,0cyclictest0-21swapper/112:25:281
1467199200,0cyclictest0-21swapper/110:10:301
1467199200,0cyclictest0-21swapper/109:30:311
14669992020,0cyclictest0-21swapper/012:00:310
14669992020,0cyclictest0-21swapper/010:32:310
14669992020,0cyclictest0-21swapper/009:10:290
14669992020,0cyclictest0-21swapper/008:45:310
14669992019,1cyclictest23107-21apt-get09:40:300
1466999200,20cyclictest0-21swapper/007:25:290
1466999200,0cyclictest0-21swapper/007:50:020
1466999200,0cyclictest0-21swapper/007:20:300
14679991919,0cyclictest29448-21apt-get07:40:303
14679991919,0cyclictest0-21swapper/309:05:283
14679991918,1cyclictest4725-21munin-run07:55:003
1467999190,0cyclictest0-21swapper/308:15:293
14674991919,0cyclictest0-21swapper/212:15:302
14674991919,0cyclictest0-21swapper/208:45:312
14674991919,0cyclictest0-21swapper/207:55:292
14674991918,1cyclictest21863-21apt-get08:30:322
14674991917,1cyclictest17330-21apt-get07:15:292
1467499190,19cyclictest0-21swapper/212:05:302
1467499190,19cyclictest0-21swapper/207:30:272
14671991919,0cyclictest0-21swapper/112:30:321
14671991919,0cyclictest0-21swapper/112:19:591
14671991919,0cyclictest0-21swapper/112:00:311
14671991919,0cyclictest0-21swapper/111:50:291
14671991919,0cyclictest0-21swapper/110:32:321
14671991919,0cyclictest0-21swapper/109:55:271
14671991919,0cyclictest0-21swapper/109:40:301
14671991919,0cyclictest0-21swapper/108:55:311
14671991918,0cyclictest0-21swapper/108:05:301
14669991919,0cyclictest12552-21apt-get12:15:300
14669991919,0cyclictest0-21swapper/011:50:290
14669991919,0cyclictest0-21swapper/010:00:300
14669991919,0cyclictest0-21swapper/008:50:280
14669991919,0cyclictest0-21swapper/008:35:320
14669991919,0cyclictest0-21swapper/008:00:270
14669991919,0cyclictest0-21swapper/007:40:290
14669991918,0cyclictest11034-21apt-get09:15:290
14669991917,1cyclictest0-21swapper/007:15:010
14679991818,0cyclictest20693-21apt-get09:35:313
14679991818,0cyclictest0-21swapper/312:15:303
14679991818,0cyclictest0-21swapper/310:50:263
14679991818,0cyclictest0-21swapper/310:30:263
14679991818,0cyclictest0-21swapper/307:34:593
14679991817,0cyclictest0-21swapper/309:50:323
14679991814,3cyclictest313-21systemd-journal12:00:013
14674991818,0cyclictest31548-21apt-get07:45:022
14674991818,0cyclictest31548-21apt-get07:45:022
14674991818,0cyclictest0-21swapper/209:00:292
14674991818,0cyclictest0-21swapper/208:15:292
14674991817,1cyclictest9812-21apt-get08:05:302
14674991817,1cyclictest0-21swapper/211:45:302
14671991816,1cyclictest0-21swapper/107:45:001
14669991818,0cyclictest0-21swapper/011:45:300
14669991818,0cyclictest0-21swapper/009:55:270
14669991818,0cyclictest0-21swapper/009:45:290
14669991818,0cyclictest0-21swapper/009:20:310
14669991818,0cyclictest0-21swapper/009:05:290
14669991816,1cyclictest19447-21grep07:20:010
14669991816,1cyclictest11951-21apt08:10:000
14679991717,0cyclictest0-21swapper/312:05:303
14679991717,0cyclictest0-21swapper/311:45:003
14679991717,0cyclictest0-21swapper/310:05:293
14679991716,1cyclictest0-21swapper/310:40:003
14679991716,0cyclictest10110-21apt-get12:10:323
1467999170,16cyclictest24964-21munin-node11:22:493
1467999170,0cyclictest0-21swapper/309:55:013
14674991717,0cyclictest0-21swapper/208:00:262
14674991717,0cyclictest0-21swapper/207:35:302
14674991717,0cyclictest0-21swapper/207:20:292
14674991716,0cyclictest0-21swapper/211:50:302
14671991717,0cyclictest24529-21apt-get07:30:271
14671991717,0cyclictest0-21swapper/111:30:251
14671991717,0cyclictest0-21swapper/107:55:301
14669991717,0cyclictest0-21swapper/009:30:300
14669991715,1cyclictest5200-21munin-run10:10:000
14669991715,1cyclictest22639-21perf11:30:000
14679991616,0cyclictest0-21swapper/309:40:303
14679991616,0cyclictest0-21swapper/309:30:313
14679991614,1cyclictest18338-21latency_hist11:10:003
14679991614,1cyclictest13129-21chrt09:19:363
1467999160,15cyclictest0-21swapper/307:20:303
14674991616,0cyclictest27894-21apt-get09:50:322
14674991614,1cyclictest19888-21diskstats08:25:312
14674991614,1cyclictest0-21swapper/209:10:352
14674991613,1cyclictest0-21swapper/210:10:302
1467499160,15cyclictest27275-21sed08:40:322
14671991616,0cyclictest0-21swapper/109:15:291
14671991615,1cyclictest2797-21dump-pmu-power10:05:001
14671991613,2cyclictest9563-21cat10:30:011
1467199160,15cyclictest8335-21sh09:10:001
1467199160,15cyclictest0-21swapper/111:37:341
14669991616,0cyclictest0-21swapper/012:05:290
14669991616,0cyclictest0-21swapper/009:25:300
14669991616,0cyclictest0-21swapper/008:40:300
14669991614,1cyclictest0-21swapper/008:20:360
14669991613,2cyclictest30051-21latency_hist09:55:000
1466999160,1cyclictest18308-21perf11:10:010
1466999160,15cyclictest0-21swapper/011:30:300
1467999153,9cyclictest21450irq/143-nvme0q407:45:383
1467999153,9cyclictest21450irq/143-nvme0q407:45:383
14679991515,0cyclictest0-21swapper/308:20:323
14679991515,0cyclictest0-21swapper/307:10:283
14679991514,1cyclictest24964-21munin-node11:59:433
1467999150,14cyclictest129852sleep310:41:043
1467499155,10cyclictest24964-21munin-node10:44:582
14674991514,1cyclictest13964-21perf10:50:002
14674991513,1cyclictest32757-21cron09:59:592
14674991513,1cyclictest0-21swapper/210:25:012
1467199157,7cyclictest151591phc2sys07:35:321
14671991513,1cyclictest17667-21turbostat11:05:001
1467199150,1cyclictest0-21swapper/111:20:011
1467199150,1cyclictest0-21swapper/107:10:351
14669991515,0cyclictest0-21swapper/012:30:310
14669991515,0cyclictest0-21swapper/007:55:300
14669991514,0cyclictest32233-21apt-get07:45:290
14669991514,0cyclictest32233-21apt-get07:45:290
1466999150,14cyclictest0-21swapper/007:30:270
14679991414,0cyclictest0-21swapper/312:25:333
14679991414,0cyclictest0-21swapper/311:00:313
14679991414,0cyclictest0-21swapper/310:55:223
14679991414,0cyclictest0-21swapper/310:22:463
14679991413,1cyclictest0-21swapper/311:50:233
1467999140,14cyclictest0-21swapper/312:32:003
1467999140,14cyclictest0-21swapper/311:30:103
1467999140,14cyclictest0-21swapper/311:25:053
1467999140,14cyclictest0-21swapper/311:15:163
1467999140,14cyclictest0-21swapper/310:47:053
1467999140,0cyclictest0-21swapper/312:20:163
1467999140,0cyclictest0-21swapper/310:25:203
1467999140,0cyclictest0-21swapper/310:18:513
1467999140,0cyclictest0-21swapper/310:00:063
1467999140,0cyclictest0-21swapper/308:35:313
14674991414,0cyclictest0-21swapper/210:25:162
1467499140,14cyclictest0-21swapper/211:20:052
1467499140,14cyclictest0-21swapper/211:16:022
1467499140,14cyclictest0-21swapper/210:54:302
1467499140,14cyclictest0-21swapper/210:35:022
1467499140,14cyclictest0-21swapper/210:16:522
1467499140,14cyclictest0-21swapper/209:20:042
1467499140,14cyclictest0-21swapper/208:50:262
1467499140,0cyclictest0-21swapper/211:25:412
1467499140,0cyclictest0-21swapper/211:00:232
1467499140,0cyclictest0-21swapper/208:55:102
14671991414,0cyclictest0-21swapper/112:20:371
14671991414,0cyclictest0-21swapper/111:05:351
14671991411,2cyclictest23424-21/usr/sbin/munin07:25:381
1467199140,14cyclictest0-21swapper/110:50:591
1467199140,14cyclictest0-21swapper/110:40:291
1467199140,14cyclictest0-21swapper/110:36:001
1467199140,14cyclictest0-21swapper/110:21:211
1467199140,14cyclictest0-21swapper/109:25:221
1467199140,14cyclictest0-21swapper/109:00:331
1467199140,14cyclictest0-21swapper/108:25:451
1467199140,0cyclictest0-21swapper/111:25:401
1467199140,0cyclictest0-21swapper/108:30:021
14669991414,0cyclictest0-21swapper/012:20:060
14669991414,0cyclictest0-21swapper/011:15:340
14669991414,0cyclictest0-21swapper/010:25:250
14669991414,0cyclictest0-21swapper/010:22:060
1466999140,14cyclictest0-21swapper/011:20:200
1466999140,14cyclictest0-21swapper/011:01:470
1466999140,14cyclictest0-21swapper/010:45:510
1466999140,14cyclictest0-21swapper/010:40:330
1466999140,14cyclictest0-21swapper/010:15:140
1466999140,0cyclictest0-21swapper/012:36:050
1466999140,0cyclictest0-21swapper/010:55:230
1466999140,0cyclictest0-21swapper/010:38:140
14679991313,0cyclictest0-21swapper/307:15:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional