You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-11 - 20:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Thu Dec 11, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3228499250,2cyclictest9417-21ls07:30:003
3227699250,23cyclictest313-21systemd-journal08:39:591
3228199240,23cyclictest14493-21apt-get12:08:552
3227199240,23cyclictest23124-21apt-get10:13:490
3228499230,22cyclictest6683-21munin-run11:50:003
3228499230,22cyclictest6683-21munin-run11:50:003
3228499230,22cyclictest4648-21sendmail-msp07:20:003
32281992323,0cyclictest0-21swapper/211:28:502
32276992323,0cyclictest26243-21apt-get09:13:521
32276992321,1cyclictest24964-21apt-get08:03:501
32276992321,1cyclictest12037-21apt-get12:03:501
32276992320,2cyclictest1608-21sendmail08:20:001
3227699230,22cyclictest2426-21apt-get11:43:501
3227699230,1cyclictest461-21munin-run09:25:001
32271992322,0cyclictest0-21swapper/011:53:500
32271992321,2cyclictest31611-21perf08:15:000
3227199230,22cyclictest20873-21sh09:00:000
3227199230,22cyclictest0-21swapper/009:33:490
3227199230,1cyclictest13871-21cron12:05:010
32284992220,1cyclictest2909-21perf09:30:003
32284992220,1cyclictest0-21swapper/308:10:003
3228499220,21cyclictest5191-21apt-get07:23:523
3228499220,21cyclictest0-21swapper/307:13:503
3228499220,1cyclictest0-21swapper/308:25:003
3228199220,21cyclictest29757-21apt-get08:13:492
3228199220,21cyclictest18331-21apt-get10:03:512
3228199220,21cyclictest16078-21latency_hist08:50:002
3228199220,1cyclictest593-21dbus-daemon07:20:002
3228199220,1cyclictest24417-21awk08:00:002
3227699220,21cyclictest25525-21apt-get10:18:521
3227699220,21cyclictest22401-21apt-get11:18:511
3227699220,21cyclictest0-21swapper/109:08:511
3227699220,1cyclictest20150-21apt-get07:53:491
3227699220,1cyclictest17771-21awk10:00:001
3227699220,1cyclictest0-21swapper/109:45:011
3227699220,1cyclictest0-21swapper/108:15:001
3227199220,21cyclictest4002-21awk08:25:000
3227199220,21cyclictest30490-21grep09:20:000
3227199220,21cyclictest0-21swapper/011:15:010
3227199220,21cyclictest0-21swapper/010:18:510
3227199220,21cyclictest0-21swapper/007:48:510
3227199220,21cyclictest0-21swapper/007:38:500
32284992121,0cyclictest32157-21apt-get08:18:503
32284992120,0cyclictest0-21swapper/312:23:513
32284992120,0cyclictest0-21swapper/310:13:503
32284992119,1cyclictest313-21systemd-journal11:20:013
32284992118,2cyclictest593-21dbus-daemon10:45:003
3228499210,21cyclictest0-21swapper/308:33:493
3228499210,21cyclictest0-21swapper/308:30:003
3228499210,20cyclictest7944-21apt-get10:48:503
3228499210,20cyclictest0-21swapper/311:58:513
3228499210,20cyclictest0-21swapper/311:03:513
3228499210,20cyclictest0-21swapper/310:58:523
3228499210,20cyclictest0-21swapper/308:43:493
3228499210,0cyclictest10341-21apt-get10:53:513
3228499210,0cyclictest0-21swapper/312:03:513
3228499210,0cyclictest0-21swapper/310:18:523
3228499210,0cyclictest0-21swapper/310:03:513
32281992121,0cyclictest0-21swapper/211:18:502
32281992120,1cyclictest13888-21perf12:05:012
32281992119,1cyclictest5002-21grep10:40:002
32281992119,1cyclictest18491-21munin-plugin-st08:55:002
32281992119,1cyclictest14625-21sh11:00:002
3228199210,20cyclictest26492-21apt-get12:33:522
3228199210,20cyclictest12486-21apt07:38:502
3228199210,1cyclictest15211-21apt-get11:03:512
3227699210,21cyclictest0-21swapper/112:28:521
3227699210,0cyclictest3452-21apt-get09:33:491
3227699210,0cyclictest0-21swapper/111:23:501
3227699210,0cyclictest0-21swapper/111:03:511
3227699210,0cyclictest0-21swapper/107:18:531
32271992121,0cyclictest0-21swapper/011:23:500
32271992121,0cyclictest0-21swapper/009:38:490
32271992121,0cyclictest0-21swapper/008:45:010
32271992120,1cyclictest0-21swapper/011:08:510
32271992119,1cyclictest0-21swapper/010:25:000
3227199210,20cyclictest0-21swapper/009:58:500
3227199210,20cyclictest0-21swapper/009:48:510
3227199210,20cyclictest0-21swapper/008:53:490
3227199210,1cyclictest22547-21apt-get07:58:490
3227199210,0cyclictest0-21swapper/008:03:490
3227199210,0cyclictest0-21swapper/007:13:500
32284992020,0cyclictest0-21swapper/311:53:503
32284992020,0cyclictest0-21swapper/311:43:513
32284992020,0cyclictest0-21swapper/311:23:503
32284992020,0cyclictest0-21swapper/310:38:513
32284992020,0cyclictest0-21swapper/308:13:493
32284992020,0cyclictest0-21swapper/307:58:493
32284992018,1cyclictest30756-21basename12:40:003
32284992017,2cyclictest30476-21plymouth09:19:593
3228499200,20cyclictest0-21swapper/312:08:553
3228499200,20cyclictest0-21swapper/309:45:013
3228499200,1cyclictest29445-21turbostat11:30:003
3228499200,0cyclictest0-21swapper/312:33:523
3228499200,0cyclictest0-21swapper/311:33:503
3228499200,0cyclictest0-21swapper/307:53:493
3228499200,0cyclictest0-21swapper/307:33:513
32281992020,0cyclictest9986-21apt-get07:33:512
32281992020,0cyclictest9363-21apt-get08:38:472
32281992020,0cyclictest21445-21apt-get09:03:492
32281992020,0cyclictest0-21swapper/212:38:532
32281992020,0cyclictest0-21swapper/210:53:512
32281992020,0cyclictest0-21swapper/209:58:502
32281992020,0cyclictest0-21swapper/208:58:502
32281992020,0cyclictest0-21swapper/208:28:512
32281992020,0cyclictest0-21swapper/208:08:502
32281992020,0cyclictest0-21swapper/207:40:022
32281992019,1cyclictest7586-21apt-get07:28:492
32281992019,0cyclictest0-21swapper/210:08:502
32281992018,1cyclictest15356-21dump-pmu-power09:55:002
3228199200,20cyclictest0-21swapper/212:28:522
3228199200,20cyclictest0-21swapper/210:18:522
3228199200,0cyclictest5547-21apt-get10:43:502
3228199200,0cyclictest4824-21apt-get11:48:512
3228199200,0cyclictest4824-21apt-get11:48:512
3228199200,0cyclictest0-21swapper/210:48:502
3228199200,0cyclictest0-21swapper/210:33:502
32276992020,0cyclictest1047-21apt-get09:28:531
32276992020,0cyclictest0-21swapper/110:13:501
32276992020,0cyclictest0-21swapper/110:03:511
32276992020,0cyclictest0-21swapper/108:48:511
32276992020,0cyclictest0-21swapper/107:48:511
32276992020,0cyclictest0-21swapper/107:40:021
32276992019,1cyclictest11784-21apt-get08:43:481
32276992018,1cyclictest24411-21munin-run08:00:001
32276992018,1cyclictest16333-21latency_hist12:10:001
3227699200,0cyclictest0-21swapper/110:38:511
32271992020,0cyclictest0-21swapper/012:08:550
32271992020,0cyclictest0-21swapper/008:28:520
32271992020,0cyclictest0-21swapper/007:40:020
32271992019,1cyclictest0-21swapper/007:53:490
32271992019,0cyclictest0-21swapper/009:03:490
32271992018,1cyclictest0-21swapper/009:30:000
3227199200,20cyclictest0-21swapper/011:28:490
3227199200,0cyclictest8270-21apt-get09:43:500
3227199200,0cyclictest0-21swapper/012:28:520
3227199200,0cyclictest0-21swapper/012:18:530
3227199200,0cyclictest0-21swapper/011:43:500
3227199200,0cyclictest0-21swapper/011:18:510
3227199200,0cyclictest0-21swapper/010:58:520
32284991919,0cyclictest0-21swapper/312:28:523
32284991919,0cyclictest0-21swapper/312:13:503
32284991919,0cyclictest0-21swapper/309:53:503
32284991919,0cyclictest0-21swapper/309:33:503
32284991919,0cyclictest0-21swapper/308:48:513
32284991919,0cyclictest0-21swapper/308:38:483
32284991919,0cyclictest0-21swapper/308:03:483
32281991919,0cyclictest0-21swapper/210:23:502
32281991919,0cyclictest0-21swapper/208:43:492
32281991919,0cyclictest0-21swapper/208:33:502
32281991919,0cyclictest0-21swapper/208:03:492
3228199190,0cyclictest0-21swapper/207:23:522
32276991919,0cyclictest0-21swapper/112:23:521
32276991919,0cyclictest0-21swapper/112:13:511
32276991919,0cyclictest0-21swapper/111:38:501
32276991919,0cyclictest0-21swapper/109:43:501
32276991918,1cyclictest5852-21apt-get09:38:501
32276991918,1cyclictest12176-21idleruntime-cro10:55:001
32276991918,1cyclictest0-21swapper/107:23:521
32276991918,0cyclictest0-21swapper/107:38:501
32276991917,1cyclictest30522-21sendmail-msp09:20:001
32271991919,0cyclictest0-21swapper/011:38:510
32271991919,0cyclictest0-21swapper/010:28:500
32271991918,1cyclictest0-21swapper/010:03:510
3227199190,13cyclictest0-21swapper/007:28:490
32284991818,0cyclictest0-21swapper/309:13:523
32284991817,0cyclictest12487-21apt-get07:38:503
32281991818,0cyclictest17608-21apt-get11:08:512
32281991818,0cyclictest0-21swapper/211:58:512
32281991818,0cyclictest0-21swapper/209:38:502
32281991818,0cyclictest0-21swapper/209:28:542
32281991818,0cyclictest0-21swapper/209:23:502
32281991818,0cyclictest0-21swapper/209:13:512
32281991818,0cyclictest0-21swapper/208:23:502
32276991818,0cyclictest32744-21apt-get10:33:501
32276991818,0cyclictest0-21swapper/108:58:511
32276991818,0cyclictest0-21swapper/108:28:521
32276991817,1cyclictest18476-21perf08:55:001
32276991817,1cyclictest0-21swapper/111:28:501
32276991817,1cyclictest0-21swapper/110:10:001
32276991817,0cyclictest0-21swapper/107:33:511
32276991816,1cyclictest30748-21ls12:40:001
32271991818,0cyclictest0-21swapper/012:13:500
32271991818,0cyclictest0-21swapper/011:58:510
32271991818,0cyclictest0-21swapper/011:33:500
32271991818,0cyclictest0-21swapper/010:43:500
32271991816,2cyclictest1619-21grep08:20:000
32271991816,1cyclictest27361-21apt-get08:08:510
32271991816,1cyclictest15373-21grep09:55:000
32284991717,0cyclictest5852-21apt-get09:38:503
32284991717,0cyclictest0-21swapper/312:18:533
32284991717,0cyclictest0-21swapper/308:53:493
32281991717,0cyclictest0-21swapper/210:13:502
32281991716,0cyclictest0-21swapper/209:33:492
32281991715,1cyclictest1588-21sendmail08:20:002
32276991717,0cyclictest0-21swapper/112:33:521
32276991717,0cyclictest0-21swapper/111:53:501
32276991717,0cyclictest0-21swapper/111:33:501
32276991717,0cyclictest0-21swapper/108:33:501
32276991715,1cyclictest609-21polkitd11:50:001
32276991715,1cyclictest609-21polkitd11:50:001
32276991715,1cyclictest0-21swapper/111:09:591
3227699170,0cyclictest9618-21apt-get11:58:511
32271991717,0cyclictest0-21swapper/010:33:500
32271991715,1cyclictest16056-21cron08:50:000
32271991714,2cyclictest28086-21cron09:15:000
32284991616,0cyclictest0-21swapper/311:13:503
32284991616,0cyclictest0-21swapper/310:28:503
32284991616,0cyclictest0-21swapper/310:08:503
32284991615,1cyclictest23844-21apt-get09:08:503
32284991614,1cyclictest32715-21tail09:23:593
32281991616,0cyclictest0-21swapper/211:43:512
32281991615,1cyclictest19283-21apt-get12:18:522
32281991614,1cyclictest30627-21sed11:33:532
32276991616,0cyclictest0-21swapper/107:28:501
32276991615,1cyclictest4030-21awk08:25:001
32271991616,0cyclictest0-21swapper/012:23:510
32271991616,0cyclictest0-21swapper/008:38:480
32271991615,0cyclictest0-21swapper/010:35:020
32271991614,1cyclictest17060-21perf11:05:000
3227199160,1cyclictest0-21swapper/010:10:000
32284991515,0cyclictest14345-21apt-get07:40:023
32284991515,0cyclictest0-21swapper/310:33:493
32284991515,0cyclictest0-21swapper/309:03:493
32284991515,0cyclictest0-21swapper/308:58:513
3228499150,1cyclictest0-21swapper/310:20:013
3228499150,1cyclictest0-21swapper/307:48:563
3228499150,0cyclictest0-21swapper/309:58:503
32281991515,0cyclictest7222-21apt-get11:53:502
32281991515,0cyclictest0-21swapper/207:53:492
3228199150,14cyclictest24194-21expr09:08:522
3227699156,8cyclictest578-21irqbalance11:11:171
32276991515,0cyclictest27941-21apt-get10:23:501
32276991515,0cyclictest0-21swapper/112:18:531
32276991515,0cyclictest0-21swapper/110:48:501
32276991513,1cyclictest0-21swapper/110:40:571
32271991515,0cyclictest470-21perf09:25:000
32271991515,0cyclictest0-21swapper/011:48:510
32271991515,0cyclictest0-21swapper/011:48:500
32271991513,1cyclictest0-21swapper/012:38:550
3227199150,15cyclictest28203-21munin-node10:51:490
32284991414,0cyclictest0-21swapper/309:40:043
3228499140,14cyclictest0-21swapper/311:36:333
32281991414,0cyclictest21698-21apt-get12:23:522
32281991414,0cyclictest0-21swapper/211:37:252
32281991413,1cyclictest10113-21apt-get09:45:012
3228199140,14cyclictest0-21swapper/212:10:402
3228199140,14cyclictest0-21swapper/211:12:192
3228199140,14cyclictest0-21swapper/210:25:262
3228199140,14cyclictest0-21swapper/209:15:512
3228199140,14cyclictest0-21swapper/207:49:462
3228199140,14cyclictest0-21swapper/207:11:202
3228199140,0cyclictest0-21swapper/209:44:022
32276991414,0cyclictest0-21swapper/109:02:511
32276991412,1cyclictest13465-21apt-get09:53:501
3227699140,14cyclictest0-21swapper/108:07:171
3227699140,14cyclictest0-21swapper/107:11:351
3227699140,0cyclictest12739-21apt-get10:58:521
3227699140,0cyclictest0-21swapper/110:25:081
32271991414,0cyclictest0-21swapper/010:47:060
3227199140,14cyclictest0-21swapper/009:05:190
3227199140,14cyclictest0-21swapper/008:30:470
3227199140,14cyclictest0-21swapper/007:30:470
3227199140,14cyclictest0-21swapper/007:22:280
3227199140,0cyclictest0-21swapper/012:30:060
3227199140,0cyclictest0-21swapper/007:19:260
32281991313,0cyclictest0-21swapper/211:20:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional