You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-16 - 17:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Tue Dec 16, 2025 12:44:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11164992929,0cyclictest131ksoftirqd/010:35:000
11174992421,2cyclictest34-21ksoftirqd/312:30:013
11164992423,0cyclictest0-21swapper/011:34:330
11174992320,1cyclictest313-21systemd-journal09:35:013
1117499230,22cyclictest29430-21sendmail-msp10:00:003
1117499230,21cyclictest313-21systemd-journal11:30:003
1116999230,2cyclictest22552-21apt08:40:012
1116999230,22cyclictest13515-21wc11:40:002
1116699230,22cyclictest6993-21dump-pmu-power10:20:011
1116699230,1cyclictest11167-21ls12:40:011
1116699230,1cyclictest11167-21ls12:40:011
1116499231,21cyclictest0-21swapper/011:49:320
1116499230,22cyclictest13540-21sendmail-msp11:40:000
1116499230,0cyclictest0-21swapper/008:54:320
11174992220,1cyclictest18400-21ls07:25:003
11174992220,1cyclictest0-21swapper/312:10:013
1117499220,21cyclictest606-21dbus-daemon08:45:013
1117499220,21cyclictest26376-21munin-run12:05:013
1117499220,21cyclictest19529-21latency_hist10:45:003
1117499220,21cyclictest15953-21perf07:20:013
1117499220,21cyclictest15912-21apt-get08:29:313
1117499220,21cyclictest0-21swapper/311:59:333
1117499220,21cyclictest0-21swapper/311:24:323
1117499220,21cyclictest0-21swapper/311:09:333
1117499220,1cyclictest19404-21apt-get07:29:313
1117499220,1cyclictest0-21swapper/312:19:293
1116999220,21cyclictest25583-21apt-get08:45:022
1116999220,21cyclictest16561-21apt-get07:24:302
1116999220,21cyclictest0-21swapper/209:09:322
11166992221,1cyclictest16527-21apt-get11:49:321
11166992220,1cyclictest3925-21munin-run11:20:011
11166992220,1cyclictest31757-21apt-get12:19:291
1116699220,21cyclictest738-21apt-get09:04:331
1116699220,21cyclictest0-21swapper/108:44:311
1116699220,1cyclictest19375-21apt-get11:54:311
11164992220,1cyclictest29428-21apt10:00:000
11164992220,1cyclictest13-21ksoftirqd/012:00:010
11164992220,1cyclictest12151-21cron09:25:010
1116499221,20cyclictest22448-21kworker/0:212:16:200
1116499220,21cyclictest2527-21cron09:05:000
1116499220,21cyclictest0-21swapper/011:30:000
1116499220,21cyclictest0-21swapper/011:24:310
1116499220,1cyclictest18367-21apt-get08:34:310
11174992119,1cyclictest606-21dbus-daemon09:10:003
1117499210,20cyclictest30031-21apt-get10:04:313
1117499210,20cyclictest20760-21apt-get08:39:323
1117499210,20cyclictest0-21swapper/312:34:303
1117499210,20cyclictest0-21swapper/311:19:333
1117499210,20cyclictest0-21swapper/310:24:313
1117499210,20cyclictest0-21swapper/308:19:303
1117499210,1cyclictest34-21ksoftirqd/309:25:013
1117499210,1cyclictest22215-21latency_hist09:45:003
11169992120,1cyclictest6915-21apt-get11:29:322
11169992120,1cyclictest24301-21sh10:55:012
11169992120,1cyclictest18378-21apt-get11:50:012
1116999210,21cyclictest0-21swapper/211:54:312
1116999210,20cyclictest6305-21apt-get08:09:322
1116999210,20cyclictest17674-21apt-get10:44:312
1116999210,0cyclictest7617-21apt-get10:24:312
1116999210,0cyclictest22159-21apt-get11:59:322
11166992121,0cyclictest18393-21apt-get07:25:021
11166992121,0cyclictest0-21swapper/110:59:321
11166992120,1cyclictest4557-21apt-get12:29:321
11166992120,0cyclictest0-21swapper/109:55:001
11166992120,0cyclictest0-21swapper/109:24:321
11166992120,0cyclictest0-21swapper/108:14:311
11166992119,1cyclictest22576-21sendmail-msp08:40:011
11166992119,1cyclictest18420-21awk07:25:001
1116699210,21cyclictest0-21swapper/110:34:321
1116699210,0cyclictest0-21swapper/111:34:331
1116699210,0cyclictest0-21swapper/110:04:321
1116699210,0cyclictest0-21swapper/109:19:311
1116699210,0cyclictest0-21swapper/109:14:321
1116699210,0cyclictest0-21swapper/108:34:311
11164992121,0cyclictest0-21swapper/012:29:320
11164992120,0cyclictest0-21swapper/012:09:320
11164992119,1cyclictest29725-21apt-get11:09:330
11164992119,1cyclictest19531-21wc10:45:000
11164992119,1cyclictest0-21swapper/008:45:010
1116499211,20cyclictest22448-21kworker/0:210:04:520
1116499210,20cyclictest19416-21apt-get09:40:030
1116499210,20cyclictest0-21swapper/007:24:300
1116499210,0cyclictest14143-21apt-get07:19:310
1116499210,0cyclictest0-21swapper/011:14:320
1116499210,0cyclictest0-21swapper/007:44:310
1116499210,0cyclictest0-21swapper/007:29:310
11174992020,0cyclictest0-21swapper/311:34:333
11174992020,0cyclictest0-21swapper/307:59:323
11174992018,1cyclictest1798-21latency_hist10:10:013
11174992018,1cyclictest14130-21apt-get11:44:323
11174992017,1cyclictest0-21swapper/311:54:313
1117499200,20cyclictest0-21swapper/311:04:323
1117499200,20cyclictest0-21swapper/311:04:323
11169992020,0cyclictest29362-21apt-get12:14:322
11169992020,0cyclictest21802-21apt-get07:34:312
11169992020,0cyclictest2106-21apt-get11:19:332
11169992020,0cyclictest0-21swapper/211:04:322
11169992020,0cyclictest0-21swapper/211:04:322
11169992020,0cyclictest0-21swapper/210:10:022
11169992020,0cyclictest0-21swapper/209:54:302
11169992020,0cyclictest0-21swapper/209:49:312
11169992020,0cyclictest0-21swapper/207:59:322
11169992019,1cyclictest26731-21perf11:00:012
1116999200,20cyclictest0-21swapper/211:14:322
1116999200,20cyclictest0-21swapper/209:29:312
1116999200,0cyclictest20124-21apt-get10:49:322
1116999200,0cyclictest13518-21apt-get08:24:302
1116999200,0cyclictest0-21swapper/210:39:322
1116999200,0cyclictest0-21swapper/210:34:322
1116999200,0cyclictest0-21swapper/208:34:312
1116999200,0cyclictest0-21swapper/207:29:312
11166992020,0cyclictest24962-21apt-get08:45:021
11166992020,0cyclictest11746-21apt-get07:14:311
11166992020,0cyclictest0-21swapper/112:04:291
11166992020,0cyclictest0-21swapper/111:39:331
11166992020,0cyclictest0-21swapper/111:29:311
11166992020,0cyclictest0-21swapper/110:35:021
11166992020,0cyclictest0-21swapper/109:34:311
11166992020,0cyclictest0-21swapper/108:54:311
11166992019,1cyclictest1481-21apt-get07:59:321
11166992019,0cyclictest21777-21apt-get11:55:021
11166992019,0cyclictest0-21swapper/108:59:321
11166992018,1cyclictest28500-21idleruntime-cro07:45:011
11166992018,1cyclictest19401-21turbostat09:40:011
1116699200,0cyclictest0-21swapper/108:24:301
11164992020,0cyclictest0-21swapper/012:04:300
11164992020,0cyclictest0-21swapper/011:44:320
11164992020,0cyclictest0-21swapper/011:04:320
11164992020,0cyclictest0-21swapper/011:04:320
11164992020,0cyclictest0-21swapper/010:54:310
11164992019,1cyclictest8066-21cron08:10:010
1116499201,19cyclictest22448-21kworker/0:211:51:320
1116499200,0cyclictest0-21swapper/010:59:320
1116499200,0cyclictest0-21swapper/010:35:490
1116499200,0cyclictest0-21swapper/009:39:320
11174991919,0cyclictest0-21swapper/312:39:323
11174991919,0cyclictest0-21swapper/312:39:323
11174991919,0cyclictest0-21swapper/311:39:323
11174991919,0cyclictest0-21swapper/310:34:323
11174991919,0cyclictest0-21swapper/309:04:323
11174991918,1cyclictest11746-21apt-get07:14:313
11174991918,1cyclictest0-21swapper/310:39:323
11174991918,1cyclictest0-21swapper/308:04:303
11174991918,1cyclictest0-21swapper/307:54:303
11174991918,1cyclictest0-21swapper/307:54:303
11174991918,0cyclictest0-21swapper/310:54:313
11169991919,0cyclictest0-21swapper/212:39:322
11169991919,0cyclictest0-21swapper/212:39:322
11169991919,0cyclictest0-21swapper/212:19:292
11169991919,0cyclictest0-21swapper/208:44:312
11169991919,0cyclictest0-21swapper/207:14:312
1116999190,5cyclictest17701-21grep08:30:012
11166991919,0cyclictest0-21swapper/111:14:321
11166991919,0cyclictest0-21swapper/110:29:311
11166991919,0cyclictest0-21swapper/110:09:321
11166991919,0cyclictest0-21swapper/109:44:311
11166991918,1cyclictest0-21swapper/107:49:301
11166991917,1cyclictest4608-21munin-run10:15:011
11166991917,1cyclictest0-21swapper/112:25:001
1116499192,17cyclictest22448-21kworker/0:210:46:400
11164991919,0cyclictest0-21swapper/008:29:310
11164991918,1cyclictest32426-21apt-get10:09:320
11164991918,1cyclictest0-21swapper/011:19:330
11164991918,1cyclictest0-21swapper/007:39:300
11164991918,0cyclictest0-21swapper/010:29:310
1116499190,19cyclictest31423-21kworker/0:007:49:240
11174991818,0cyclictest0-21swapper/312:24:323
11174991818,0cyclictest0-21swapper/310:59:323
11174991818,0cyclictest0-21swapper/309:29:313
11174991818,0cyclictest0-21swapper/308:24:313
11174991817,1cyclictest34-21ksoftirqd/310:20:013
11174991817,1cyclictest0-21swapper/309:19:313
11169991818,0cyclictest30755-21apt-get08:59:312
11169991818,0cyclictest0-21swapper/212:09:322
11169991818,0cyclictest0-21swapper/209:34:312
11169991818,0cyclictest0-21swapper/208:54:322
11169991818,0cyclictest0-21swapper/207:19:312
11169991817,1cyclictest605-21avahi-daemon09:40:012
11169991816,1cyclictest26717-21apt-get07:44:312
1116999180,17cyclictest0-21swapper/210:29:312
11166991818,0cyclictest11097-21apt-get08:19:301
11166991818,0cyclictest0-21swapper/110:24:311
11166991816,1cyclictest606-21dbus-daemon12:35:011
11166991815,2cyclictest31511-21basename11:10:011
1116699180,17cyclictest0-21swapper/107:34:311
11164991818,0cyclictest6953-21apt-get12:34:300
11164991818,0cyclictest0-21swapper/010:24:310
11164991818,0cyclictest0-21swapper/008:39:320
11164991818,0cyclictest0-21swapper/008:19:300
11164991818,0cyclictest0-21swapper/007:34:310
11164991818,0cyclictest0-21swapper/007:14:310
11164991817,1cyclictest16952-21apt09:35:010
11174991717,0cyclictest0-21swapper/308:34:313
11174991717,0cyclictest0-21swapper/308:14:313
11174991717,0cyclictest0-21swapper/308:09:323
11174991714,2cyclictest7354-21latency_hist09:15:013
11169991717,0cyclictest24573-21apt-get12:04:292
11169991717,0cyclictest10368-21apt-get09:24:322
11169991717,0cyclictest0-21swapper/209:19:312
11169991716,1cyclictest5555-21apt-get09:14:322
11169991716,0cyclictest9316-21apt-get11:34:332
11169991715,1cyclictest2552-21cron09:05:002
11166991717,0cyclictest0-21swapper/111:44:311
11166991717,0cyclictest0-21swapper/109:49:311
11166991717,0cyclictest0-21swapper/108:05:011
11166991716,0cyclictest0-21swapper/112:14:321
1116699170,16cyclictest6177-21grep11:24:421
11164991717,0cyclictest0-21swapper/007:59:320
11164991716,1cyclictest0-21swapper/008:50:010
11164991715,1cyclictest1-21systemd09:10:000
11174991614,1cyclictest28361-21apt-get08:54:313
1117499160,15cyclictest866-21grep11:14:383
11169991616,0cyclictest0-21swapper/210:19:322
11169991615,1cyclictest4521-21apt-get11:24:312
11169991614,1cyclictest0-21swapper/212:24:362
11169991613,3cyclictest29426-21munin-run10:00:002
11166991616,0cyclictest27330-21apt-get11:04:321
11166991616,0cyclictest27330-21apt-get11:04:321
11166991616,0cyclictest0-21swapper/108:10:011
1116699160,0cyclictest0-21swapper/107:54:301
1116699160,0cyclictest0-21swapper/107:54:301
11164991616,0cyclictest9348-21apt-get12:39:320
11164991616,0cyclictest9348-21apt-get12:39:320
11164991614,1cyclictest0-21swapper/012:15:000
1116499160,0cyclictest0-21swapper/009:19:310
11174991515,0cyclictest0-21swapper/310:14:313
11174991514,0cyclictest0-21swapper/310:26:013
11174991513,1cyclictest31122-21cron12:15:003
11174991513,1cyclictest18358-21cron11:50:013
1117499150,1cyclictest0-21swapper/308:57:023
11169991515,0cyclictest31832-21perf10:05:012
11169991515,0cyclictest24301-21apt-get07:39:312
1116999150,14cyclictest0-21swapper/212:34:302
11166991515,0cyclictest0-21swapper/107:19:321
11166991514,1cyclictest29389-21cron10:00:001
1116499153,7cyclictest151rcu_preempt08:10:010
11164991515,0cyclictest0-21swapper/009:29:310
11164991515,0cyclictest0-21swapper/008:04:300
11164991515,0cyclictest0-21swapper/007:54:300
11164991515,0cyclictest0-21swapper/007:54:300
11174991414,0cyclictest0-21swapper/309:50:023
11174991414,0cyclictest0-21swapper/307:41:343
1117499140,14cyclictest0-21swapper/309:45:023
1117499140,14cyclictest0-21swapper/309:35:103
1117499140,14cyclictest0-21swapper/308:46:153
1117499140,14cyclictest0-21swapper/307:45:153
1117499140,14cyclictest0-21swapper/307:35:103
1117499140,14cyclictest0-21swapper/307:30:083
1117499140,0cyclictest0-21swapper/310:46:203
11169991414,0cyclictest0-21swapper/212:29:322
11169991414,0cyclictest0-21swapper/209:40:032
1116999140,14cyclictest0-21swapper/208:11:112
1116999140,14cyclictest0-21swapper/208:00:372
1116999140,14cyclictest0-21swapper/207:50:122
1116999140,14cyclictest0-21swapper/207:50:122
1116999140,14cyclictest0-21swapper/207:45:342
1116999140,0cyclictest0-21swapper/211:40:192
1116999140,0cyclictest0-21swapper/211:05:422
1116999140,0cyclictest0-21swapper/210:06:182
1116999140,0cyclictest0-21swapper/208:15:102
1116699140,14cyclictest0-21swapper/112:05:041
1116699140,14cyclictest0-21swapper/109:25:571
11164991414,0cyclictest0-21swapper/012:20:380
11164991414,0cyclictest0-21swapper/010:10:060
1116499140,14cyclictest0-21swapper/010:15:120
1116499140,14cyclictest0-21swapper/009:50:350
1116499140,14cyclictest0-21swapper/009:45:250
1116499140,14cyclictest0-21swapper/009:10:020
1116499140,14cyclictest0-21swapper/008:20:020
1116499140,0cyclictest0-21swapper/008:55:310
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional