You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-26 - 20:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue May 26, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6656995437,11cyclictest35966irq/58-eth012:25:113
662899533,39cyclictest28883-21ssh09:30:130
664599529,18cyclictest27-21ksoftirqd/212:35:142
6656994920,20cyclictest35966irq/58-eth010:05:133
662899480,38cyclictest8425-21ssh11:25:130
6656994738,5cyclictest35966irq/58-eth011:05:133
665699466,29cyclictest35966irq/58-eth012:35:133
6645994527,11cyclictest24781-21diskmemload11:10:122
664599451,35cyclictest19915-21ssh09:25:142
662899451,35cyclictest25191-21ssh09:10:140
6656994310,23cyclictest35966irq/58-eth009:35:123
664599411,17cyclictest0-21swapper/209:05:112
6628994120,4cyclictest12-21ksoftirqd/010:45:140
662899403,19cyclictest131rcu_preempt11:35:130
664599382,18cyclictest11894-21sshd11:45:142
664599382,18cyclictest11894-21sshd11:45:132
665699374,23cyclictest35966irq/58-eth011:40:123
665699372,22cyclictest35966irq/58-eth009:55:133
665699372,22cyclictest35966irq/58-eth007:15:113
665699365,16cyclictest0-21swapper/310:40:133
665699353,29cyclictest35966irq/58-eth008:10:153
663699351,31cyclictest10876-21ssh09:20:131
6656993417,12cyclictest35966irq/58-eth011:15:143
6656993417,11cyclictest35966irq/58-eth011:25:133
6645993410,17cyclictest6620-21cyclictest08:25:122
665699332,23cyclictest35966irq/58-eth009:25:143
664599330,22cyclictest32401-21ssh09:50:112
664599330,22cyclictest32401-21ssh09:50:112
665699323,21cyclictest35966irq/58-eth011:45:133
665699323,21cyclictest35966irq/58-eth011:45:133
665699322,18cyclictest35966irq/58-eth012:05:143
665699313,20cyclictest35966irq/58-eth010:50:143
6645993119,6cyclictest27-21ksoftirqd/212:20:132
665699305,19cyclictest35966irq/58-eth012:15:133
665699303,17cyclictest35966irq/58-eth009:45:113
665699303,17cyclictest35966irq/58-eth009:45:113
6656993015,10cyclictest35966irq/58-eth011:30:133
6656993012,12cyclictest35966irq/58-eth012:00:133
663699303,23cyclictest14409-21ssh09:40:121
663699302,23cyclictest1855-21ssh09:15:141
6636993017,9cyclictest21-21ksoftirqd/109:00:131
662899301,21cyclictest779-21ssh12:15:130
665699299,12cyclictest35966irq/58-eth010:45:143
6656992912,7cyclictest0-21swapper/307:35:123
663699292,23cyclictest24329-21sshd12:10:151
663699287,14cyclictest11869-21ssh11:45:131
663699287,14cyclictest11869-21ssh11:45:121
663699285,20cyclictest17980-21ssh10:00:121
663699284,20cyclictest201rcuc/110:10:121
6636992822,3cyclictest32420-21sh09:50:131
6636992822,3cyclictest32420-21sh09:50:121
6636992810,13cyclictest29861-21diskmemload11:55:131
663699277,16cyclictest7161-21ssh10:30:131
6628992714,8cyclictest0-21swapper/011:30:120
662899271,20cyclictest9762-21ssh12:20:140
6656992613,8cyclictest35966irq/58-eth012:30:143
664599268,15cyclictest0-21swapper/210:55:012
664599262,17cyclictest0-21swapper/210:35:152
664599261,16cyclictest9015-21ssh09:55:122
6645992610,8cyclictest0-21swapper/211:05:132
662899261,18cyclictest22931-21ssh11:15:140
665699256,13cyclictest35966irq/58-eth009:15:143
6656992511,9cyclictest35966irq/58-eth011:50:143
664599251,21cyclictest3339-21ls10:10:032
664599250,18cyclictest17652-21cpuspeed07:55:132
664599250,18cyclictest17652-21cpuspeed07:55:132
663699256,15cyclictest21524-21ssh10:20:131
663699256,15cyclictest21524-21ssh10:20:131
663699254,18cyclictest5086-21sh11:05:141
663699253,16cyclictest1738-21ssh10:45:141
6636992513,9cyclictest31483-21cron11:20:021
663699251,18cyclictest5479-21sh09:35:141
663699251,15cyclictest31180-21usage_EthcTask11:35:311
6636992511,11cyclictest18730-21sh12:25:111
662899251,12cyclictest1109-21tune2fs08:45:150
6656992413,8cyclictest321rcuc/309:50:113
6656992413,8cyclictest321rcuc/309:50:113
664599246,7cyclictest18729-21diskmemload12:25:112
6636992414,7cyclictest16993-21cron11:30:011
662899241,18cyclictest16249-21ssh10:35:150
665699233,12cyclictest0-21swapper/309:20:133
665699233,11cyclictest0-21swapper/310:10:133
664599231,8cyclictest0-21swapper/208:45:132
6645992312,9cyclictest23082-21ssh10:00:502
6645992311,9cyclictest1985-21ssh11:01:422
663699232,6cyclictest14144-21memory10:50:231
6636992313,7cyclictest28256-21cron08:40:011
663699231,18cyclictest17392-21ssh11:30:131
663699231,16cyclictest6422-21sshd12:00:141
662899231,15cyclictest20855-21ssh11:50:140
665699229,9cyclictest33-21ksoftirqd/312:10:133
665699228,8cyclictest0-21swapper/311:35:143
665699223,8cyclictest0-21swapper/309:10:143
665699223,15cyclictest0-21swapper/309:40:123
665699222,12cyclictest35966irq/58-eth011:55:133
664599227,12cyclictest22292-21cut11:50:212
664599222,11cyclictest32523-21kthreadcore11:55:212
664599221,8cyclictest22918-21rm11:15:132
664599221,6cyclictest27-21ksoftirqd/211:30:122
664599221,4cyclictest0-21swapper/209:13:262
664599221,4cyclictest0-21swapper/209:05:022
6645992213,6cyclictest0-21swapper/209:35:182
6645992211,9cyclictest22270-21ssh12:08:032
664599221,17cyclictest0-21swapper/207:30:142
664599221,13cyclictest0-21swapper/210:20:212
664599221,13cyclictest0-21swapper/210:20:212
664599221,12cyclictest26635-21kthreadcore09:45:212
664599221,12cyclictest26635-21kthreadcore09:45:212
6645992210,5cyclictest9016-21ssh11:42:022
6645992210,4cyclictest4293-21kthreadcore10:10:202
663699228,4cyclictest131rcu_preempt09:55:161
663699223,8cyclictest131rcu_preempt08:15:121
663699222,15cyclictest2927-21sh11:40:141
663699221,9cyclictest32070-21fschecks_count11:20:151
663699221,17cyclictest21-21ksoftirqd/107:40:131
6628992215,5cyclictest12-21ksoftirqd/009:35:130
662899221,17cyclictest31910-21ssh11:20:130
662899221,16cyclictest32127-21ls11:00:240
665699212,15cyclictest35966irq/58-eth009:00:133
665699212,13cyclictest35966irq/58-eth010:20:133
665699212,13cyclictest35966irq/58-eth010:20:133
6656992110,7cyclictest35966irq/58-eth011:00:143
664599219,5cyclictest24709-21grep12:10:192
664599215,3cyclictest131rcu_preempt08:05:192
664599214,9cyclictest607-21klogd11:25:012
664599211,6cyclictest9428-21kthreadcore11:25:192
664599211,5cyclictest15236-21latency_hist09:00:012
664599211,4cyclictest2775-21kthreadcore12:15:212
6645992112,3cyclictest0-21swapper/208:35:262
664599211,17cyclictest30460-21kthreadcore08:40:202
6645992110,9cyclictest19350-21ssh10:18:002
6645992110,9cyclictest1552-21ssh11:38:522
663699215,13cyclictest21-21ksoftirqd/108:50:121
663699212,17cyclictest25164-21ssh10:40:131
6636992112,6cyclictest24284-21sshd10:39:291
663699211,17cyclictest7836-21ssh12:35:261
662899212,2cyclictest0-21swapper/009:25:140
662899211,5cyclictest24782-21idleruntime-cro09:10:010
6628992113,6cyclictest0-21swapper/007:40:140
6628992112,6cyclictest0-21swapper/008:05:310
662899211,15cyclictest4487-21find08:15:110
662899211,15cyclictest11893-21ssh11:45:130
662899211,15cyclictest11893-21ssh11:45:130
665699207,8cyclictest33-21ksoftirqd/311:23:503
664599208,9cyclictest23929-21kthreadcore07:25:222
664599208,9cyclictest23929-21kthreadcore07:25:212
664599205,13cyclictest11833-21grep07:45:292
664599201,9cyclictest5769-21threads09:15:292
664599201,9cyclictest25555-21irqcore10:40:182
664599201,7cyclictest1885866ssh09:40:302
664599201,4cyclictest131rcu_preempt12:00:222
664599201,4cyclictest10417-21sed07:10:262
664599201,11cyclictest9813-21sed08:20:152
664599201,10cyclictest30948-21grep07:35:152
664599201,10cyclictest28936-21kthreadcore12:30:192
664599201,10cyclictest21992-21kthreadcore10:55:202
663699208,9cyclictest27737-21apt-get12:30:131
663699206,8cyclictest131rcu_preempt08:40:121
663699206,11cyclictest23657-21rm09:45:161
663699206,11cyclictest23657-21rm09:45:151
6636992013,4cyclictest31142-21uptime10:05:311
6636992011,7cyclictest12733-21fw_packets10:15:141
662899209,7cyclictest18700-21apt-get12:25:100
662899201,2cyclictest0-21swapper/011:05:250
6628992011,6cyclictest30578-21cron07:35:010
662899201,15cyclictest30601-21ssh10:05:260
662899201,15cyclictest27762-21rm12:30:140
662899201,15cyclictest12265-21latency_hist10:15:020
662899201,10cyclictest9006-21dpkg09:55:110
6628992010,7cyclictest12-21ksoftirqd/009:15:140
665699198,8cyclictest0-21swapper/310:59:183
665699193,13cyclictest0-21swapper/310:30:133
665699192,11cyclictest0-21swapper/307:50:133
665699191,13cyclictest0-21swapper/308:35:293
665699191,13cyclictest0-21swapper/308:30:223
665699191,12cyclictest0-21swapper/308:15:153
6656991910,7cyclictest0-21swapper/308:05:173
6656991910,6cyclictest35966irq/58-eth011:10:133
6656991910,6cyclictest35966irq/58-eth010:00:133
664599199,7cyclictest0-21swapper/210:30:192
664599197,10cyclictest12781-21kthreadcore07:15:182
664599193,14cyclictest24065-21kthreadcore08:00:202
664599191,9cyclictest32284-21fschecks_time08:10:172
664599191,9cyclictest2234-21sed10:45:192
664599191,9cyclictest19434-21kthreadcore07:20:222
664599191,6cyclictest24781-21diskmemload10:26:442
664599191,3cyclictest6620-21cyclictest08:50:132
664599191,3cyclictest0-21swapper/209:30:142
664599191,15cyclictest16413-21wc07:50:262
664599191,10cyclictest7012-21kthreadcore08:15:202
664599191,10cyclictest18900-21apt-get08:30:132
663699198,3cyclictest21-21ksoftirqd/111:00:161
663699192,15cyclictest21-21ksoftirqd/108:00:121
663699191,16cyclictest10038-21ls07:10:211
663699191,14cyclictest28892-21ssh09:30:141
663699191,14cyclictest15390-21sh12:05:151
663699191,13cyclictest24787-21latency_hist09:10:011
663699191,13cyclictest20845-21rm11:50:131
6636991910,7cyclictest24151-21cat09:25:291
6636991910,6cyclictest135-21systemd-journal12:20:071
662899199,8cyclictest21428-21perl07:25:160
662899199,8cyclictest21428-21perl07:25:160
662899194,12cyclictest22095-21kthreadcore10:55:200
662899191,9cyclictest16505-21cat07:50:270
6628991913,4cyclictest0-21swapper/008:55:150
6628991912,4cyclictest0-21swapper/011:10:190
662899191,16cyclictest0-21swapper/009:20:120
662899191,15cyclictest30577-21ssh10:25:120
6628991910,6cyclictest5536-21kthreadcore12:35:200
6628991910,6cyclictest0-21swapper/008:50:140
665699182,10cyclictest0-21swapper/308:45:133
665699182,10cyclictest0-21swapper/308:00:053
665699182,10cyclictest0-21swapper/307:45:143
665699181,10cyclictest0-21swapper/312:20:143
6656991810,6cyclictest0-21swapper/309:30:513
664599183,7cyclictest27-21ksoftirqd/209:20:192
664599181,14cyclictest3552-21irqcore07:40:162
663699189,7cyclictest1952-21tr08:10:221
663699183,3cyclictest131rcu_preempt08:05:161
663699181,9cyclictest10319-21kthreadcore08:20:171
6636991811,4cyclictest14305-21cut07:15:201
663699181,13cyclictest21051-21ntp_states07:55:251
663699181,13cyclictest21051-21ntp_states07:55:251
663699181,13cyclictest12969-21cut07:50:151
6636991810,5cyclictest1528-21kthreadcore12:15:191
662899189,6cyclictest18992-21ntpq12:05:260
662899185,10cyclictest22254-21apt-key08:00:020
662899185,10cyclictest22254-21apt-key08:00:020
662899182,13cyclictest29862-21ssh11:55:130
662899181,9cyclictest11754-21fschecks_time07:15:150
662899181,8cyclictest21474-21/usr/sbin/munin10:20:110
662899181,8cyclictest21474-21/usr/sbin/munin10:20:110
662899181,3cyclictest210111kthreadcore10:00:210
6628991811,5cyclictest0-21swapper/009:00:270
6628991811,5cyclictest0-21swapper/007:10:270
6628991811,4cyclictest0-21swapper/008:20:140
662899181,13cyclictest6396-21apt-get12:00:120
6628991810,6cyclictest24749-21cron10:40:020
6628991810,5cyclictest27793-21sh09:45:290
6628991810,5cyclictest27793-21sh09:45:290
6628991810,5cyclictest0-21swapper/012:10:180
665699179,6cyclictest0-21swapper/310:35:083
665699172,10cyclictest0-21swapper/310:25:053
665699172,10cyclictest0-21swapper/307:40:123
665699171,13cyclictest0-21swapper/307:20:143
665699171,10cyclictest0-21swapper/308:55:203
663699178,6cyclictest23474-21cron08:35:011
663699177,7cyclictest2970-21cron07:40:011
663699177,6cyclictest630-21Async09:36:311
663699175,3cyclictest131rcu_preempt11:12:471
663699172,8cyclictest0-21swapper/108:55:111
663699172,12cyclictest21877-21uname07:25:191
663699172,12cyclictest21877-21uname07:25:191
663699171,13cyclictest31567-21kthreadcore10:25:181
663699171,11cyclictest18675-21grep08:30:011
663699171,10cyclictest25576-21sed09:10:181
663699170,13cyclictest19572-21apt-get10:55:111
663699170,11cyclictest11362-21ls07:45:231
662899178,7cyclictest32288-21fw_packets08:10:170
662899171,8cyclictest16531-21kthreadcore09:40:200
6628991711,4cyclictest10645-21/usr/sbin/munin10:50:150
662899171,13cyclictest32533-21ssh09:50:140
662899171,13cyclictest32533-21ssh09:50:140
662899171,13cyclictest22487-21apt-get08:00:130
662899171,13cyclictest20995-21apt07:25:010
662899171,12cyclictest3607-21cut11:40:180
6628991710,5cyclictest9171-21kthreadcore10:30:220
6628991710,4cyclictest14447-21kthreadcore10:15:200
662899170,14cyclictest31162-21tr08:40:210
665699168,6cyclictest33-21ksoftirqd/310:15:063
665699162,9cyclictest0-21swapper/308:50:133
665699162,9cyclictest0-21swapper/308:40:143
664599161,6cyclictest6722-21latency_hist07:10:012
663699167,6cyclictest26019-21sort07:30:131
663699160,5cyclictest16411-21acpi07:20:121
662899169,4cyclictest0-21swapper/008:30:270
662899168,5cyclictest10958-21cut07:45:220
662899162,9cyclictest0-21swapper/008:25:140
662899161,12cyclictest23709-21apt-get08:35:140
665699151,8cyclictest0-21swapper/308:20:133
665699151,11cyclictest0-21swapper/307:30:143
665699151,11cyclictest0-21swapper/307:25:183
665699151,11cyclictest0-21swapper/307:25:183
6656991510,3cyclictest0-21swapper/309:10:013
662899151,11cyclictest30966-21fschecks_count07:35:150
665699141,7cyclictest0-21swapper/307:55:123
665699141,7cyclictest0-21swapper/307:55:123
663699141,10cyclictest6741-21latency_hist07:10:011
665699137,4cyclictest0-21swapper/307:10:203
662899131,9cyclictest6689-21latency_hist07:10:010
665699122,6cyclictest0-21swapper/308:25:143
665699111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional