You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 13:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Mar 27, 2026 00:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
53858322290,8sleep20-21swapper/219:05:122
62002421250,4sleep13056-21PK-Backend23:15:011
60655121190,2sleep00-21swapper/022:35:140
63728221000,3sleep00-21swapper/000:05:010
139099730,58rtkit-daemon0-21swapper/319:05:383
53994727050,13sleep10-21swapper/119:08:411
53972726646,14sleep00-21swapper/019:05:510
5519052610,4sleep13056-21PK-Backend19:40:171
6104112500,2sleep30-21swapper/322:45:203
6205602490,2sleep13056-21PK-Backend23:15:181
5748772430,4sleep10-21swapper/120:45:211
6482952410,4sleep20-21swapper/200:35:132
5854062320,3sleep23056-21PK-Backend21:15:212
5761302320,1sleep30-21swapper/320:50:113
54019099296,21cyclictest883-21dbus-daemon23:42:550
54019099281,23cyclictest568601-21/usr/sbin/munin20:30:240
54019099278,17cyclictest610422-21awk22:45:200
54019099268,16cyclictest632710-21idleruntime23:50:150
54019099268,16cyclictest597214-21tr22:00:110
54019099267,17cyclictest883-21dbus-daemon21:30:000
54019099267,17cyclictest645019-21cat00:25:180
54019099267,17cyclictest605247-21users22:30:230
54019099260,19cyclictest0-21swapper/022:43:560
540198992520,4cyclictest34-21ksoftirqd/222:07:492
540198992516,7cyclictest2638-21gdbus23:37:542
54019099259,14cyclictest581631-21grep21:05:150
54019099257,16cyclictest565372-21ls20:20:140
54019099256,17cyclictest646811-21ls00:30:190
54019099255,5cyclictest627429-21expr23:35:120
540190992511,12cyclictest641226-21expr00:15:130
139099250,16rtkit-daemon2627-21gmain19:25:292
6322912240,6chrt0-21swapper/223:50:102
540203992421,2cyclictest411rcuc/320:01:433
540203992417,4cyclictest605646-21apt-get22:35:043
54020399240,4cyclictest555619-21python319:50:213
54020399240,15cyclictest3056-21PK-Backend00:36:093
54019899240,22cyclictest2630-21gdbus22:37:502
54019099247,15cyclictest625960-21missed_timers23:30:180
54019099247,15cyclictest586720-21kernelversion21:20:170
540190992419,4cyclictest0-21swapper/021:51:580
540190992418,4cyclictest624283-21cut23:25:170
54020399234,3cyclictest3056-21PK-Backend21:20:003
54020399234,3cyclictest3056-21PK-Backend00:23:093
54020399232,8cyclictest641466-21irqstats00:15:163
54020399232,8cyclictest629222-21hddtemp_smartct23:40:153
54020399232,8cyclictest624472-21python323:25:193
54020399232,8cyclictest589432-21munin-run21:30:013
54020399231,9cyclictest604548-21diskstats22:30:153
540203992314,7cyclictest575160-21users20:45:233
54020399231,10cyclictest626274-21switchtime23:30:233
54020399230,11cyclictest627058-21/usr/sbin/munin23:35:173
54019099238,13cyclictest620380-21cat23:15:130
54019099238,13cyclictest564853-21sshd20:19:450
54019099236,14cyclictest436-20systemd-journal21:19:550
54019099231,20cyclictest1-21systemd00:00:000
540190992310,11cyclictest552325-21timerwakeupswit19:40:210
54019099230,22cyclictest540868-21irqstats19:10:150
139099230,15rtkit-daemon3056-21PK-Backend20:24:583
54020399223,8cyclictest612672-21runrttasks22:52:443
540203992214,6cyclictest436-20systemd-journal19:49:593
540203992213,6cyclictest573853-21sed20:45:003
540203992213,6cyclictest436-20systemd-journal19:30:013
540198992220,1cyclictest0-21swapper/220:02:322
540198992220,1cyclictest0-21swapper/219:57:032
540198992213,7cyclictest605703-21systemd22:35:012
54019899220,2cyclictest0-21swapper/223:05:222
540193992220,1cyclictest0-21swapper/100:20:151
54019099227,13cyclictest552750-21dump-pmu-power19:49:540
54019099226,14cyclictest642935-21cstates00:20:130
54019099224,16cyclictest0-21swapper/020:25:190
540190992220,1cyclictest0-21swapper/021:48:070
540190992220,1cyclictest0-21swapper/021:48:060
540190992220,1cyclictest0-21swapper/020:56:210
540190992220,1cyclictest0-21swapper/019:16:510
54019099221,7cyclictest560665-21ntpq20:05:200
54019099221,6cyclictest576527-21awk20:50:170
54019099221,19cyclictest590349-21needreboot21:30:170
54019099220,7cyclictest612071-21grep22:50:160
54020399212,14cyclictest600710-21apt22:20:143
54020399211,4cyclictest581018-21apt21:05:103
540203992112,7cyclictest1-21systemd21:00:013
540203992110,3cyclictest42-21ksoftirqd/321:40:143
54020399210,6cyclictest42-21ksoftirqd/322:42:223
54020399210,20cyclictest0-21swapper/320:07:243
54020399210,19cyclictest598247-21sshd22:01:453
54020399210,18cyclictest548179-21df19:30:123
540198992119,1cyclictest3056-21PK-Backend00:20:192
540198992119,1cyclictest0-21swapper/223:59:302
540198992119,1cyclictest0-21swapper/223:22:122
540198992119,1cyclictest0-21swapper/222:58:532
540198992119,1cyclictest0-21swapper/222:27:172
540198992119,1cyclictest0-21swapper/221:36:472
540198992119,1cyclictest0-21swapper/219:41:422
540198992119,1cyclictest0-21swapper/200:25:322
540198992119,1cyclictest0-21swapper/200:14:032
540198992114,5cyclictest1-21systemd22:00:002
54019899210,20cyclictest0-21swapper/220:21:442
540193992119,1cyclictest3056-21PK-Backend00:12:311
540193992119,1cyclictest0-21swapper/123:40:201
540193992117,3cyclictest26-21ksoftirqd/122:35:011
54019099216,14cyclictest5594872chrt20:02:370
54019099215,4cyclictest637269-21perf00:09:550
54019099215,4cyclictest550540-21wc19:35:220
54019099213,4cyclictest615358-21kernelversion23:00:160
54019099212,3cyclictest0-21swapper/019:26:560
540190992119,1cyclictest0-21swapper/023:21:000
540190992119,1cyclictest0-21swapper/023:10:260
540190992119,1cyclictest0-21swapper/022:57:480
540190992119,1cyclictest0-21swapper/000:38:540
54019099211,8cyclictest571528-21chrt20:36:250
54019099211,7cyclictest640655-21chrt00:14:220
54019099211,7cyclictest601410-21awk22:20:210
54019099210,8cyclictest573179-21awk20:40:210
54019099210,7cyclictest631117-21cat23:45:150
54019099210,7cyclictest592909-21awk21:39:590
54019099210,7cyclictest548668-21grep19:30:190
54019099210,20cyclictest0-21swapper/021:56:510
54019099210,20cyclictest0-21swapper/021:44:140
54019099210,19cyclictest3056-21PK-Backend20:46:000
139099210,15rtkit-daemon3056-21PK-Backend19:10:562
139099210,15rtkit-daemon0-21swapper/323:57:583
139099210,14rtkit-daemon0-21swapper/220:08:102
139099210,14rtkit-daemon0-21swapper/200:34:232
540203992018,1cyclictest411rcuc/321:47:143
540203992018,1cyclictest411rcuc/321:47:143
540203992018,1cyclictest0-21swapper/322:25:343
540203992018,1cyclictest0-21swapper/322:13:443
540203992015,3cyclictest582922-21grep21:10:113
54020399201,4cyclictest545207-21python319:20:193
540203992014,4cyclictest886-21NetworkManager00:10:003
54020399201,17cyclictest557959-21munin-run20:00:013
54020399200,6cyclictest42-21ksoftirqd/320:31:263
54020399200,1cyclictest3056-21PK-Backend00:14:573
54020399200,1cyclictest0-21swapper/323:01:513
54020399200,1cyclictest0-21swapper/319:17:223
54020399200,19cyclictest3056-21PK-Backend20:15:083
54020399200,19cyclictest0-21swapper/323:17:183
54020399200,19cyclictest0-21swapper/323:10:363
54020399200,19cyclictest0-21swapper/321:01:593
54020399200,19cyclictest0-21swapper/320:14:063
54019899209,9cyclictest883-21dbus-daemon23:17:522
540198992018,1cyclictest3056-21PK-Backend20:32:592
540198992018,1cyclictest0-21swapper/223:47:132
540198992018,1cyclictest0-21swapper/223:14:422
540198992018,1cyclictest0-21swapper/222:42:182
540198992018,1cyclictest0-21swapper/222:24:262
540198992018,1cyclictest0-21swapper/222:11:152
540198992018,1cyclictest0-21swapper/221:42:152
540198992018,1cyclictest0-21swapper/220:37:272
540198992018,1cyclictest0-21swapper/219:53:252
540198992018,1cyclictest0-21swapper/219:49:472
540198992018,1cyclictest0-21swapper/200:00:312
540198992013,5cyclictest637863-21cat00:05:142
54019899200,19cyclictest3056-21PK-Backend21:29:312
54019899200,19cyclictest0-21swapper/221:51:212
54019899200,19cyclictest0-21swapper/221:00:522
54019899200,19cyclictest0-21swapper/219:39:542
54019899200,19cyclictest0-21swapper/219:24:122
540193992018,1cyclictest0-21swapper/120:35:191
540193992014,4cyclictest1095-21runrttasks20:25:201
540193992013,6cyclictest608828-21sed22:40:201
54019099204,3cyclictest0-21swapper/022:17:300
54019099203,2cyclictest0-21swapper/021:10:050
54019099202,3cyclictest0-21swapper/019:56:350
540190992018,1cyclictest0-21swapper/023:06:340
540190992018,1cyclictest0-21swapper/019:22:540
54019099201,5cyclictest579261-21/usr/sbin/munin21:00:200
54019099200,7cyclictest603212-21proc_pri22:25:220
54019099200,6cyclictest555883-21chrt19:50:510
54019099200,19cyclictest0-21swapper/022:12:330
139099200,14rtkit-daemon0-21swapper/219:15:032
54020399193,5cyclictest3056-21PK-Backend00:27:333
54020399193,14cyclictest600392-21taskset22:19:383
54020399192,6cyclictest3056-21PK-Backend23:48:383
54020399192,5cyclictest3056-21PK-Backend19:10:123
540203991917,1cyclictest0-21swapper/323:06:443
540203991917,1cyclictest0-21swapper/322:57:583
540203991917,1cyclictest0-21swapper/322:06:003
540203991917,1cyclictest0-21swapper/321:51:073
54020399191,6cyclictest42-21ksoftirqd/323:50:313
540203991916,2cyclictest3056-21PK-Backend00:31:343
540203991914,3cyclictest978-21snmpd19:42:003
540203991913,5cyclictest636081-21ls00:00:133
54020399190,1cyclictest0-21swapper/321:35:523
54020399190,18cyclictest3056-21PK-Backend20:25:303
54020399190,18cyclictest0-21swapper/321:34:513
54020399190,17cyclictest622740-21smart_sda23:20:203
540198991917,1cyclictest0-21swapper/223:03:332
540198991917,1cyclictest0-21swapper/222:47:452
540198991917,1cyclictest0-21swapper/220:47:192
540198991917,1cyclictest0-21swapper/220:26:582
540198991912,6cyclictest576846-21sed20:50:202
540198991910,7cyclictest436-20systemd-journal22:05:002
54019899190,1cyclictest0-21swapper/220:41:522
54019899190,18cyclictest3056-21PK-Backend00:15:542
54019899190,18cyclictest0-21swapper/222:18:452
54019899190,18cyclictest0-21swapper/221:21:402
54019899190,18cyclictest0-21swapper/221:10:582
54019899190,18cyclictest0-21swapper/221:08:092
540193991911,7cyclictest591165-21cron21:35:001
540193991911,6cyclictest645169-21ntp_kernel_pll_00:25:201
540193991911,6cyclictest588687-21ntp_states21:25:171
540190991915,3cyclictest3056-21PK-Backend20:10:320
54019099191,4cyclictest2156-21gnome-shell22:05:090
540203991816,1cyclictest0-21swapper/319:39:593
54020399181,5cyclictest0-21swapper/321:20:183
540203991811,5cyclictest570469-21cat20:35:123
54020399180,1cyclictest0-21swapper/321:57:023
540198991816,1cyclictest0-21swapper/223:43:362
540198991816,1cyclictest0-21swapper/222:50:352
540198991816,1cyclictest0-21swapper/221:33:102
540198991816,1cyclictest0-21swapper/219:32:032
540198991814,3cyclictest34-21ksoftirqd/223:30:142
540198991812,5cyclictest578554-21sed20:55:212
54019899180,17cyclictest0-21swapper/220:11:372
54019399189,7cyclictest595897-21cron21:55:001
540193991816,1cyclictest3056-21PK-Backend19:45:171
540193991813,3cyclictest596525-21munin-plugin-st22:00:001
540193991811,6cyclictest627457-21diskstats23:35:131
540193991811,5cyclictest646189-21awk00:30:121
54019399181,14cyclictest568628-21apt20:30:111
54019399180,6cyclictest3056-21PK-Backend22:25:001
540198991715,1cyclictest0-21swapper/221:47:422
540198991715,1cyclictest0-21swapper/221:47:422
540198991710,5cyclictest623923-21cat23:25:112
54019399179,6cyclictest632308-21apt23:50:121
54019399172,13cyclictest623335-21cron23:25:001
540193991715,1cyclictest3056-21PK-Backend21:39:141
540193991715,1cyclictest0-21swapper/123:49:401
540193991715,1cyclictest0-21swapper/123:30:471
540193991715,1cyclictest0-21swapper/121:17:241
540193991713,3cyclictest648894-21timerandwakeup00:35:211
540193991712,4cyclictest564127-21ntp_kernel_err20:15:201
540193991712,3cyclictest565742-21missed_timers20:20:171
540193991711,5cyclictest587073-21gzip21:20:201
540193991711,4cyclictest561556-21sed20:10:001
540193991710,6cyclictest540763-21df19:10:141
540193991710,5cyclictest634637-21sed23:55:191
540193991710,5cyclictest624325-21ntpq23:25:161
540193991614,1cyclictest3056-21PK-Backend22:54:451
540193991612,2cyclictest610472-21processes22:45:211
540193991611,3cyclictest573558-21latency_hist20:45:011
540193991610,4cyclictest544872-21sed19:20:141
54019899155,2cyclictest141rcu_preempt20:17:582
54019399159,5cyclictest641855-21swap00:15:211
54019399157,6cyclictest607776-21latency_hist22:40:001
54019399152,11cyclictest546214-21tr19:25:111
540193991513,1cyclictest3056-21PK-Backend19:51:251
540193991513,1cyclictest3056-21PK-Backend19:35:581
540193991513,1cyclictest3056-21PK-Backend00:02:241
540193991513,1cyclictest0-21swapper/120:50:471
540193991511,3cyclictest603903-21latency_hist22:30:011
540193991511,3cyclictest3056-21PK-Backend22:19:441
540193991511,3cyclictest3056-21PK-Backend21:00:281
54019399150,7cyclictest3056-21PK-Backend21:50:001
54019399150,7cyclictest3056-21PK-Backend21:50:001
54019399150,14cyclictest3056-21PK-Backend20:14:521
54019399150,14cyclictest0-21swapper/121:41:381
54019399149,3cyclictest557027-21latency19:55:151
540193991412,1cyclictest3056-21PK-Backend22:03:411
540193991412,1cyclictest3056-21PK-Backend20:58:311
540193991412,1cyclictest3056-21PK-Backend20:01:331
540193991412,1cyclictest3056-21PK-Backend00:05:351
54019399140,1cyclictest3056-21PK-Backend21:13:261
54019399140,1cyclictest3056-21PK-Backend19:15:511
54019399140,13cyclictest3056-21PK-Backend19:30:371
54019399139,3cyclictest3056-21PK-Backend23:05:061
54019399139,3cyclictest3056-21PK-Backend22:13:501
54019399135,6cyclictest614600-21cron23:00:001
54019399128,3cyclictest3056-21PK-Backend22:05:521
54019399117,3cyclictest3056-21PK-Backend21:05:011
54019399116,3cyclictest6161062sleep123:02:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional