You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-23 - 10:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Jul 23, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
34354821590,3sleep00-21swapper/021:50:020
36859121280,2sleep00-21swapper/023:05:160
37558421270,3sleep10-21swapper/123:25:131
1390991020,76rtkit-daemon0-21swapper/219:06:492
28638827839,14sleep30-21swapper/319:06:233
28643827739,32sleep10-21swapper/119:07:021
28644727346,17sleep00-21swapper/019:07:090
2988052660,5sleep10-21swapper/119:40:191
3864292520,3sleep33056-21PK-Backend23:55:193
3637722470,2sleep13056-21PK-Backend22:50:211
3837692460,3sleep20-21swapper/223:50:012
3701622460,2sleep00-21swapper/023:10:130
3270442460,3sleep00-21swapper/021:00:170
28686799381,24cyclictest304483-21python319:55:223
3620302360,2sleep20-21swapper/222:45:212
286867992913,14cyclictest348846-21sed22:05:013
28686099291,1cyclictest141rcu_preempt19:13:111
3747142270,9chrt34-21ksoftirqd/223:20:452
3168662260,3sleep00-21swapper/020:30:210
286867992612,12cyclictest353932-21sh22:20:003
286867992612,12cyclictest353932-21sh22:20:003
286867992611,13cyclictest362809-21sed22:50:013
286867992611,13cyclictest335262-21tr21:25:133
28686799260,6cyclictest327179-21pluginstate21:00:193
28686799260,1cyclictest0-21swapper/323:50:033
286858992616,8cyclictest357696-21unin-run22:30:000
3529842250,3sleep0161rcuc/022:15:150
3529842250,3sleep0161rcuc/022:15:150
3443312250,3sleep00-21swapper/021:50:170
28686799256,17cyclictest333877-21sed21:20:173
28686799251,21cyclictest362912-21apt22:50:123
28686799251,21cyclictest315148-21taskset20:25:233
28686799250,1cyclictest0-21swapper/321:58:163
286858992521,3cyclictest13-21ksoftirqd/000:00:010
28685899250,23cyclictest372723-21/usr/sbin/munin23:15:230
3232892240,9sleep234-21ksoftirqd/220:50:162
28686799248,14cyclictest354612-21ls22:20:163
28686799246,16cyclictest395252-21date00:25:013
28686799244,19cyclictest338892-21expr21:35:143
286867992410,12cyclictest1788620-21WebKitWebProces22:06:133
28685899241,3cyclictest299022-21python319:40:210
286858992411,5cyclictest13-21ksoftirqd/023:50:000
3651542230,2sleep33056-21PK-Backend22:55:163
28686799237,14cyclictest303466-21wc19:55:013
286867992318,4cyclictest3056-21PK-Backend23:34:563
28686799230,21cyclictest398182-21missed_timers00:30:163
286864992317,5cyclictest387187-21cron00:00:002
286864992315,6cyclictest399014-21cron00:35:002
28686499230,4cyclictest307529-21awk20:05:182
28686099239,2cyclictest141rcu_preempt20:40:001
28686099233,6cyclictest26-21ksoftirqd/121:35:001
28686099230,2cyclictest141rcu_preempt22:05:221
28685899231,20cyclictest380260-21munin-run23:40:000
3845572220,5sleep23056-21PK-Backend23:50:172
28686799227,13cyclictest396366-21processes00:25:183
286867992219,2cyclictest3056-21PK-Backend19:34:323
28686799221,2cyclictest330715-21ps21:10:203
286867992211,10cyclictest3056-21PK-Backend20:55:393
28686799220,2cyclictest393034-21chrt00:16:483
28686799220,2cyclictest393034-21chrt00:16:483
28686799220,20cyclictest3056-21PK-Backend20:23:053
28686799220,18cyclictest3056-21PK-Backend00:05:113
286864992220,1cyclictest0-21swapper/220:45:462
286864992220,1cyclictest0-21swapper/219:34:552
286864992217,3cyclictest335336-21expr21:25:132
28686499220,21cyclictest0-21swapper/221:09:532
286860992220,1cyclictest0-21swapper/122:28:131
286860992220,1cyclictest0-21swapper/119:47:041
286860992220,1cyclictest0-21swapper/119:15:271
286860992220,1cyclictest0-21swapper/100:16:551
286860992220,1cyclictest0-21swapper/100:16:551
286860992214,7cyclictest310310-21unin-run20:15:001
28686099220,21cyclictest385216-21taskset23:52:071
286858992218,3cyclictest378399-21sshd23:33:350
286858992215,6cyclictest1-21systemd23:45:000
286858992214,7cyclictest883-21dbus-daemon21:45:000
28685899220,21cyclictest3056-21PK-Backend20:10:420
28685899220,21cyclictest0-21swapper/021:17:070
28685899220,21cyclictest0-21swapper/000:09:090
28685899220,10cyclictest3651572wc22:55:160
139099220,14rtkit-daemon3056-21PK-Backend23:25:023
3995592210,5sleep00-21swapper/000:35:140
3921692210,4sleep20-21swapper/200:15:112
3921692210,4sleep20-21swapper/200:15:112
28686799215,15cyclictest338132-21sshd21:33:243
28686799213,16cyclictest297320-21sort19:35:223
28686799212,7cyclictest368801-21cut23:05:173
28686799212,17cyclictest2156-21gnome-shell20:06:093
286867992119,1cyclictest3056-21PK-Backend23:39:223
286867992119,1cyclictest3056-21PK-Backend22:32:443
286867992119,1cyclictest3056-21PK-Backend21:54:583
286867992119,1cyclictest3056-21PK-Backend20:17:513
286867992118,2cyclictest3056-21PK-Backend21:08:073
28686799210,20cyclictest3056-21PK-Backend20:46:103
28686799210,20cyclictest3056-21PK-Backend20:34:003
28686799210,20cyclictest3056-21PK-Backend20:01:433
28686799210,20cyclictest0-21swapper/321:15:513
28686799210,19cyclictest3056-21PK-Backend19:23:513
28686799210,19cyclictest3056-21PK-Backend19:23:513
286864992119,1cyclictest0-21swapper/222:15:362
286864992119,1cyclictest0-21swapper/222:15:362
286864992119,1cyclictest0-21swapper/222:06:232
286864992119,1cyclictest0-21swapper/221:51:572
286864992119,1cyclictest0-21swapper/221:47:172
286864992119,1cyclictest0-21swapper/221:02:352
286864992119,1cyclictest0-21swapper/220:33:222
286864992119,1cyclictest0-21swapper/219:54:562
286864992119,1cyclictest0-21swapper/200:27:512
286864992119,1cyclictest0-21swapper/200:13:582
28686499210,20cyclictest3056-21PK-Backend19:45:362
28686499210,20cyclictest0-21swapper/221:11:142
28686499210,20cyclictest0-21swapper/220:42:002
28686499210,20cyclictest0-21swapper/200:21:562
28686099218,2cyclictest141rcu_preempt23:13:251
28686099217,4cyclictest3435492latency_hist21:50:011
286860992119,1cyclictest3056-21PK-Backend21:52:231
286860992119,1cyclictest0-21swapper/121:07:281
286860992119,1cyclictest0-21swapper/120:51:321
286860992119,1cyclictest0-21swapper/119:51:301
28686099210,20cyclictest0-21swapper/120:40:241
28686099210,20cyclictest0-21swapper/119:22:101
28686099210,20cyclictest0-21swapper/119:22:101
28686099210,19cyclictest0-21swapper/122:31:301
139099210,15rtkit-daemon3056-21PK-Backend19:10:223
139099210,15rtkit-daemon0-21swapper/221:55:422
139099210,14rtkit-daemon0-21swapper/320:40:283
139099210,13rtkit-daemon3056-21PK-Backend19:28:363
139099210,13rtkit-daemon0-21swapper/219:24:152
139099210,13rtkit-daemon0-21swapper/219:24:152
28686799203,15cyclictest340966-21netstat21:40:183
28686799201,9cyclictest360307-21python322:40:193
28686799201,8cyclictest356974-21/usr/sbin/munin22:25:193
286867992018,1cyclictest3056-21PK-Backend23:41:113
286867992018,1cyclictest3056-21PK-Backend23:19:073
28686799201,7cyclictest309894-21/usr/sbin/munin20:10:223
286867992017,2cyclictest3056-21PK-Backend19:47:373
28686799200,1cyclictest3056-21PK-Backend23:24:283
28686799200,19cyclictest3056-21PK-Backend23:11:233
28686799200,19cyclictest3056-21PK-Backend23:02:043
28686799200,19cyclictest3056-21PK-Backend19:40:343
28686799200,18cyclictest3056-21PK-Backend21:45:103
286864992018,1cyclictest3056-21PK-Backend20:21:392
286864992018,1cyclictest0-21swapper/223:41:202
286864992018,1cyclictest0-21swapper/222:41:312
286864992018,1cyclictest0-21swapper/222:02:522
286864992018,1cyclictest0-21swapper/220:37:422
286864992018,1cyclictest0-21swapper/220:29:502
286864992018,1cyclictest0-21swapper/220:03:552
286864992014,4cyclictest304291-21grep19:55:182
286864992013,6cyclictest338278-21/usr/sbin/munin21:35:002
286864992013,5cyclictest297752-21sshd19:38:242
286864992012,6cyclictest294034-21users19:25:212
286864992012,6cyclictest287997-21awk19:10:192
28686499200,19cyclictest0-21swapper/222:53:072
28686499200,19cyclictest0-21swapper/222:32:052
28686499200,19cyclictest0-21swapper/222:21:172
28686499200,19cyclictest0-21swapper/221:41:152
28686499200,19cyclictest0-21swapper/220:12:202
28686499200,18cyclictest0-21swapper/222:25:362
28686099203,11cyclictest397026-21apt-get00:30:031
286860992018,1cyclictest0-21swapper/122:35:351
286860992018,1cyclictest0-21swapper/122:10:211
286860992018,1cyclictest0-21swapper/121:56:491
286860992018,1cyclictest0-21swapper/121:35:121
286860992018,1cyclictest0-21swapper/100:10:261
286860992017,2cyclictest377539-21irqstats23:30:171
28686099200,19cyclictest0-21swapper/122:41:031
28686099200,19cyclictest0-21swapper/120:26:591
28686099200,19cyclictest0-21swapper/100:26:471
28685899207,4cyclictest13-21ksoftirqd/021:25:010
28685899204,6cyclictest141rcu_preempt00:04:380
286858992018,1cyclictest3056-21PK-Backend21:39:450
286858992018,1cyclictest0-21swapper/023:52:340
286858992018,1cyclictest0-21swapper/022:07:360
286858992018,1cyclictest0-21swapper/021:09:570
286858992018,1cyclictest0-21swapper/000:15:190
286858992018,1cyclictest0-21swapper/000:15:190
286858992016,3cyclictest13-21ksoftirqd/022:20:230
286858992014,5cyclictest335261-21tr21:25:130
286858992013,6cyclictest307667-21memory20:05:180
286858992013,5cyclictest396495-21cat00:25:200
286858992012,6cyclictest436-20systemd-journal22:35:000
28685899200,1cyclictest0-21swapper/020:16:440
28685899200,1cyclictest0-21swapper/000:11:130
28685899200,19cyclictest0-21swapper/023:20:090
28685899200,19cyclictest0-21swapper/020:41:310
28685899200,19cyclictest0-21swapper/019:56:290
139099200,14rtkit-daemon3056-21PK-Backend23:46:563
139099200,14rtkit-daemon0-21swapper/223:04:372
139099200,14rtkit-daemon0-21swapper/221:20:012
139099200,14rtkit-daemon0-21swapper/219:43:542
3515082190,4chrt0-21swapper/322:10:213
28686799195,12cyclictest0-21swapper/320:35:213
286867991917,1cyclictest3056-21PK-Backend22:38:253
28686799190,1cyclictest3056-21PK-Backend19:16:413
28686799190,1cyclictest3056-21PK-Backend00:36:053
286864991917,1cyclictest0-21swapper/220:55:392
286864991917,1cyclictest0-21swapper/219:17:512
286864991911,6cyclictest436-20systemd-journal21:17:002
286864991911,6cyclictest352184-21cron22:15:002
28686499190,1cyclictest0-21swapper/222:36:242
28686499190,1cyclictest0-21swapper/200:01:072
286860991917,1cyclictest0-21swapper/123:56:191
286860991917,1cyclictest0-21swapper/120:18:411
286860991917,1cyclictest0-21swapper/119:30:411
286860991917,1cyclictest0-21swapper/100:39:321
286860991914,3cyclictest307533-21if_enp2s020:05:171
286860991913,5cyclictest883-21dbus-daemon00:10:001
286860991912,4cyclictest381989-21sh23:45:001
286860991912,2cyclictest26-21ksoftirqd/123:04:481
286860991911,2cyclictest26-21ksoftirqd/122:58:171
286860991911,2cyclictest26-21ksoftirqd/121:43:151
286860991911,2cyclictest26-21ksoftirqd/121:24:281
286860991911,2cyclictest26-21ksoftirqd/120:32:461
28686099190,1cyclictest3056-21PK-Backend22:01:491
28686099190,1cyclictest0-21swapper/122:15:001
28686099190,1cyclictest0-21swapper/122:15:001
28686099190,1cyclictest0-21swapper/120:20:371
28685899197,4cyclictest13-21ksoftirqd/019:55:020
286858991917,1cyclictest3056-21PK-Backend20:57:200
286858991917,1cyclictest0-21swapper/022:38:460
286858991917,1cyclictest0-21swapper/022:04:050
286858991917,1cyclictest0-21swapper/020:51:590
286858991917,1cyclictest0-21swapper/019:18:170
286858991917,1cyclictest0-21swapper/000:24:040
286858991912,6cyclictest306355-21sed20:00:220
286858991912,6cyclictest301293-21sshd19:49:230
286858991912,5cyclictest397026-21apt-get00:30:010
286858991912,5cyclictest318764-21ls20:35:230
286858991912,5cyclictest291598-21df_inode19:20:140
286858991912,5cyclictest291598-21df_inode19:20:130
286858991911,5cyclictest286694-21apt-get19:10:040
28685899190,1cyclictest3056-21PK-Backend20:26:030
28685899190,1cyclictest0-21swapper/020:23:210
28685899190,18cyclictest0-21swapper/023:01:170
28685899190,18cyclictest0-21swapper/022:45:150
28685899190,18cyclictest0-21swapper/022:40:550
28685899190,18cyclictest0-21swapper/022:11:080
28685899190,18cyclictest0-21swapper/021:13:490
286867991816,1cyclictest3056-21PK-Backend00:04:093
28686799180,6cyclictest323125-21tr20:50:163
286864991816,1cyclictest0-21swapper/223:34:512
286864991816,1cyclictest0-21swapper/200:06:012
286864991810,6cyclictest339613-21awk21:35:232
28686099189,2cyclictest141rcu_preempt22:46:501
28686099187,2cyclictest141rcu_preempt23:49:551
286860991816,1cyclictest0-21swapper/123:18:481
286860991816,1cyclictest0-21swapper/123:09:081
286860991816,1cyclictest0-21swapper/119:57:381
286860991816,1cyclictest0-21swapper/100:04:451
286860991812,4cyclictest26-21ksoftirqd/120:50:011
286860991811,2cyclictest26-21ksoftirqd/119:38:051
286860991811,2cyclictest26-21ksoftirqd/100:22:371
28686099180,7cyclictest3056-21PK-Backend21:00:191
28685899185,5cyclictest13-21ksoftirqd/019:25:160
286858991813,4cyclictest13-21ksoftirqd/020:50:000
286858991812,5cyclictest296188-21systemd-run19:35:000
286858991811,6cyclictest13-21ksoftirqd/021:31:240
286858991811,5cyclictest436-20systemd-journal22:53:340
286858991811,5cyclictest375329-21apt23:25:130
28686799170,15cyclictest3056-21PK-Backend00:10:083
286864991711,4cyclictest310685-21cpuspeed_turbos20:15:122
28686099172,12cyclictest329873-21apt21:10:121
286860991711,5cyclictest335598-21sed21:25:171
286860991711,5cyclictest26-21ksoftirqd/119:25:181
286860991710,5cyclictest374044-21cat23:20:171
286860991710,2cyclictest26-21ksoftirqd/120:03:101
28686099170,3cyclictest3056-21PK-Backend22:20:231
286858991710,5cyclictest345755-21cat21:55:140
286864991614,1cyclictest0-21swapper/223:35:532
286864991614,1cyclictest0-21swapper/200:37:382
286864991613,2cyclictest3056-21PK-Backend23:05:172
286864991611,4cyclictest375943-21grep23:25:182
286864991611,4cyclictest370025-21cpuspeed_turbos23:10:122
28686099168,7cyclictest379531-21ntp_states23:35:191
286860991610,5cyclictest332541-21cat21:15:221
28685899163,2cyclictest0-21swapper/019:37:490
28686499158,5cyclictest883-21dbus-daemon23:15:002
28686499156,7cyclictest365003-21df22:55:132
286860991513,1cyclictest3056-21PK-Backend20:58:491
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional