You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-12 - 13:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sun Apr 12, 2026 00:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
119076127145,17sleep00-21swapper/019:05:220
119005426947,15sleep30-21swapper/319:05:093
119141126445,13sleep20-21swapper/219:08:292
119135726443,14sleep10-21swapper/119:07:491
13009182520,5sleep31300923-21needreboot00:15:173
12644902430,9sleep234-21ksoftirqd/222:34:552
12426752310,4sleep23056-21PK-Backend21:30:162
1191672992916,11cyclictest1198315-21ntp_states19:25:190
139099280,4rtkit-daemon1389-21rtkit-daemon20:02:520
139099280,4rtkit-daemon1389-21rtkit-daemon00:18:041
119168799281,7cyclictest1304210-21/usr/sbin/munin00:25:173
139099270,4rtkit-daemon1389-21rtkit-daemon00:03:201
12571512270,1sleep30-21swapper/322:10:223
1191682992723,3cyclictest331rcuc/222:15:332
1191682992721,5cyclictest34-21ksoftirqd/200:04:512
1191674992715,10cyclictest1264542-21unin-run22:35:001
139099260,4rtkit-daemon1389-21rtkit-daemon21:54:081
119168799262,19cyclictest0-21swapper/319:10:103
119168799241,10cyclictest1298288-21dump-pmu-power00:10:003
119168299244,10cyclictest34-21ksoftirqd/223:10:212
1191682992410,6cyclictest141rcu_preempt21:20:062
1191672992416,6cyclictest1241721-21apt-get21:30:010
139099230,4rtkit-daemon1389-21rtkit-daemon20:52:460
119168299236,9cyclictest34-21ksoftirqd/200:10:182
1191682992319,3cyclictest331rcuc/221:10:222
1191682992317,5cyclictest1261033-21cron22:25:002
1191682992314,7cyclictest1234757-21cron21:10:002
1191672992321,1cyclictest0-21swapper/023:08:140
139099220,4rtkit-daemon1389-21rtkit-daemon21:16:390
139099220,4rtkit-daemon1389-21rtkit-daemon20:09:380
139099220,19rtkit-daemon1788643-21eadedCompositor19:23:070
11997362220,5sleep234-21ksoftirqd/219:30:152
1191687992220,1cyclictest0-21swapper/320:42:243
1191687992220,1cyclictest0-21swapper/319:56:213
119168799220,20cyclictest1213943-21expr20:10:133
119168299228,3cyclictest141rcu_preempt23:04:042
1191682992220,1cyclictest0-21swapper/223:50:352
1191682992220,1cyclictest0-21swapper/223:19:062
1191682992220,1cyclictest0-21swapper/222:46:142
1191682992220,1cyclictest0-21swapper/219:37:212
1191682992220,1cyclictest0-21swapper/219:22:542
1191682992220,1cyclictest0-21swapper/200:20:572
119168299221,8cyclictest34-21ksoftirqd/223:30:102
1191682992211,3cyclictest141rcu_preempt23:43:452
1191682992210,3cyclictest141rcu_preempt21:58:402
119168299220,21cyclictest0-21swapper/220:10:192
119168299220,1cyclictest1297267-21/usr/sbin/munin00:05:162
1191674992220,1cyclictest3056-21PK-Backend19:14:451
1191674992220,1cyclictest0-21swapper/123:54:421
1191674992220,1cyclictest0-21swapper/123:05:291
1191674992219,2cyclictest3056-21PK-Backend23:39:291
119167499220,20cyclictest3056-21PK-Backend23:22:041
119167499220,20cyclictest3056-21PK-Backend00:38:021
1191672992220,1cyclictest3056-21PK-Backend23:20:030
1191672992220,1cyclictest0-21swapper/023:15:510
1191672992220,1cyclictest0-21swapper/021:36:210
139099210,4rtkit-daemon1389-21rtkit-daemon22:21:290
1191687992119,1cyclictest0-21swapper/323:20:343
1191687992119,1cyclictest0-21swapper/320:55:363
1191687992119,1cyclictest0-21swapper/319:20:053
1191687992119,1cyclictest0-21swapper/300:32:273
1191687992118,2cyclictest3056-21PK-Backend23:34:473
1191687992115,4cyclictest1299402-21python300:10:193
119168799211,17cyclictest1224368-21munin-run20:39:593
119168799210,20cyclictest0-21swapper/321:01:303
119168299219,3cyclictest141rcu_preempt22:14:552
119168299218,5cyclictest34-21ksoftirqd/221:45:122
119168299218,2cyclictest141rcu_preempt20:15:452
119168299213,9cyclictest34-21ksoftirqd/223:35:162
1191682992119,1cyclictest3056-21PK-Backend20:05:072
1191682992119,1cyclictest0-21swapper/223:23:182
1191682992119,1cyclictest0-21swapper/223:06:282
1191682992119,1cyclictest0-21swapper/221:53:012
1191682992119,1cyclictest0-21swapper/220:43:322
1191682992119,1cyclictest0-21swapper/220:22:502
1191682992119,1cyclictest0-21swapper/220:03:172
1191682992119,1cyclictest0-21swapper/219:46:202
1191682992117,3cyclictest331rcuc/220:48:482
1191682992117,3cyclictest331rcuc/220:32:432
1191682992111,2cyclictest141rcu_preempt22:02:042
1191682992111,2cyclictest141rcu_preempt21:35:062
1191682992110,2cyclictest141rcu_preempt22:44:512
1191682992110,2cyclictest141rcu_preempt22:25:592
119168299210,20cyclictest978-21snmpd21:43:412
119168299210,20cyclictest978-21snmpd21:43:412
119168299210,20cyclictest0-21swapper/223:48:462
119168299210,20cyclictest0-21swapper/222:35:392
119168299210,20cyclictest0-21swapper/220:39:342
119168299210,20cyclictest0-21swapper/200:19:082
1191674992119,1cyclictest3056-21PK-Backend23:57:191
1191674992119,1cyclictest3056-21PK-Backend20:27:241
1191674992119,1cyclictest3056-21PK-Backend20:23:591
1191674992119,1cyclictest3056-21PK-Backend19:58:041
1191674992119,1cyclictest3056-21PK-Backend00:33:501
1191674992119,1cyclictest3056-21PK-Backend00:29:371
1191674992119,1cyclictest0-21swapper/123:42:531
1191674992119,1cyclictest0-21swapper/123:32:051
1191674992119,1cyclictest0-21swapper/121:56:331
1191674992119,1cyclictest0-21swapper/120:19:331
1191674992119,1cyclictest0-21swapper/119:53:041
1191674992119,1cyclictest0-21swapper/100:21:121
1191674992118,2cyclictest3056-21PK-Backend19:49:251
1191674992114,5cyclictest883-21dbus-daemon21:20:011
1191674992113,6cyclictest883-21dbus-daemon23:19:511
1191674992113,6cyclictest1299286-21ntp_states00:10:181
119167499210,20cyclictest0-21swapper/120:02:161
1191672992119,1cyclictest0-21swapper/023:52:420
1191672992119,1cyclictest0-21swapper/023:45:520
1191672992119,1cyclictest0-21swapper/023:34:500
1191672992119,1cyclictest0-21swapper/023:29:020
1191672992119,1cyclictest0-21swapper/022:55:500
1191672992119,1cyclictest0-21swapper/022:51:380
1191672992119,1cyclictest0-21swapper/020:47:270
1191672992119,1cyclictest0-21swapper/020:10:090
1191672992119,1cyclictest0-21swapper/019:49:410
1191672992119,1cyclictest0-21swapper/019:42:580
1191672992119,1cyclictest0-21swapper/019:17:510
1191672992119,1cyclictest0-21swapper/019:10:540
1191672992119,1cyclictest0-21swapper/000:15:330
1191672992119,1cyclictest0-21swapper/000:12:080
1191672992119,1cyclictest0-21swapper/000:07:560
1191672992118,2cyclictest0-21swapper/021:33:580
1191672992114,6cyclictest1288401-21sed23:40:170
1191672992114,5cyclictest2156-21gnome-shell22:29:090
119167299210,20cyclictest3056-21PK-Backend22:11:020
119167299210,20cyclictest0-21swapper/023:02:260
119167299210,20cyclictest0-21swapper/020:20:380
119167299210,20cyclictest0-21swapper/000:24:460
119167299210,20cyclictest0-21swapper/000:04:310
1191687992018,1cyclictest3056-21PK-Backend23:51:233
1191687992018,1cyclictest3056-21PK-Backend23:49:483
1191687992018,1cyclictest0-21swapper/322:42:083
1191687992018,1cyclictest0-21swapper/319:15:173
1191687992015,3cyclictest1255836-21sh22:10:003
1191687992014,4cyclictest1253309-21strings22:00:163
1191687992013,6cyclictest1213577-21sed20:10:003
119168799200,8cyclictest1258321-21taskset22:15:173
119168799200,19cyclictest0-21swapper/322:23:173
119168799200,19cyclictest0-21swapper/320:33:453
119168299202,8cyclictest34-21ksoftirqd/219:25:192
1191682992018,1cyclictest0-21swapper/200:39:502
1191682992018,1cyclictest0-21swapper/200:33:002
1191682992018,1cyclictest0-21swapper/200:26:592
1191682992017,2cyclictest331rcuc/220:55:582
1191682992016,3cyclictest331rcuc/222:07:402
1191682992016,3cyclictest331rcuc/221:01:562
1191682992016,3cyclictest331rcuc/220:27:592
1191682992016,3cyclictest331rcuc/219:19:062
1191682992010,3cyclictest34-21ksoftirqd/219:50:152
119168299200,8cyclictest1270689-21ntp_kernel_err22:50:182
119168299200,19cyclictest0-21swapper/223:28:192
119168299200,19cyclictest0-21swapper/221:25:022
119168299200,19cyclictest0-21swapper/221:17:182
119168299200,19cyclictest0-21swapper/219:57:152
1191674992018,1cyclictest0-21swapper/123:27:041
1191674992018,1cyclictest0-21swapper/123:11:161
1191674992018,1cyclictest0-21swapper/121:06:181
1191674992018,1cyclictest0-21swapper/119:39:381
1191674992018,1cyclictest0-21swapper/119:31:071
1191674992017,2cyclictest3056-21PK-Backend20:08:181
1191674992017,2cyclictest3056-21PK-Backend19:16:201
1191674992011,7cyclictest883-21dbus-daemon22:55:001
119167499200,19cyclictest3056-21PK-Backend22:06:421
119167499200,19cyclictest3056-21PK-Backend21:21:181
119167499200,19cyclictest3056-21PK-Backend21:12:131
119167499200,19cyclictest3056-21PK-Backend20:38:531
119167499200,19cyclictest3056-21PK-Backend20:32:451
119167499200,19cyclictest0-21swapper/119:40:411
1191672992018,1cyclictest0-21swapper/022:00:550
1191672992018,1cyclictest0-21swapper/021:06:540
1191672992018,1cyclictest0-21swapper/020:32:470
1191672992018,1cyclictest0-21swapper/020:28:140
1191672992018,1cyclictest0-21swapper/019:54:560
1191672992015,3cyclictest1306938-21sh00:35:000
119167299200,1cyclictest3056-21PK-Backend00:35:340
119167299200,19cyclictest0-21swapper/022:42:390
119167299200,19cyclictest0-21swapper/021:49:130
119167299200,19cyclictest0-21swapper/021:03:090
119167299200,19cyclictest0-21swapper/020:16:120
1191687991917,1cyclictest3056-21PK-Backend23:13:043
1191687991917,1cyclictest3056-21PK-Backend21:58:563
1191687991917,1cyclictest3056-21PK-Backend21:45:033
1191687991917,1cyclictest0-21swapper/321:12:253
1191687991917,1cyclictest0-21swapper/319:51:013
1191687991917,1cyclictest0-21swapper/319:31:403
119168799191,6cyclictest1293924-21/usr/sbin/munin23:55:233
1191687991916,2cyclictest3056-21PK-Backend20:18:583
1191687991914,3cyclictest1296314-21munin-run00:05:003
119168799190,1cyclictest3056-21PK-Backend22:33:163
119168799190,1cyclictest3056-21PK-Backend22:25:253
119168799190,1cyclictest3056-21PK-Backend19:29:033
119168799190,1cyclictest0-21swapper/319:42:553
119168799190,18cyclictest3056-21PK-Backend22:53:443
119168799190,18cyclictest0-21swapper/323:07:093
119168799190,18cyclictest0-21swapper/322:38:233
119168799190,18cyclictest0-21swapper/300:38:353
1191682991916,2cyclictest331rcuc/223:58:352
1191682991916,2cyclictest331rcuc/219:42:242
1191682991915,3cyclictest978-21snmpd20:53:212
1191682991911,6cyclictest1192746-21awk19:10:202
1191682991910,2cyclictest141rcu_preempt22:59:192
119167499193,14cyclictest1257972-21cpuspeed_turbos22:15:131
1191674991917,1cyclictest3056-21PK-Backend20:41:171
1191674991917,1cyclictest0-21swapper/122:48:521
1191674991916,2cyclictest3056-21PK-Backend20:10:561
119167499190,1cyclictest3056-21PK-Backend21:02:321
119167499190,18cyclictest0-21swapper/123:01:161
1191672991917,1cyclictest0-21swapper/023:11:390
1191672991917,1cyclictest0-21swapper/022:09:260
1191672991917,1cyclictest0-21swapper/020:41:190
1191672991917,1cyclictest0-21swapper/019:39:060
1191672991917,1cyclictest0-21swapper/019:31:360
1191672991910,6cyclictest436-20systemd-journal23:40:000
119167299190,1cyclictest0-21swapper/023:56:540
119167299190,18cyclictest0-21swapper/021:44:390
119167299190,18cyclictest0-21swapper/021:44:390
139099180,14rtkit-daemon26-21ksoftirqd/122:24:011
119168799189,6cyclictest436-20systemd-journal20:25:003
1191687991816,1cyclictest0-21swapper/320:45:213
1191687991816,1cyclictest0-21swapper/300:23:073
1191687991813,3cyclictest1279909-21ntp_states23:15:193
1191687991813,3cyclictest1237727-21timerandwakeup21:15:223
1191687991813,3cyclictest1228465-21df20:50:133
1191687991811,5cyclictest1286716-21cat23:35:163
119168799181,15cyclictest1283020-21diskstats23:25:143
1191687991811,5cyclictest1272893-21cat22:55:213
119168799180,6cyclictest1204901-21diskstats19:45:143
119168799180,1cyclictest3056-21PK-Backend21:05:363
119168799180,1cyclictest0-21swapper/321:20:513
1191674991816,1cyclictest3056-21PK-Backend00:06:461
1191674991816,1cyclictest0-21swapper/121:45:041
1191674991816,1cyclictest0-21swapper/121:32:471
1191674991816,1cyclictest0-21swapper/121:26:391
1191674991816,1cyclictest0-21swapper/119:26:271
1191674991812,5cyclictest1266694-21expr22:40:121
1191674991812,5cyclictest1245991-21expr21:40:131
1191674991812,5cyclictest1245991-21expr21:40:121
1191674991811,5cyclictest436-20systemd-journal21:39:531
1191674991810,6cyclictest883-21dbus-daemon22:28:011
1191672991816,1cyclictest0-21swapper/021:59:190
1191672991816,1cyclictest0-21swapper/020:38:550
1191672991816,1cyclictest0-21swapper/019:59:550
1191672991812,4cyclictest2156-21gnome-shell21:53:090
1191672991811,5cyclictest883-21dbus-daemon21:14:500
119167299180,1cyclictest0-21swapper/022:39:280
1191687991715,1cyclictest3056-21PK-Backend22:47:503
1191687991712,3cyclictest1242621-21diskstats21:30:143
1191687991712,3cyclictest1210089-21df20:00:123
119168799170,5cyclictest1201051-21cat19:35:093
119168799170,16cyclictest3056-21PK-Backend21:29:503
119168799170,15cyclictest1249914-21ntp_states21:50:193
119168799170,15cyclictest1245447-21munin-plugin-st21:40:003
119167499179,7cyclictest1253101-21cat22:00:151
119167499179,6cyclictest1229562-21cron20:55:001
1191674991715,1cyclictest0-21swapper/123:45:301
1191674991714,2cyclictest3056-21PK-Backend22:10:391
119167299178,7cyclictest436-20systemd-journal22:20:010
1191672991710,5cyclictest1263205-21tr22:30:140
1191687991614,1cyclictest3056-21PK-Backend23:04:123
1191687991613,2cyclictest3056-21PK-Backend23:40:083
1191687991611,3cyclictest1219259-21cpuspeed20:25:143
119167499169,5cyclictest1196827-21cat19:20:191
1191674991611,3cyclictest1271910-21sh22:55:011
1191672991612,3cyclictest13-21ksoftirqd/022:50:000
119167299161,13cyclictest1303911-21cpuspeed_turbos00:25:130
1191672991611,3cyclictest1230046-21cstates20:55:130
119168799150,14cyclictest3056-21PK-Backend21:41:323
119168799150,14cyclictest3056-21PK-Backend21:41:323
119167499150,14cyclictest1227843-21idleruntime-cro20:50:011
119167499149,4cyclictest1265693-21grep22:35:231
119167299141,10cyclictest1238520-21apt21:20:130
119167499106,3cyclictest3056-21PK-Backend20:59:351
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional