You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-31 - 08:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun May 31, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
376581622130,5sleep10-21swapper/119:05:141
380470521360,3sleep30-21swapper/320:55:003
380470521360,3sleep30-21swapper/320:54:593
387197821090,3sleep234-21ksoftirqd/200:05:132
387061421080,4sleep33056-21PK-Backend00:00:183
376545527747,22sleep30-21swapper/319:05:133
376568826853,9sleep00-21swapper/019:05:140
376674826444,14sleep20-21swapper/219:08:052
38735462360,2sleep33056-21PK-Backend00:10:133
38791182340,3sleep00-21swapper/000:25:130
38423022320,4sleep13056-21PK-Backend22:40:121
38423022320,4sleep13056-21PK-Backend22:40:121
376709199271,21cyclictest978-21snmpd20:30:093
376708699276,4cyclictest26-21ksoftirqd/123:05:011
3767091992611,9cyclictest42-21ksoftirqd/319:40:213
139099260,4rtkit-daemon1389-21rtkit-daemon22:00:460
376709199251,20cyclictest3792511-21apt20:20:113
139099250,3rtkit-daemon1389-21rtkit-daemon22:09:410
3767088992413,8cyclictest883-21dbus-daemon20:48:322
376708699240,7cyclictest3056-21PK-Backend20:50:211
376708699240,7cyclictest3056-21PK-Backend20:50:211
3767081992414,8cyclictest3809877-21unin-run21:09:590
139099240,3rtkit-daemon1389-21rtkit-daemon20:46:411
376708699237,8cyclictest3056-21PK-Backend19:57:161
376708699235,10cyclictest26-21ksoftirqd/122:30:011
3767086992320,2cyclictest0-21swapper/122:02:441
376708699230,6cyclictest3789043-21apt20:10:101
139099230,15rtkit-daemon3056-21PK-Backend00:23:443
3767091992220,1cyclictest0-21swapper/320:28:563
376709199220,20cyclictest3056-21PK-Backend22:12:393
376708699225,6cyclictest26-21ksoftirqd/119:15:001
376708699224,16cyclictest3787487-21grep20:05:121
376708699222,6cyclictest3056-21PK-Backend20:55:211
3767086992220,1cyclictest3056-21PK-Backend21:14:171
3767086992220,1cyclictest0-21swapper/123:31:531
3767086992219,2cyclictest251rcuc/123:20:401
3767086992219,2cyclictest251rcuc/123:07:531
3767086992219,2cyclictest0-21swapper/121:25:521
376708699220,8cyclictest26-21ksoftirqd/122:05:101
376708699220,21cyclictest0-21swapper/120:16:311
376708199220,21cyclictest0-21swapper/019:10:170
139099220,15rtkit-daemon3056-21PK-Backend23:31:163
139099220,14rtkit-daemon3056-21PK-Backend23:24:333
3767091992119,1cyclictest3056-21PK-Backend22:03:533
3767091992119,1cyclictest3056-21PK-Backend21:50:213
3767091992119,1cyclictest3056-21PK-Backend20:42:213
3767091992119,1cyclictest0-21swapper/323:42:493
3767091992119,1cyclictest0-21swapper/322:47:473
3767091992119,1cyclictest0-21swapper/321:11:343
3767091992119,1cyclictest0-21swapper/319:39:143
3767091992119,1cyclictest0-21swapper/319:17:383
3767091992119,1cyclictest0-21swapper/300:39:413
3767091992119,1cyclictest0-21swapper/300:27:503
3767091992118,2cyclictest3056-21PK-Backend19:22:113
3767091992114,5cyclictest42-21ksoftirqd/319:45:133
3767091992114,4cyclictest2156-21gnome-shell20:18:083
3767091992113,6cyclictest3784945-21sshd19:55:323
3767091992112,6cyclictest436-20systemd-journal19:32:323
3767091992112,6cyclictest3783567-21cron19:55:003
3767091992111,7cyclictest886-21NetworkManager20:48:333
3767091992111,7cyclictest3809834-21cron21:10:003
376709199210,2cyclictest3056-21PK-Backend21:40:003
376709199210,20cyclictest3056-21PK-Backend23:11:193
376709199210,20cyclictest3056-21PK-Backend22:59:013
376709199210,20cyclictest3056-21PK-Backend20:09:163
376709199210,20cyclictest3056-21PK-Backend00:17:373
376709199210,19cyclictest3056-21PK-Backend22:09:543
376708899210,2cyclictest3780356-21/usr/sbin/munin19:45:142
376708899210,20cyclictest0-21swapper/222:59:212
376708899210,20cyclictest0-21swapper/222:00:472
376708899210,20cyclictest0-21swapper/219:31:422
376708899210,20cyclictest0-21swapper/200:22:212
376708699216,9cyclictest26-21ksoftirqd/121:25:011
376708699214,7cyclictest26-21ksoftirqd/119:30:041
376708699214,10cyclictest26-21ksoftirqd/119:24:551
3767086992119,1cyclictest3056-21PK-Backend21:33:031
3767086992119,1cyclictest0-21swapper/122:17:441
3767086992119,1cyclictest0-21swapper/121:05:391
3767086992119,1cyclictest0-21swapper/120:04:421
3767086992119,1cyclictest0-21swapper/100:21:071
3767086992118,2cyclictest3056-21PK-Backend23:40:031
3767086992118,2cyclictest3056-21PK-Backend22:48:461
3767086992118,2cyclictest3056-21PK-Backend19:40:081
3767086992118,2cyclictest3056-21PK-Backend00:32:091
3767086992113,6cyclictest1-21systemd23:00:001
376708699210,20cyclictest3056-21PK-Backend19:19:061
376708699210,20cyclictest0-21swapper/123:49:361
376708699210,20cyclictest0-21swapper/123:26:531
376708699210,20cyclictest0-21swapper/100:13:161
3767081992115,5cyclictest3859180-21/usr/sbin/munin23:29:590
376708199210,20cyclictest0-21swapper/019:39:580
376708199210,1cyclictest0-21swapper/022:24:450
139099210,14rtkit-daemon3056-21PK-Backend23:54:183
139099210,13rtkit-daemon3056-21PK-Backend23:58:483
3767091992018,1cyclictest3056-21PK-Backend23:06:333
3767091992018,1cyclictest3056-21PK-Backend21:56:303
3767091992018,1cyclictest3056-21PK-Backend20:02:263
3767091992018,1cyclictest0-21swapper/322:32:333
3767091992018,1cyclictest0-21swapper/300:31:163
3767091992017,2cyclictest3056-21PK-Backend21:43:393
3767091992017,2cyclictest3056-21PK-Backend20:12:003
376709199200,1cyclictest3056-21PK-Backend20:38:163
376709199200,19cyclictest3056-21PK-Backend23:18:293
376709199200,19cyclictest0-21swapper/320:57:493
376709199200,19cyclictest0-21swapper/319:12:113
376709199200,18cyclictest3816980-21dpkg21:30:013
3767088992018,1cyclictest3056-21PK-Backend21:25:252
3767088992018,1cyclictest0-21swapper/222:44:062
3767088992018,1cyclictest0-21swapper/222:44:052
3767088992018,1cyclictest0-21swapper/222:16:342
3767088992018,1cyclictest0-21swapper/221:47:152
3767088992018,1cyclictest0-21swapper/221:47:142
3767088992018,1cyclictest0-21swapper/221:32:002
3767088992018,1cyclictest0-21swapper/221:24:032
3767088992018,1cyclictest0-21swapper/220:11:502
3767088992017,2cyclictest3840253-21latency_hist22:35:002
3767088992013,6cyclictest3878970-21expr00:25:132
3767088992013,5cyclictest3785497-21cut20:00:012
3767088992012,6cyclictest3882677-21cat00:35:152
376708899200,1cyclictest0-21swapper/223:10:162
376708899200,19cyclictest0-21swapper/223:05:282
376708899200,19cyclictest0-21swapper/222:50:152
376708899200,19cyclictest0-21swapper/222:06:142
376708899200,19cyclictest0-21swapper/220:35:362
376708899200,19cyclictest0-21swapper/219:51:022
376708899200,19cyclictest0-21swapper/219:35:412
376708899200,19cyclictest0-21swapper/219:25:192
376708899200,19cyclictest0-21swapper/219:13:102
376708899200,19cyclictest0-21swapper/200:03:002
376708699204,6cyclictest26-21ksoftirqd/123:18:311
376708699202,7cyclictest26-21ksoftirqd/121:17:071
3767086992018,1cyclictest0-21swapper/122:12:171
3767086992017,2cyclictest3056-21PK-Backend20:30:441
3767086992017,2cyclictest251rcuc/122:37:131
376708699200,1cyclictest0-21swapper/100:19:181
376708699200,19cyclictest3056-21PK-Backend22:32:591
376708699200,19cyclictest3056-21PK-Backend20:26:041
376708699200,19cyclictest0-21swapper/123:11:311
376708699200,19cyclictest0-21swapper/122:23:111
376708699200,19cyclictest0-21swapper/121:54:461
376708699200,19cyclictest0-21swapper/121:48:381
376708699200,19cyclictest0-21swapper/121:48:381
376708699200,19cyclictest0-21swapper/120:37:191
3767081992018,1cyclictest0-21swapper/022:16:330
3767081992018,1cyclictest0-21swapper/020:21:280
3767081992016,3cyclictest13-21ksoftirqd/020:49:590
3767081992012,6cyclictest3788785-21cron20:10:000
3767081992011,7cyclictest3849773-21ntp_states23:00:190
376708199200,1cyclictest0-21swapper/023:51:160
376708199200,1cyclictest0-21swapper/019:31:260
376708199200,19cyclictest0-21swapper/022:54:310
376708199200,19cyclictest0-21swapper/022:46:140
376708199200,19cyclictest0-21swapper/021:29:420
376708199200,19cyclictest0-21swapper/021:16:440
376708199200,19cyclictest0-21swapper/020:02:220
376708199200,19cyclictest0-21swapper/019:17:340
139099200,14rtkit-daemon3056-21PK-Backend23:45:253
3767091991917,1cyclictest0-21swapper/321:32:173
3767091991917,1cyclictest0-21swapper/321:21:493
376709199191,5cyclictest3806832-21cpuspeed_turbos21:00:133
3767091991911,5cyclictest3842082-21/usr/sbin/munin22:40:123
3767091991911,5cyclictest3842082-21/usr/sbin/munin22:40:123
376709199190,1cyclictest0-21swapper/322:37:053
376709199190,1cyclictest0-21swapper/322:29:353
376709199190,18cyclictest3056-21PK-Backend22:24:083
3767088991917,1cyclictest3056-21PK-Backend19:23:442
3767088991917,1cyclictest0-21swapper/221:44:452
3767088991917,1cyclictest0-21swapper/221:38:022
3767088991917,1cyclictest0-21swapper/220:20:022
3767088991917,1cyclictest0-21swapper/200:30:112
3767088991917,1cyclictest0-21swapper/200:17:072
3767088991912,5cyclictest3831317-21tr22:10:132
376708899190,1cyclictest0-21swapper/223:28:142
376708899190,1cyclictest0-21swapper/220:53:482
376708899190,1cyclictest0-21swapper/220:53:482
376708899190,1cyclictest0-21swapper/220:28:132
376708899190,18cyclictest0-21swapper/223:39:012
376708899190,18cyclictest0-21swapper/223:39:012
376708899190,18cyclictest0-21swapper/222:38:462
376708899190,18cyclictest0-21swapper/222:25:342
376708899190,18cyclictest0-21swapper/221:52:492
376708899190,18cyclictest0-21swapper/221:16:192
376708899190,18cyclictest0-21swapper/220:02:512
3767086991917,1cyclictest3056-21PK-Backend00:37:091
3767086991917,1cyclictest0-21swapper/121:40:471
3767086991916,2cyclictest3056-21PK-Backend19:47:181
3767086991916,2cyclictest251rcuc/121:55:261
3767086991916,2cyclictest251rcuc/119:25:501
376708699191,5cyclictest26-21ksoftirqd/100:00:011
3767086991910,6cyclictest1095-21runrttasks23:50:221
3767086991910,3cyclictest26-21ksoftirqd/122:50:171
376708699190,1cyclictest3056-21PK-Backend21:01:121
376708699190,1cyclictest3056-21PK-Backend19:50:291
376708699190,1cyclictest3056-21PK-Backend00:28:451
376708699190,1cyclictest0-21swapper/120:42:471
376708699190,1cyclictest0-21swapper/100:08:031
3767081991917,1cyclictest0-21swapper/022:56:070
3767081991917,1cyclictest0-21swapper/022:11:060
3767081991917,1cyclictest0-21swapper/021:57:270
3767081991917,1cyclictest0-21swapper/020:51:300
3767081991917,1cyclictest0-21swapper/020:51:300
3767081991917,1cyclictest0-21swapper/020:35:410
3767081991917,1cyclictest0-21swapper/020:31:020
3767081991917,1cyclictest0-21swapper/020:29:400
3767081991917,1cyclictest0-21swapper/019:29:510
3767081991917,1cyclictest0-21swapper/000:06:170
376708199190,1cyclictest0-21swapper/023:57:390
376708199190,1cyclictest0-21swapper/023:34:540
376708199190,1cyclictest0-21swapper/022:41:410
376708199190,1cyclictest0-21swapper/022:41:410
376708199190,1cyclictest0-21swapper/021:35:430
376708199190,1cyclictest0-21swapper/000:13:070
376708199190,18cyclictest3056-21PK-Backend21:31:040
376708199190,18cyclictest0-21swapper/020:11:550
3767091991816,1cyclictest3056-21PK-Backend22:16:443
3767091991816,1cyclictest3056-21PK-Backend21:46:433
3767091991816,1cyclictest3056-21PK-Backend21:46:433
3767091991816,1cyclictest3056-21PK-Backend19:28:193
3767091991812,4cyclictest3873073-21turbostat00:10:013
3767091991811,5cyclictest3845747-21grep22:50:113
3767088991816,1cyclictest0-21swapper/223:54:362
3767088991816,1cyclictest0-21swapper/223:31:382
3767088991816,1cyclictest0-21swapper/223:04:412
3767088991816,1cyclictest0-21swapper/221:11:532
3767088991816,1cyclictest0-21swapper/220:56:442
3767088991816,1cyclictest0-21swapper/220:32:532
3767088991816,1cyclictest0-21swapper/220:07:172
3767088991816,1cyclictest0-21swapper/219:18:242
3767088991816,1cyclictest0-21swapper/200:11:542
3767088991815,2cyclictest3056-21PK-Backend19:44:262
3767088991811,5cyclictest3863382-21awk23:40:162
376708899180,1cyclictest0-21swapper/222:48:392
3767086991815,2cyclictest3056-21PK-Backend20:20:371
3767086991815,2cyclictest251rcuc/123:37:291
3767086991815,2cyclictest251rcuc/123:37:291
3767086991814,3cyclictest251rcuc/121:36:151
3767086991812,5cyclictest3870933-21sed00:00:221
376708699180,8cyclictest3776888-21runrttasks19:35:101
376708199183,5cyclictest13-21ksoftirqd/019:55:120
3767081991816,1cyclictest0-21swapper/023:09:460
3767081991816,1cyclictest0-21swapper/021:14:480
3767081991816,1cyclictest0-21swapper/000:18:210
3767081991813,4cyclictest1095-21runrttasks23:45:000
3767081991811,5cyclictest3852678-21grep23:10:120
3767081991811,5cyclictest3840893-21expr22:35:170
376708199180,1cyclictest0-21swapper/021:48:010
376708199180,1cyclictest0-21swapper/021:48:010
139099180,2rtkit-daemon1389-21rtkit-daemon20:40:530
3767091991715,1cyclictest0-21swapper/323:00:383
376709199170,15cyclictest38615602sleep323:35:163
376709199170,15cyclictest38615602sleep323:35:163
376709199170,15cyclictest38132462sleep321:19:223
3767088991715,1cyclictest0-21swapper/220:44:352
3767088991712,3cyclictest3807593-21timerandwakeup21:00:222
3767088991711,5cyclictest3810032-21sed21:10:012
3767088991711,4cyclictest3836693-21dpkg22:25:012
3767081991715,1cyclictest0-21swapper/022:25:320
3767081991715,1cyclictest0-21swapper/021:44:210
3767081991715,1cyclictest0-21swapper/021:00:560
3767081991715,1cyclictest0-21swapper/020:55:490
3767081991715,1cyclictest0-21swapper/019:22:550
3767081991715,1cyclictest0-21swapper/000:01:040
3767081991712,4cyclictest13-21ksoftirqd/023:20:010
3767081991711,5cyclictest3781834-21cron19:50:000
3767081991710,5cyclictest3876886-21ls00:20:100
3767081991710,5cyclictest3791329-21cat20:15:160
3767091991611,3cyclictest3859164-21turbostat.cron23:29:593
376708199169,5cyclictest3864642-21apt23:45:130
376708199169,5cyclictest3824248-21grep21:50:110
3767081991614,1cyclictest3056-21PK-Backend00:36:260
3767081991610,4cyclictest3782198-21cpuspeed19:50:130
376708899158,6cyclictest883-21dbus-daemon23:25:012
3767088991513,1cyclictest0-21swapper/223:18:272
3767088991513,1cyclictest0-21swapper/221:56:412
376708899151,12cyclictest3864782-21cpuspeed_turbos23:45:132
376708199159,5cyclictest3779534-21grep19:40:210
376708199157,6cyclictest436-20systemd-journal23:22:320
3767081991511,3cyclictest3838617-21/usr/sbin/munin22:30:190
3767081991511,3cyclictest13-21ksoftirqd/023:35:170
3767081991511,3cyclictest13-21ksoftirqd/023:35:170
3767081991510,3cyclictest1095-21runrttasks00:31:270
3767088991411,2cyclictest1095-21runrttasks23:59:032
376708899140,5cyclictest0-21swapper/220:15:322
3767081991412,1cyclictest0-21swapper/021:20:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional