You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-09 - 09:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat May 09, 2026 00:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
391940321190,2sleep30-21swapper/319:50:003
397256221110,3sleep30-21swapper/322:20:013
395689821030,3sleep03056-21PK-Backend21:35:110
39484322980,3sleep20-21swapper/221:10:192
390487528038,35sleep30-21swapper/319:09:593
139099770,67rtkit-daemon0-21swapper/119:08:181
390463026942,21sleep00-21swapper/019:06:510
39042892690,7sleep2331rcuc/219:05:232
39357022490,3sleep33056-21PK-Backend20:35:113
139099420,3rtkit-daemon0-21swapper/222:53:022
390504399383,24cyclictest3928070-21seq20:13:111
3905043993813,12cyclictest4011330-21/usr/sbin/munin00:10:171
390504399361,25cyclictest4018255-21runrttasks00:30:021
390504399361,18cyclictest4016164-21taskset00:23:361
390504399340,32cyclictest3988877-21expr23:05:131
390504399340,19cyclictest4003723-21seq23:48:281
39393402330,4sleep0131ksoftirqd/020:45:150
390504399331,17cyclictest3970353-21runrttasks22:11:521
390504399330,20cyclictest3917351-21seq19:42:191
390504399330,19cyclictest3993262-21taskset23:17:401
390504399330,18cyclictest4017680-21taskset00:25:491
390504399330,18cyclictest3915545-21seq19:36:261
390504399330,17cyclictest3977593-21runrttasks22:31:451
390504399321,18cyclictest3947048-21taskset21:06:561
390504399321,16cyclictest3968656-21runrttasks22:07:191
390504399321,16cyclictest3929722-21runrttasks20:17:081
390504399320,19cyclictest3987907-21taskset23:00:401
390504399320,19cyclictest3980985-21taskset22:40:471
390504399320,18cyclictest3948966-21taskset21:14:181
390504399320,18cyclictest3943670-21taskset20:58:021
390504399320,17cyclictest3979370-21seq22:37:181
390504399320,17cyclictest3950626-21taskset21:18:191
390504399320,17cyclictest3913920-21seq19:32:501
40074142310,14chrt34-21ksoftirqd/200:00:012
390504399310,17cyclictest4014438-21runrttasks00:18:391
390504399310,17cyclictest3956121-21seq21:33:471
390504399310,17cyclictest3952197-21taskset21:21:161
390504399310,17cyclictest3941870-21taskset20:52:121
390504399310,16cyclictest3933112-21runrttasks20:26:101
390504399301,16cyclictest3917824-21/usr/sbin/munin19:45:111
390504399301,15cyclictest3996712-21seq23:27:311
390504399301,15cyclictest0-21swapper/120:46:351
390504399300,18cyclictest3945451-21taskset21:03:401
390504399300,17cyclictest4007243-21taskset23:59:021
390504399300,17cyclictest4001935-21seq23:42:461
390504399300,16cyclictest3991405-21taskset23:11:071
390504399300,16cyclictest3963204-21taskset21:52:231
390504399300,16cyclictest3961512-21seq21:47:541
390504399280,16cyclictest4000229-21seq23:38:051
390504399280,15cyclictest3973973-21seq22:23:071
390504399270,24cyclictest1895067-21snapd21:41:131
390504399261,22cyclictest3935795-21df20:35:121
3905043992610,14cyclictest3964670-21awk21:55:171
3905040992515,7cyclictest3943826-21cron21:00:000
3905040992511,11cyclictest3960361-21cpuspeed_turbos21:45:110
3905045992418,5cyclictest34-21ksoftirqd/220:10:002
390504399249,14cyclictest3975947-21apt-config22:30:011
390504399249,14cyclictest26-21ksoftirqd/122:59:371
390504099242,6cyclictest4016291-21apt00:25:010
39216052230,3chrt0-21swapper/219:55:122
390504399237,14cyclictest3954000-21sed21:25:211
390504399233,18cyclictest4020698-21uname00:35:181
3905043992318,4cyclictest0-21swapper/122:17:001
3905050992219,2cyclictest3056-21PK-Backend22:26:303
390505099220,3cyclictest978-21snmpd21:54:333
3905045992220,1cyclictest0-21swapper/223:22:582
3905045992220,1cyclictest0-21swapper/220:18:382
3905045992211,4cyclictest34-21ksoftirqd/222:40:142
390504399220,20cyclictest3921503-21wc19:55:121
390504399220,1cyclictest0-21swapper/122:01:221
3905040992220,1cyclictest3056-21PK-Backend19:17:540
3905040992220,1cyclictest0-21swapper/021:01:090
3905040992220,1cyclictest0-21swapper/020:17:020
3905040992220,1cyclictest0-21swapper/000:19:210
3905040992215,5cyclictest3977854-21cron22:35:000
390504099220,20cyclictest3056-21PK-Backend20:25:060
390504099220,20cyclictest3056-21PK-Backend19:46:470
390504099220,19cyclictest3947310-21munin-run21:10:000
139099220,16rtkit-daemon3056-21PK-Backend22:10:143
139099220,15rtkit-daemon3056-21PK-Backend20:25:362
139099220,14rtkit-daemon3056-21PK-Backend22:42:293
3905050992119,1cyclictest3056-21PK-Backend23:29:373
3905050992119,1cyclictest3056-21PK-Backend20:19:443
3905050992119,1cyclictest0-21swapper/323:09:233
3905050992119,1cyclictest0-21swapper/322:05:423
3905050992119,1cyclictest0-21swapper/320:45:313
3905050992119,1cyclictest0-21swapper/320:24:163
3905050992119,1cyclictest0-21swapper/319:50:023
3905050992119,1cyclictest0-21swapper/300:07:363
3905050992119,1cyclictest0-21swapper/300:00:403
390505099210,20cyclictest3056-21PK-Backend20:53:153
390505099210,20cyclictest3056-21PK-Backend00:31:013
390505099210,20cyclictest0-21swapper/322:04:403
390505099210,20cyclictest0-21swapper/321:55:343
390505099210,20cyclictest0-21swapper/320:02:533
390505099210,19cyclictest3056-21PK-Backend22:37:253
390505099210,19cyclictest3056-21PK-Backend21:25:403
3905045992119,1cyclictest0-21swapper/223:53:402
3905045992119,1cyclictest0-21swapper/223:03:312
3905045992119,1cyclictest0-21swapper/222:45:472
3905045992119,1cyclictest0-21swapper/222:35:462
3905045992119,1cyclictest0-21swapper/221:32:192
3905045992119,1cyclictest0-21swapper/221:28:542
3905045992119,1cyclictest0-21swapper/220:31:422
3905045992119,1cyclictest0-21swapper/220:23:112
3905045992119,1cyclictest0-21swapper/220:03:582
3905045992119,1cyclictest0-21swapper/219:51:422
3905045992119,1cyclictest0-21swapper/219:14:502
3905045992119,1cyclictest0-21swapper/200:32:402
3905045992119,1cyclictest0-21swapper/200:24:432
3905045992119,1cyclictest0-21swapper/200:10:502
3905045992115,4cyclictest34-21ksoftirqd/221:20:012
3905045992112,8cyclictest34-21ksoftirqd/222:00:162
390504599210,20cyclictest0-21swapper/223:28:052
390504599210,20cyclictest0-21swapper/222:12:412
390504599210,20cyclictest0-21swapper/222:06:122
390504599210,20cyclictest0-21swapper/221:36:322
390504599210,20cyclictest0-21swapper/221:02:312
390504399216,13cyclictest0-21swapper/123:54:591
390504399212,17cyclictest3956668-21cmp21:35:011
390504399211,18cyclictest3921067-21munin-run19:55:001
390504399211,18cyclictest1-21systemd23:31:351
390504399211,17cyclictest4007883-21apt00:00:111
390504399211,17cyclictest3982335-21ntp_states22:45:181
390504399210,19cyclictest3905498-21expr19:10:121
3905040992119,1cyclictest3056-21PK-Backend23:51:160
3905040992119,1cyclictest3056-21PK-Backend22:56:140
3905040992119,1cyclictest3056-21PK-Backend20:43:180
3905040992119,1cyclictest3056-21PK-Backend20:08:370
3905040992119,1cyclictest3056-21PK-Backend00:14:280
3905040992119,1cyclictest0-21swapper/023:46:230
3905040992119,1cyclictest0-21swapper/023:44:200
3905040992119,1cyclictest0-21swapper/022:46:410
3905040992119,1cyclictest0-21swapper/022:11:390
3905040992119,1cyclictest0-21swapper/021:21:510
3905040992119,1cyclictest0-21swapper/019:39:230
3905040992112,6cyclictest4016706-21cut00:25:110
3905040992112,6cyclictest1095-21runrttasks21:10:190
390504099210,20cyclictest3056-21PK-Backend22:17:540
390504099210,20cyclictest0-21swapper/022:36:540
390504099210,19cyclictest3056-21PK-Backend20:24:180
390504099210,19cyclictest3056-21PK-Backend19:29:360
390504099210,19cyclictest3056-21PK-Backend19:11:050
390504099210,19cyclictest0-21swapper/021:43:130
390504099210,19cyclictest0-21swapper/000:35:510
39165352200,3sleep00-21swapper/019:40:150
3905050992018,1cyclictest3056-21PK-Backend23:45:463
3905050992018,1cyclictest3056-21PK-Backend23:15:313
3905050992018,1cyclictest3056-21PK-Backend20:11:593
3905050992018,1cyclictest3056-21PK-Backend00:37:443
3905050992018,1cyclictest3056-21PK-Backend00:27:093
3905050992018,1cyclictest0-21swapper/322:55:573
3905050992018,1cyclictest0-21swapper/320:06:313
3905050992018,1cyclictest0-21swapper/319:28:253
3905050992017,2cyclictest3056-21PK-Backend23:55:463
3905050992017,2cyclictest3056-21PK-Backend22:32:463
3905050992017,2cyclictest3056-21PK-Backend21:33:033
3905050992015,3cyclictest3958494-21apt21:40:103
3905050992014,4cyclictest2156-21gnome-shell00:11:093
3905050992012,6cyclictest3933972-21expr20:30:123
3905050992012,6cyclictest3932048-21ls20:25:113
390505099201,17cyclictest3997915-21/usr/sbin/munin23:30:193
390505099200,1cyclictest3056-21PK-Backend21:15:543
390505099200,1cyclictest3056-21PK-Backend20:57:483
390505099200,19cyclictest3056-21PK-Backend23:43:433
390505099200,19cyclictest3056-21PK-Backend23:12:273
390505099200,19cyclictest0-21swapper/319:20:553
390505099200,19cyclictest0-21swapper/319:10:153
390504599209,4cyclictest34-21ksoftirqd/220:14:422
3905045992018,1cyclictest0-21swapper/222:31:072
3905045992018,1cyclictest0-21swapper/221:54:162
3905045992018,1cyclictest0-21swapper/219:48:162
3905045992018,1cyclictest0-21swapper/219:43:162
3905045992018,1cyclictest0-21swapper/200:26:462
3905045992013,5cyclictest4021477-21cron00:40:002
3905045992010,3cyclictest34-21ksoftirqd/220:58:012
390504599200,1cyclictest0-21swapper/223:10:272
390504599200,1cyclictest0-21swapper/221:56:052
390504599200,1cyclictest0-21swapper/221:46:592
390504599200,1cyclictest0-21swapper/221:05:422
390504599200,1cyclictest0-21swapper/219:30:522
390504599200,19cyclictest0-21swapper/220:51:372
390504599200,19cyclictest0-21swapper/220:46:022
390504599200,19cyclictest0-21swapper/219:25:592
390504599200,19cyclictest0-21swapper/200:17:462
390504599200,19cyclictest0-21swapper/200:07:462
390504399209,10cyclictest4010011-21hddtemp_smartct00:05:151
390504399200,18cyclictest3994434-21grep23:20:161
3905040992018,1cyclictest3056-21PK-Backend21:56:320
3905040992018,1cyclictest3056-21PK-Backend20:37:090
3905040992018,1cyclictest3056-21PK-Backend19:59:580
3905040992018,1cyclictest0-21swapper/023:32:300
3905040992018,1cyclictest0-21swapper/023:11:280
3905040992018,1cyclictest0-21swapper/023:01:550
3905040992018,1cyclictest0-21swapper/022:08:010
3905040992018,1cyclictest0-21swapper/020:54:130
390504099201,17cyclictest3914115-21perf19:35:000
390504099200,1cyclictest3056-21PK-Backend23:55:080
390504099200,1cyclictest0-21swapper/020:13:040
390504099200,19cyclictest3056-21PK-Backend23:22:300
390504099200,19cyclictest3056-21PK-Backend22:50:330
390504099200,19cyclictest0-21swapper/021:30:020
139099200,14rtkit-daemon3056-21PK-Backend23:35:413
139099200,14rtkit-daemon0-21swapper/322:51:143
139099200,14rtkit-daemon0-21swapper/223:39:532
40081612190,4sleep23056-21PK-Backend00:00:132
39875222190,4sleep33056-21PK-Backend23:00:193
3905050991917,1cyclictest3056-21PK-Backend21:06:203
3905050991917,1cyclictest3056-21PK-Backend21:04:453
3905050991917,1cyclictest0-21swapper/319:56:513
3905050991916,2cyclictest3056-21PK-Backend22:20:153
3905050991916,2cyclictest3056-21PK-Backend21:10:193
3905050991916,2cyclictest3056-21PK-Backend19:33:193
3905050991913,4cyclictest3937166-21apt20:40:103
390505099190,1cyclictest3056-21PK-Backend21:47:303
390505099190,1cyclictest3056-21PK-Backend00:15:333
390504599199,8cyclictest3998269-21users23:30:232
3905045991917,1cyclictest0-21swapper/223:05:342
3905045991917,1cyclictest0-21swapper/222:58:382
3905045991917,1cyclictest0-21swapper/222:22:492
3905045991917,1cyclictest0-21swapper/221:24:422
3905045991914,3cyclictest3907232-21proc_pri19:15:022
390504599190,1cyclictest0-21swapper/221:40:442
390504599190,1cyclictest0-21swapper/219:24:582
390504599190,18cyclictest0-21swapper/223:49:342
390504399198,10cyclictest0-21swapper/119:23:361
390504399198,10cyclictest0-21swapper/119:19:061
390504399191,16cyclictest3984419-21chrt22:50:221
390504399191,16cyclictest3933673-21/usr/sbin/munin20:30:251
390504399190,18cyclictest0-21swapper/120:22:091
390504399190,17cyclictest3937257-21ls20:40:111
390504399190,17cyclictest3911635-21ntpq19:25:181
3905040991917,1cyclictest0-21swapper/023:15:340
3905040991917,1cyclictest0-21swapper/022:22:200
3905040991916,2cyclictest3056-21PK-Backend00:05:560
3905040991912,5cyclictest3954074-21ls21:25:210
390504099190,1cyclictest3056-21PK-Backend00:03:530
390504099190,1cyclictest0-21swapper/022:25:590
390504099190,1cyclictest0-21swapper/021:52:060
390504099190,18cyclictest3056-21PK-Backend22:04:220
390504099190,18cyclictest0-21swapper/000:31:590
3905050991816,1cyclictest0-21swapper/321:38:043
3905050991815,2cyclictest3907438-21apt19:15:093
3905050991815,2cyclictest3056-21PK-Backend19:41:043
390505099181,13cyclictest82332-21update-manager23:51:333
390504599187,4cyclictest34-21ksoftirqd/222:29:312
390504599187,2cyclictest141rcu_preempt20:40:062
3905045991811,5cyclictest34-21ksoftirqd/223:45:002
390504399180,17cyclictest3924135-21chrt20:00:301
390504399180,16cyclictest3926487-21apt-config20:10:011
3905040991816,1cyclictest0-21swapper/023:25:340
3905040991811,4cyclictest3950965-21latency_hist21:20:010
390504099180,1cyclictest3056-21PK-Backend20:34:460
39717232170,4sleep23056-21PK-Backend22:15:192
3905050991715,1cyclictest3056-21PK-Backend19:39:143
3905050991710,5cyclictest3993702-21/usr/sbin/munin23:20:113
390504599179,6cyclictest34-21ksoftirqd/219:35:102
390504599178,3cyclictest34-21ksoftirqd/220:35:352
3905045991714,2cyclictest331rcuc/223:17:392
3905040991715,1cyclictest3056-21PK-Backend20:02:350
3905040991715,1cyclictest3056-21PK-Backend19:54:580
3905040991715,1cyclictest0-21swapper/023:38:250
3905040991715,1cyclictest0-21swapper/022:42:490
3905040991714,2cyclictest3056-21PK-Backend23:09:250
390505099160,14cyclictest3056-21PK-Backend21:25:003
3905040991614,1cyclictest3056-21PK-Backend19:24:090
3905050991512,2cyclictest3056-21PK-Backend00:24:193
390505099150,13cyclictest3056-21PK-Backend22:45:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional