You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-17 - 08:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Tue Mar 17, 2026 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
366487499489448,19cyclictest3735751-21turbostat22:30:013
366487499481457,20cyclictest3764048-21turbostat23:49:593
366487499466457,5cyclictest3716713-21turbostat21:39:553
366487499463455,5cyclictest3730550-21turbostat22:19:543
366487499463455,5cyclictest3690142-21turbostat20:24:553
366487499462455,4cyclictest3697372-21turbostat20:44:553
366487499462455,4cyclictest3679148-21turbostat19:54:553
366487499461441,17cyclictest3673960-21turbostat19:35:003
366487499460451,6cyclictest3725363-21turbostat22:04:563
366487499459451,5cyclictest3718446-21turbostat21:44:563
366487499459451,5cyclictest3718446-21turbostat21:44:563
366487499457450,4cyclictest3778210-21turbostat00:34:553
366487499457436,18cyclictest3684645-21turbostat20:05:003
366487499455448,4cyclictest3776179-21turbostat00:29:563
366487499452444,5cyclictest3781678-21turbostat00:39:593
366487499452444,5cyclictest3725363-21turbostat22:00:003
366487499451442,6cyclictest3735751-21turbostat22:34:563
366487499450443,4cyclictest3709786-21turbostat21:19:563
366487499450442,5cyclictest3695641-21turbostat20:39:553
366487499449440,6cyclictest3680881-21turbostat19:59:553
366487399449441,5cyclictest3779937-21turbostat00:39:552
366487499448440,5cyclictest3670495-21turbostat19:24:593
366487499446439,4cyclictest3716713-21turbostat21:35:003
366487499446439,4cyclictest3677410-21turbostat19:45:013
366487499444436,5cyclictest3734016-21turbostat22:25:013
366487499442435,4cyclictest3760578-21turbostat23:40:013
366487499441431,7cyclictest3714967-21turbostat21:29:593
366487499441419,19cyclictest3679148-21turbostat19:49:593
366487499440431,6cyclictest3765776-21turbostat23:54:593
366487499438431,4cyclictest3695641-21turbostat20:34:593
366487499438430,5cyclictest3670495-21turbostat19:29:553
366487499438429,6cyclictest3730550-21turbostat22:15:003
366487499438429,6cyclictest3700834-21turbostat20:50:013
366487499437428,6cyclictest3776179-21turbostat00:25:013
366487499434427,4cyclictest3692178-21turbostat20:29:563
366487499433426,4cyclictest3740943-21turbostat22:49:543
366487499433426,4cyclictest3704595-21turbostat20:59:593
366487499432425,4cyclictest3740943-21turbostat22:44:593
366487499431424,4cyclictest3753353-21turbostat23:24:553
366487499430422,5cyclictest3688414-21turbostat20:19:553
366487499430399,14cyclictest3774454-21turbostat00:19:593
366487499429422,4cyclictest3702560-21turbostat20:55:013
366487499429422,4cyclictest3702560-21turbostat20:55:013
366487499427420,4cyclictest3765776-21turbostat23:59:553
366487499427420,4cyclictest3666536-21turbostat19:19:553
366487499412405,4cyclictest3769242-21turbostat00:09:563
366487499412405,4cyclictest3769242-21turbostat00:09:563
366486699412404,5cyclictest3713243-21turbostat21:29:540
366487499410403,4cyclictest3757118-21turbostat23:34:553
366487399408401,4cyclictest3700834-21turbostat20:54:562
366487399408401,4cyclictest3700834-21turbostat20:54:562
366487499405397,5cyclictest3753353-21turbostat23:20:003
366487399402393,6cyclictest3762319-21turbostat23:49:562
366486999402391,8cyclictest3760578-21turbostat23:44:561
366487499401394,4cyclictest3746419-21turbostat23:00:003
366487499401394,4cyclictest3720172-21turbostat21:49:553
366487499400393,4cyclictest3762319-21turbostat23:45:013
366487499400392,5cyclictest3749896-21turbostat23:10:003
366487499399391,5cyclictest3723631-21turbostat21:54:593
366487499396389,4cyclictest3746419-21turbostat23:04:553
366487499396389,4cyclictest3704595-21turbostat21:04:553
366487499395387,5cyclictest3711514-21turbostat21:24:563
366487499395372,20cyclictest3664749-21turbostat19:14:553
366487499390381,6cyclictest3742671-21turbostat22:54:563
366487399389378,8cyclictest3774454-21turbostat00:24:542
366487499388379,6cyclictest3755079-21turbostat23:29:553
366487499388368,17cyclictest3708054-21turbostat21:10:003
366487499380373,4cyclictest3727094-21turbostat22:09:563
366487499379372,4cyclictest3767506-21turbostat00:04:563
366487399375353,19cyclictest3744396-21turbostat22:59:552
366487499369362,4cyclictest3686687-21turbostat20:14:563
366487499362354,5cyclictest3772714-21turbostat00:14:593
366487499356347,6cyclictest3675687-21turbostat19:40:003
366487499350344,3cyclictest3686687-21turbostat20:10:013
366487499350344,3cyclictest3686687-21turbostat20:10:003
366487499345338,4cyclictest3709786-21turbostat21:15:013
366487399345338,4cyclictest3734016-21turbostat22:29:552
366487499333313,17cyclictest3751627-21turbostat23:15:013
366487499318311,4cyclictest3737476-21turbostat22:39:553
366414422240,9sleep04497-21PK-Backend19:05:240
3664626213572,53sleep20-21swapper/219:08:402
3663252210873,23sleep30-21swapper/319:05:083
3664483210774,24sleep10-21swapper/119:06:501
197399720,5rtkit-daemon0-21swapper/120:11:531
37124722670,5chrt0-21swapper/121:20:231
36758082670,5chrt4497-21PK-Backend19:40:022
197399610,8rtkit-daemon1972-21rtkit-daemon19:25:052
37653912560,5sleep1251rcuc/123:50:311
36813652540,6sleep1251rcuc/119:55:141
37232182510,19chrt26-21ksoftirqd/121:50:291
36768112510,6sleep24497-21PK-Backend19:40:272
37469542500,5chrt0-21swapper/023:00:150
36843802460,11sleep013-21ksoftirqd/020:02:180
3664873994628,13cyclictest1-21systemd22:33:062
3664873994625,17cyclictest3699726-21df20:45:142
197399460,31rtkit-daemon3705600-21ntp_states21:00:232
3664873994513,28cyclictest976-21rs:main2
3664869994523,18cyclictest3689547-21python320:15:301
3664869994511,7cyclictest141rcu_preempt21:05:261
3664866994527,14cyclictest3685566-40apt-check20:05:230
3664866994527,14cyclictest3685566-40apt-check20:05:220
3664869994413,6cyclictest141rcu_preempt20:20:311
3664873994323,6cyclictest34-21ksoftirqd/219:15:022
3664873994323,12cyclictest34-21ksoftirqd/200:30:142
366486999433,14cyclictest0-21swapper/121:00:021
3664869994316,5cyclictest26-21ksoftirqd/122:40:281
3664869994313,11cyclictest26-21ksoftirqd/123:20:211
366486699431,21cyclictest3768510-21/usr/sbin/munin00:00:240
197399430,36rtkit-daemon927-21gmain19:51:282
37300252420,6sleep0161rcuc/022:10:270
3664873994223,6cyclictest34-21ksoftirqd/220:25:022
3664869994231,8cyclictest26-21ksoftirqd/120:05:021
366486999421,20cyclictest3721020-21chrt21:45:221
3664866994213,7cyclictest141rcu_preempt20:20:270
3664866994212,12cyclictest13-21ksoftirqd/020:45:250
37090712410,7sleep04497-21PK-Backend21:10:240
366487399418,17cyclictest141rcu_preempt21:14:552
3664873994130,6cyclictest34-21ksoftirqd/220:40:212
3664873994123,6cyclictest34-21ksoftirqd/223:25:022
3664873994123,6cyclictest34-21ksoftirqd/222:55:022
3664873994122,6cyclictest34-21ksoftirqd/222:35:242
3664869994125,12cyclictest3694648-21awk20:30:191
3664869994123,13cyclictest26-21ksoftirqd/122:20:561
3664869994121,13cyclictest26-21ksoftirqd/122:49:191
3664869994116,5cyclictest26-21ksoftirqd/122:50:351
366486999411,17cyclictest3701576-21chrt20:50:201
366486999411,17cyclictest3701576-21chrt20:50:191
3664869994110,8cyclictest26-21ksoftirqd/122:08:371
366486699415,7cyclictest141rcu_preempt22:05:130
3664866994122,15cyclictest3766399-21df23:55:140
36678812400,5sleep0161rcuc/019:15:200
366487399402,33cyclictest3717349-21diskstats21:35:152
3664873994022,13cyclictest34-21ksoftirqd/221:05:292
3664869994024,12cyclictest923-21dbus-daemon23:10:211
3664869994023,6cyclictest26-21ksoftirqd/123:17:531
3664869994023,5cyclictest26-21ksoftirqd/122:29:441
3664869994022,6cyclictest26-21ksoftirqd/122:39:181
3664869994022,5cyclictest26-21ksoftirqd/121:35:021
3664869994021,10cyclictest26-21ksoftirqd/100:10:011
3664869994021,10cyclictest26-21ksoftirqd/100:10:011
3664869994013,6cyclictest141rcu_preempt00:30:131
366486999401,22cyclictest26-21ksoftirqd/100:30:001
3664866994017,18cyclictest3751980-21/usr/sbin/munin23:15:130
3664866994011,8cyclictest2069-21gnome-shell19:31:590
3664866994011,13cyclictest141rcu_preempt00:10:190
197399400,28rtkit-daemon4497-21PK-Backend23:28:062
197399400,27rtkit-daemon0-21swapper/200:19:102
366487399393,11cyclictest141rcu_preempt20:20:022
3664873993914,10cyclictest141rcu_preempt23:05:142
3664869993923,5cyclictest26-21ksoftirqd/123:47:061
3664869993923,5cyclictest26-21ksoftirqd/122:56:071
3664869993923,5cyclictest26-21ksoftirqd/119:17:511
3664869993923,5cyclictest26-21ksoftirqd/119:14:121
3664869993922,6cyclictest26-21ksoftirqd/123:38:571
3664869993922,6cyclictest26-21ksoftirqd/123:30:311
3664869993922,6cyclictest26-21ksoftirqd/123:27:101
3664869993922,6cyclictest26-21ksoftirqd/122:02:511
3664869993922,6cyclictest26-21ksoftirqd/121:35:301
3664869993922,6cyclictest26-21ksoftirqd/119:48:051
3664869993922,6cyclictest26-21ksoftirqd/100:37:541
3664869993922,6cyclictest26-21ksoftirqd/100:15:571
3664869993922,6cyclictest26-21ksoftirqd/100:03:451
3664869993921,14cyclictest26-21ksoftirqd/121:10:121
3664869993916,19cyclictest3685566-40apt-check20:05:211
3664869993916,19cyclictest3685566-40apt-check20:05:211
366486999391,14cyclictest37316682rt-features22:15:271
366486999391,13cyclictest4497-21PK-Backend21:28:181
366486699394,9cyclictest13-21ksoftirqd/023:45:190
3664866993920,14cyclictest3769875-21df00:05:130
3664866993920,14cyclictest3769875-21df00:05:120
3664866993914,8cyclictest13-21ksoftirqd/000:25:020
3664866993912,7cyclictest3765801-21munin-run23:55:010
3664866993911,8cyclictest2069-21gnome-shell20:42:590
3664873993821,6cyclictest34-21ksoftirqd/220:40:022
3664873993821,5cyclictest34-21ksoftirqd/219:50:012
366487399381,11cyclictest3725355-21apt21:59:592
3664869993822,6cyclictest26-21ksoftirqd/119:25:141
3664869993822,5cyclictest26-21ksoftirqd/122:11:241
3664869993822,5cyclictest26-21ksoftirqd/121:59:331
3664869993822,5cyclictest26-21ksoftirqd/121:43:551
3664869993822,5cyclictest26-21ksoftirqd/121:43:541
3664869993822,5cyclictest26-21ksoftirqd/119:42:371
3664869993822,5cyclictest26-21ksoftirqd/119:21:571
3664869993822,5cyclictest26-21ksoftirqd/100:22:411
3664869993822,5cyclictest26-21ksoftirqd/100:10:331
3664869993821,6cyclictest26-21ksoftirqd/121:03:191
3664869993821,6cyclictest26-21ksoftirqd/120:48:181
3664869993820,6cyclictest26-21ksoftirqd/122:30:131
366486999381,10cyclictest4497-21PK-Backend23:57:391
366486699388,5cyclictest141rcu_preempt20:00:000
366486699387,10cyclictest141rcu_preempt23:10:000
3664866993828,6cyclictest3672207-21cron19:30:000
3664866993824,11cyclictest3781667-21munin-plugin-st00:40:000
3664866993823,11cyclictest3757100-21cron23:30:000
3664866993821,13cyclictest13-21ksoftirqd/022:05:030
3664866993821,12cyclictest3719777-21uptime21:40:300
3664866993821,12cyclictest3719777-21uptime21:40:290
3664866993821,11cyclictest923-21dbus-daemon22:55:020
3664866993812,9cyclictest13-21ksoftirqd/000:20:010
3664866993811,9cyclictest13-21ksoftirqd/021:55:120
3664866993811,8cyclictest2069-21gnome-shell00:33:580
366486699381,13cyclictest36885982wc20:15:020
3664866993811,11cyclictest3777288-21/usr/sbin/munin00:25:220
3664866993810,7cyclictest13-21ksoftirqd/019:20:260
3664866993810,10cyclictest3715245-21/usr/sbin/munin21:30:110
197399380,27rtkit-daemon4497-21PK-Backend23:44:222
197399380,26rtkit-daemon4497-21PK-Backend21:49:222
3664873993720,13cyclictest34-21ksoftirqd/221:30:272
3664873993719,6cyclictest34-21ksoftirqd/200:00:012
3664869993723,10cyclictest3711259-21cron21:17:001
3664869993721,6cyclictest26-21ksoftirqd/123:02:091
3664869993720,6cyclictest26-21ksoftirqd/123:06:411
3664869993720,5cyclictest26-21ksoftirqd/119:35:261
366486699379,10cyclictest13-21ksoftirqd/022:20:240
3664866993723,5cyclictest13-21ksoftirqd/022:55:140
3664866993721,5cyclictest13-21ksoftirqd/021:45:270
3664866993721,12cyclictest13-21ksoftirqd/019:45:010
3664866993720,13cyclictest3696274-21df_inode20:35:160
3664866993720,13cyclictest13-21ksoftirqd/020:15:290
3664866993720,12cyclictest13-21ksoftirqd/020:25:270
3664866993719,6cyclictest13-21ksoftirqd/020:50:210
3664866993719,6cyclictest13-21ksoftirqd/020:50:210
3664866993712,7cyclictest13-21ksoftirqd/022:45:130
3664866993710,7cyclictest3736410-21/usr/sbin/munin22:30:190
3664866993710,7cyclictest13-21ksoftirqd/023:40:300
366487399364,13cyclictest141rcu_preempt21:29:262
3664873993624,8cyclictest3692624-21tr20:25:132
3664873993621,11cyclictest3776934-21expr00:25:142
3664873993621,11cyclictest3770244-21ntpq00:05:212
3664873993621,11cyclictest3770244-21ntpq00:05:212
3664873993620,5cyclictest34-21ksoftirqd/223:40:022
3664873993619,6cyclictest34-21ksoftirqd/219:15:232
3664873993619,12cyclictest3771523-21cat00:10:142
3664873993615,17cyclictest141rcu_preempt00:04:192
3664873993611,6cyclictest141rcu_preempt21:43:272
3664873993611,6cyclictest141rcu_preempt21:43:272
3664873993611,11cyclictest141rcu_preempt23:55:002
366486999364,12cyclictest1137-21runrttasks20:25:081
3664869993621,6cyclictest26-21ksoftirqd/119:50:281
3664869993620,5cyclictest26-21ksoftirqd/120:42:141
366486999361,14cyclictest4497-21PK-Backend19:30:301
3664866993626,7cyclictest3722138-21/usr/sbin/munin21:50:110
3664866993621,5cyclictest13-21ksoftirqd/023:20:220
3664866993621,12cyclictest13-21ksoftirqd/020:30:130
3664866993621,11cyclictest3750915-21awk23:10:250
3664866993620,12cyclictest3678549-21python319:45:270
3664866993619,12cyclictest13-21ksoftirqd/021:00:210
3664866993619,12cyclictest13-21ksoftirqd/020:56:250
3664866993619,12cyclictest13-21ksoftirqd/019:50:160
3664866993615,6cyclictest141rcu_preempt23:35:300
366486699361,14cyclictest3739561-21sort22:40:140
3664866993610,13cyclictest141rcu_preempt22:25:200
197399360,29rtkit-daemon0-21swapper/219:30:502
3664873993525,6cyclictest34-21ksoftirqd/220:02:202
3664873993525,5cyclictest34-21ksoftirqd/219:55:212
3664873993520,11cyclictest3695446-21seq20:32:452
366487399351,30cyclictest4497-21PK-Backend22:09:032
366487399351,15cyclictest4497-21PK-Backend22:50:012
3664869993519,5cyclictest26-21ksoftirqd/120:35:061
3664866993520,5cyclictest13-21ksoftirqd/021:05:300
3664866993520,11cyclictest3710760-21ntp_kernel_pll_21:15:220
3664866993520,11cyclictest13-21ksoftirqd/021:35:230
3664866993519,12cyclictest3731990-21sshd22:16:310
366487399345,8cyclictest141rcu_preempt22:15:022
3664873993423,6cyclictest34-21ksoftirqd/221:20:272
3664873993423,6cyclictest34-21ksoftirqd/220:56:312
3664873993423,6cyclictest34-21ksoftirqd/219:20:212
3664873993419,11cyclictest3758891-21ls23:35:012
3664873993416,6cyclictest34-21ksoftirqd/222:40:252
3664873993412,8cyclictest141rcu_preempt22:00:152
3664866993418,5cyclictest13-21ksoftirqd/021:20:260
3664866993418,12cyclictest3757768-21entropy23:30:180
3664866993417,5cyclictest13-21ksoftirqd/022:35:260
3664866993417,12cyclictest3665754-21netstat19:10:230
366487399335,10cyclictest141rcu_preempt21:18:352
3664873993324,5cyclictest4497-21PK-Backend23:15:212
3664873993323,5cyclictest34-21ksoftirqd/221:50:182
3664866993322,6cyclictest13-21ksoftirqd/019:35:150
3664873993222,6cyclictest34-21ksoftirqd/222:20:022
3664873993216,12cyclictest34-21ksoftirqd/220:10:012
3664873993216,12cyclictest34-21ksoftirqd/220:10:012
3664873993210,5cyclictest141rcu_preempt20:10:202
3664873993121,6cyclictest34-21ksoftirqd/223:00:122
366487399311,27cyclictest4497-21PK-Backend23:13:282
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional