You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-07 - 16:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Jun 07, 2026 12:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
379299022990,22sleep10-21swapper/107:05:211
380091721180,3sleep00-21swapper/007:25:210
381868721080,4sleep13056-21PK-Backend08:15:231
379363627640,13sleep30-21swapper/307:07:523
379290027245,17sleep00-21swapper/007:05:200
379352626643,16sleep20-21swapper/207:06:272
38580342620,10sleep013-21ksoftirqd/010:10:100
38300272510,3sleep234-21ksoftirqd/208:50:002
139099310,2rtkit-daemon0-21swapper/009:57:400
379394599299,6cyclictest3853185-21cat09:55:151
379394599293,8cyclictest3808588-40unattended-upgr07:49:541
3793945992911,6cyclictest3862049-21grep10:20:171
3793945992911,4cyclictest3877323-21find11:05:001
3793945992910,5cyclictest3890559-21cat11:40:161
139099290,4rtkit-daemon1389-21rtkit-daemon08:23:370
379394599287,5cyclictest3798289-21cpuspeed_turbos07:20:111
3793945992813,3cyclictest3803022-21cron07:35:001
3793945992811,5cyclictest3868438-21date10:40:011
379394599281,14cyclictest3823433-21grep08:30:101
379394599281,10cyclictest3795848-21munin-node07:15:011
379394399284,6cyclictest0-21swapper/007:33:080
38188702270,3sleep20-21swapper/208:15:242
379394599277,5cyclictest3906314-21sort12:25:161
379394599277,4cyclictest3902477-21cpuspeed_turbos12:15:121
379394599276,6cyclictest3892771-21sshd11:45:361
379394599275,7cyclictest3848407-21python309:40:211
3793945992710,4cyclictest3859523-21latency_hist10:15:011
379394399275,8cyclictest3877260-21apt-get11:05:030
38780662260,5chrt3056-21PK-Backend11:05:132
379394599269,4cyclictest3895395-21cstates11:55:141
379394599268,5cyclictest883-21dbus-daemon09:10:011
379394599267,4cyclictest3842520-21cpuspeed_turbos09:25:101
379394599265,7cyclictest2164-21gdbus07:35:351
379394599262,8cyclictest82332-21update-manager10:55:371
379394599262,8cyclictest3808897-21apt07:50:111
379394599262,5cyclictest0-21swapper/111:50:121
3793945992610,3cyclictest3900465-21latency_hist12:10:011
3793945992610,3cyclictest3840345-21latency_hist09:20:011
379394399266,8cyclictest0-21swapper/009:40:000
379394399263,14cyclictest0-21swapper/007:45:130
139099260,4rtkit-daemon1389-21rtkit-daemon08:46:521
379394599259,3cyclictest3821604-21sort08:25:111
379394599258,5cyclictest2175-21JS10:27:481
379394599251,9cyclictest3869209-21missed_timers10:40:171
379394599251,9cyclictest3869209-21missed_timers10:40:171
379394599251,8cyclictest3896999-21cpuspeed_turbos12:00:121
3793945992510,3cyclictest1095-21runrttasks11:13:311
379394599250,10cyclictest3878537-21chrt11:05:211
379394599247,4cyclictest0-21swapper/112:10:181
379394599241,9cyclictest3871797-21munin-run10:50:001
379394599241,8cyclictest3813100-21/usr/sbin/munin08:00:171
379394599241,7cyclictest436-20systemd-journal12:40:001
3793945992410,5cyclictest26-21ksoftirqd/112:34:141
379394599240,9cyclictest3842097-21basename09:25:001
379394399243,8cyclictest0-21swapper/010:30:040
379394399243,8cyclictest0-21swapper/010:15:190
3793943992421,2cyclictest3795611-21turbostat07:15:010
3793948992314,7cyclictest3800570-21ntp_states07:25:192
379394599236,4cyclictest3864071-21/usr/sbin/munin10:25:201
379394599235,6cyclictest3827357-21/usr/sbin/munin08:40:181
379394599235,6cyclictest3827357-21/usr/sbin/munin08:40:171
379394599231,6cyclictest3866583-21cron10:35:001
379394399238,13cyclictest436-20systemd-journal11:00:370
379394399235,6cyclictest3861190-21kworker/0:210:44:370
379394399235,6cyclictest3861190-21kworker/0:210:44:370
379394399233,8cyclictest0-21swapper/008:00:140
3793943992315,6cyclictest3837721-21memory09:10:170
3793950992220,1cyclictest3056-21PK-Backend10:36:303
3793950992219,1cyclictest3056-21PK-Backend11:18:283
379395099220,20cyclictest3056-21PK-Backend09:15:403
379395099220,20cyclictest1788620-21WebKitWebProces08:57:143
3793948992220,1cyclictest0-21swapper/210:52:412
3793948992220,1cyclictest0-21swapper/208:40:042
3793948992220,1cyclictest0-21swapper/208:40:042
3793948992220,1cyclictest0-21swapper/208:13:282
3793948992220,1cyclictest0-21swapper/207:50:022
379394599222,6cyclictest3805966-21python307:40:221
3793945992219,2cyclictest0-21swapper/108:08:331
3793945992219,1cyclictest0-21swapper/111:32:251
379394599220,8cyclictest436-20systemd-journal11:30:001
379394599220,8cyclictest3873509-21cron10:55:001
379394599220,21cyclictest0-21swapper/107:55:541
379394599220,20cyclictest3056-21PK-Backend08:12:591
379394399227,13cyclictest3874918-21smartctl10:55:210
379394399224,7cyclictest3901702-21sshd12:10:360
379394399223,11cyclictest0-21swapper/012:33:370
379394399222,7cyclictest0-21swapper/007:43:400
3793943992217,3cyclictest3872373-21cstates10:50:130
3793943992212,8cyclictest3893398-21apt11:50:120
3793943992211,9cyclictest3798076-21/usr/sbin/munin07:20:230
38970372210,2sleep33056-21PK-Backend12:00:123
3793950992119,1cyclictest3056-21PK-Backend10:58:003
3793950992119,1cyclictest3056-21PK-Backend10:18:403
3793950992119,1cyclictest0-21swapper/308:48:093
3793950992118,2cyclictest3056-21PK-Backend07:43:193
3793950992118,1cyclictest3056-21PK-Backend07:50:293
3793950992113,6cyclictest436-20systemd-journal09:45:003
379395099210,20cyclictest3056-21PK-Backend11:20:173
379395099210,20cyclictest3056-21PK-Backend10:01:353
379395099210,19cyclictest3056-21PK-Backend12:20:403
379395099210,19cyclictest3056-21PK-Backend12:15:123
379395099210,19cyclictest3056-21PK-Backend11:50:113
379395099210,19cyclictest3056-21PK-Backend11:48:223
379395099210,19cyclictest3056-21PK-Backend10:21:303
379395099210,19cyclictest3056-21PK-Backend09:25:193
379395099210,19cyclictest3056-21PK-Backend09:11:133
379395099210,19cyclictest3056-21PK-Backend08:37:263
379395099210,19cyclictest3056-21PK-Backend08:00:563
379395099210,19cyclictest3056-21PK-Backend07:35:213
379395099210,19cyclictest3056-21PK-Backend07:30:283
3793948992119,1cyclictest0-21swapper/212:20:542
3793948992119,1cyclictest0-21swapper/212:15:282
3793948992119,1cyclictest0-21swapper/212:06:212
3793948992119,1cyclictest0-21swapper/211:35:392
3793948992119,1cyclictest0-21swapper/211:22:412
3793948992119,1cyclictest0-21swapper/210:39:282
3793948992119,1cyclictest0-21swapper/210:02:182
3793948992119,1cyclictest0-21swapper/209:48:522
3793948992119,1cyclictest0-21swapper/208:27:422
3793948992119,1cyclictest0-21swapper/207:31:042
3793948992119,1cyclictest0-21swapper/207:21:432
3793948992118,2cyclictest0-21swapper/211:18:022
379394899210,20cyclictest0-21swapper/212:39:132
379394899210,20cyclictest0-21swapper/211:41:062
379394899210,20cyclictest0-21swapper/210:22:252
379394899210,20cyclictest0-21swapper/208:00:312
379394899210,20cyclictest0-21swapper/207:13:392
379394599212,6cyclictest3904921-21python312:20:211
3793945992119,1cyclictest0-21swapper/110:09:391
3793945992119,1cyclictest0-21swapper/109:48:431
3793945992116,3cyclictest3837153-21wc09:10:111
379394599210,20cyclictest0-21swapper/109:51:331
379394399213,7cyclictest0-21swapper/007:16:000
3793943992112,7cyclictest3871807-21awk10:50:000
3793943992112,7cyclictest3845562-21awk09:34:590
3793943992112,7cyclictest3814880-21sed08:05:160
3793943992112,6cyclictest3890306-21diskstats11:40:140
3793943992111,9cyclictest3812112-21basename08:00:000
3793943992110,9cyclictest3885031-21cat11:25:120
3793943992110,9cyclictest3823029-21awk08:30:000
3793943992110,9cyclictest0-21swapper/010:25:110
3793943992110,8cyclictest0-21swapper/008:40:110
3793943992110,8cyclictest0-21swapper/008:40:110
379394399210,6cyclictest3843820-21basename09:29:590
379394399210,1cyclictest0-21swapper/008:31:290
39095832200,5sleep342-21ksoftirqd/312:35:143
3793950992018,1cyclictest3056-21PK-Backend12:31:083
3793950992018,1cyclictest3056-21PK-Backend12:09:463
3793950992018,1cyclictest3056-21PK-Backend11:14:503
3793950992018,1cyclictest3056-21PK-Backend11:00:503
3793950992018,1cyclictest3056-21PK-Backend10:33:393
3793950992018,1cyclictest3056-21PK-Backend10:12:583
3793950992018,1cyclictest3056-21PK-Backend10:05:273
3793950992018,1cyclictest3056-21PK-Backend08:22:133
3793950992018,1cyclictest3056-21PK-Backend08:05:363
3793950992018,1cyclictest3056-21PK-Backend07:46:243
3793950992018,1cyclictest3056-21PK-Backend07:18:323
3793950992018,1cyclictest0-21swapper/308:51:473
3793950992017,2cyclictest3056-21PK-Backend11:34:503
3793950992017,1cyclictest3056-21PK-Backend12:13:233
379395099200,7cyclictest3833677-21apt09:00:113
379395099200,1cyclictest3056-21PK-Backend09:06:333
379395099200,19cyclictest3056-21PK-Backend09:51:153
379395099200,19cyclictest3056-21PK-Backend09:36:143
379395099200,19cyclictest3056-21PK-Backend07:22:583
3793948992018,1cyclictest0-21swapper/211:54:392
3793948992018,1cyclictest0-21swapper/211:25:322
3793948992018,1cyclictest0-21swapper/211:14:232
3793948992018,1cyclictest0-21swapper/210:45:112
3793948992018,1cyclictest0-21swapper/210:15:422
3793948992018,1cyclictest0-21swapper/210:07:582
3793948992018,1cyclictest0-21swapper/209:58:262
3793948992018,1cyclictest0-21swapper/209:41:222
3793948992018,1cyclictest0-21swapper/209:20:202
3793948992018,1cyclictest0-21swapper/208:31:342
3793948992012,6cyclictest3815214-21python308:05:212
379394899200,1cyclictest0-21swapper/212:14:262
379394899200,19cyclictest0-21swapper/212:33:592
379394899200,19cyclictest0-21swapper/211:56:282
379394899200,19cyclictest0-21swapper/211:34:382
379394899200,19cyclictest0-21swapper/210:57:202
379394899200,19cyclictest0-21swapper/210:33:472
379394899200,19cyclictest0-21swapper/209:51:432
379394899200,19cyclictest0-21swapper/209:14:522
379394899200,19cyclictest0-21swapper/208:56:282
379394899200,19cyclictest0-21swapper/208:51:482
3793945992018,1cyclictest3056-21PK-Backend08:53:471
3793945992018,1cyclictest0-21swapper/111:16:501
3793945992018,1cyclictest0-21swapper/110:17:221
3793945992012,5cyclictest3821023-21latency_hist08:25:001
379394599200,19cyclictest0-21swapper/107:11:211
379394399209,10cyclictest3887974-21sed11:35:000
379394399208,9cyclictest0-21swapper/012:35:120
3793943992018,1cyclictest0-21swapper/009:16:520
3793943992018,1cyclictest0-21swapper/009:02:040
3793943992018,1cyclictest0-21swapper/008:55:350
3793943992013,5cyclictest3889143-21expr11:35:220
3793943992013,5cyclictest3868199-21sshd10:38:360
3793943992013,3cyclictest3891666-21apt11:45:120
3793943992011,7cyclictest3896626-21unin-run12:00:000
3793943992011,7cyclictest3847547-21/usr/sbin/munin09:40:110
3793943992011,7cyclictest3810371-21sh07:55:010
3793943992010,8cyclictest3907253-21cron12:29:590
3793943992010,8cyclictest3899261-21/usr/sbin/munin12:05:170
3793943992010,8cyclictest3855330-21runrttasks10:00:190
379394399200,19cyclictest0-21swapper/011:19:130
379394399200,19cyclictest0-21swapper/009:21:170
379394399200,19cyclictest0-21swapper/008:15:140
379394399200,19cyclictest0-21swapper/008:12:230
3793950991917,1cyclictest3056-21PK-Backend11:40:193
3793950991917,1cyclictest3056-21PK-Backend11:27:333
3793950991917,1cyclictest3056-21PK-Backend10:48:403
3793950991917,1cyclictest3056-21PK-Backend10:42:123
3793950991917,1cyclictest3056-21PK-Backend10:42:113
3793950991917,1cyclictest3056-21PK-Backend09:56:563
3793950991917,1cyclictest3056-21PK-Backend08:27:533
3793950991917,1cyclictest3056-21PK-Backend07:13:173
3793950991917,1cyclictest0-21swapper/308:19:223
3793950991916,2cyclictest3056-21PK-Backend10:53:203
3793950991913,4cyclictest3816236-21cpuspeed_turbos08:10:123
379395099190,1cyclictest3056-21PK-Backend11:57:143
379395099190,1cyclictest3056-21PK-Backend11:05:313
379395099190,1cyclictest3056-21PK-Backend09:20:543
379395099190,1cyclictest3056-21PK-Backend08:33:223
379395099190,18cyclictest3056-21PK-Backend09:46:023
379395099190,18cyclictest3056-21PK-Backend08:41:533
379395099190,18cyclictest3056-21PK-Backend08:41:523
379395099190,18cyclictest3056-21PK-Backend07:29:263
379394899192,15cyclictest3826466-21cron08:39:592
3793948991917,1cyclictest0-21swapper/211:01:122
3793948991917,1cyclictest0-21swapper/210:11:502
3793948991917,1cyclictest0-21swapper/209:27:242
3793948991917,1cyclictest0-21swapper/208:20:102
3793948991917,1cyclictest0-21swapper/207:35:562
3793948991914,3cyclictest3846457-21tail09:35:182
379394899190,1cyclictest0-21swapper/212:02:432
379394899190,1cyclictest0-21swapper/209:19:182
379394899190,1cyclictest0-21swapper/209:01:072
379394899190,18cyclictest0-21swapper/209:31:012
3793945991917,1cyclictest3056-21PK-Backend08:56:241
3793945991911,5cyclictest436-20systemd-journal07:29:551
379394599190,1cyclictest0-21swapper/110:02:551
379394599190,18cyclictest0-21swapper/109:31:531
379394399199,8cyclictest3829150-21cut08:45:180
379394399199,8cyclictest0-21swapper/010:05:100
3793943991917,1cyclictest0-21swapper/012:15:500
3793943991912,5cyclictest3879689-21expr11:10:130
3793943991912,5cyclictest3862051-21cut10:20:170
3793943991911,6cyclictest3849396-21grep09:45:140
3793943991911,6cyclictest3836131-21ntpq09:05:180
3793943991911,6cyclictest3825533-21wc08:35:150
3793943991911,6cyclictest3803492-21tr07:35:120
3793943991910,7cyclictest3883641-21cut11:20:180
379394399190,18cyclictest0-21swapper/008:52:450
3793950991816,1cyclictest3056-21PK-Backend11:38:293
3793950991816,1cyclictest3056-21PK-Backend10:29:003
3793950991816,1cyclictest3056-21PK-Backend07:59:553
379395099181,15cyclictest3907023-21chrt12:27:113
3793948991816,1cyclictest3056-21PK-Backend10:40:302
3793948991816,1cyclictest3056-21PK-Backend10:40:302
3793948991812,5cyclictest3811069-21awk07:55:162
3793948991811,5cyclictest3835781-21df09:05:132
379394899180,1cyclictest0-21swapper/211:46:342
3793945991810,6cyclictest3888933-21ntp_states11:35:191
3793945991810,6cyclictest3846330-21ls09:35:181
379394399189,7cyclictest3851247-21expr09:50:120
3793943991810,7cyclictest3897135-21expr12:00:120
3793943991810,6cyclictest3904066-21/usr/sbin/munin12:20:110
3793948991715,1cyclictest0-21swapper/210:29:092
3793948991715,1cyclictest0-21swapper/207:40:502
3793950991613,2cyclictest3056-21PK-Backend09:30:463
3793948991611,4cyclictest3796959-21cat07:15:172
379394599169,5cyclictest3825720-21ntpq08:35:181
3793948991513,1cyclictest0-21swapper/207:45:572
3793948991412,1cyclictest0-21swapper/212:27:102
3793945991412,1cyclictest3056-21PK-Backend09:02:271
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional