You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 14:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Mar 25, 2026 00:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
361751321330,4sleep20-21swapper/219:15:132
372267821200,3sleep20-21swapper/200:10:202
365694321200,2sleep00-21swapper/021:05:170
139099790,70rtkit-daemon0-21swapper/119:08:561
361471926444,13sleep00-21swapper/019:08:170
361458926341,14sleep30-21swapper/319:06:363
361453426247,9sleep20-21swapper/219:06:062
37244062500,2sleep13056-21PK-Backend00:15:201
36888412490,3sleep30-21swapper/322:35:183
361501299442,40cyclictest2330-21gdbus20:43:242
361501599279,16cyclictest3640293-21date20:20:003
361501299269,5cyclictest0-21swapper/221:45:132
361501299259,14cyclictest3668132-21switchtime21:35:212
361501299258,15cyclictest3710432-21awk23:35:232
3615012992522,2cyclictest0-21swapper/221:24:212
3615012992412,10cyclictest3702179-21cat23:15:012
37315222230,2sleep13056-21PK-Backend00:35:211
36189532230,2sleep13056-21PK-Backend19:20:001
361501599238,3cyclictest3729400-21latency00:30:153
361501299234,17cyclictest2005-21goa-daemon21:28:362
3615012992319,3cyclictest0-21swapper/200:24:582
3615012992315,6cyclictest436-20systemd-journal20:18:002
3615012992312,9cyclictest3709003-21anacron23:32:242
3615012992312,9cyclictest3658902-21taskset21:10:132
3615012992311,9cyclictest978-21snmpd22:25:202
361501299230,10cyclictest3704173-21sed23:20:012
36444592220,3sleep20-21swapper/220:30:142
361501599226,14cyclictest0-21swapper/323:40:203
3615015992220,1cyclictest3056-21PK-Backend20:07:013
3615015992220,1cyclictest0-21swapper/322:06:263
3615015992220,1cyclictest0-21swapper/319:30:053
3615015992217,3cyclictest3690879-21ntp_states22:40:193
3615015992216,4cyclictest3666367-21python321:30:213
3615015992215,5cyclictest436-20systemd-journal20:13:243
361501599221,18cyclictest3633010-21unattended-upgr19:58:263
361501599220,20cyclictest3719314-21/usr/sbin/munin00:00:193
361501299229,12cyclictest0-21swapper/223:05:002
361501299222,19cyclictest0-21swapper/221:00:572
3615012992220,1cyclictest0-21swapper/221:30:052
3615012992220,1cyclictest0-21swapper/219:29:452
3615012992214,6cyclictest3718158-21cron00:00:002
3615012992214,6cyclictest34-21ksoftirqd/222:20:202
361501299220,3cyclictest3641037-21hddtemp_smartct20:20:152
361501299220,20cyclictest3657612-21awk21:05:242
361500799220,20cyclictest3056-21PK-Backend22:31:491
361500799220,20cyclictest3056-21PK-Backend20:01:561
36341962210,8chrt42-21ksoftirqd/320:00:213
361501599215,14cyclictest957-21gdbus20:53:303
3615015992119,1cyclictest0-21swapper/323:54:143
3615015992119,1cyclictest0-21swapper/323:30:553
3615015992119,1cyclictest0-21swapper/323:00:063
3615015992119,1cyclictest0-21swapper/322:59:063
3615015992119,1cyclictest0-21swapper/322:10:523
3615015992119,1cyclictest0-21swapper/322:04:363
3615015992119,1cyclictest0-21swapper/321:20:153
3615015992119,1cyclictest0-21swapper/320:45:073
3615015992119,1cyclictest0-21swapper/320:40:553
3615015992119,1cyclictest0-21swapper/320:25:543
3615015992119,1cyclictest0-21swapper/300:39:503
3615015992118,2cyclictest0-21swapper/322:50:123
361501599210,20cyclictest0-21swapper/323:23:253
3615012992119,1cyclictest0-21swapper/223:48:142
3615012992119,1cyclictest0-21swapper/223:43:342
3615012992119,1cyclictest0-21swapper/222:14:322
3615012992119,1cyclictest0-21swapper/221:54:112
3615012992119,1cyclictest0-21swapper/221:43:432
3615012992119,1cyclictest0-21swapper/220:58:202
3615012992119,1cyclictest0-21swapper/220:53:072
3615012992119,1cyclictest0-21swapper/220:45:292
3615012992119,1cyclictest0-21swapper/220:13:062
3615012992119,1cyclictest0-21swapper/219:48:392
3615012992119,1cyclictest0-21swapper/219:34:252
3615012992119,1cyclictest0-21swapper/219:10:402
3615012992117,3cyclictest0-21swapper/222:32:312
3615012992111,8cyclictest3688405-21expr22:35:132
3615012992111,8cyclictest3630290-21hddtemp_smartct19:50:142
361501299210,20cyclictest0-21swapper/222:50:282
361501299210,20cyclictest0-21swapper/219:37:162
361501299210,20cyclictest0-21swapper/200:02:282
3615007992118,2cyclictest3056-21PK-Backend20:24:411
361500799210,20cyclictest3056-21PK-Backend21:58:361
361500799210,19cyclictest3056-21PK-Backend23:58:551
361500799210,19cyclictest3056-21PK-Backend23:44:091
361500799210,19cyclictest3056-21PK-Backend21:49:581
361500799210,19cyclictest3056-21PK-Backend21:23:071
361500799210,19cyclictest3056-21PK-Backend21:07:191
361500799210,19cyclictest3056-21PK-Backend19:30:191
3615003992119,1cyclictest0-21swapper/020:31:040
3615003992116,3cyclictest3689623-21sh22:40:000
361500399210,20cyclictest0-21swapper/020:25:230
139099210,4rtkit-daemon1389-21rtkit-daemon00:30:360
3615015992018,1cyclictest0-21swapper/323:26:163
3615015992018,1cyclictest0-21swapper/321:48:553
3615015992018,1cyclictest0-21swapper/319:11:453
3615015992018,1cyclictest0-21swapper/300:06:373
3615015992015,4cyclictest42-21ksoftirqd/322:25:133
3615015992012,7cyclictest883-21dbus-daemon19:42:443
3615015992012,6cyclictest3657262-21ntpq21:05:203
361501599201,17cyclictest3653173-21df20:55:113
361501599201,16cyclictest883-21dbus-daemon19:47:333
3615015992010,8cyclictest436-20systemd-journal19:52:333
361501599200,1cyclictest0-21swapper/323:06:353
361501599200,19cyclictest0-21swapper/323:55:153
361501599200,19cyclictest0-21swapper/321:56:463
361501599200,19cyclictest0-21swapper/321:42:533
361501599200,19cyclictest0-21swapper/321:28:273
361501299209,10cyclictest0-21swapper/221:15:152
3615012992018,1cyclictest0-21swapper/223:26:442
3615012992018,1cyclictest0-21swapper/220:37:052
3615012992018,1cyclictest0-21swapper/219:23:302
3615012992018,1cyclictest0-21swapper/200:06:192
3615012992016,3cyclictest0-21swapper/223:08:532
3615012992013,5cyclictest3642529-21grep20:25:122
3615012992012,6cyclictest978-21snmpd19:55:122
3615012992012,6cyclictest3725118-21cron00:20:002
3615012992011,7cyclictest3731722-21update-manager00:35:362
3615012992011,7cyclictest3727663-21sed00:25:172
3615012992011,7cyclictest3634182-21awk20:00:202
3615012992011,7cyclictest2270-21gdbus19:42:242
3615012992010,8cyclictest3691875-21apt22:45:112
3615012992010,8cyclictest3676228-21cat22:00:132
3615012992010,8cyclictest3635318-21expr20:05:132
361501299200,19cyclictest0-21swapper/222:16:212
3615007992018,1cyclictest3056-21PK-Backend23:05:421
3615007992017,2cyclictest3056-21PK-Backend20:33:391
361500799200,19cyclictest3056-21PK-Backend23:50:371
361500799200,19cyclictest3056-21PK-Backend23:49:351
361500799200,19cyclictest3056-21PK-Backend22:01:131
361500799200,19cyclictest3056-21PK-Backend21:52:351
361500799200,19cyclictest3056-21PK-Backend20:52:401
361500799200,18cyclictest3056-21PK-Backend23:12:581
361500799200,18cyclictest3056-21PK-Backend22:38:521
361500799200,18cyclictest3056-21PK-Backend20:55:161
361500799200,18cyclictest3056-21PK-Backend20:10:411
361500799200,18cyclictest3056-21PK-Backend19:40:261
361500799200,18cyclictest3056-21PK-Backend00:20:111
3615003992018,1cyclictest0-21swapper/022:27:430
3615003992018,1cyclictest0-21swapper/019:19:580
3615003992017,2cyclictest0-21swapper/019:35:130
3615003992013,5cyclictest3693338-21cron22:49:590
361500399200,19cyclictest0-21swapper/021:45:180
361501599199,7cyclictest3640107-21apt-get20:20:033
361501599191,4cyclictest3618290-21python319:15:203
3615015991914,3cyclictest3667738-21kernelversion21:35:163
3615015991914,3cyclictest3658483-21cnf-update-db21:10:013
3615015991914,3cyclictest2156-21gnome-shell23:47:093
3615015991912,5cyclictest3703202-21grep23:15:163
3615015991911,6cyclictest436-20systemd-journal00:15:003
361501599191,16cyclictest3681925-21ntp_states22:15:183
361501599190,1cyclictest0-21swapper/300:24:153
361501599190,18cyclictest0-21swapper/321:01:243
3615012991913,4cyclictest3678556-21cut22:05:212
3615012991911,7cyclictest2627-21gmain23:20:012
3615012991911,6cyclictest3689986-21find22:40:012
3615012991910,8cyclictest3715030-21expr23:50:122
3615012991910,7cyclictest3729197-21expr00:30:122
3615012991910,7cyclictest3696273-21cat22:55:212
3615012991910,7cyclictest3674962-21ntpq21:55:182
3615007991917,1cyclictest3056-21PK-Backend22:24:461
3615007991917,1cyclictest3056-21PK-Backend22:06:271
3615007991917,1cyclictest3056-21PK-Backend21:38:561
3615007991917,1cyclictest3056-21PK-Backend20:46:511
3615007991917,1cyclictest3056-21PK-Backend20:39:271
3615007991917,1cyclictest3056-21PK-Backend19:51:361
3615007991917,1cyclictest3056-21PK-Backend00:31:061
3615007991913,4cyclictest1095-21runrttasks21:00:121
361500799190,1cyclictest3056-21PK-Backend23:34:011
361500799190,1cyclictest3056-21PK-Backend23:28:331
361500799190,1cyclictest3056-21PK-Backend22:12:301
361500799190,1cyclictest3056-21PK-Backend21:12:331
361500799190,1cyclictest3056-21PK-Backend20:16:371
361500799190,1cyclictest3056-21PK-Backend19:58:041
361500799190,1cyclictest3056-21PK-Backend00:01:461
361500799190,1cyclictest0-21swapper/120:08:381
361500799190,18cyclictest3056-21PK-Backend19:28:301
3615003991918,1cyclictest0-21swapper/020:58:210
3615003991918,1cyclictest0-21swapper/020:02:510
3615003991914,3cyclictest3666988-21munin-run21:35:000
3615003991913,5cyclictest3679184-21awk22:10:000
361500399190,1cyclictest0-21swapper/021:13:550
37134612180,3sleep00-21swapper/023:45:160
361501599188,8cyclictest883-21dbus-daemon23:35:313
361501599183,13cyclictest3686383-21find22:30:113
3615015991816,1cyclictest0-21swapper/323:14:053
3615015991816,1cyclictest0-21swapper/322:24:113
3615015991813,3cyclictest3644863-21ntp_states20:30:213
3615015991813,3cyclictest3620911-21apt19:25:113
3615015991811,5cyclictest3692140-21expr22:45:123
3615015991811,5cyclictest3625389-21cat19:35:233
3615015991810,6cyclictest3619883-21ntpq19:20:173
361501599180,6cyclictest3661773-21systemd-run21:20:003
361501599180,5cyclictest3727439-21cstates00:25:143
361501599180,16cyclictest3672588-21wc21:50:133
3615007991816,1cyclictest3056-21PK-Backend23:21:171
3615007991816,1cyclictest3056-21PK-Backend22:45:071
3615007991816,1cyclictest3056-21PK-Backend00:29:171
3615007991815,2cyclictest3056-21PK-Backend22:52:101
361500399185,12cyclictest3668613-21spatcher21:38:510
3615003991817,1cyclictest0-21swapper/022:04:240
3615003991816,1cyclictest0-21swapper/020:48:540
3615003991814,2cyclictest37158422sleep023:50:570
3615003991813,3cyclictest3663939-21cpuspeed_turbos21:25:120
3615003991812,4cyclictest3694334-21grep22:50:200
3615003991811,5cyclictest3691613-21grep22:45:000
3615003991811,5cyclictest3629448-21chmod19:50:010
361500399181,14cyclictest3673357-21chrt21:50:210
3615015991715,1cyclictest0-21swapper/320:35:073
3615007991715,1cyclictest3056-21PK-Backend23:35:021
3615007991715,1cyclictest3056-21PK-Backend22:29:121
3615007991715,1cyclictest3056-21PK-Backend22:16:551
3615007991712,3cyclictest3698544-21sh23:05:001
361500399179,6cyclictest436-20systemd-journal20:08:230
361500399179,6cyclictest436-20systemd-journal00:20:000
361500399172,13cyclictest3624123-21cron19:35:010
3615003991715,1cyclictest0-21swapper/021:16:050
3615003991713,3cyclictest3702001-21cron23:15:000
3615003991713,3cyclictest3655535-21ntp_offset21:00:190
3615003991712,3cyclictest3722387-21if_err_enp2s000:10:170
3615003991711,5cyclictest3637044-21expr20:10:150
3615003991711,4cyclictest3702987-21cat23:15:140
361500399171,14cyclictest3633164-21turbostat20:00:010
3615003991711,4cyclictest3616672-21perf19:14:590
361501599160,3cyclictest3724138-21irqstats00:15:173
3615007991614,1cyclictest3056-21PK-Backend21:42:221
3615007991614,1cyclictest3056-21PK-Backend00:12:551
361500799161,2cyclictest141rcu_preempt21:35:001
361500399168,6cyclictest3706010-21apt23:25:130
361500399163,11cyclictest3679571-21tr22:10:140
3615003991614,1cyclictest0-21swapper/020:38:530
3615003991614,1cyclictest0-21swapper/019:23:430
3615003991612,2cyclictest3697058-21/usr/sbin/munin23:00:150
3615003991611,4cyclictest13-21ksoftirqd/000:05:330
361500399161,13cyclictest3638797-21tr20:15:100
3615003991610,5cyclictest3682090-21sed22:15:210
361500799158,5cyclictest3643857-21cron20:30:001
361500799152,2cyclictest141rcu_preempt19:11:291
3615007991513,1cyclictest3056-21PK-Backend21:29:431
3615007991513,1cyclictest3056-21PK-Backend19:46:071
3615007991513,1cyclictest3056-21PK-Backend00:07:271
3615007991513,1cyclictest0-21swapper/121:15:111
361500399159,5cyclictest13-21ksoftirqd/019:30:010
361500399159,4cyclictest3704466-21cpuspeed_turbos23:20:130
361500399158,6cyclictest3651191-21grep20:50:120
361500399158,5cyclictest3696813-21apt23:00:000
361500399152,8cyclictest3669011-21apt21:40:120
3615003991514,1cyclictest0-21swapper/021:56:200
3615003991514,1cyclictest0-21swapper/000:36:470
3615003991514,1cyclictest0-21swapper/000:29:510
3615003991512,2cyclictest3710976-21turbostat23:40:000
3615003991511,3cyclictest2235-21evolution-calen23:59:590
3615003991510,4cyclictest3726047-21tr00:20:190
3615003991510,4cyclictest3626325-21cpuspeed_turbos19:40:120
3615003991510,3cyclictest3699606-21grep23:05:200
361500399150,14cyclictest0-21swapper/020:43:470
36477412140,4sleep13056-21PK-Backend20:40:121
3615007991412,1cyclictest3056-21PK-Backend19:22:291
361500799140,2cyclictest3056-21PK-Backend23:00:001
361500399149,4cyclictest3708237-21sed23:30:170
361500399149,3cyclictest436-20systemd-journal23:44:550
361500399149,3cyclictest3662985-21timerandwakeup21:20:220
361500399148,5cyclictest3630409-21grep19:50:160
361500399142,7cyclictest3640646-21apt20:20:120
3615003991410,3cyclictest3683127-21cpuspeed_turbos22:20:120
361500399140,13cyclictest0-21swapper/022:34:260
361500399140,13cyclictest0-21swapper/000:03:280
361500799139,3cyclictest3056-21PK-Backend22:40:061
361500799139,3cyclictest3056-21PK-Backend19:35:241
3615007991311,1cyclictest3056-21PK-Backend23:15:491
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional