You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 07:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Jul 07, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
375178823020,10sleep30-21swapper/319:05:173
382142621030,2sleep20-21swapper/222:25:242
139099750,63rtkit-daemon0-21swapper/119:07:101
375228327440,28sleep00-21swapper/019:07:120
375246026545,14sleep20-21swapper/219:09:322
37564562570,8sleep126-21ksoftirqd/119:20:201
139099530,3rtkit-daemon0-21swapper/123:30:590
139099510,3rtkit-daemon0-21swapper/120:05:250
3752638993313,6cyclictest34-21ksoftirqd/223:45:002
3752638993212,4cyclictest34-21ksoftirqd/221:05:012
3752638993210,4cyclictest34-21ksoftirqd/223:50:002
37545752310,2sleep10-21swapper/119:15:161
3752638993110,9cyclictest34-21ksoftirqd/220:45:002
139099310,6rtkit-daemon3808289-21cat21:50:131
38537772280,4sleep33056-21PK-Backend00:00:043
38351282270,4sleep23056-21PK-Backend23:05:222
3752642992614,3cyclictest42-21ksoftirqd/323:05:013
3752638992620,2cyclictest34-21ksoftirqd/200:00:012
375263599260,25cyclictest1389-21rtkit-daemon00:10:311
375263899254,5cyclictest34-21ksoftirqd/222:15:022
139099250,4rtkit-daemon1389-21rtkit-daemon23:54:550
375263899247,5cyclictest34-21ksoftirqd/222:00:012
375263899245,6cyclictest34-21ksoftirqd/200:20:002
375263899242,3cyclictest141rcu_preempt20:10:222
3752638992420,3cyclictest331rcuc/219:45:092
3752638992410,2cyclictest141rcu_preempt23:30:172
3752635992416,5cyclictest3848066-21sh23:45:001
3752635992415,7cyclictest3827009-21cron22:44:591
375263599240,22cyclictest1788620-21WebKitWebProces00:20:091
375264299230,5cyclictest3811155-21grep21:59:593
375263899232,9cyclictest141rcu_preempt00:15:002
3752638992317,3cyclictest3828734-21cron22:50:002
375263899230,3cyclictest3804846-21df21:40:132
375263899230,2cyclictest141rcu_preempt20:35:012
375263599235,4cyclictest26-21ksoftirqd/123:55:221
3752635992315,6cyclictest3837394-21cron23:15:011
37960672220,6sleep0161rcuc/021:15:130
375264299229,4cyclictest42-21ksoftirqd/322:15:023
375264299227,13cyclictest3793846-21unin-run21:10:013
375263899229,2cyclictest141rcu_preempt00:05:192
375263899226,5cyclictest34-21ksoftirqd/220:55:002
3752638992220,1cyclictest0-21swapper/220:21:052
3752638992220,1cyclictest0-21swapper/219:16:152
3752638992220,1cyclictest0-21swapper/219:13:382
3752638992214,6cyclictest3863948-21/usr/sbin/munin00:30:002
3752638992211,4cyclictest34-21ksoftirqd/221:45:132
3752638992211,4cyclictest34-21ksoftirqd/221:21:412
375263899220,11cyclictest34-21ksoftirqd/219:40:012
375263899220,11cyclictest34-21ksoftirqd/219:40:012
3752635992211,7cyclictest436-20systemd-journal21:05:011
139099220,14rtkit-daemon0-21swapper/200:38:182
375264299219,3cyclictest42-21ksoftirqd/322:35:183
375264299213,8cyclictest3056-21PK-Backend20:10:233
375264299212,1cyclictest141rcu_preempt20:55:013
3752642992114,5cyclictest3778459-21sshd20:25:083
3752642992113,6cyclictest3796815-21sed21:15:223
3752642992110,3cyclictest42-21ksoftirqd/322:15:213
3752642992110,10cyclictest978-21snmpd20:15:063
375264299210,3cyclictest3842612-21munin-run23:30:003
375264299210,1cyclictest3056-21PK-Backend21:45:353
375263899219,3cyclictest34-21ksoftirqd/200:33:552
375263899218,2cyclictest141rcu_preempt23:24:592
375263899216,4cyclictest34-21ksoftirqd/221:50:122
375263899214,5cyclictest34-21ksoftirqd/222:10:002
375263899212,9cyclictest34-21ksoftirqd/223:40:012
375263899212,10cyclictest34-21ksoftirqd/200:05:012
3752638992119,1cyclictest0-21swapper/223:28:082
3752638992119,1cyclictest0-21swapper/222:21:512
3752638992119,1cyclictest0-21swapper/220:59:112
3752638992114,5cyclictest3833648-21sshd23:01:262
3752638992112,6cyclictest3823553-21awk22:34:592
3752638992112,6cyclictest3823553-21awk22:34:592
375263899210,2cyclictest3056-21PK-Backend19:55:202
375263899210,20cyclictest3056-21PK-Backend23:52:482
375263899210,20cyclictest3056-21PK-Backend19:27:452
375263899210,20cyclictest0-21swapper/222:53:132
375263899210,20cyclictest0-21swapper/222:36:382
375263899210,20cyclictest0-21swapper/221:34:332
375263899210,20cyclictest0-21swapper/219:24:072
3752635992112,7cyclictest3823544-21cron22:35:001
3752635992112,7cyclictest3823544-21cron22:35:001
3752632992116,3cyclictest3812006-21memory22:00:170
3752632992115,5cyclictest883-21dbus-daemon20:50:000
3752632992113,6cyclictest3827002-21cron22:44:590
3752632992113,6cyclictest3769303-21/usr/sbin/munin20:00:000
375263299210,20cyclictest0-21swapper/023:18:510
375263299210,19cyclictest0-21swapper/019:12:330
139099210,3rtkit-daemon1389-21rtkit-daemon00:07:201
38487322200,2sleep10-21swapper/123:45:151
375264299208,4cyclictest42-21ksoftirqd/320:35:013
375264299208,10cyclictest3826288-21ntp_states22:40:203
375264299207,8cyclictest42-21ksoftirqd/320:55:243
3752642992018,1cyclictest3056-21PK-Backend19:50:513
3752642992015,3cyclictest978-21snmpd22:06:073
3752642992012,2cyclictest42-21ksoftirqd/321:25:163
375264299200,1cyclictest3056-21PK-Backend20:00:023
375264299200,1cyclictest3056-21PK-Backend19:14:343
375264299200,19cyclictest3056-21PK-Backend20:49:043
375264299200,19cyclictest3056-21PK-Backend19:55:443
375264299200,19cyclictest0-21swapper/322:33:003
375264299200,19cyclictest0-21swapper/322:32:593
375264299200,19cyclictest0-21swapper/321:39:203
375264299200,19cyclictest0-21swapper/300:35:553
375263899208,3cyclictest34-21ksoftirqd/221:36:222
375263899205,4cyclictest34-21ksoftirqd/220:30:012
375263899202,4cyclictest34-21ksoftirqd/222:15:172
3752638992018,1cyclictest0-21swapper/222:41:042
3752638992014,4cyclictest3766289-21cpuspeed_turbos19:50:122
375263899201,10cyclictest34-21ksoftirqd/220:05:192
375263899200,9cyclictest3860922-21tr00:20:122
375263899200,1cyclictest0-21swapper/223:10:522
375263899200,1cyclictest0-21swapper/220:17:192
375263899200,19cyclictest0-21swapper/222:03:522
375263899200,19cyclictest0-21swapper/219:43:462
375263599202,14cyclictest3797553-21apt21:20:121
3752635992012,6cyclictest3767571-21cron19:54:591
3752635992012,5cyclictest3802657-21latency_hist21:35:011
3752635992012,2cyclictest26-21ksoftirqd/100:20:021
375263599201,18cyclictest3769777-21cstates20:00:131
3752635992011,7cyclictest3839772-21df23:20:141
3752635992011,7cyclictest3811146-21cron22:00:001
3752632992018,1cyclictest0-21swapper/022:35:390
3752632992013,5cyclictest436-20systemd-journal21:35:000
375263299200,1cyclictest0-21swapper/021:45:370
375263299200,19cyclictest3056-21PK-Backend23:37:300
375263299200,19cyclictest3056-21PK-Backend22:20:310
375263299200,19cyclictest3056-21PK-Backend21:36:310
375263299200,19cyclictest0-21swapper/022:49:170
375263299200,19cyclictest0-21swapper/022:09:160
375263299200,19cyclictest0-21swapper/021:24:000
375263299200,19cyclictest0-21swapper/019:20:110
375263299200,19cyclictest0-21swapper/000:36:040
38504382190,3sleep33056-21PK-Backend23:50:143
375264299197,8cyclictest42-21ksoftirqd/323:55:233
375264299197,5cyclictest141rcu_preempt20:07:583
375264299192,15cyclictest3802481-21munin-plugin-st21:34:593
3752642991917,1cyclictest3056-21PK-Backend23:45:333
3752642991917,1cyclictest3056-21PK-Backend23:30:123
3752642991917,1cyclictest3056-21PK-Backend23:21:533
3752642991917,1cyclictest3056-21PK-Backend00:15:483
3752642991917,1cyclictest0-21swapper/322:53:343
3752642991917,1cyclictest0-21swapper/321:43:193
3752642991917,1cyclictest0-21swapper/321:12:023
3752642991917,1cyclictest0-21swapper/320:38:153
3752642991912,5cyclictest3834940-21ntpq23:05:193
3752642991912,5cyclictest3819946-21sshd22:23:263
3752642991912,2cyclictest42-21ksoftirqd/319:40:213
375264299190,5cyclictest3760475-21chrt19:33:103
375264299190,1cyclictest3056-21PK-Backend00:25:353
375264299190,1cyclictest0-21swapper/323:37:563
375264299190,1cyclictest0-21swapper/322:29:553
375264299190,18cyclictest3056-21PK-Backend23:19:573
3752638991917,1cyclictest0-21swapper/219:32:252
3752638991916,2cyclictest0-21swapper/221:12:552
375263899190,1cyclictest0-21swapper/220:04:562
375263899190,18cyclictest0-21swapper/223:18:022
375263899190,18cyclictest0-21swapper/220:36:132
375263599198,4cyclictest141rcu_preempt21:15:011
3752635991913,4cyclictest3818349-21dump-pmu-power22:20:001
3752635991913,4cyclictest26-21ksoftirqd/119:45:001
3752635991912,5cyclictest3806938-21ntpq21:45:181
3752635991911,6cyclictest3797279-21sh21:20:001
3752635991911,6cyclictest3768501-21strings19:55:191
3752635991911,6cyclictest26-21ksoftirqd/119:25:131
375263599190,1cyclictest141rcu_preempt23:40:011
375263599190,1cyclictest141rcu_preempt20:10:011
375263599190,1cyclictest0-21swapper/100:27:391
375263599190,18cyclictest0-21swapper/120:41:221
375263299195,5cyclictest3822251-21cpuspeed_turbos22:30:140
375263299195,5cyclictest3822251-21cpuspeed_turbos22:30:140
3752632991917,1cyclictest0-21swapper/023:47:310
3752632991917,1cyclictest0-21swapper/023:41:490
3752632991917,1cyclictest0-21swapper/020:12:020
3752632991917,1cyclictest0-21swapper/019:44:030
3752632991917,1cyclictest0-21swapper/000:14:150
3752632991915,3cyclictest13-21ksoftirqd/019:40:010
3752632991915,3cyclictest13-21ksoftirqd/019:40:010
375263299191,3cyclictest141rcu_preempt22:00:000
3752632991912,5cyclictest3771181-21wc20:05:000
3752632991912,5cyclictest3754372-21basename19:15:140
3752632991911,6cyclictest13-21ksoftirqd/020:50:130
375263299190,1cyclictest3056-21PK-Backend22:55:060
375263299190,1cyclictest0-21swapper/023:11:070
375263299190,1cyclictest0-21swapper/023:01:280
375263299190,1cyclictest0-21swapper/022:27:000
375263299190,1cyclictest0-21swapper/000:22:190
375263299190,1cyclictest0-21swapper/000:19:220
375263299190,1cyclictest0-21swapper/000:02:380
375263299190,18cyclictest0-21swapper/023:20:140
375263299190,18cyclictest0-21swapper/021:14:000
375263299190,18cyclictest0-21swapper/020:40:550
375263299190,18cyclictest0-21swapper/020:21:280
375263299190,18cyclictest0-21swapper/000:31:440
139099190,13rtkit-daemon3056-21PK-Backend22:26:521
3752642991816,1cyclictest3056-21PK-Backend00:12:163
3752642991815,2cyclictest3056-21PK-Backend19:48:413
3752642991811,5cyclictest3778292-21date20:25:013
375264299180,1cyclictest0-21swapper/323:40:053
375264299180,17cyclictest3865692-21idleruntime-cro00:34:593
375264299180,17cyclictest0-21swapper/319:27:463
375264299180,16cyclictest38560232sleep300:05:173
375264299180,16cyclictest3831152-21hddtemp_smartct22:55:203
375263899182,3cyclictest3713082-21kworker/2:121:25:192
3752638991816,1cyclictest0-21swapper/222:56:312
3752638991811,4cyclictest331rcuc/221:05:112
375263899180,1cyclictest0-21swapper/221:15:202
375263599189,7cyclictest3765224-21python319:45:181
375263599187,9cyclictest3844372-21unin-run23:34:591
375263599181,4cyclictest141rcu_preempt22:10:131
3752635991811,5cyclictest3828327-21cat22:45:221
3752635991811,5cyclictest3788555-21sshd20:54:251
3752635991811,5cyclictest3775605-21wc20:15:171
3752635991811,5cyclictest1-21systemd00:40:011
375263599181,12cyclictest1-21systemd19:12:051
3752635991810,6cyclictest3854424-21grep00:00:151
3752635991810,6cyclictest3811787-21df22:00:131
3752635991810,6cyclictest26-21ksoftirqd/123:55:001
375263599180,5cyclictest3805947-21turbostat.cron21:44:591
375263599180,17cyclictest0-21swapper/100:32:251
375263299182,7cyclictest141rcu_preempt00:00:020
3752632991816,1cyclictest0-21swapper/021:29:480
3752632991816,1cyclictest0-21swapper/019:53:300
3752632991816,1cyclictest0-21swapper/000:27:250
3752632991816,1cyclictest0-21swapper/000:05:350
3752632991815,2cyclictest0-21swapper/020:35:470
3752632991813,3cyclictest3792889-21latency21:05:170
3752632991813,3cyclictest3778227-21latency_hist20:25:020
3752632991812,5cyclictest3792087-21cron21:05:000
3752632991812,4cyclictest3790336-21cron21:00:000
3752632991811,6cyclictest3760192-21sed19:30:210
3752632991811,5cyclictest3765121-21awk19:45:170
3752632991810,6cyclictest3841888-21ntpq23:25:220
375264299179,6cyclictest3783741-21awk20:40:123
375264299179,2cyclictest42-21ksoftirqd/319:20:153
375264299178,6cyclictest42-21ksoftirqd/323:12:263
375264299176,3cyclictest42-21ksoftirqd/319:37:573
375264299176,3cyclictest42-21ksoftirqd/319:37:573
375264299175,5cyclictest141rcu_preempt21:21:593
375264299175,5cyclictest141rcu_preempt00:21:593
375264299172,2cyclictest141rcu_preempt19:15:203
3752642991713,3cyclictest3812896-21idleruntime-cro22:05:003
3752642991712,4cyclictest3809497-21date21:55:013
3752642991710,5cyclictest3827880-21cat22:45:163
375263899176,2cyclictest141rcu_preempt20:45:142
375263599178,2cyclictest141rcu_preempt23:10:001
375263599171,1cyclictest141rcu_preempt23:00:011
375263599171,12cyclictest3790363-21awk20:59:591
375263599171,12cyclictest3781694-21awk20:35:001
3752635991710,6cyclictest3785550-21grep20:45:141
3752635991710,5cyclictest3800744-21cron21:30:001
3752635991710,4cyclictest3761441-21latency19:35:171
3752635991710,4cyclictest3761441-21latency19:35:171
3752635991710,2cyclictest26-21ksoftirqd/123:20:011
375263599170,1cyclictest141rcu_preempt22:10:011
3752632991715,1cyclictest3056-21PK-Backend22:50:400
3752632991712,3cyclictest3809044-21vmstat21:50:210
3752642991614,1cyclictest3056-21PK-Backend21:03:163
375263599169,5cyclictest3832643-21cut23:00:131
375263599162,8cyclictest3841147-21apt23:25:141
375263599161,12cyclictest3759351-21grep19:30:121
375263599161,11cyclictest3830490-21munin-run22:55:011
3752635991610,4cyclictest3803413-21needreboot21:35:211
3752635991610,4cyclictest3773420-21cpuspeed_turbos20:10:121
375263599160,4cyclictest141rcu_preempt22:20:211
375263299164,2cyclictest141rcu_preempt22:10:100
3752632991614,1cyclictest0-21swapper/021:40:300
3752632991614,1cyclictest0-21swapper/020:16:210
3752632991613,2cyclictest3834212-21apt23:05:110
3752632991610,4cyclictest3816984-21sort22:15:130
375263599158,4cyclictest3792531-21cpuspeed_turbos21:05:131
375263599150,1cyclictest141rcu_preempt20:25:171
375263299152,4cyclictest436-20systemd-journal20:34:590
3752632991511,3cyclictest13-21ksoftirqd/019:30:000
375263599143,2cyclictest141rcu_preempt20:35:131
375263599143,2cyclictest141rcu_preempt20:25:011
3752635991411,2cyclictest3056-21PK-Backend22:36:061
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional