You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 14:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Tue Mar 24, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304595527741,9sleep00-21swapper/019:05:130
304717527442,25sleep30-21swapper/319:06:473
304597927145,16sleep10-21swapper/119:05:131
304723126839,22sleep20-21swapper/219:07:312
31490502430,3sleep23056-21PK-Backend23:55:132
31312622400,3sleep20-21swapper/223:05:132
31345842380,2sleep00-21swapper/023:15:010
3047599992814,4cyclictest42-21ksoftirqd/300:15:023
3047599992814,4cyclictest42-21ksoftirqd/300:15:023
30710992270,3chrt0-21swapper/020:15:010
3047599992713,4cyclictest42-21ksoftirqd/323:45:593
139099270,4rtkit-daemon1389-21rtkit-daemon19:55:361
31594162260,3sleep00-21swapper/000:25:140
3047599992622,3cyclictest42-21ksoftirqd/321:45:003
304759999261,9cyclictest30571942/usr/sbin/munin19:35:223
3047599992614,9cyclictest883-21dbus-daemon19:33:583
3047599992614,9cyclictest883-21dbus-daemon19:33:573
139099260,4rtkit-daemon1389-21rtkit-daemon21:40:400
3047599992512,4cyclictest42-21ksoftirqd/320:00:323
3047599992511,4cyclictest42-21ksoftirqd/322:55:543
31040332240,3sleep13056-21PK-Backend21:45:201
304759999244,10cyclictest42-21ksoftirqd/320:10:023
3047599992418,4cyclictest42-21ksoftirqd/319:20:013
3047599992413,7cyclictest1095-21runrttasks22:35:113
3047593992421,1cyclictest26-21ksoftirqd/123:20:001
139099240,3rtkit-daemon1389-21rtkit-daemon22:40:220
3047599992317,5cyclictest42-21ksoftirqd/320:34:183
3047599992317,5cyclictest42-21ksoftirqd/320:34:173
3047599992311,8cyclictest3162479-21munin-plugin-st00:35:003
3047599992311,4cyclictest42-21ksoftirqd/323:50:443
3047599992310,4cyclictest42-21ksoftirqd/323:55:533
304759999230,7cyclictest42-21ksoftirqd/323:40:203
304759999230,6cyclictest42-21ksoftirqd/321:30:213
3047596992315,6cyclictest3080513-21fw_packets20:40:122
3047593992316,6cyclictest26-21ksoftirqd/119:54:061
3047592992321,1cyclictest0-21swapper/000:35:090
139099230,4rtkit-daemon1389-21rtkit-daemon21:26:510
139099230,4rtkit-daemon1389-21rtkit-daemon19:49:211
304759999228,5cyclictest42-21ksoftirqd/322:34:093
304759999226,14cyclictest3097496-21apt-get21:30:003
3047599992220,1cyclictest0-21swapper/323:32:013
3047599992220,1cyclictest0-21swapper/320:25:313
3047599992219,2cyclictest0-21swapper/321:56:023
3047599992219,2cyclictest0-21swapper/321:51:023
3047599992217,4cyclictest42-21ksoftirqd/323:30:003
304759999221,17cyclictest2156-21gnome-shell21:04:233
3047599992211,4cyclictest42-21ksoftirqd/319:50:173
304759699220,20cyclictest3098000-21/usr/sbin/munin21:30:162
304759399220,3cyclictest3151343-21ntp_states00:00:181
3047592992220,1cyclictest0-21swapper/023:33:180
3047592992220,1cyclictest0-21swapper/022:37:280
3047592992220,1cyclictest0-21swapper/021:52:390
3047592992220,1cyclictest0-21swapper/020:17:010
3047592992220,1cyclictest0-21swapper/019:22:190
3047592992220,1cyclictest0-21swapper/000:21:440
3047592992220,1cyclictest0-21swapper/000:16:240
3047592992220,1cyclictest0-21swapper/000:16:240
304759299220,2cyclictest3136187-21munin-run23:20:000
304759999217,6cyclictest42-21ksoftirqd/320:52:213
304759999217,4cyclictest42-21ksoftirqd/300:05:033
3047599992119,1cyclictest3056-21PK-Backend23:11:003
3047599992119,1cyclictest3056-21PK-Backend00:25:213
3047599992119,1cyclictest0-21swapper/322:17:063
3047599992119,1cyclictest0-21swapper/320:35:253
304759999211,7cyclictest3094254-21/usr/sbin/munin21:20:143
3047599992112,5cyclictest1095-21runrttasks23:35:203
3047599992110,4cyclictest42-21ksoftirqd/319:55:223
304759999210,20cyclictest0-21swapper/323:21:473
304759999210,20cyclictest0-21swapper/322:53:493
304759999210,20cyclictest0-21swapper/320:12:133
304759999210,20cyclictest0-21swapper/319:42:253
3047596992119,1cyclictest0-21swapper/200:16:172
3047596992119,1cyclictest0-21swapper/200:16:172
3047596992118,2cyclictest3056-21PK-Backend19:50:192
3047596992115,4cyclictest34-21ksoftirqd/220:39:202
3047596992113,6cyclictest3136440-21awk23:20:122
3047596992112,7cyclictest3141035-21python323:30:212
304759699210,9cyclictest31611852expr00:30:112
304759699210,2cyclictest141rcu_preempt22:55:172
3047593992114,5cyclictest26-21ksoftirqd/123:45:001
3047593992113,6cyclictest26-21ksoftirqd/100:25:151
3047593992112,7cyclictest26-21ksoftirqd/123:00:011
3047593992112,6cyclictest26-21ksoftirqd/122:40:111
304759399210,20cyclictest3056-21PK-Backend20:09:211
304759399210,20cyclictest3056-21PK-Backend19:15:401
304759399210,20cyclictest3056-21PK-Backend00:30:201
304759399210,19cyclictest3056-21PK-Backend23:31:391
304759399210,19cyclictest3056-21PK-Backend21:31:411
3047592992119,1cyclictest0-21swapper/023:55:490
3047592992119,1cyclictest0-21swapper/023:53:320
3047592992119,1cyclictest0-21swapper/023:43:180
3047592992119,1cyclictest0-21swapper/022:33:490
3047592992119,1cyclictest0-21swapper/022:21:190
3047592992119,1cyclictest0-21swapper/021:48:280
3047592992119,1cyclictest0-21swapper/021:09:340
3047592992119,1cyclictest0-21swapper/020:49:330
3047592992119,1cyclictest0-21swapper/020:06:340
3047592992119,1cyclictest0-21swapper/019:29:090
3047592992119,1cyclictest0-21swapper/000:12:040
304759299210,20cyclictest0-21swapper/021:19:550
304759299210,20cyclictest0-21swapper/020:30:340
304759299210,20cyclictest0-21swapper/020:30:330
304759999209,3cyclictest42-21ksoftirqd/322:40:293
304759999202,13cyclictest3071159-21apt20:15:123
3047599992018,1cyclictest3056-21PK-Backend00:00:553
3047599992018,1cyclictest0-21swapper/322:46:323
3047599992018,1cyclictest0-21swapper/322:21:173
3047599992018,1cyclictest0-21swapper/322:00:153
3047599992018,1cyclictest0-21swapper/320:41:473
3047599992017,2cyclictest3056-21PK-Backend19:48:203
3047599992010,2cyclictest141rcu_preempt00:23:493
304759999200,6cyclictest3093995-21cron21:20:003
304759999200,1cyclictest3056-21PK-Backend23:07:203
304759999200,19cyclictest0-21swapper/321:35:273
304759999200,19cyclictest0-21swapper/319:11:103
304759999200,19cyclictest0-21swapper/300:10:343
304759999200,18cyclictest3056-21PK-Backend23:01:053
304759999200,11cyclictest42-21ksoftirqd/320:20:033
3047596992018,1cyclictest0-21swapper/223:53:122
3047596992018,1cyclictest0-21swapper/223:38:532
3047596992018,1cyclictest0-21swapper/222:37:282
3047596992018,1cyclictest0-21swapper/222:33:162
3047596992018,1cyclictest0-21swapper/221:02:462
3047596992018,1cyclictest0-21swapper/200:38:352
3047596992017,2cyclictest0-21swapper/221:11:102
3047596992015,4cyclictest34-21ksoftirqd/221:44:532
3047596992014,4cyclictest34-21ksoftirqd/220:55:002
3047596992013,5cyclictest883-21dbus-daemon20:29:162
3047596992012,6cyclictest3115255-21wc22:20:002
304759699200,1cyclictest0-21swapper/223:00:332
304759699200,1cyclictest0-21swapper/219:44:112
304759699200,19cyclictest0-21swapper/222:11:052
304759699200,19cyclictest0-21swapper/221:50:512
304759699200,19cyclictest0-21swapper/220:16:482
304759699200,19cyclictest0-21swapper/220:15:002
304759699200,19cyclictest0-21swapper/219:36:262
304759699200,19cyclictest0-21swapper/200:14:012
3047593992018,1cyclictest0-21swapper/100:15:201
3047593992018,1cyclictest0-21swapper/100:15:191
3047593992017,2cyclictest3056-21PK-Backend21:56:011
3047593992017,2cyclictest3056-21PK-Backend20:51:391
3047593992017,2cyclictest3056-21PK-Backend19:30:481
3047593992017,2cyclictest3056-21PK-Backend19:30:481
3047593992012,6cyclictest26-21ksoftirqd/123:10:171
3047593992012,6cyclictest26-21ksoftirqd/123:00:131
3047593992012,6cyclictest26-21ksoftirqd/120:04:101
3047593992012,6cyclictest26-21ksoftirqd/100:20:181
3047593992012,2cyclictest26-21ksoftirqd/121:30:001
3047593992012,2cyclictest26-21ksoftirqd/119:12:391
3047593992011,7cyclictest26-21ksoftirqd/122:05:201
3047593992011,3cyclictest26-21ksoftirqd/100:06:091
304759399200,1cyclictest3056-21PK-Backend23:29:371
304759399200,19cyclictest3056-21PK-Backend21:04:371
304759399200,19cyclictest3056-21PK-Backend20:11:581
304759399200,19cyclictest3056-21PK-Backend00:36:551
304759399200,19cyclictest0-21swapper/123:55:591
304759399200,19cyclictest0-21swapper/123:48:221
304759399200,18cyclictest3056-21PK-Backend22:51:031
304759399200,18cyclictest3056-21PK-Backend22:20:481
3047592992018,1cyclictest3056-21PK-Backend22:01:040
3047592992018,1cyclictest0-21swapper/023:36:230
3047592992018,1cyclictest0-21swapper/023:03:570
3047592992018,1cyclictest0-21swapper/022:58:300
3047592992018,1cyclictest0-21swapper/022:53:020
3047592992018,1cyclictest0-21swapper/022:27:340
3047592992018,1cyclictest0-21swapper/022:17:550
3047592992018,1cyclictest0-21swapper/022:14:290
3047592992018,1cyclictest0-21swapper/022:06:050
3047592992018,1cyclictest0-21swapper/020:23:170
3047592992018,1cyclictest0-21swapper/019:38:420
3047592992018,1cyclictest0-21swapper/019:16:250
3047592992018,1cyclictest0-21swapper/000:30:370
3047592992018,1cyclictest0-21swapper/000:08:470
304759299200,1cyclictest0-21swapper/020:25:060
304759299200,19cyclictest0-21swapper/023:45:210
304759299200,19cyclictest0-21swapper/023:29:120
304759299200,19cyclictest0-21swapper/023:23:040
304759299200,19cyclictest0-21swapper/021:14:200
304759999198,3cyclictest141rcu_preempt21:48:273
3047599991917,1cyclictest0-21swapper/300:36:433
3047599991916,2cyclictest411rcuc/321:10:523
3047599991912,4cyclictest3086810-21perf21:00:003
304759999190,1cyclictest0-21swapper/322:05:153
304759999190,1cyclictest0-21swapper/319:27:593
304759699192,5cyclictest0-21swapper/200:25:202
3047596991917,1cyclictest0-21swapper/223:48:542
3047596991917,1cyclictest0-21swapper/223:16:232
3047596991917,1cyclictest0-21swapper/222:54:052
3047596991917,1cyclictest0-21swapper/222:22:072
3047596991917,1cyclictest0-21swapper/221:45:032
3047596991917,1cyclictest0-21swapper/219:55:262
3047596991917,1cyclictest0-21swapper/219:32:342
3047596991917,1cyclictest0-21swapper/219:32:342
3047596991917,1cyclictest0-21swapper/219:21:202
3047596991915,2cyclictest883-21dbus-daemon20:35:002
3047596991915,2cyclictest883-21dbus-daemon20:35:002
3047596991914,4cyclictest2156-21gnome-shell21:07:092
3047596991912,5cyclictest3134429-21cron23:15:002
304759699190,1cyclictest0-21swapper/223:29:402
304759699190,1cyclictest0-21swapper/222:49:252
304759699190,1cyclictest0-21swapper/222:41:072
304759699190,1cyclictest0-21swapper/222:03:282
304759699190,18cyclictest0-21swapper/223:40:562
304759699190,18cyclictest0-21swapper/220:21:142
304759699190,18cyclictest0-21swapper/220:05:332
304759699190,18cyclictest0-21swapper/200:20:372
304759699190,18cyclictest0-21swapper/200:09:422
304759699190,18cyclictest0-21swapper/200:02:052
304759399199,6cyclictest26-21ksoftirqd/121:07:281
3047593991917,1cyclictest3056-21PK-Backend20:34:561
3047593991917,1cyclictest3056-21PK-Backend20:34:561
3047593991911,7cyclictest26-21ksoftirqd/119:38:011
3047593991911,6cyclictest26-21ksoftirqd/122:25:131
3047593991911,2cyclictest26-21ksoftirqd/122:30:241
3047593991911,2cyclictest26-21ksoftirqd/122:15:151
3047593991911,2cyclictest26-21ksoftirqd/121:35:161
3047593991911,2cyclictest26-21ksoftirqd/120:28:091
3047593991911,2cyclictest26-21ksoftirqd/120:22:101
3047593991911,2cyclictest26-21ksoftirqd/120:15:091
3047593991911,2cyclictest26-21ksoftirqd/119:42:091
3047593991911,2cyclictest26-21ksoftirqd/119:26:101
3047593991911,2cyclictest26-21ksoftirqd/100:13:091
304759399190,1cyclictest3056-21PK-Backend23:21:381
304759399190,1cyclictest3056-21PK-Backend21:22:551
304759399190,1cyclictest3056-21PK-Backend21:12:001
304759399190,1cyclictest3056-21PK-Backend19:20:471
304759399190,1cyclictest0-21swapper/122:35:421
304759399190,18cyclictest3056-21PK-Backend20:35:441
3047592991916,2cyclictest3056-21PK-Backend00:04:270
3047592991914,4cyclictest13-21ksoftirqd/019:13:550
3047592991912,6cyclictest3094980-21awk21:20:180
3047592991911,6cyclictest3064573-21df19:55:130
3047592991911,6cyclictest3061508-21awk19:45:200
304759299190,1cyclictest0-21swapper/019:42:340
304759299190,18cyclictest0-21swapper/023:07:370
304759299190,18cyclictest0-21swapper/020:55:210
30828942180,2sleep30-21swapper/320:45:233
3047599991816,1cyclictest0-21swapper/322:27:323
304759999181,4cyclictest3112646-21ntp_states22:10:203
3047599991810,6cyclictest3051972-21munin-node19:20:103
304759999180,6cyclictest42-21ksoftirqd/323:15:143
304759999180,16cyclictest0-21swapper/321:05:043
3047596991816,1cyclictest0-21swapper/222:06:052
3047596991810,7cyclictest3048506-21ntp_states19:10:162
3047596991810,6cyclictest3082721-21python320:45:222
3047596991810,6cyclictest3050299-21cat19:15:142
304759699180,1cyclictest0-21swapper/220:59:212
3047593991812,4cyclictest3081846-21sort20:45:141
3047593991811,6cyclictest26-21ksoftirqd/123:05:221
3047593991811,2cyclictest26-21ksoftirqd/122:11:101
3047593991811,2cyclictest26-21ksoftirqd/121:50:181
3047593991810,2cyclictest26-21ksoftirqd/123:51:391
3047593991810,2cyclictest26-21ksoftirqd/123:35:151
3047593991810,2cyclictest26-21ksoftirqd/122:00:151
304759399180,1cyclictest3056-21PK-Backend22:47:251
304759299188,8cyclictest2237-21gdbus19:33:580
304759299188,8cyclictest2237-21gdbus19:33:580
3047592991816,1cyclictest0-21swapper/021:56:040
3047592991816,1cyclictest0-21swapper/021:34:280
3047592991816,1cyclictest0-21swapper/020:03:100
3047592991813,3cyclictest3063413-21timerandwakeup19:50:200
304759299180,1cyclictest0-21swapper/021:01:090
304759699178,3cyclictest141rcu_preempt21:55:112
3047596991715,1cyclictest0-21swapper/221:25:302
3047596991715,1cyclictest0-21swapper/221:15:082
3047596991715,1cyclictest0-21swapper/219:48:162
3047596991711,5cyclictest34-21ksoftirqd/221:20:222
3047596991710,5cyclictest3053744-21grep19:25:112
304759399179,2cyclictest26-21ksoftirqd/120:40:131
3047593991711,2cyclictest26-21ksoftirqd/120:59:131
3047593991710,2cyclictest26-21ksoftirqd/121:41:191
3047596991614,1cyclictest0-21swapper/220:00:332
3047593991613,2cyclictest3056-21PK-Backend21:15:111
304759299169,5cyclictest436-20systemd-journal20:39:190
304759299168,6cyclictest3124520-21cat22:45:180
304759699158,6cyclictest3117400-21expr22:25:142
304759699151,11cyclictest3099757-21apt21:35:112
3047592991510,3cyclictest1095-21runrttasks21:35:170
3047592991411,2cyclictest3083579-21/usr/sbin/munin20:50:180
3047592991411,2cyclictest1095-21runrttasks20:42:500
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional