You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-08 - 04:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Jul 08, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16233821040,4sleep03056-21PK-Backend20:45:210
139099760,61rtkit-daemon0-21swapper/319:06:353
12718726344,13sleep10-21swapper/119:09:231
12703826340,16sleep20-21swapper/219:07:272
12576026242,14sleep00-21swapper/019:05:070
1447272520,2sleep0144725-21wc19:55:210
1918562440,4sleep23056-21PK-Backend22:10:162
2296792390,4sleep2886-21NetworkManager00:00:112
1834132380,4chrt3056-21PK-Backend21:45:190
127381992824,3cyclictest34-21ksoftirqd/221:32:082
127381992824,3cyclictest34-21ksoftirqd/221:32:072
12738099271,5cyclictest188676-21python322:00:211
139099260,4rtkit-daemon1389-21rtkit-daemon00:35:060
2146742250,3chrt0-21swapper/123:15:201
139099250,4rtkit-daemon1389-21rtkit-daemon23:13:260
139099250,4rtkit-daemon1389-21rtkit-daemon22:16:351
1377462250,4sleep03056-21PK-Backend19:35:190
127381992516,7cyclictest193371-21df22:15:152
127386992421,1cyclictest3056-21PK-Backend00:30:583
127381992421,2cyclictest0-21swapper/222:52:362
127381992415,8cyclictest136761-21unin-run19:35:012
139099230,4rtkit-daemon1389-21rtkit-daemon20:15:480
139099230,19rtkit-daemon127370-21cyclictest21:50:171
139099230,18rtkit-daemon127370-21cyclictest22:20:220
127386992320,2cyclictest3056-21PK-Backend23:09:253
127386992320,2cyclictest3056-21PK-Backend22:56:153
127386992320,2cyclictest3056-21PK-Backend21:42:273
12738699230,21cyclictest3056-21PK-Backend23:36:223
12738699230,21cyclictest3056-21PK-Backend23:25:083
12738699230,21cyclictest3056-21PK-Backend22:36:343
12738699230,21cyclictest3056-21PK-Backend22:15:043
127381992321,1cyclictest0-21swapper/222:56:142
127381992321,1cyclictest0-21swapper/221:42:272
127381992320,2cyclictest0-21swapper/223:33:052
127381992320,2cyclictest0-21swapper/222:42:292
127381992320,2cyclictest0-21swapper/222:42:292
12738199230,21cyclictest0-21swapper/223:00:272
127380992315,6cyclictest1-21systemd19:20:011
139099220,4rtkit-daemon1389-21rtkit-daemon21:09:171
127386992220,1cyclictest3056-21PK-Backend22:52:373
127386992220,1cyclictest0-21swapper/300:07:063
127386992219,2cyclictest3056-21PK-Backend22:48:573
127386992219,2cyclictest3056-21PK-Backend19:45:333
127386992219,2cyclictest3056-21PK-Backend19:45:323
127386992213,6cyclictest42-21ksoftirqd/320:15:223
12738699220,21cyclictest3056-21PK-Backend23:55:573
12738699220,21cyclictest3056-21PK-Backend23:14:463
12738699220,21cyclictest3056-21PK-Backend22:11:403
12738699220,21cyclictest3056-21PK-Backend22:01:193
12738699220,20cyclictest3056-21PK-Backend21:57:003
127381992220,1cyclictest0-21swapper/223:18:242
127381992220,1cyclictest0-21swapper/223:09:262
127381992220,1cyclictest0-21swapper/222:48:572
127381992219,2cyclictest0-21swapper/223:57:522
127381992214,6cyclictest436-20systemd-journal21:30:012
127381992210,3cyclictest34-21ksoftirqd/220:25:012
12738199220,21cyclictest0-21swapper/223:28:262
12738199220,21cyclictest0-21swapper/223:14:472
12738199220,21cyclictest0-21swapper/222:36:342
12738199220,21cyclictest0-21swapper/222:01:182
12738199220,21cyclictest0-21swapper/221:56:592
12738199220,1cyclictest0-21swapper/221:39:362
127380992220,1cyclictest0-21swapper/122:37:051
127380992216,4cyclictest239836-21perf00:30:011
12738099220,21cyclictest173937-21apt21:20:121
12738099220,21cyclictest0-21swapper/123:12:001
12738099220,21cyclictest0-21swapper/122:54:431
127376992214,6cyclictest202254-21memory22:40:160
127376992214,6cyclictest202254-21memory22:40:160
127376992214,6cyclictest187532-21cron22:00:000
127376992210,11cyclictest236115-21cron00:17:000
12737699220,21cyclictest225982-21munin-run23:50:010
2018712210,3sleep33056-21PK-Backend22:40:123
2018712210,3sleep33056-21PK-Backend22:40:123
139099210,3rtkit-daemon1389-21rtkit-daemon23:38:201
139099210,3rtkit-daemon1389-21rtkit-daemon19:48:260
139099210,3rtkit-daemon1389-21rtkit-daemon19:48:260
139099210,3rtkit-daemon1389-21rtkit-daemon19:32:200
127386992119,1cyclictest3056-21PK-Backend23:33:053
127386992119,1cyclictest0-21swapper/319:36:003
127386992119,1cyclictest0-21swapper/319:33:163
127386992118,2cyclictest3056-21PK-Backend23:52:053
127386992118,2cyclictest3056-21PK-Backend20:39:063
12738699210,20cyclictest3056-21PK-Backend00:21:113
12738699210,20cyclictest0-21swapper/300:26:053
12738699210,1cyclictest3056-21PK-Backend23:46:503
12738699210,1cyclictest0-21swapper/322:30:403
12738699210,1cyclictest0-21swapper/300:19:013
12738699210,19cyclictest3056-21PK-Backend21:26:103
12738699210,19cyclictest3056-21PK-Backend19:22:483
127381992119,1cyclictest0-21swapper/223:52:042
127381992119,1cyclictest0-21swapper/200:14:092
12738199211,4cyclictest145407-21munin-plugin-st20:00:002
12738199210,1cyclictest0-21swapper/223:36:232
127380992119,1cyclictest0-21swapper/123:44:451
127380992119,1cyclictest0-21swapper/123:33:151
127380992119,1cyclictest0-21swapper/121:17:561
127380992119,1cyclictest0-21swapper/121:12:361
127380992119,1cyclictest0-21swapper/120:34:511
127380992119,1cyclictest0-21swapper/120:19:031
127380992119,1cyclictest0-21swapper/120:06:321
127380992119,1cyclictest0-21swapper/120:04:021
127380992114,5cyclictest136717-21cron19:35:001
127380992113,6cyclictest202280-21cat22:40:171
127380992113,6cyclictest202280-21cat22:40:171
127380992113,6cyclictest196868-21fw_packets22:25:161
12738099210,20cyclictest0-21swapper/123:26:261
12738099210,20cyclictest0-21swapper/121:59:481
12738099210,20cyclictest0-21swapper/119:45:091
12738099210,20cyclictest0-21swapper/119:45:091
12738099210,20cyclictest0-21swapper/100:15:411
12738099210,20cyclictest0-21swapper/100:06:141
12738099210,19cyclictest3056-21PK-Backend00:11:211
12737699218,5cyclictest141rcu_preempt21:15:200
127376992115,4cyclictest185842-21pluginstate21:55:000
12737699210,20cyclictest3056-21PK-Backend22:32:410
12737699210,20cyclictest3056-21PK-Backend20:43:580
12737699210,20cyclictest0-21swapper/023:57:580
12737699210,20cyclictest0-21swapper/022:26:460
139099200,16rtkit-daemon133476-21cstates19:25:130
127386992018,1cyclictest3056-21PK-Backend23:18:243
127386992018,1cyclictest3056-21PK-Backend21:13:413
127386992018,1cyclictest3056-21PK-Backend20:07:153
127386992018,1cyclictest3056-21PK-Backend00:01:243
12738699201,1cyclictest141rcu_preempt20:02:133
12738699200,1cyclictest3056-21PK-Backend23:00:273
12738699200,1cyclictest0-21swapper/300:10:363
12738699200,19cyclictest3056-21PK-Backend21:06:033
12738699200,19cyclictest3056-21PK-Backend20:31:563
12738699200,19cyclictest3056-21PK-Backend19:26:403
12738699200,19cyclictest0-21swapper/320:55:353
12738199203,2cyclictest141rcu_preempt22:06:402
12738199203,15cyclictest197943-21sh22:30:002
127381992017,2cyclictest0-21swapper/200:16:182
127381992016,3cyclictest28650irq/123-ahci[0019:40:002
127381992015,3cyclictest183196-21memory21:45:172
127381992013,5cyclictest34-21ksoftirqd/221:00:012
12738199200,1cyclictest0-21swapper/200:21:112
127380992018,1cyclictest0-21swapper/123:47:291
127380992018,1cyclictest0-21swapper/120:11:321
127380992018,1cyclictest0-21swapper/119:56:101
127380992018,1cyclictest0-21swapper/119:12:311
127380992018,1cyclictest0-21swapper/100:37:511
127380992017,2cyclictest0-21swapper/100:00:191
127380992013,5cyclictest168495-21chmod21:05:011
127380992012,6cyclictest210495-21expr23:05:131
12738099200,1cyclictest0-21swapper/100:23:321
12738099200,19cyclictest3056-21PK-Backend21:33:041
12738099200,19cyclictest3056-21PK-Backend21:33:041
12738099200,19cyclictest0-21swapper/121:29:401
12738099200,19cyclictest0-21swapper/120:27:071
12738099200,19cyclictest0-21swapper/119:20:291
12738099200,19cyclictest0-21swapper/100:33:321
127376992018,1cyclictest0-21swapper/000:31:450
127376992017,2cyclictest3056-21PK-Backend00:10:210
127376992013,6cyclictest883-21dbus-daemon22:50:000
127376992012,6cyclictest193403-21fw_packets22:15:160
127376992011,7cyclictest436-20systemd-journal19:14:590
12737699200,1cyclictest3056-21PK-Backend21:00:140
12737699200,1cyclictest0-21swapper/021:25:410
12737699200,1cyclictest0-21swapper/019:20:440
12737699200,19cyclictest3056-21PK-Backend21:30:150
12737699200,19cyclictest3056-21PK-Backend21:30:140
12737699200,19cyclictest0-21swapper/023:41:140
12737699200,19cyclictest0-21swapper/023:07:350
12737699200,19cyclictest0-21swapper/022:06:590
12737699200,19cyclictest0-21swapper/020:54:320
12737699200,19cyclictest0-21swapper/000:00:080
139099190,16rtkit-daemon0-21swapper/123:51:531
12738699192,15cyclictest3056-21PK-Backend19:59:293
127386991917,1cyclictest3056-21PK-Backend20:22:513
127386991916,2cyclictest3056-21PK-Backend23:42:583
127386991914,3cyclictest183284-21strings21:45:173
127386991913,4cyclictest167070-21cstates21:00:123
127386991911,6cyclictest42-21ksoftirqd/319:50:133
12738699190,1cyclictest3056-21PK-Backend21:39:363
12738699190,1cyclictest3056-21PK-Backend19:18:213
12738199195,3cyclictest141rcu_preempt23:20:152
12738199194,6cyclictest156438-21diskstats20:30:152
12738199193,3cyclictest141rcu_preempt21:23:342
12738199193,3cyclictest141rcu_preempt20:26:112
12738199192,3cyclictest141rcu_preempt19:10:222
127381991915,3cyclictest173672-21turbostat.cron21:20:012
127381991914,3cyclictest170622-21grep21:10:122
12738199191,3cyclictest141rcu_preempt19:43:442
127381991912,6cyclictest141322-21python319:45:222
127381991912,6cyclictest141322-21python319:45:212
127381991911,6cyclictest34-21ksoftirqd/220:35:222
127381991911,6cyclictest34-21ksoftirqd/200:35:222
12738199190,6cyclictest884-21gmain23:40:402
127380991918,1cyclictest0-21swapper/121:37:021
127380991917,1cyclictest0-21swapper/123:02:471
127380991917,1cyclictest0-21swapper/122:56:381
127380991917,1cyclictest0-21swapper/122:23:541
127380991917,1cyclictest0-21swapper/119:50:031
127380991912,6cyclictest216597-21which23:20:191
12738099190,1cyclictest0-21swapper/123:56:481
12738099190,1cyclictest0-21swapper/122:49:091
12738099190,1cyclictest0-21swapper/121:41:011
12738099190,18cyclictest0-21swapper/120:20:581
12738099190,18cyclictest0-21swapper/119:28:131
127376991917,1cyclictest0-21swapper/022:55:530
127376991917,1cyclictest0-21swapper/022:14:220
127376991917,1cyclictest0-21swapper/020:31:410
127376991917,1cyclictest0-21swapper/020:09:510
127376991917,1cyclictest0-21swapper/019:51:060
127376991917,1cyclictest0-21swapper/019:18:480
127376991913,5cyclictest13-21ksoftirqd/023:15:120
127376991913,5cyclictest13-21ksoftirqd/022:04:090
12737699190,1cyclictest0-21swapper/023:23:310
12737699190,1cyclictest0-21swapper/022:50:530
12737699190,1cyclictest0-21swapper/020:25:390
12737699190,1cyclictest0-21swapper/020:12:220
12737699190,18cyclictest0-21swapper/023:34:250
12737699190,18cyclictest0-21swapper/022:36:540
12737699190,18cyclictest0-21swapper/021:23:590
12737699190,18cyclictest0-21swapper/020:03:420
12738699186,3cyclictest1618152expr20:45:133
127386991816,1cyclictest3056-21PK-Backend23:23:123
127386991816,1cyclictest3056-21PK-Backend20:40:493
127386991816,1cyclictest3056-21PK-Backend19:40:043
127386991816,1cyclictest3056-21PK-Backend19:14:293
127386991815,2cyclictest3056-21PK-Backend22:28:233
127386991815,2cyclictest3056-21PK-Backend00:35:523
12738699181,3cyclictest141rcu_preempt22:10:013
12738199186,4cyclictest34-21ksoftirqd/219:18:032
12738199184,6cyclictest132277-21ntp_states19:20:182
12738199183,3cyclictest141rcu_preempt00:34:592
12738199182,2cyclictest141rcu_preempt20:41:502
127381991816,1cyclictest0-21swapper/219:54:102
127381991813,4cyclictest34-21ksoftirqd/221:05:012
127381991810,6cyclictest34-21ksoftirqd/221:05:162
127381991810,2cyclictest34-21ksoftirqd/222:30:332
12738199180,3cyclictest141rcu_preempt20:05:532
12738199180,1cyclictest0-21swapper/220:10:192
127380991816,1cyclictest0-21swapper/122:33:261
127380991816,1cyclictest0-21swapper/120:37:421
127380991811,5cyclictest137646-21tr19:35:181
12738099180,1cyclictest0-21swapper/122:09:471
127376991816,1cyclictest0-21swapper/023:51:230
127376991816,1cyclictest0-21swapper/023:01:270
127376991816,1cyclictest0-21swapper/021:10:080
127376991816,1cyclictest0-21swapper/000:23:060
127376991811,6cyclictest158354-21sed20:35:170
127386991713,3cyclictest163627-21df20:50:133
127386991712,3cyclictest172331-21tr21:15:143
127386991711,5cyclictest195349-21sed22:20:173
127386991710,5cyclictest178154-21ntpq21:30:203
127386991710,5cyclictest178154-21ntpq21:30:193
12738199172,2cyclictest141rcu_preempt21:50:192
12738199172,12cyclictest194746-21apt22:20:112
127381991715,1cyclictest0-21swapper/200:05:442
12738199171,2cyclictest141rcu_preempt20:00:102
12738199171,2cyclictest141rcu_preempt19:25:132
127381991712,4cyclictest239234-21python300:25:222
127381991710,2cyclictest34-21ksoftirqd/220:16:292
12738099178,7cyclictest883-21dbus-daemon20:42:071
12737699179,6cyclictest165311-21expr20:55:130
12737699179,6cyclictest139442-21ntpq19:40:170
127376991715,1cyclictest0-21swapper/000:29:350
127376991710,5cyclictest218297-21ntpq23:25:190
12738699169,5cyclictest42-21ksoftirqd/321:50:163
12738699162,2cyclictest141rcu_preempt20:13:023
127386991614,1cyclictest3056-21PK-Backend20:27:373
12738699161,3cyclictest141rcu_preempt21:20:563
12738199169,2cyclictest34-21ksoftirqd/223:49:392
12738199168,7cyclictest163730-21cat20:50:162
127380991612,3cyclictest164973-21/usr/sbin/munin20:55:311
127380991610,4cyclictest161675-21tr20:45:121
12737699161,4cyclictest141rcu_preempt20:20:110
127376991614,1cyclictest3056-21PK-Backend00:06:020
127376991611,3cyclictest182346-21latency_hist21:45:010
127376991611,3cyclictest13-21ksoftirqd/023:35:110
127381991511,3cyclictest161405-21apt-key20:45:012
127380991511,2cyclictest192738-21sh22:15:001
127380991510,4cyclictest26-21ksoftirqd/121:49:481
12737699151,11cyclictest179154-21apt21:35:120
12737699150,14cyclictest1389-21rtkit-daemon21:09:090
12738099149,3cyclictest139089-21df19:40:131
12738099148,5cyclictest164245-21grep20:50:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional