You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-13 - 12:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Jul 13, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
297834021320,8sleep342-21ksoftirqd/319:15:123
297510927044,16sleep30-21swapper/319:05:243
297543226545,14sleep00-21swapper/019:05:450
297565526444,14sleep20-21swapper/219:08:382
297569126240,15sleep10-21swapper/119:09:011
30163112440,4sleep13056-21PK-Backend21:00:201
139099330,4rtkit-daemon2637-21gmain21:58:251
30064332300,3sleep00-21swapper/020:35:000
139099270,4rtkit-daemon1389-21rtkit-daemon19:42:080
139099270,4rtkit-daemon1389-21rtkit-daemon19:42:080
30260272260,7sleep3421ksoftirqd/321:30:123
29979982260,4sleep00-21swapper/020:10:150
297589699257,3cyclictest141rcu_preempt19:54:022
139099250,4rtkit-daemon1389-21rtkit-daemon22:13:431
2975893992417,5cyclictest2992510-21cstates19:55:151
297589999239,12cyclictest3008097-21unin-run20:40:003
297589999230,21cyclictest3064459-21users23:15:243
2975896992319,3cyclictest34-21ksoftirqd/223:35:012
2975896992315,6cyclictest436-20systemd-journal21:34:592
2975891992320,2cyclictest0-21swapper/023:00:320
2975891992320,2cyclictest0-21swapper/020:07:210
2975891992313,8cyclictest3048964-21cron22:35:000
297589999225,15cyclictest3086547-21wc00:20:013
297589999225,15cyclictest0-21swapper/323:30:143
297589999222,6cyclictest3035465-21python321:55:223
297589999222,18cyclictest3062391-21ntp_states23:10:203
2975899992220,1cyclictest0-21swapper/321:27:223
2975899992220,1cyclictest0-21swapper/300:07:223
2975899992219,2cyclictest3056-21PK-Backend22:38:483
297589999221,8cyclictest3028272-21python321:35:223
2975899992216,4cyclictest3043527-21idleruntime-cro22:20:003
297589999220,8cyclictest30804942timerandwakeup00:00:253
297589999220,20cyclictest3040685-21df22:10:143
297589699229,3cyclictest141rcu_preempt22:37:502
297589699224,9cyclictest34-21ksoftirqd/221:50:152
2975896992220,1cyclictest0-21swapper/223:20:002
2975896992220,1cyclictest0-21swapper/221:16:242
2975896992220,1cyclictest0-21swapper/221:14:072
2975896992220,1cyclictest0-21swapper/200:39:292
2975896992218,3cyclictest4176944-21snapd20:59:142
2975896992218,3cyclictest3000875-21apt20:20:012
2975896992210,10cyclictest34-21ksoftirqd/222:05:002
297589399220,20cyclictest3056-21PK-Backend23:15:571
297589399220,20cyclictest3056-21PK-Backend20:46:381
2975891992220,1cyclictest0-21swapper/023:42:230
2975891992220,1cyclictest0-21swapper/023:39:250
2975891992220,1cyclictest0-21swapper/022:42:540
2975891992220,1cyclictest0-21swapper/022:35:380
2975891992220,1cyclictest0-21swapper/021:43:050
2975891992220,1cyclictest0-21swapper/021:27:510
2975891992220,1cyclictest0-21swapper/019:46:120
2975891992220,1cyclictest0-21swapper/000:18:400
2975891992216,3cyclictest2995398-21latency_hist20:05:010
2975891992211,9cyclictest3076315-21latency23:50:170
30104292210,3sleep3411rcuc/320:45:143
30038972210,8sleep126-21ksoftirqd/120:25:191
297589999213,16cyclictest3083941-21strings00:10:193
297589999212,16cyclictest3053572-21python322:45:203
2975899992119,1cyclictest3056-21PK-Backend20:15:583
2975899992119,1cyclictest0-21swapper/323:55:263
2975899992119,1cyclictest0-21swapper/321:22:083
2975899992119,1cyclictest0-21swapper/321:10:533
2975899992119,1cyclictest0-21swapper/321:03:083
2975899992119,1cyclictest0-21swapper/320:40:393
2975899992119,1cyclictest0-21swapper/320:30:103
2975899992119,1cyclictest0-21swapper/319:21:573
2975899992119,1cyclictest0-21swapper/300:30:543
2975899992114,5cyclictest3058383-21/usr/sbin/munin23:00:143
2975899992110,8cyclictest3045521-21sh22:24:593
297589999210,20cyclictest0-21swapper/323:05:163
297589999210,20cyclictest0-21swapper/320:06:323
297589999210,19cyclictest3056-21PK-Backend19:55:223
297589999210,19cyclictest3056-21PK-Backend19:52:523
297589999210,19cyclictest0-21swapper/323:40:183
297589699218,3cyclictest141rcu_preempt00:13:532
297589699217,5cyclictest34-21ksoftirqd/200:30:012
297589699217,2cyclictest141rcu_preempt20:34:072
297589699213,7cyclictest34-21ksoftirqd/222:20:122
2975896992119,1cyclictest0-21swapper/223:53:122
2975896992119,1cyclictest0-21swapper/222:59:452
2975896992119,1cyclictest0-21swapper/222:06:542
2975896992119,1cyclictest0-21swapper/221:23:412
2975896992119,1cyclictest0-21swapper/221:06:232
2975896992119,1cyclictest0-21swapper/221:00:212
2975896992119,1cyclictest0-21swapper/220:08:572
2975896992119,1cyclictest0-21swapper/219:16:112
2975896992119,1cyclictest0-21swapper/200:32:262
2975896992117,3cyclictest331rcuc/223:45:002
2975896992114,5cyclictest2980256-21expr19:20:132
297589699210,2cyclictest0-21swapper/200:20:162
297589699210,20cyclictest0-21swapper/223:04:592
297589699210,20cyclictest0-21swapper/220:50:352
297589399210,20cyclictest3056-21PK-Backend20:24:421
297589399210,20cyclictest3056-21PK-Backend19:24:381
297589399210,20cyclictest3056-21PK-Backend00:09:581
297589399210,20cyclictest0-21swapper/120:04:411
297589399210,20cyclictest0-21swapper/100:14:511
297589399210,19cyclictest3056-21PK-Backend23:53:551
297589399210,19cyclictest3056-21PK-Backend23:06:371
297589399210,19cyclictest3056-21PK-Backend22:30:341
297589399210,19cyclictest3056-21PK-Backend20:56:121
297589399210,19cyclictest3056-21PK-Backend20:12:241
297589399210,19cyclictest3056-21PK-Backend19:35:411
297589399210,19cyclictest3056-21PK-Backend00:35:271
297589399210,19cyclictest3056-21PK-Backend00:29:441
297589399210,19cyclictest3056-21PK-Backend00:02:201
2975891992119,1cyclictest0-21swapper/023:30:400
2975891992119,1cyclictest0-21swapper/023:13:570
2975891992119,1cyclictest0-21swapper/022:22:470
2975891992119,1cyclictest0-21swapper/022:14:220
2975891992119,1cyclictest0-21swapper/021:31:030
2975891992119,1cyclictest0-21swapper/021:17:230
2975891992119,1cyclictest0-21swapper/020:52:430
2975891992119,1cyclictest0-21swapper/020:18:290
2975891992119,1cyclictest0-21swapper/019:51:460
2975891992119,1cyclictest0-21swapper/019:16:520
2975891992119,1cyclictest0-21swapper/000:34:210
2975891992119,1cyclictest0-21swapper/000:01:020
2975891992110,8cyclictest883-21dbus-daemon20:00:000
297589199210,4cyclictest3042750-21awk22:15:160
297589199210,20cyclictest3056-21PK-Backend20:20:110
297589199210,20cyclictest0-21swapper/000:37:520
2975899992018,1cyclictest3056-21PK-Backend20:04:483
2975899992018,1cyclictest0-21swapper/323:29:243
2975899992018,1cyclictest0-21swapper/300:37:573
2975899992011,7cyclictest436-20systemd-journal21:50:003
297589999200,19cyclictest3056-21PK-Backend19:44:143
297589999200,19cyclictest3056-21PK-Backend19:44:133
297589999200,19cyclictest0-21swapper/323:51:073
297589999200,19cyclictest0-21swapper/323:46:473
297589999200,19cyclictest0-21swapper/322:53:203
297589999200,19cyclictest0-21swapper/321:53:113
297589999200,19cyclictest0-21swapper/321:08:373
297589699209,3cyclictest141rcu_preempt19:41:132
297589699209,3cyclictest141rcu_preempt19:41:132
297589699207,3cyclictest141rcu_preempt19:47:162
297589699204,9cyclictest34-21ksoftirqd/222:15:152
2975896992018,1cyclictest0-21swapper/222:33:292
2975896992018,1cyclictest0-21swapper/220:41:012
2975896992018,1cyclictest0-21swapper/200:05:222
297589699201,5cyclictest34-21ksoftirqd/223:37:292
2975896992013,5cyclictest3066708-21sed23:25:002
2975896992011,2cyclictest141rcu_preempt20:03:192
297589699200,8cyclictest34-21ksoftirqd/219:55:152
297589699200,6cyclictest3023997-21cpuspeed21:25:112
297589699200,19cyclictest3056-21PK-Backend21:59:292
297589699200,19cyclictest0-21swapper/223:58:542
297589699200,19cyclictest0-21swapper/223:13:112
297589699200,19cyclictest0-21swapper/222:49:242
297589699200,19cyclictest0-21swapper/221:42:062
297589699200,19cyclictest0-21swapper/220:46:502
297589699200,19cyclictest0-21swapper/200:04:352
2975893992017,1cyclictest3056-21PK-Backend21:39:311
2975893992017,1cyclictest3056-21PK-Backend21:25:391
297589399200,1cyclictest3056-21PK-Backend23:03:531
297589399200,1cyclictest3056-21PK-Backend00:33:171
297589399200,19cyclictest3056-21PK-Backend21:09:091
297589399200,19cyclictest3056-21PK-Backend19:17:301
297589399200,19cyclictest3056-21PK-Backend19:12:361
297589399200,19cyclictest3056-21PK-Backend00:22:411
297589399200,19cyclictest3056-21PK-Backend00:15:381
297589399200,19cyclictest0-21swapper/122:23:521
297589399200,19cyclictest0-21swapper/119:40:281
297589399200,19cyclictest0-21swapper/119:40:271
297589399200,18cyclictest3056-21PK-Backend20:43:271
2975891992018,1cyclictest3056-21PK-Backend23:48:530
2975891992018,1cyclictest0-21swapper/023:58:520
2975891992018,1cyclictest0-21swapper/023:24:390
2975891992018,1cyclictest0-21swapper/023:05:120
2975891992018,1cyclictest0-21swapper/022:01:040
2975891992018,1cyclictest0-21swapper/020:36:210
2975891992018,1cyclictest0-21swapper/019:35:500
2975891992018,1cyclictest0-21swapper/000:14:210
2975891992013,6cyclictest3088477-21sshd00:25:110
2975891992013,4cyclictest2983266-21sh19:30:000
297589199200,1cyclictest0-21swapper/019:32:120
297589199200,19cyclictest0-21swapper/023:26:010
297589199200,19cyclictest0-21swapper/023:15:190
297589199200,19cyclictest0-21swapper/021:22:030
297589199200,19cyclictest0-21swapper/020:48:240
297589199200,19cyclictest0-21swapper/020:42:020
30466552190,4sleep00-21swapper/022:25:200
2975899991917,1cyclictest0-21swapper/322:32:053
2975899991917,1cyclictest0-21swapper/321:19:313
2975899991917,1cyclictest0-21swapper/320:14:163
2975899991917,1cyclictest0-21swapper/300:20:553
2975899991914,3cyclictest3070042-21apt-get23:35:033
2975899991911,6cyclictest3065829-21ntpq23:20:183
297589999191,16cyclictest2982666-21python319:25:213
297589999190,1cyclictest0-21swapper/320:52:483
297589999190,18cyclictest3056-21PK-Backend19:47:193
297589999190,18cyclictest0-21swapper/321:40:213
297589999190,18cyclictest0-21swapper/319:10:063
297589699198,3cyclictest141rcu_preempt22:40:552
297589699195,8cyclictest34-21ksoftirqd/219:31:462
2975896991917,1cyclictest0-21swapper/223:27:572
2975896991917,1cyclictest0-21swapper/222:51:542
2975896991917,1cyclictest0-21swapper/220:10:392
2975896991917,1cyclictest0-21swapper/219:10:572
297589699191,6cyclictest3001771-21df_inode20:20:142
2975896991913,5cyclictest2982055-21expr19:25:132
297589699190,1cyclictest0-21swapper/223:46:432
297589699190,1cyclictest0-21swapper/200:18:062
2975893991917,1cyclictest3056-21PK-Backend23:41:121
2975893991917,1cyclictest3056-21PK-Backend23:34:561
2975893991917,1cyclictest3056-21PK-Backend23:24:281
2975893991917,1cyclictest0-21swapper/119:47:091
2975893991916,2cyclictest3056-21PK-Backend23:45:301
2975893991916,2cyclictest3056-21PK-Backend23:12:401
2975893991916,2cyclictest3056-21PK-Backend22:54:401
297589399190,1cyclictest3056-21PK-Backend23:58:491
297589399190,1cyclictest3056-21PK-Backend22:18:321
297589399190,1cyclictest3056-21PK-Backend22:08:241
297589399190,1cyclictest3056-21PK-Backend21:40:051
297589399190,1cyclictest3056-21PK-Backend21:15:451
297589399190,1cyclictest3056-21PK-Backend20:52:261
297589399190,1cyclictest3056-21PK-Backend20:06:581
297589399190,1cyclictest0-21swapper/122:38:251
297589399190,1cyclictest0-21swapper/119:50:141
297589199194,13cyclictest3040067-21awk22:10:000
2975891991917,1cyclictest0-21swapper/021:11:230
2975891991917,1cyclictest0-21swapper/021:09:400
2975891991915,3cyclictest1895009-21snapd22:46:130
2975891991914,4cyclictest2980738-21grep19:20:190
2975891991911,5cyclictest1095-21runrttasks00:05:030
297589199190,1cyclictest0-21swapper/019:11:580
297589199190,18cyclictest0-21swapper/022:53:500
297589999182,12cyclictest3003137-21apt20:25:113
2975899991816,1cyclictest0-21swapper/322:25:573
2975899991814,3cyclictest3038959-21diskstats22:05:143
2975899991811,6cyclictest3051841-21python322:40:213
2975899991811,5cyclictest3057152-21ntpq22:55:193
297589999181,15cyclictest3037453-21python322:00:213
2975899991810,6cyclictest2984299-21awk19:30:193
297589699188,2cyclictest141rcu_preempt21:38:342
297589699186,2cyclictest141rcu_preempt20:36:272
2975896991816,1cyclictest0-21swapper/221:45:312
2975896991816,1cyclictest0-21swapper/219:36:192
2975896991814,3cyclictest331rcuc/223:05:152
2975896991811,5cyclictest3040766-21cat22:10:172
2975893991816,1cyclictest3056-21PK-Backend21:50:061
2975893991816,1cyclictest3056-21PK-Backend21:10:171
2975893991815,2cyclictest3056-21PK-Backend23:27:461
2975893991815,2cyclictest3056-21PK-Backend22:30:001
2975893991813,3cyclictest3006329-21latency_hist20:35:011
2975893991811,6cyclictest3026253-21memory21:30:181
2975891991816,1cyclictest3056-21PK-Backend21:59:220
2975891991816,1cyclictest0-21swapper/021:45:220
297589199181,12cyclictest3057908-21cron23:00:010
2975899991712,3cyclictest3013512-21/usr/sbin/munin20:55:233
2975899991712,3cyclictest2985813-21latency19:35:163
297589999171,14cyclictest3000875-21apt-get20:20:033
2975896991710,5cyclictest3004585-21cron20:30:002
2975893991715,1cyclictest3056-21PK-Backend22:49:551
2975893991715,1cyclictest0-21swapper/122:00:051
2975893991714,2cyclictest3056-21PK-Backend21:20:591
2975893991710,5cyclictest3072039-21cron23:40:001
2975893991710,5cyclictest3000411-21cat20:15:221
2975893991710,5cyclictest26-21ksoftirqd/122:40:211
2975891991715,1cyclictest0-21swapper/021:03:040
2975891991715,1cyclictest0-21swapper/020:57:360
2975899991614,1cyclictest0-21swapper/300:28:443
2975896991610,4cyclictest2156-21gnome-shell22:26:092
297589399167,7cyclictest436-20systemd-journal20:35:001
2975893991614,1cyclictest3056-21PK-Backend22:58:531
2975893991614,1cyclictest3056-21PK-Backend21:48:571
297589199162,5cyclictest13-21ksoftirqd/021:35:120
2975891991513,1cyclictest0-21swapper/000:21:370
2975891991512,2cyclictest3003296-21tr20:25:110
2975893991410,3cyclictest3056-21PK-Backend19:25:151
297589199131,11cyclictest3033009-21tr21:50:120
297589399110,2cyclictest3056-21PK-Backend19:33:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional