You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-05 - 03:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Fri Jun 05, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
246780621350,5sleep12467805-21sed22:30:001
24601152950,2sleep00-21swapper/022:05:210
139099940,79rtkit-daemon0-21swapper/319:06:093
239533628247,24sleep10-21swapper/119:05:101
239672626940,22sleep00-21swapper/019:08:550
239649226140,15sleep20-21swapper/219:05:562
24299802490,4sleep20-21swapper/220:40:192
24315772360,3sleep3411rcuc/320:45:173
23975282360,4sleep13056-21PK-Backend19:10:121
24666602310,3sleep30-21swapper/322:25:183
24132752300,3sleep20-21swapper/219:55:002
2397077992910,4cyclictest2433489-21sed20:50:213
24046452280,4chrt3056-21PK-Backend19:30:011
239707799285,8cyclictest883-21dbus-daemon20:24:483
2397077992810,5cyclictest42-21ksoftirqd/300:30:153
239707799279,4cyclictest2487444-21grep23:25:013
239707799279,4cyclictest2446133-21switchtime21:25:223
239707799278,5cyclictest2416761-21date20:05:003
239707799277,6cyclictest2495354-21pluginstate23:45:213
24257232260,8sleep126-21ksoftirqd/120:30:121
239707799269,5cyclictest2450930-21uname21:40:173
2397077992613,3cyclictest2417094-21cpuspeed_turbos20:05:133
24367712250,9sleep013-21ksoftirqd/021:00:180
24307532250,3sleep33056-21PK-Backend20:45:013
2397077992510,3cyclictest2475986-21ntp_states22:50:203
239707799250,8cyclictest2447717-21ntp_states21:30:193
239707799250,3cyclictest2504101-21memory00:10:163
239707799250,1cyclictest0-21swapper/320:11:513
239707799250,14cyclictest2436026-21/usr/sbin/munin21:00:123
2397047992516,7cyclictest2497771-21apt-get23:55:000
25112212240,8sleep013-21ksoftirqd/000:30:170
2397077992410,2cyclictest2513657-21latency_hist00:40:003
239707799240,4cyclictest2423924-21apt20:25:113
239707799240,15cyclictest2505991-21ntp_states00:15:203
239707799240,13cyclictest2507202-21cstates00:20:133
239706399240,23cyclictest2413832-21expr19:55:121
24894962230,3sleep13056-21PK-Backend23:30:121
239707799231,7cyclictest2453911-21apt21:50:123
2397077992317,5cyclictest0-21swapper/300:08:013
239707799231,11cyclictest2441485-21/usr/sbin/munin21:15:243
2397077992310,3cyclictest886-21NetworkManager23:29:513
239707799230,13cyclictest2510017-21chrt00:28:003
239707799230,13cyclictest2471027-21chrt22:37:253
2397072992313,7cyclictest436-20systemd-journal23:30:012
2397047992315,6cyclictest2430575-21cron20:45:010
2397047992313,8cyclictest436-20systemd-journal19:20:000
139099230,15rtkit-daemon0-21swapper/323:10:193
2397077992219,2cyclictest3056-21PK-Backend19:47:103
239707799220,20cyclictest3056-21PK-Backend19:43:323
239707799220,20cyclictest24131052sleep319:53:263
239707799220,13cyclictest2439753-21/usr/sbin/munin21:10:213
239707299228,12cyclictest2499891-21date00:00:012
2397072992220,1cyclictest0-21swapper/223:46:342
2397072992220,1cyclictest0-21swapper/223:04:512
2397072992220,1cyclictest0-21swapper/222:29:422
2397072992220,1cyclictest0-21swapper/221:31:422
2397072992220,1cyclictest0-21swapper/221:24:322
2397072992220,1cyclictest0-21swapper/219:58:272
2397072992214,6cyclictest436-20systemd-journal00:05:002
239707299220,20cyclictest3056-21PK-Backend21:05:332
239707299220,20cyclictest0-21swapper/220:48:372
2397047992220,1cyclictest0-21swapper/020:34:230
2397047992220,1cyclictest0-21swapper/020:24:020
2397047992220,1cyclictest0-21swapper/019:44:350
2397047992220,1cyclictest0-21swapper/019:10:400
2397047992217,3cyclictest2496046-21ls23:50:000
239704799220,21cyclictest0-21swapper/020:47:330
239704799220,21cyclictest0-21swapper/020:11:040
139099220,14rtkit-daemon3056-21PK-Backend19:15:442
2397077992119,1cyclictest3056-21PK-Backend22:45:293
2397077992119,1cyclictest3056-21PK-Backend22:09:263
2397077992119,1cyclictest3056-21PK-Backend20:19:213
2397077992119,1cyclictest0-21swapper/323:59:243
2397077992119,1cyclictest0-21swapper/323:53:023
2397077992119,1cyclictest0-21swapper/321:55:203
2397077992119,1cyclictest0-21swapper/320:30:443
2397077992119,1cyclictest0-21swapper/300:04:433
239707799211,8cyclictest2443510-21apt21:20:123
2397077992114,1cyclictest0-21swapper/323:09:273
2397077992110,2cyclictest2476978-21apt22:55:133
2397077992110,2cyclictest2476978-21apt22:55:133
239707799210,8cyclictest2427781-21fw_packets20:35:153
2397072992119,1cyclictest0-21swapper/223:33:232
2397072992119,1cyclictest0-21swapper/223:23:432
2397072992119,1cyclictest0-21swapper/222:17:592
2397072992119,1cyclictest0-21swapper/222:00:082
2397072992119,1cyclictest0-21swapper/221:36:502
2397072992119,1cyclictest0-21swapper/220:53:302
2397072992119,1cyclictest0-21swapper/220:26:392
2397072992119,1cyclictest0-21swapper/220:01:182
2397072992119,1cyclictest0-21swapper/219:21:572
2397072992119,1cyclictest0-21swapper/200:25:142
2397072992110,9cyclictest2492568-21cron23:40:012
239707299210,20cyclictest0-21swapper/221:49:542
239707299210,20cyclictest0-21swapper/220:55:332
239707299210,20cyclictest0-21swapper/220:55:322
239707299210,20cyclictest0-21swapper/220:39:032
239707299210,20cyclictest0-21swapper/220:30:312
239707299210,20cyclictest0-21swapper/219:47:192
239707299210,20cyclictest0-21swapper/219:36:242
239707299210,20cyclictest0-21swapper/200:37:452
2397063992117,3cyclictest2422593-21ls20:20:151
2397063992113,6cyclictest2463310-21cat22:15:181
239706399210,1cyclictest3056-21PK-Backend00:07:051
239706399210,19cyclictest3056-21PK-Backend23:55:351
239706399210,19cyclictest3056-21PK-Backend23:14:461
239706399210,19cyclictest3056-21PK-Backend22:53:031
239706399210,19cyclictest3056-21PK-Backend21:35:441
239706399210,19cyclictest3056-21PK-Backend20:35:401
2397047992119,1cyclictest0-21swapper/023:10:580
2397047992119,1cyclictest0-21swapper/021:43:030
2397047992119,1cyclictest0-21swapper/021:31:010
2397047992119,1cyclictest0-21swapper/021:22:020
2397047992119,1cyclictest0-21swapper/021:06:190
2397047992119,1cyclictest0-21swapper/020:58:230
2397047992119,1cyclictest0-21swapper/020:58:230
2397047992119,1cyclictest0-21swapper/020:25:040
2397047992119,1cyclictest0-21swapper/019:59:080
2397047992119,1cyclictest0-21swapper/000:01:400
239704799211,5cyclictest883-21dbus-daemon19:55:000
2397047992114,4cyclictest2466723-21strings22:25:180
239704799210,20cyclictest0-21swapper/022:52:060
239704799210,20cyclictest0-21swapper/022:16:370
239704799210,20cyclictest0-21swapper/021:40:000
239704799210,20cyclictest0-21swapper/021:26:550
239704799210,20cyclictest0-21swapper/019:26:160
139099210,3rtkit-daemon0-21swapper/222:57:042
139099210,3rtkit-daemon0-21swapper/222:57:042
139099210,15rtkit-daemon3056-21PK-Backend19:36:393
139099210,15rtkit-daemon0-21swapper/222:35:322
139099210,13rtkit-daemon0-21swapper/223:42:502
2397077992017,1cyclictest3056-21PK-Backend19:20:203
2397077992013,5cyclictest2397661-21expr19:10:123
2397077992012,6cyclictest2453039-21python321:45:203
239707799200,1cyclictest3056-21PK-Backend19:17:013
239707799200,19cyclictest3056-21PK-Backend22:11:283
239707799200,19cyclictest3056-21PK-Backend22:02:163
239707799200,19cyclictest3056-21PK-Backend21:07:213
239707799200,19cyclictest3056-21PK-Backend19:57:043
239707799200,19cyclictest0-21swapper/322:15:343
2397072992018,1cyclictest0-21swapper/221:57:032
2397072992018,1cyclictest0-21swapper/221:26:352
2397072992013,6cyclictest2404594-21/usr/sbin/munin19:29:592
2397072992013,5cyclictest2423645-21cron20:25:002
2397072992012,6cyclictest883-21dbus-daemon19:14:482
239707299200,19cyclictest0-21swapper/222:23:192
239707299200,19cyclictest0-21swapper/221:16:352
239707299200,19cyclictest0-21swapper/221:14:322
2397063992018,1cyclictest3056-21PK-Backend23:39:471
2397063992018,1cyclictest3056-21PK-Backend23:05:341
2397063992018,1cyclictest0-21swapper/122:31:191
2397063992018,1cyclictest0-21swapper/119:41:401
2397063992017,2cyclictest3056-21PK-Backend21:57:341
239706399200,2cyclictest2438911-21smartctl21:05:221
239706399200,1cyclictest3056-21PK-Backend21:48:351
239706399200,1cyclictest3056-21PK-Backend00:38:011
239706399200,1cyclictest0-21swapper/122:14:441
239706399200,19cyclictest3056-21PK-Backend00:33:551
239706399200,19cyclictest0-21swapper/120:17:021
239706399200,18cyclictest3056-21PK-Backend23:47:111
239706399200,18cyclictest3056-21PK-Backend20:06:061
2397047992019,1cyclictest0-21swapper/023:34:570
2397047992019,1cyclictest0-21swapper/020:51:260
2397047992018,1cyclictest0-21swapper/022:12:300
2397047992018,1cyclictest0-21swapper/020:05:370
2397047992018,1cyclictest0-21swapper/000:35:060
2397047992018,1cyclictest0-21swapper/000:18:310
2397047992013,4cyclictest2402858-21idleruntime-cro19:25:000
239704799200,19cyclictest0-21swapper/023:56:210
239704799200,19cyclictest0-21swapper/023:19:100
239704799200,19cyclictest0-21swapper/021:54:060
239704799200,19cyclictest0-21swapper/021:45:070
239704799200,19cyclictest0-21swapper/019:32:180
139099200,14rtkit-daemon0-21swapper/223:50:012
139099200,14rtkit-daemon0-21swapper/223:08:122
2397077991917,1cyclictest0-21swapper/323:34:423
2397077991917,1cyclictest0-21swapper/323:19:563
2397077991913,5cyclictest2480247-21wc23:05:013
2397077991910,7cyclictest2405220-21df_inode19:30:133
239707799190,18cyclictest3056-21PK-Backend22:23:463
2397072991917,1cyclictest0-21swapper/220:17:072
2397072991917,1cyclictest0-21swapper/200:09:122
2397072991911,5cyclictest2485169-21timerwakeupswit23:15:212
239707299190,1cyclictest0-21swapper/219:44:282
239707299190,18cyclictest0-21swapper/223:11:132
2397063991917,1cyclictest3056-21PK-Backend21:17:591
2397063991917,1cyclictest3056-21PK-Backend21:10:151
2397063991917,1cyclictest3056-21PK-Backend00:16:301
2397063991917,1cyclictest0-21swapper/119:47:551
2397063991917,1cyclictest0-21swapper/119:36:251
239706399190,1cyclictest3056-21PK-Backend23:28:051
239706399190,1cyclictest3056-21PK-Backend23:23:581
239706399190,1cyclictest3056-21PK-Backend21:33:401
239706399190,1cyclictest3056-21PK-Backend21:04:191
239706399190,1cyclictest3056-21PK-Backend20:53:321
239706399190,1cyclictest3056-21PK-Backend20:02:291
239706399190,1cyclictest3056-21PK-Backend19:30:251
239706399190,1cyclictest3056-21PK-Backend00:27:461
239706399190,1cyclictest3056-21PK-Backend00:23:401
239706399190,1cyclictest0-21swapper/122:37:281
239706399190,1cyclictest0-21swapper/122:04:291
239706399190,1cyclictest0-21swapper/120:42:101
239706399190,18cyclictest3056-21PK-Backend21:42:401
2397047991918,1cyclictest0-21swapper/021:17:080
2397047991917,1cyclictest0-21swapper/020:35:240
2397047991917,1cyclictest0-21swapper/020:19:210
2397047991917,1cyclictest0-21swapper/019:45:350
2397047991914,3cyclictest2477148-21tr22:55:140
2397047991914,3cyclictest2477148-21tr22:55:140
2397047991912,5cyclictest4176944-21snapd22:36:130
2397047991912,5cyclictest2457352-21/usr/sbin/munin22:00:130
2397047991911,6cyclictest436-20systemd-journal23:30:000
2397047991911,6cyclictest2407214-21cat19:35:170
239704799190,1cyclictest0-21swapper/022:33:130
239704799190,1cyclictest0-21swapper/021:56:090
239704799190,1cyclictest0-21swapper/000:24:400
239704799190,18cyclictest0-21swapper/020:04:350
139099190,14rtkit-daemon0-21swapper/323:43:383
139099190,14rtkit-daemon0-21swapper/221:43:162
24196462180,5sleep2331rcuc/220:10:202
2397077991814,2cyclictest3056-21PK-Backend22:40:213
2397077991813,3cyclictest2491852-21ntp_states23:35:183
239707799180,16cyclictest2448968-21cstates21:35:133
2397072991816,1cyclictest0-21swapper/221:54:002
2397072991816,1cyclictest0-21swapper/221:04:322
2397072991816,1cyclictest0-21swapper/219:31:102
239707299181,12cyclictest2503523-21apt00:10:102
239707299180,1cyclictest0-21swapper/222:50:032
2397063991816,1cyclictest3056-21PK-Backend20:59:271
2397063991816,1cyclictest3056-21PK-Backend20:59:271
2397063991816,1cyclictest0-21swapper/119:17:001
2397063991810,7cyclictest2472252-21ntp_states22:40:191
2397063991810,6cyclictest2473828-21memory22:45:181
239706399180,1cyclictest3056-21PK-Backend23:17:511
239706399180,1cyclictest3056-21PK-Backend21:21:511
239706399180,1cyclictest3056-21PK-Backend20:26:221
2397047991816,1cyclictest0-21swapper/023:39:170
239707799173,12cyclictest2434329-21cpu20:55:103
239707799173,12cyclictest2434329-21cpu20:55:103
2397077991711,5cyclictest2404400-21seq19:27:553
239707299179,6cyclictest2418051-21awk20:05:222
2397072991715,1cyclictest0-21swapper/222:08:332
2397072991710,5cyclictest2468762-21ntp_kernel_pll_22:30:192
239706399179,6cyclictest2497777-21cron23:54:591
2397063991715,1cyclictest3056-21PK-Backend21:27:461
2397063991715,1cyclictest3056-21PK-Backend20:13:241
2397063991714,2cyclictest3056-21PK-Backend21:51:391
2397063991711,4cyclictest2402894-21cut19:25:011
2397063991710,5cyclictest2465651-21sshd22:23:291
2397047991715,1cyclictest0-21swapper/023:44:380
2397047991714,2cyclictest0-21swapper/022:45:570
2397047991712,3cyclictest2465786-21sh22:25:000
2397047991711,5cyclictest2440155-21entropy21:10:140
239704799171,12cyclictest2471518-21apt22:40:100
24776922160,2sleep13056-21PK-Backend22:55:181
24776922160,2sleep13056-21PK-Backend22:55:181
239707299169,5cyclictest2473912-21grep22:45:172
239707299162,12cyclictest2508075-21vmstat00:20:232
2397072991614,1cyclictest0-21swapper/222:44:432
2397072991611,3cyclictest2506324-21uptime00:15:222
239706399169,5cyclictest2494324-21basename23:45:001
239706399168,6cyclictest2412041-21expr19:50:121
2397063991614,1cyclictest3056-21PK-Backend20:49:401
2397047991610,5cyclictest2481987-21systemd23:10:000
2397072991510,3cyclictest2511461-21timerwakeupswit00:30:202
2397063991512,2cyclictest3056-21PK-Backend00:04:011
239704799159,5cyclictest2502279-21cut00:05:180
239704799159,5cyclictest2486041-21grep23:20:120
2397047991510,3cyclictest1095-21runrttasks00:10:070
239706399148,5cyclictest2459365-21cat22:05:131
2397063991412,1cyclictest3056-21PK-Backend00:12:261
239704799142,11cyclictest2479437-21grep23:00:180
239707799130,1cyclictest3056-21PK-Backend22:30:153
239707299136,5cyclictest2461885-21wc22:10:222
2397063991310,2cyclictest3056-21PK-Backend23:00:271
239704799139,3cyclictest3056-21PK-Backend00:25:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional