You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-27 - 03:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Jul 27, 2026 00:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
254826328147,23sleep10-21swapper/119:05:121
254914426440,16sleep30-21swapper/319:05:263
254953726242,14sleep20-21swapper/219:07:342
254950326243,13sleep00-21swapper/019:07:080
25919732470,3sleep20-21swapper/221:05:232
26509632450,2sleep20-21swapper/223:55:132
25941812440,4sleep13056-21PK-Backend21:15:001
26524132390,3sleep13056-21PK-Backend00:00:011
26638782340,3sleep30-21swapper/300:30:223
254985499321,29cyclictest2639633-21nm-dispatcher23:23:370
2549859993113,16cyclictest2625363-21cron22:45:001
26154372290,9chrt42-21ksoftirqd/322:15:133
254986599262,20cyclictest2583764-21apt20:45:103
2549865992621,3cyclictest436-20systemd-journal19:11:353
254986599261,20cyclictest2571366-21apt20:10:133
254985999269,15cyclictest2654943-21awk00:05:211
2549859992612,12cyclictest2657396-21latency_hist00:15:001
2549859992610,14cyclictest2646104-21missed_timers23:40:191
254985999260,19cyclictest3056-21PK-Backend20:15:211
26346942250,2chrt0-21swapper/123:10:011
254986599253,8cyclictest2565627-21taskset19:51:493
254986599253,14cyclictest0-21swapper/320:21:323
2549863992518,5cyclictest34-21ksoftirqd/223:23:372
254985999252,21cyclictest2561160-21wc19:40:131
2549859992519,5cyclictest3056-21PK-Backend19:59:531
254986599242,14cyclictest0-21swapper/322:37:543
254986599241,21cyclictest2552670-21diskstats19:15:153
2549863992410,12cyclictest34-21ksoftirqd/221:00:002
2549859992415,5cyclictest25857212sleep120:50:031
254985999240,9cyclictest3056-21PK-Backend00:30:011
254985999240,1cyclictest3056-21PK-Backend23:20:071
254986599235,15cyclictest2580813-21taskset20:35:183
254986599232,6cyclictest2587492-21apt20:55:123
2549865992315,4cyclictest2636552-21apt23:15:113
254985999236,3cyclictest2580043-21munin-run20:35:001
254985999235,4cyclictest2556472-21ntp_kernel_pll_19:25:181
2549859992320,2cyclictest3056-21PK-Backend22:30:061
2549859992319,3cyclictest3056-21PK-Backend21:17:331
254985999230,21cyclictest3056-21PK-Backend19:21:271
2549854992320,2cyclictest3056-21PK-Backend00:10:420
139099230,20rtkit-daemon2637-21gmain23:16:170
25817912220,4chrt3056-21PK-Backend20:40:001
254986599222,9cyclictest0-21swapper/323:28:513
2549865992214,6cyclictest2646207-21ntp_kernel_pll_23:40:203
2549865992214,6cyclictest2557212-21cron19:30:003
2549863992214,6cyclictest2646356-21python323:40:212
254985999227,4cyclictest2552972-21needreboot19:15:191
254985999227,2cyclictest25762702chrt20:21:311
254985999226,2cyclictest25965572sleep121:20:181
2549859992220,1cyclictest3056-21PK-Backend23:56:311
2549859992219,2cyclictest3056-21PK-Backend23:30:491
2549859992217,4cyclictest3056-21PK-Backend22:01:541
254985999220,7cyclictest2601357-21chrt21:35:131
254985999220,20cyclictest3056-21PK-Backend21:50:181
2549854992220,1cyclictest0-21swapper/023:58:370
2549854992220,1cyclictest0-21swapper/022:12:120
2549854992220,1cyclictest0-21swapper/021:55:430
2549854992219,2cyclictest0-21swapper/020:51:210
2549854992218,3cyclictest0-21swapper/021:30:010
254985499220,21cyclictest0-21swapper/023:09:240
254985499220,21cyclictest0-21swapper/022:48:080
254985499220,21cyclictest0-21swapper/019:23:270
139099220,4rtkit-daemon1389-21rtkit-daemon23:26:180
139099220,4rtkit-daemon1389-21rtkit-daemon20:03:300
139099220,19rtkit-daemon852-21rpcbind19:55:040
139099220,19rtkit-daemon2162-21gmain20:08:120
254986599217,12cyclictest42-21ksoftirqd/320:28:353
254986599216,14cyclictest0-21swapper/321:31:063
254986599215,14cyclictest2598797-21switchtime21:25:233
254986599213,5cyclictest0-21swapper/319:30:363
254986599212,16cyclictest2156-21gnome-shell21:02:093
2549865992118,2cyclictest0-21swapper/323:30:133
2549865992116,3cyclictest2629934-21memory22:55:183
2549865992115,4cyclictest2585760-21apt20:50:123
254986599210,20cyclictest0-21swapper/323:23:303
2549863992117,3cyclictest1095-21runrttasks20:37:182
254986399210,20cyclictest953-21gmain20:03:092
254986399210,20cyclictest0-21swapper/223:15:402
254986399210,1cyclictest2626298-21strings22:45:182
254985999218,11cyclictest26044232chrt21:43:231
254985999217,3cyclictest2615395-21tr22:15:131
254985999216,3cyclictest2614964-21latency_hist22:15:011
2549859992119,1cyclictest3056-21PK-Backend23:12:041
2549859992119,1cyclictest3056-21PK-Backend22:56:081
2549859992119,1cyclictest3056-21PK-Backend22:23:171
2549859992119,1cyclictest3056-21PK-Backend21:31:121
2549859992119,1cyclictest3056-21PK-Backend20:01:491
2549859992119,1cyclictest3056-21PK-Backend19:54:191
2549859992119,1cyclictest3056-21PK-Backend19:30:121
2549859992119,1cyclictest3056-21PK-Backend00:37:481
2549859992118,2cyclictest3056-21PK-Backend23:35:221
254985999211,5cyclictest3056-21PK-Backend20:05:131
254985999211,4cyclictest2650434-21munin-run23:55:001
254985999210,6cyclictest2582935-21sed20:40:221
254985999210,20cyclictest3056-21PK-Backend21:07:331
254985999210,19cyclictest3056-21PK-Backend23:04:471
254985999210,19cyclictest3056-21PK-Backend22:26:001
254985999210,19cyclictest3056-21PK-Backend20:29:401
254985999210,19cyclictest3056-21PK-Backend20:12:031
2549854992119,1cyclictest0-21swapper/023:54:050
2549854992119,1cyclictest0-21swapper/023:35:060
2549854992119,1cyclictest0-21swapper/023:33:290
2549854992119,1cyclictest0-21swapper/023:12:210
2549854992119,1cyclictest0-21swapper/022:54:030
2549854992119,1cyclictest0-21swapper/022:19:010
2549854992119,1cyclictest0-21swapper/021:46:370
2549854992119,1cyclictest0-21swapper/021:14:260
2549854992119,1cyclictest0-21swapper/019:54:230
2549854992119,1cyclictest0-21swapper/019:41:250
2549854992119,1cyclictest0-21swapper/019:26:100
2549854992119,1cyclictest0-21swapper/000:20:350
2549854992116,3cyclictest2619366-21grep22:25:190
254985499210,20cyclictest0-21swapper/023:04:160
254985499210,20cyclictest0-21swapper/021:07:300
254985499210,20cyclictest0-21swapper/000:37:110
254985499210,19cyclictest0-21swapper/000:15:140
139099210,4rtkit-daemon1389-21rtkit-daemon00:26:570
254986599206,12cyclictest2612136-21diskstats22:05:123
254986599203,4cyclictest2562772-21cat19:45:123
254986599202,10cyclictest0-21swapper/320:03:043
2549865992018,1cyclictest0-21swapper/321:22:513
254986599201,7cyclictest2650965-21cstates23:55:133
2549865992012,6cyclictest2620823-21awk22:30:173
2549865992012,4cyclictest2657660-21apt00:15:133
2549865992011,7cyclictest2659395-21apt00:20:123
254986599200,8cyclictest0-21swapper/320:30:153
254986599200,1cyclictest0-21swapper/323:00:253
254986599200,19cyclictest42-21ksoftirqd/319:40:373
254986599200,19cyclictest0-21swapper/323:45:543
254986599200,19cyclictest0-21swapper/323:13:363
254986599200,19cyclictest0-21swapper/322:53:423
254986599200,19cyclictest0-21swapper/321:41:233
254986599200,19cyclictest0-21swapper/319:39:493
254986599200,19cyclictest0-21swapper/300:09:263
2549863992018,1cyclictest0-21swapper/222:12:062
2549863992018,1cyclictest0-21swapper/221:45:432
2549863992018,1cyclictest0-21swapper/220:34:522
2549863992018,1cyclictest0-21swapper/219:34:362
2549863992018,1cyclictest0-21swapper/219:28:222
2549863992018,1cyclictest0-21swapper/200:00:152
2549863992016,2cyclictest2642511-21ntp_states23:30:202
2549863992013,5cyclictest34-21ksoftirqd/222:35:142
2549863992013,5cyclictest2583100-21uptime20:40:232
2549863992012,6cyclictest436-20systemd-journal21:56:072
2549863992012,6cyclictest2632831-21/usr/sbin/munin23:05:012
254986399200,1cyclictest0-21swapper/220:17:422
254986399200,19cyclictest3056-21PK-Backend19:49:162
254986399200,19cyclictest0-21swapper/223:35:412
254986399200,19cyclictest0-21swapper/223:14:522
254986399200,19cyclictest0-21swapper/222:42:212
254986399200,19cyclictest0-21swapper/222:21:052
254986399200,19cyclictest0-21swapper/221:36:372
254986399200,19cyclictest0-21swapper/220:46:562
254986399200,19cyclictest0-21swapper/219:41:052
254986399200,19cyclictest0-21swapper/219:22:542
254986399200,19cyclictest0-21swapper/219:16:182
254986399200,19cyclictest0-21swapper/200:34:422
254986399200,19cyclictest0-21swapper/200:29:212
254985999207,3cyclictest3056-21PK-Backend20:57:001
2549859992018,1cyclictest3056-21PK-Backend23:26:171
2549859992018,1cyclictest3056-21PK-Backend23:16:231
2549859992018,1cyclictest3056-21PK-Backend22:48:041
2549859992018,1cyclictest3056-21PK-Backend22:05:461
2549859992018,1cyclictest3056-21PK-Backend21:45:521
2549859992018,1cyclictest3056-21PK-Backend20:45:291
2549859992018,1cyclictest3056-21PK-Backend19:39:111
2549859992018,1cyclictest3056-21PK-Backend00:34:031
254985999201,3cyclictest2598751-21python321:25:221
254985999200,6cyclictest2564233-21wc19:50:001
254985999200,6cyclictest2551177-21chrt19:10:381
254985999200,18cyclictest3056-21PK-Backend22:38:511
254985999200,18cyclictest3056-21PK-Backend00:22:001
2549854992018,1cyclictest0-21swapper/021:16:560
2549854992018,1cyclictest0-21swapper/020:48:170
2549854992018,1cyclictest0-21swapper/019:47:410
2549854992018,1cyclictest0-21swapper/000:33:260
2549854992011,7cyclictest2612833-21users22:05:220
254985499200,1cyclictest0-21swapper/023:48:450
254985499200,1cyclictest0-21swapper/021:36:570
254985499200,1cyclictest0-21swapper/020:10:590
254985499200,19cyclictest3056-21PK-Backend23:40:270
254985499200,19cyclictest3056-21PK-Backend19:11:170
254985499200,19cyclictest0-21swapper/021:54:210
254985499200,19cyclictest0-21swapper/021:23:040
254985499200,19cyclictest0-21swapper/019:37:050
139099200,17rtkit-daemon931-21wpa_supplicant00:03:020
139099200,16rtkit-daemon2549849-21cyclictest22:58:010
254986599199,8cyclictest2554689-21needreboot19:20:163
254986599198,8cyclictest2594429-21chrt21:15:143
2549865991917,1cyclictest0-21swapper/322:47:483
2549865991917,1cyclictest0-21swapper/321:57:393
2549865991917,1cyclictest0-21swapper/321:54:213
2549865991917,1cyclictest0-21swapper/321:35:343
2549865991917,1cyclictest0-21swapper/319:55:443
2549865991917,1cyclictest0-21swapper/300:35:363
254986599191,6cyclictest2574428-21users20:15:233
254986599191,4cyclictest2605698-21python321:45:223
2549865991914,3cyclictest2644364-21python323:35:233
2549865991912,4cyclictest2614360-21python322:10:213
2549865991911,6cyclictest2656330-21idleruntime00:10:193
254986599191,16cyclictest2633955-21python323:05:223
254986599190,1cyclictest0-21swapper/323:51:153
254986599190,1cyclictest0-21swapper/322:26:043
254986599190,1cyclictest0-21swapper/320:44:313
254986599190,18cyclictest0-21swapper/300:29:273
254986599190,18cyclictest0-21swapper/300:01:553
2549863991917,1cyclictest0-21swapper/223:07:362
2549863991917,1cyclictest0-21swapper/222:30:042
2549863991917,1cyclictest0-21swapper/222:03:212
2549863991917,1cyclictest0-21swapper/221:52:532
2549863991917,1cyclictest0-21swapper/221:32:452
2549863991917,1cyclictest0-21swapper/221:23:532
2549863991917,1cyclictest0-21swapper/221:16:232
2549863991917,1cyclictest0-21swapper/200:22:252
2549863991917,1cyclictest0-21swapper/200:14:412
2549863991913,5cyclictest2604272-21seq21:41:262
2549863991912,5cyclictest2598031-21grep21:25:132
2549863991911,6cyclictest2616191-21sed22:15:222
254986399190,1cyclictest0-21swapper/222:52:012
254986399190,1cyclictest0-21swapper/222:08:352
254986399190,1cyclictest0-21swapper/200:16:172
254986399190,18cyclictest0-21swapper/221:14:272
254986399190,18cyclictest0-21swapper/219:51:472
254986399190,18cyclictest0-21swapper/219:12:262
2549859991917,1cyclictest3056-21PK-Backend23:45:151
254985999190,1cyclictest3056-21PK-Backend21:58:361
2549854991917,1cyclictest3056-21PK-Backend21:32:300
2549854991917,1cyclictest0-21swapper/022:40:170
2549854991917,1cyclictest0-21swapper/021:02:300
2549854991917,1cyclictest0-21swapper/020:20:180
2549854991917,1cyclictest0-21swapper/020:15:380
2549854991917,1cyclictest0-21swapper/019:18:130
2549854991917,1cyclictest0-21swapper/000:06:560
254985499190,1cyclictest3056-21PK-Backend22:24:290
254985499190,1cyclictest3056-21PK-Backend20:40:260
254985499190,1cyclictest0-21swapper/021:40:490
139099190,17rtkit-daemon945-21thermald20:55:520
2549865991812,4cyclictest2571080-21cron20:10:003
2549865991811,5cyclictest2591261-21expr21:05:143
254986599180,15cyclictest2593065-21diskstats21:10:153
2549863991816,1cyclictest0-21swapper/220:13:022
2549863991816,1cyclictest0-21swapper/219:58:432
2549863991816,1cyclictest0-21swapper/200:05:352
2549863991815,2cyclictest0-21swapper/221:05:012
2549863991812,5cyclictest2640159-21expr23:25:142
2549863991811,5cyclictest436-20systemd-journal20:25:002
2549863991811,5cyclictest2664693-21cat00:35:112
254986399180,1cyclictest0-21swapper/220:53:382
2549859991816,1cyclictest3056-21PK-Backend21:01:581
2549859991816,1cyclictest3056-21PK-Backend00:15:161
2549854991816,1cyclictest0-21swapper/022:00:570
2549854991816,1cyclictest0-21swapper/019:30:160
254986599171,14cyclictest2623899-21apt22:40:123
2549865991710,3cyclictest2610034-21apt22:00:123
254986599170,15cyclictest2617331-21df_inode22:20:143
254986399178,7cyclictest2636-21xdg-desktop-por20:28:352
2549863991715,1cyclictest0-21swapper/219:35:242
2549863991712,3cyclictest2629829-21interrupts22:55:172
2549863991710,6cyclictest2648571-21gdbus23:48:382
2549863991710,5cyclictest883-21dbus-daemon22:30:012
254986399170,16cyclictest2649928-21sed23:50:212
2549854991715,1cyclictest0-21swapper/022:36:120
2549854991712,3cyclictest2578745-21cpuspeed_turbos20:30:120
2549863991610,4cyclictest2571242-21sed20:10:012
254985499161,13cyclictest2581762-21turbostat.cron20:40:000
2549854991610,5cyclictest2621208-21which22:30:220
254985999140,5cyclictest2628854-21chrt22:52:111
2549854991411,2cyclictest2576797-21/usr/sbin/munin20:25:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional