You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-02 - 05:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Sep 02, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
246822221270,2sleep20-21swapper/222:55:022
240191921080,3sleep00-21swapper/019:45:150
139099840,70rtkit-daemon0-21swapper/019:07:110
238786327144,19sleep30-21swapper/319:05:253
238837026939,24sleep10-21swapper/119:09:041
238809526240,14sleep20-21swapper/219:05:322
24247612540,3sleep00-21swapper/020:50:160
24899252520,1chrt2489360-21/usr/sbin/munin23:55:172
24899252520,1chrt2489360-21/usr/sbin/munin23:55:162
24477232520,4sleep10-21swapper/121:55:181
24577232430,3sleep13056-21PK-Backend22:25:131
24367042350,7sleep2341ksoftirqd/221:25:102
238858599309,6cyclictest2504350-21awk00:35:231
2388585993015,13cyclictest26-21ksoftirqd/121:10:091
24285832290,4sleep33056-21PK-Backend21:00:223
2388585992912,6cyclictest2461915-21cat22:35:161
2388585992910,5cyclictest2500183-21df00:25:141
238858599289,5cyclictest2451080-21df22:05:131
238858599288,5cyclictest2395435-21cat19:25:221
238858599279,4cyclictest2418586-21wc20:35:001
238858599278,5cyclictest2459648-21cat22:30:141
238858599278,5cyclictest2416813-21wc20:30:011
238858599276,19cyclictest2409830-21unin-run20:09:591
238858599270,1cyclictest0-21swapper/119:40:011
238858599268,3cyclictest2419641-21switchtime20:35:201
238858599267,5cyclictest2421828-21seq20:43:501
238858599262,12cyclictest2490337-21grep23:55:221
238858599262,12cyclictest2490337-21grep23:55:221
2388585992610,5cyclictest2495161-21cat00:10:171
238858599259,5cyclictest2404613-21awk19:55:011
238858599257,4cyclictest2503038-21sh00:35:001
238858599257,4cyclictest0-21swapper/100:09:591
238858599256,5cyclictest2410251-21expr20:10:141
238858599241,10cyclictest2412839-21cat20:15:231
23932432230,4sleep33056-21PK-Backend19:20:163
2388594992321,1cyclictest0-21swapper/322:51:133
238858599231,9cyclictest2465224-21diskstats22:45:151
2388585992311,3cyclictest2392715-21/usr/sbin/munin19:20:231
2388585992310,3cyclictest1095-21runrttasks00:15:051
2388585992310,2cyclictest2398306-21df_inode19:35:121
2388583992314,6cyclictest2444689-21sshd21:46:260
2388583992311,9cyclictest883-21dbus-daemon21:55:000
24694082220,3sleep00-21swapper/022:55:270
2388594992220,1cyclictest0-21swapper/323:56:563
2388594992220,1cyclictest0-21swapper/323:56:563
2388594992220,1cyclictest0-21swapper/323:35:473
2388594992220,1cyclictest0-21swapper/323:22:033
2388594992220,1cyclictest0-21swapper/320:46:413
2388594992218,3cyclictest2487379-21sh23:50:013
238859499220,21cyclictest3056-21PK-Backend21:15:003
238859499220,21cyclictest2401406-21apt19:45:103
238859499220,20cyclictest3056-21PK-Backend22:06:053
2388589992220,1cyclictest0-21swapper/223:39:202
2388589992220,1cyclictest0-21swapper/222:47:232
2388589992220,1cyclictest0-21swapper/222:31:062
2388589992220,1cyclictest0-21swapper/222:14:512
2388589992220,1cyclictest0-21swapper/221:32:252
2388589992220,1cyclictest0-21swapper/219:27:002
2388589992220,1cyclictest0-21swapper/200:24:082
2388589992220,1cyclictest0-21swapper/200:14:092
238858999220,21cyclictest0-21swapper/220:31:082
238858599229,3cyclictest3056-21PK-Backend20:02:221
238858599225,5cyclictest2436586-21/usr/sbin/munin21:25:101
238858599221,9cyclictest2428035-21diskstats21:00:141
238858399222,17cyclictest2397675-21latency_hist19:35:000
2388583992220,1cyclictest0-21swapper/023:31:110
2388583992220,1cyclictest0-21swapper/022:34:340
2388583992220,1cyclictest0-21swapper/022:28:260
2388583992220,1cyclictest0-21swapper/021:17:070
2388583992220,1cyclictest0-21swapper/021:00:450
2388583992220,1cyclictest0-21swapper/020:08:060
2388583992220,1cyclictest0-21swapper/019:26:290
2388583992217,3cyclictest2413283-21idleruntime-cro20:19:590
2388583992214,6cyclictest883-21dbus-daemon20:14:590
2388583992213,7cyclictest883-21dbus-daemon23:59:590
2388583992213,7cyclictest883-21dbus-daemon00:00:000
2388583992211,7cyclictest436-20systemd-journal20:25:000
238858399220,7cyclictest2438058-21perf21:30:000
139099220,16rtkit-daemon0-21swapper/020:33:350
2388594992119,1cyclictest3056-21PK-Backend22:32:473
2388594992119,1cyclictest3056-21PK-Backend20:20:323
2388594992119,1cyclictest3056-21PK-Backend20:06:133
2388594992119,1cyclictest3056-21PK-Backend00:30:443
2388594992119,1cyclictest0-21swapper/322:19:443
2388594992118,2cyclictest3056-21PK-Backend22:02:203
2388594992118,2cyclictest3056-21PK-Backend20:12:493
2388594992118,2cyclictest3056-21PK-Backend19:12:063
2388594992118,2cyclictest3056-21PK-Backend19:12:063
2388594992115,5cyclictest883-21dbus-daemon23:18:043
238859499210,20cyclictest3056-21PK-Backend21:05:353
238859499210,19cyclictest3056-21PK-Backend20:28:173
238859499210,19cyclictest3056-21PK-Backend19:17:543
2388589992119,1cyclictest0-21swapper/223:45:092
2388589992119,1cyclictest0-21swapper/222:29:312
2388589992119,1cyclictest0-21swapper/222:16:272
2388589992119,1cyclictest0-21swapper/221:48:492
2388589992119,1cyclictest0-21swapper/221:08:132
2388589992119,1cyclictest0-21swapper/220:16:152
2388589992119,1cyclictest0-21swapper/220:01:132
2388589992119,1cyclictest0-21swapper/219:58:372
2388589992111,8cyclictest2393247-21kernelversion19:20:172
238858999210,20cyclictest0-21swapper/223:31:102
238858999210,20cyclictest0-21swapper/223:19:472
238858999210,20cyclictest0-21swapper/219:44:112
238858599211,9cyclictest2439187-21python321:30:221
2388585992115,5cyclictest3056-21PK-Backend23:35:591
2388583992119,1cyclictest0-21swapper/023:12:060
2388583992119,1cyclictest0-21swapper/022:50:490
2388583992119,1cyclictest0-21swapper/021:39:390
2388583992119,1cyclictest0-21swapper/021:06:540
2388583992119,1cyclictest0-21swapper/020:55:590
2388583992119,1cyclictest0-21swapper/020:48:480
2388583992119,1cyclictest0-21swapper/020:40:160
2388583992119,1cyclictest0-21swapper/019:22:290
2388583992119,1cyclictest0-21swapper/000:37:300
2388583992119,1cyclictest0-21swapper/000:23:310
2388583992119,1cyclictest0-21swapper/000:10:180
2388583992111,7cyclictest436-20systemd-journal20:30:000
238858399210,20cyclictest0-21swapper/023:08:550
238858399210,20cyclictest0-21swapper/022:48:270
238858399210,20cyclictest0-21swapper/021:40:260
238858399210,20cyclictest0-21swapper/020:37:540
238858399210,20cyclictest0-21swapper/019:11:560
238858399210,20cyclictest0-21swapper/019:11:550
238858399210,20cyclictest0-21swapper/000:05:180
238858399210,1cyclictest3056-21PK-Backend23:45:460
238858399210,19cyclictest3056-21PK-Backend19:40:150
139099210,14rtkit-daemon0-21swapper/022:11:020
24989612200,2sleep10-21swapper/100:20:221
2388594992018,1cyclictest3056-21PK-Backend22:10:383
2388594992018,1cyclictest3056-21PK-Backend20:38:583
2388594992018,1cyclictest3056-21PK-Backend00:24:073
2388594992018,1cyclictest0-21swapper/323:29:113
2388594992018,1cyclictest0-21swapper/322:48:503
2388594992018,1cyclictest0-21swapper/322:27:273
2388594992018,1cyclictest0-21swapper/322:21:183
2388594992018,1cyclictest0-21swapper/300:04:213
2388594992018,1cyclictest0-21swapper/300:04:213
2388594992015,3cyclictest2396208-21apt19:30:123
238859499200,1cyclictest3056-21PK-Backend00:15:563
238859499200,19cyclictest3056-21PK-Backend21:58:343
238859499200,19cyclictest3056-21PK-Backend21:50:293
238859499200,19cyclictest0-21swapper/319:55:123
238859499200,18cyclictest3056-21PK-Backend21:29:003
238859499200,18cyclictest3056-21PK-Backend21:20:073
2388589992018,1cyclictest0-21swapper/219:36:482
2388589992018,1cyclictest0-21swapper/219:19:512
2388589992013,5cyclictest2420855-21forks20:40:162
2388589992012,6cyclictest2446577-21sshd21:53:272
238858999200,1cyclictest0-21swapper/223:23:462
238858999200,1cyclictest0-21swapper/220:54:552
238858999200,19cyclictest0-21swapper/222:44:112
238858999200,19cyclictest0-21swapper/222:38:022
238858999200,19cyclictest0-21swapper/221:39:572
238858999200,19cyclictest0-21swapper/221:11:112
238858999200,19cyclictest0-21swapper/219:48:232
238858999200,19cyclictest0-21swapper/200:19:572
2388585992019,1cyclictest0-21swapper/121:08:491
2388585992018,1cyclictest3056-21PK-Backend23:41:391
2388585992018,1cyclictest3056-21PK-Backend00:04:241
2388585992018,1cyclictest3056-21PK-Backend00:04:241
2388585992017,2cyclictest3056-21PK-Backend23:26:471
2388585992013,5cyclictest2463183-21cat22:40:111
238858599200,19cyclictest3056-21PK-Backend23:20:301
238858599200,19cyclictest0-21swapper/121:50:141
238858599200,19cyclictest0-21swapper/121:44:451
238858399209,8cyclictest883-21dbus-daemon22:06:340
2388583992018,1cyclictest0-21swapper/022:36:100
2388583992012,6cyclictest2500536-21ntpq00:25:190
2388583992012,6cyclictest2404209-21awk19:50:220
238858399200,1cyclictest0-21swapper/022:19:530
238858399200,1cyclictest0-21swapper/021:59:330
238858399200,1cyclictest0-21swapper/000:30:070
238858399200,19cyclictest0-21swapper/022:42:180
238858399200,19cyclictest0-21swapper/021:31:210
238858399200,19cyclictest0-21swapper/021:23:020
238858399200,19cyclictest0-21swapper/020:02:050
238858399200,19cyclictest0-21swapper/019:18:310
139099200,14rtkit-daemon0-21swapper/023:15:290
139099200,14rtkit-daemon0-21swapper/019:39:210
139099200,13rtkit-daemon0-21swapper/023:29:170
238859499192,15cyclictest2398474-21cat19:35:173
2388594991917,1cyclictest3056-21PK-Backend21:37:193
2388594991917,1cyclictest0-21swapper/323:00:463
2388594991917,1cyclictest0-21swapper/322:55:593
2388594991917,1cyclictest0-21swapper/320:02:493
2388594991916,2cyclictest3056-21PK-Backend21:43:133
238859499191,16cyclictest2426794-21python320:55:213
238859499190,1cyclictest3056-21PK-Backend21:49:423
238859499190,1cyclictest3056-21PK-Backend21:32:463
238859499190,18cyclictest3056-21PK-Backend23:14:533
238859499190,18cyclictest3056-21PK-Backend22:37:213
2388589991917,1cyclictest0-21swapper/200:39:572
2388589991917,1cyclictest0-21swapper/200:34:092
2388589991917,1cyclictest0-21swapper/200:00:432
2388589991917,1cyclictest0-21swapper/200:00:432
2388589991915,3cyclictest978-21snmpd22:56:402
2388589991914,3cyclictest2419452-21grep20:35:182
2388589991912,5cyclictest1095-21runrttasks23:50:122
2388589991911,7cyclictest2456667-21ntpq22:20:202
2388589991911,6cyclictest2472643-21python323:05:202
238858999190,1cyclictest0-21swapper/221:40:452
238858999190,18cyclictest0-21swapper/220:12:162
238858599191,8cyclictest2433724-21python321:15:251
2388585991917,1cyclictest0-21swapper/122:15:141
2388585991914,4cyclictest26-21ksoftirqd/123:10:201
238858599190,18cyclictest0-21swapper/123:15:371
238858399193,14cyclictest2478449-21cron23:25:000
2388583991917,1cyclictest0-21swapper/021:14:100
2388583991917,1cyclictest0-21swapper/019:58:400
2388583991917,1cyclictest0-21swapper/000:16:540
238858399190,1cyclictest0-21swapper/022:03:180
238858399190,18cyclictest0-21swapper/022:21:290
139099190,12rtkit-daemon0-21swapper/123:46:101
2388594991816,1cyclictest3056-21PK-Backend20:40:343
2388594991813,4cyclictest2395800-21sshd19:28:283
2388594991812,4cyclictest2484339-21tr23:40:133
2388594991811,5cyclictest42-21ksoftirqd/320:50:103
238859499180,17cyclictest2481333-21wc23:30:173
2388589991816,1cyclictest0-21swapper/221:58:282
2388589991816,1cyclictest0-21swapper/221:21:252
2388589991816,1cyclictest0-21swapper/221:16:182
2388589991816,1cyclictest0-21swapper/219:50:132
2388589991816,1cyclictest0-21swapper/200:28:212
2388589991814,3cyclictest2413845-21ls20:20:142
2388589991813,3cyclictest2415413-21cpuspeed_turbos20:25:132
2388589991810,6cyclictest2397703-21unin-run19:35:002
238858999180,1cyclictest0-21swapper/223:14:122
238858999180,1cyclictest0-21swapper/219:11:062
238858999180,1cyclictest0-21swapper/219:11:062
2388585991816,1cyclictest3056-21PK-Backend23:54:241
2388585991816,1cyclictest0-21swapper/120:23:351
2388585991814,3cyclictest3056-21PK-Backend22:56:521
2388585991813,3cyclictest2441523-21latency_hist21:40:001
2388585991811,5cyclictest2402924-21sed19:50:011
2388583991816,1cyclictest0-21swapper/023:40:110
2388594991715,1cyclictest3056-21PK-Backend22:43:303
2388594991715,1cyclictest3056-21PK-Backend20:19:443
2388594991714,2cyclictest3056-21PK-Backend00:08:333
2388594991712,3cyclictest2472640-21python323:05:213
238859499171,12cyclictest2399673-21apt19:40:113
2388589991715,1cyclictest0-21swapper/223:00:062
2388589991715,1cyclictest0-21swapper/222:00:032
2388589991715,1cyclictest0-21swapper/220:07:162
2388589991715,1cyclictest0-21swapper/200:09:092
238858599171,5cyclictest2453318-21python322:10:221
2388585991713,3cyclictest2473254-21turbostat.cron23:10:001
2388585991710,5cyclictest2444975-21cron21:50:011
2388585991710,5cyclictest2392501-21pluginstate19:20:011
238858599170,1cyclictest3056-21PK-Backend22:51:591
2388583991710,5cyclictest2487633-21find23:50:110
238859499169,5cyclictest2487944-21cstates23:50:123
2388594991614,1cyclictest3056-21PK-Backend21:12:523
238859499160,14cyclictest24042972sleep319:51:163
238858999169,6cyclictest2484783-21cat23:40:172
238858999162,12cyclictest2426100-21grep20:55:122
238858599169,6cyclictest2424368-21tr20:50:121
238858599168,6cyclictest2388970-21expr19:10:131
238858599168,6cyclictest2388970-21expr19:10:131
2388585991614,1cyclictest3056-21PK-Backend23:33:501
2388585991614,1cyclictest3056-21PK-Backend19:57:121
2388585991611,3cyclictest1095-21runrttasks20:58:441
238858399169,5cyclictest2482898-21cat23:35:160
139099160,8rtkit-daemon2449074-21cpuspeed_turbos22:00:131
2388594991513,1cyclictest3056-21PK-Backend20:30:403
238858999159,4cyclictest2423056-21ls20:45:192
238858999158,5cyclictest2479655-21cat23:25:212
238858999151,12cyclictest2428523-21python321:00:202
2388585991513,1cyclictest3056-21PK-Backend22:22:041
238858599150,13cyclictest2422398-21gdbus20:45:021
2388594991412,1cyclictest3056-21PK-Backend00:29:563
2388594991410,3cyclictest3056-21PK-Backend00:12:133
2388589991412,1cyclictest0-21swapper/222:05:582
238858599148,5cyclictest2435015-21grep21:20:121
238858599140,1cyclictest3056-21PK-Backend19:34:551
238858399148,4cyclictest2470245-21seq23:00:100
238858399140,13cyclictest2490868-21perf00:04:550
238858399140,13cyclictest2490868-21perf00:04:550
238859499130,3cyclictest978-21snmpd00:36:243
238858599127,3cyclictest24713292sleep123:02:521
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional