You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 10:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Thu Mar 26, 2026 00:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8941621250,3sleep23056-21PK-Backend23:55:162
139099950,60rtkit-daemon0-21swapper/219:07:492
417991926944,16sleep00-21swapper/019:05:110
418118226742,18sleep30-21swapper/319:07:393
418129526343,14sleep10-21swapper/119:08:581
628062530,3sleep03056-21PK-Backend22:40:240
954952480,3sleep13056-21PK-Backend00:15:001
812872480,10sleep234-21ksoftirqd/223:34:552
639062430,4sleep03056-21PK-Backend22:45:130
41875212410,4sleep33056-21PK-Backend19:25:113
306912390,4sleep20-21swapper/221:10:122
908812360,3sleep00-21swapper/000:00:110
139099280,4rtkit-daemon1389-21rtkit-daemon20:54:473
4181511992714,11cyclictest4185027-21sshd19:15:230
139099270,4rtkit-daemon1389-21rtkit-daemon22:38:453
139099260,4rtkit-daemon1389-21rtkit-daemon21:24:012
139099260,4rtkit-daemon1389-21rtkit-daemon00:28:362
418151599240,1cyclictest0-21swapper/100:39:441
418151599240,19cyclictest3056-21PK-Backend21:03:471
418151599240,19cyclictest3056-21PK-Backend21:03:471
418151599240,18cyclictest3056-21PK-Backend22:38:371
139099240,4rtkit-daemon1389-21rtkit-daemon21:36:412
4181515992318,3cyclictest11580-21expr20:15:141
418151599230,21cyclictest3056-21PK-Backend22:51:011
4181511992314,7cyclictest12430-21sshd20:15:230
4181521992220,1cyclictest3056-21PK-Backend23:10:383
4181521992220,1cyclictest0-21swapper/300:14:533
4181521992219,2cyclictest3056-21PK-Backend19:40:223
418152199220,21cyclictest0-21swapper/300:03:033
418152199220,20cyclictest53374-21hddtemp_smartct22:15:153
4181519992220,1cyclictest3056-21PK-Backend20:15:282
4181519992220,1cyclictest0-21swapper/222:42:442
4181519992220,1cyclictest0-21swapper/222:21:082
4181519992220,1cyclictest0-21swapper/222:00:402
4181519992220,1cyclictest0-21swapper/221:41:272
4181519992220,1cyclictest0-21swapper/200:08:562
4181519992217,3cyclictest20670-21sed20:40:172
4181519992216,5cyclictest95506-21systemd-run00:15:002
4181519992215,4cyclictest52714-21munin-plugin-st22:15:002
4181519992210,8cyclictest436-20systemd-journal21:59:592
418151999220,21cyclictest0-21swapper/220:51:592
4181515992216,1cyclictest3056-21PK-Backend20:43:461
4181515992214,6cyclictest14192-21sshd20:20:231
418151599220,20cyclictest3056-21PK-Backend20:48:131
4181511992215,4cyclictest74105-21awk23:15:000
4181511992211,10cyclictest90266-21grep00:00:000
139099220,4rtkit-daemon1389-21rtkit-daemon20:46:132
960182210,3sleep20-21swapper/200:15:132
4181521992119,1cyclictest3056-21PK-Backend22:49:493
4181521992119,1cyclictest3056-21PK-Backend21:42:433
4181521992119,1cyclictest3056-21PK-Backend20:13:003
4181521992119,1cyclictest3056-21PK-Backend19:20:413
4181521992119,1cyclictest0-21swapper/323:18:353
4181521992119,1cyclictest0-21swapper/323:01:403
4181521992119,1cyclictest0-21swapper/322:09:473
4181521992119,1cyclictest0-21swapper/322:04:203
4181521992119,1cyclictest0-21swapper/321:36:283
4181521992119,1cyclictest0-21swapper/320:56:073
4181521992119,1cyclictest0-21swapper/319:11:493
4181521992119,1cyclictest0-21swapper/300:05:063
4181521992118,2cyclictest3056-21PK-Backend22:44:083
4181521992118,2cyclictest3056-21PK-Backend19:19:543
418152199211,7cyclictest66159-21ntp_states22:50:183
418152199211,5cyclictest59354-21python322:30:183
4181521992114,5cyclictest5465-21seq19:58:043
4181521992113,6cyclictest436-20systemd-journal22:59:353
418152199210,20cyclictest3056-21PK-Backend21:09:173
418152199210,20cyclictest0-21swapper/323:27:353
418152199210,20cyclictest0-21swapper/322:20:563
418152199210,20cyclictest0-21swapper/322:13:263
418152199210,20cyclictest0-21swapper/320:20:173
418152199210,20cyclictest0-21swapper/300:15:543
418152199210,19cyclictest3056-21PK-Backend20:15:033
4181519992119,1cyclictest3056-21PK-Backend20:30:222
4181519992119,1cyclictest0-21swapper/223:40:102
4181519992119,1cyclictest0-21swapper/223:21:582
4181519992119,1cyclictest0-21swapper/222:51:302
4181519992119,1cyclictest0-21swapper/222:32:572
4181519992119,1cyclictest0-21swapper/222:19:052
4181519992119,1cyclictest0-21swapper/221:52:222
4181519992119,1cyclictest0-21swapper/221:46:542
4181519992119,1cyclictest0-21swapper/221:09:222
4181519992119,1cyclictest0-21swapper/220:05:022
4181519992116,3cyclictest70377-21sleep23:05:002
4181519992112,7cyclictest25236-21apt20:55:122
418151999210,20cyclictest0-21swapper/222:07:562
418151999210,20cyclictest0-21swapper/220:24:072
418151999210,20cyclictest0-21swapper/219:40:422
418151999210,19cyclictest3056-21PK-Backend19:10:062
4181515992116,3cyclictest84789-21latency_hist23:45:001
4181515992116,3cyclictest80636-21wc23:30:191
4181515992114,5cyclictest9244-21sed20:10:001
4181515992113,6cyclictest75091-21awk23:15:181
4181515992110,3cyclictest141rcu_preempt21:50:001
418151599210,20cyclictest0-21swapper/123:54:491
418151599210,19cyclictest3056-21PK-Backend22:05:591
4181511992119,1cyclictest0-21swapper/022:24:460
4181511992119,1cyclictest0-21swapper/021:06:240
4181511992119,1cyclictest0-21swapper/020:21:300
418151199210,20cyclictest0-21swapper/023:49:080
418151199210,20cyclictest0-21swapper/020:10:200
418151199210,19cyclictest3056-21PK-Backend22:15:120
4181521992018,1cyclictest3056-21PK-Backend21:15:193
4181521992018,1cyclictest3056-21PK-Backend19:50:563
4181521992018,1cyclictest0-21swapper/323:50:263
4181521992018,1cyclictest0-21swapper/323:34:303
4181521992018,1cyclictest0-21swapper/323:20:383
4181521992018,1cyclictest0-21swapper/321:21:223
4181521992018,1cyclictest0-21swapper/300:33:123
4181521992017,2cyclictest3056-21PK-Backend20:09:563
418152199201,17cyclictest46420-21diskstats21:55:153
418152199200,7cyclictest26968-21apt21:00:103
418152199200,7cyclictest26968-21apt21:00:103
418152199200,1cyclictest3056-21PK-Backend19:46:033
418152199200,1cyclictest0-21swapper/300:36:023
418152199200,19cyclictest3056-21PK-Backend22:25:503
418152199200,19cyclictest3056-21PK-Backend21:47:093
418152199200,19cyclictest3056-21PK-Backend19:36:303
418152199200,19cyclictest0-21swapper/323:36:333
418152199200,19cyclictest0-21swapper/321:26:363
418152199200,19cyclictest0-21swapper/320:41:123
418152199200,19cyclictest0-21swapper/320:37:343
418152199200,19cyclictest0-21swapper/320:30:323
418152199200,19cyclictest0-21swapper/319:32:393
418152199200,19cyclictest0-21swapper/300:29:193
418151999204,5cyclictest886-21NetworkManager23:51:162
4181519992018,1cyclictest0-21swapper/223:19:552
4181519992018,1cyclictest0-21swapper/221:28:422
4181519992018,1cyclictest0-21swapper/219:50:412
4181519992018,1cyclictest0-21swapper/219:27:502
4181519992018,1cyclictest0-21swapper/219:16:002
4181519992012,6cyclictest32168-21awk21:15:112
418151999200,19cyclictest0-21swapper/223:25:022
418151999200,19cyclictest0-21swapper/222:27:032
418151999200,19cyclictest0-21swapper/220:39:152
418151999200,19cyclictest0-21swapper/219:38:392
418151999200,19cyclictest0-21swapper/219:22:562
418151999200,19cyclictest0-21swapper/200:22:482
4181515992018,1cyclictest3056-21PK-Backend20:36:431
4181515992018,1cyclictest0-21swapper/123:59:421
4181515992018,1cyclictest0-21swapper/121:41:471
4181515992017,2cyclictest3056-21PK-Backend21:25:031
418151599200,1cyclictest3056-21PK-Backend22:59:461
418151599200,19cyclictest3056-21PK-Backend22:30:051
418151599200,19cyclictest3056-21PK-Backend20:34:541
418151599200,19cyclictest3056-21PK-Backend19:58:511
418151599200,19cyclictest0-21swapper/121:13:481
418151599200,19cyclictest0-21swapper/120:25:271
418151599200,19cyclictest0-21swapper/120:01:551
418151599200,19cyclictest0-21swapper/120:01:551
418151599200,19cyclictest0-21swapper/119:51:561
418151599200,19cyclictest0-21swapper/119:42:081
418151599200,18cyclictest3056-21PK-Backend22:12:151
4181511992018,1cyclictest0-21swapper/023:04:060
4181511992018,1cyclictest0-21swapper/021:53:020
4181511992018,1cyclictest0-21swapper/021:15:160
4181511992017,2cyclictest3056-21PK-Backend00:15:440
418151199201,5cyclictest66912-21munin-plugin-st22:54:590
418151199200,19cyclictest0-21swapper/023:44:070
4922190,5sleep20-21swapper/219:45:112
4181521991916,2cyclictest3056-21PK-Backend20:04:503
4181521991916,2cyclictest3056-21PK-Backend20:04:503
418152199190,1cyclictest3056-21PK-Backend23:08:353
418152199190,1cyclictest0-21swapper/321:32:503
418152199190,19cyclictest0-21swapper/323:55:193
418152199190,18cyclictest3056-21PK-Backend21:12:423
4181519991917,1cyclictest0-21swapper/220:28:332
4181519991917,1cyclictest0-21swapper/219:32:432
4181519991917,1cyclictest0-21swapper/200:32:212
4181519991917,1cyclictest0-21swapper/200:03:012
4181519991912,4cyclictest38240-21python321:30:212
418151999190,1cyclictest0-21swapper/223:36:052
418151999190,1cyclictest0-21swapper/223:13:472
418151999190,1cyclictest0-21swapper/223:09:412
418151999190,1cyclictest0-21swapper/222:49:272
4181515991918,1cyclictest0-21swapper/119:15:191
4181515991917,1cyclictest0-21swapper/123:09:471
4181515991917,1cyclictest0-21swapper/122:27:151
4181515991917,1cyclictest0-21swapper/119:33:231
4181515991917,1cyclictest0-21swapper/119:26:401
4181515991916,2cyclictest3056-21PK-Backend00:26:461
4181515991912,5cyclictest30206-21systemd-run21:10:001
418151599190,1cyclictest3056-21PK-Backend23:46:031
418151599190,1cyclictest3056-21PK-Backend22:18:311
418151599190,1cyclictest3056-21PK-Backend20:50:011
418151599190,1cyclictest0-21swapper/119:47:021
418151599190,18cyclictest0-21swapper/123:27:311
418151599190,18cyclictest0-21swapper/122:48:111
418151599190,18cyclictest0-21swapper/119:10:581
4181511991917,1cyclictest0-21swapper/000:37:140
4181511991916,2cyclictest0-21swapper/020:25:150
4181511991911,6cyclictest4185676-21apt19:20:120
4181511991911,5cyclictest4192054-21sshd19:36:220
4181511991911,5cyclictest3094-21needreboot19:50:160
4181511991910,7cyclictest883-21dbus-daemon19:33:560
418151199190,1cyclictest0-21swapper/023:54:080
418151199190,1cyclictest0-21swapper/020:09:190
418151199190,1cyclictest0-21swapper/019:56:280
4181521991816,1cyclictest0-21swapper/323:45:323
4181521991816,1cyclictest0-21swapper/321:52:373
4181521991816,1cyclictest0-21swapper/320:48:293
418152199180,6cyclictest83682-21df23:40:133
4181519991816,1cyclictest3056-21PK-Backend23:47:062
4181515991816,1cyclictest3056-21PK-Backend22:20:321
4181515991816,1cyclictest3056-21PK-Backend21:57:551
4181515991816,1cyclictest3056-21PK-Backend21:51:391
4181515991816,1cyclictest3056-21PK-Backend00:23:551
4181515991816,1cyclictest3056-21PK-Backend00:06:381
4181515991816,1cyclictest0-21swapper/121:15:361
4181515991816,1cyclictest0-21swapper/119:38:161
4181515991816,1cyclictest0-21swapper/119:23:501
4181515991810,6cyclictest9747-21cat20:10:151
418151599180,1cyclictest3056-21PK-Backend23:21:361
418151199189,6cyclictest41006-21grep21:40:110
418151199189,6cyclictest2185-21latency_hist19:50:010
418151199184,12cyclictest23228-21apt20:50:000
418151199182,12cyclictest97482-21apt00:20:120
4181511991811,5cyclictest71671-21awk23:05:190
418151199180,1cyclictest0-21swapper/020:38:390
4181521991715,1cyclictest3056-21PK-Backend20:27:073
4181521991711,5cyclictest98447-21cat00:20:223
418151999172,11cyclictest82332-21update-manager21:03:582
418151999172,11cyclictest82332-21update-manager21:03:572
4181519991715,1cyclictest0-21swapper/200:36:132
418151599172,13cyclictest62626-21grep22:40:211
4181515991715,1cyclictest3056-21PK-Backend22:01:331
4181515991715,1cyclictest3056-21PK-Backend21:37:201
4181515991715,1cyclictest0-21swapper/100:31:261
4181515991715,1cyclictest0-21swapper/100:01:451
418151199172,13cyclictest43368-21grep21:45:180
4181511991715,1cyclictest3056-21PK-Backend22:05:180
4181511991712,3cyclictest51220-21/usr/sbin/munin22:10:160
4181511991711,4cyclictest2156-21gnome-shell23:27:090
418151199171,11cyclictest31901-21sh21:15:000
4181511991710,5cyclictest39814-21awk21:35:170
418151999168,6cyclictest4862-21awk19:55:182
418151999165,8cyclictest883-21dbus-daemon22:59:352
4181519991611,4cyclictest34-21ksoftirqd/220:00:112
4181519991611,4cyclictest34-21ksoftirqd/220:00:112
4181519991610,4cyclictest60652-21basename22:35:162
418151599169,5cyclictest82093-21grep23:35:151
4181515991614,1cyclictest3056-21PK-Backend23:03:521
4181515991611,3cyclictest73205-21load23:10:151
418151199169,5cyclictest76385-21cat23:20:120
418151199167,7cyclictest4181900-21cpuspeed_turbos19:10:120
4181511991614,1cyclictest0-21swapper/022:33:440
4181511991614,1cyclictest0-21swapper/021:00:430
4181511991614,1cyclictest0-21swapper/021:00:430
418151199161,13cyclictest80205-21df23:30:130
418151199161,11cyclictest436-20systemd-journal19:30:010
4181511991610,5cyclictest46532-21cut21:55:170
4181511991610,4cyclictest883-21dbus-daemon00:28:030
418151199160,15cyclictest0-21swapper/020:34:060
4181519991510,4cyclictest2019-21gmain20:15:002
4181515991513,1cyclictest3056-21PK-Backend21:21:381
4181515991510,4cyclictest96186-21awk00:15:161
4181515991510,3cyclictest1095-21runrttasks21:30:211
418151199159,4cyclictest6090-21tr20:00:130
418151199159,4cyclictest6090-21tr20:00:120
418151199156,8cyclictest4192793-21cpuspeed_turbos19:40:110
4181511991513,1cyclictest3056-21PK-Backend20:55:500
4181511991513,1cyclictest0-21swapper/022:03:430
4181511991513,1cyclictest0-21swapper/021:32:260
4181511991511,2cyclictest58232-21perf22:30:000
418151199151,10cyclictest95483-21perf00:15:000
4181511991510,4cyclictest81317-21perf23:39:550
4181511991510,3cyclictest34066-21tr21:20:120
418151199150,1cyclictest0-21swapper/020:53:400
418151199150,14cyclictest3056-21PK-Backend00:31:390
418151199149,3cyclictest93749-21munin-plugin-st00:10:000
418151199148,5cyclictest20328-21expr20:40:110
418151199140,13cyclictest0-21swapper/023:15:080
418151199140,13cyclictest0-21swapper/021:27:060
418151599137,5cyclictest25723-21cut20:55:161
418151199139,3cyclictest3056-21PK-Backend22:35:110
418151199136,5cyclictest68059-21cat22:55:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional