You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-25 - 14:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Apr 25, 2026 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29018421140,3sleep13056-21PK-Backend23:50:191
21375721040,4sleep13056-21PK-Backend20:15:141
23255321010,3sleep00-21swapper/021:10:100
18980927940,14sleep20-21swapper/219:09:152
18961927743,28sleep30-21swapper/319:06:483
139099730,60rtkit-daemon0-21swapper/019:08:060
18839727245,17sleep10-21swapper/119:05:121
2292392500,2sleep00-21swapper/021:00:120
2842152480,2sleep13056-21PK-Backend23:35:111
2736462470,4sleep13056-21PK-Backend23:05:111
139099430,5rtkit-daemon3056-21PK-Backend19:24:051
139099430,5rtkit-daemon3056-21PK-Backend19:24:051
2005022390,3sleep10-21swapper/119:35:201
139099390,3rtkit-daemon3056-21PK-Backend22:04:241
2400322370,1sleep2240033-21cat21:30:132
139099370,3rtkit-daemon3056-21PK-Backend20:44:111
139099370,3rtkit-daemon3056-21PK-Backend20:12:051
139099370,3rtkit-daemon3056-21PK-Backend19:41:561
2547362360,2sleep30-21swapper/322:10:123
190011993228,2cyclictest226478-21python320:50:202
190011993114,10cyclictest285637-21taskset23:39:292
19001199305,23cyclictest0-21swapper/219:15:032
19001499291,26cyclictest291897-21ntp_states23:55:203
19001499291,26cyclictest291897-21ntp_states23:55:203
19001199286,9cyclictest300481-21taskset00:20:152
19001199275,6cyclictest285963-21apt23:40:122
19001199274,14cyclictest0-21swapper/222:20:212
19001199269,15cyclictest241767-21expr21:35:132
19001199268,16cyclictest280521-21unin-run23:25:002
19001199265,6cyclictest190517-21expr19:10:122
19000799260,7cyclictest3056-21PK-Backend23:55:191
19000799260,7cyclictest3056-21PK-Backend23:55:191
19001199259,14cyclictest219021-21expr20:30:142
19001199259,14cyclictest212393-21taskset20:10:182
19001199253,14cyclictest0-21swapper/222:25:212
190011992512,11cyclictest221208-21ps20:35:202
2330112240,4sleep13056-21PK-Backend21:10:161
19001499240,4cyclictest305688-21/usr/sbin/munin00:35:143
19001199248,14cyclictest2156-21gnome-shell22:45:092
19001199245,7cyclictest0-21swapper/200:25:222
190011992422,1cyclictest253955-21apt-get22:10:032
190011992416,5cyclictest294846-21cnf-update-db00:05:012
190011992416,5cyclictest294846-21cnf-update-db00:05:012
190011992411,10cyclictest290898-21munin-run23:55:012
1911562230,6chrt191158-21cat19:10:200
190014992318,3cyclictest282855-21df23:30:123
190014992317,4cyclictest290759-21runrttasks23:53:303
190014992315,5cyclictest219551-21python320:30:213
19001499231,19cyclictest0-21swapper/320:40:103
19001199238,14cyclictest2176-21JS10:27:482
19001199234,18cyclictest208262-21/usr/sbin/munin20:00:152
190011992316,5cyclictest34-21ksoftirqd/220:20:102
190011992316,5cyclictest34-21ksoftirqd/220:20:102
190011992315,5cyclictest223011-21python320:40:192
190011992314,7cyclictest245618-21diskstats21:45:152
190011992314,5cyclictest298370-21apt00:15:132
190011992313,6cyclictest227326-21/usr/sbin/munin20:55:232
190011992310,11cyclictest232432-21cat21:10:012
190011992310,11cyclictest230538-21cron21:05:002
19000799230,21cyclictest3056-21PK-Backend19:53:191
190004992315,6cyclictest238897-21python321:25:200
139099230,16rtkit-daemon3056-21PK-Backend23:32:321
2532282220,6sleep30-21swapper/322:05:193
2532282220,6sleep30-21swapper/322:05:193
2039582220,7sleep013-21ksoftirqd/019:45:210
19001499225,15cyclictest285281-21timerwakeupswit23:35:223
190014992214,6cyclictest192822-21fw_packets19:15:153
19001499220,3cyclictest216328-21taskset20:20:383
19001499220,3cyclictest216328-21taskset20:20:383
19001199228,13cyclictest196637-21uname19:25:162
19001199228,12cyclictest273315-21unin-run23:05:002
190011992220,1cyclictest0-21swapper/223:32:092
190011992220,1cyclictest0-21swapper/221:59:352
190011992220,1cyclictest0-21swapper/200:14:132
190011992218,3cyclictest1895000-21snapd22:19:132
190011992213,5cyclictest270088-21apt22:55:112
190011992212,6cyclictest223889-21apt20:45:132
190011992211,8cyclictest194429-21apt19:20:122
190011992211,8cyclictest194429-21apt19:20:122
190011992210,10cyclictest202808-21cron19:45:012
190011992210,10cyclictest1095-21runrttasks19:32:022
19001199220,21cyclictest0-21swapper/223:27:292
19001199220,20cyclictest0-21swapper/219:54:022
190007992219,2cyclictest3056-21PK-Backend23:10:291
19000799220,20cyclictest3056-21PK-Backend21:04:221
190004992220,1cyclictest0-21swapper/022:07:530
190004992220,1cyclictest0-21swapper/022:07:530
19000499220,21cyclictest0-21swapper/020:38:100
19001499216,3cyclictest232614-21cpuspeed21:10:113
190014992118,2cyclictest3056-21PK-Backend00:28:183
190014992114,5cyclictest243193-21mktemp21:40:013
190014992114,5cyclictest229792-21ntp_kernel_pll_21:00:173
19001199219,10cyclictest210468-21if_err_enp2s020:05:162
19001199218,3cyclictest275429-21apt-config23:10:012
19001199212,15cyclictest82332-21update-manager23:18:522
190011992119,1cyclictest0-21swapper/221:44:142
190011992119,1cyclictest0-21swapper/221:18:252
190011992119,1cyclictest0-21swapper/221:18:252
190011992119,1cyclictest0-21swapper/220:26:132
190011992118,2cyclictest0-21swapper/221:13:592
190011992114,5cyclictest303787-21cstates00:30:132
190011992113,3cyclictest213485-21apt20:15:122
190011992112,7cyclictest253956-21turbostat22:10:002
190011992112,7cyclictest253956-21turbostat22:10:002
190011992112,5cyclictest238901-21python321:25:212
190011992111,8cyclictest291554-21/usr/sbin/munin23:55:162
190011992111,8cyclictest291554-21/usr/sbin/munin23:55:162
190011992111,8cyclictest289319-21ls23:50:012
190011992111,8cyclictest262280-21python322:30:212
190011992111,8cyclictest237519-21sh21:25:002
190011992111,8cyclictest206931-21/usr/sbin/munin19:55:142
190011992110,9cyclictest436-20systemd-journal22:35:232
190011992110,8cyclictest1-21systemd22:43:512
190007992119,1cyclictest3056-21PK-Backend23:40:101
190007992119,1cyclictest3056-21PK-Backend21:56:071
190007992119,1cyclictest3056-21PK-Backend20:54:291
190007992119,1cyclictest3056-21PK-Backend00:33:231
190007992119,1cyclictest0-21swapper/122:18:581
190007992119,1cyclictest0-21swapper/121:21:051
190007992118,2cyclictest3056-21PK-Backend23:46:391
190007992118,2cyclictest3056-21PK-Backend21:36:481
190007992118,2cyclictest3056-21PK-Backend21:08:011
190007992118,2cyclictest3056-21PK-Backend00:02:131
190007992115,4cyclictest258097-21tr22:20:111
19000799210,20cyclictest3056-21PK-Backend21:16:391
19000799210,20cyclictest3056-21PK-Backend21:16:391
19000799210,20cyclictest3056-21PK-Backend20:09:201
19000799210,20cyclictest0-21swapper/100:26:271
19000799210,19cyclictest3056-21PK-Backend21:31:331
190004992119,1cyclictest0-21swapper/023:24:110
190004992119,1cyclictest0-21swapper/022:03:410
190004992119,1cyclictest0-21swapper/020:06:530
190004992119,1cyclictest0-21swapper/019:24:280
190004992119,1cyclictest0-21swapper/019:24:280
190004992119,1cyclictest0-21swapper/000:22:380
190004992119,1cyclictest0-21swapper/000:01:150
19000499210,20cyclictest0-21swapper/022:51:190
19000499210,20cyclictest0-21swapper/022:46:530
19000499210,20cyclictest0-21swapper/021:37:040
19000499210,20cyclictest0-21swapper/020:45:260
139099210,15rtkit-daemon3056-21PK-Backend21:55:011
2080912200,10sleep013-21ksoftirqd/020:00:010
190014992018,1cyclictest0-21swapper/320:51:133
190014992018,1cyclictest0-21swapper/319:41:243
190014992018,1cyclictest0-21swapper/300:30:213
190014992013,5cyclictest198337-21cat19:30:163
19001499201,16cyclictest236890-21python321:20:193
19001499200,19cyclictest0-21swapper/323:04:093
19001499200,19cyclictest0-21swapper/322:04:483
19001499200,19cyclictest0-21swapper/319:56:383
190011992018,1cyclictest0-21swapper/222:50:312
190011992018,1cyclictest0-21swapper/219:39:362
190011992013,5cyclictest250677-21/usr/sbin/munin22:00:132
190011992013,5cyclictest247179-21cnf-update-db21:50:012
190011992013,5cyclictest203951-21python319:45:202
190011992011,7cyclictest273699-21expr23:05:122
190007992018,1cyclictest3056-21PK-Backend22:32:511
190007992018,1cyclictest3056-21PK-Backend20:32:181
190007992018,1cyclictest3056-21PK-Backend00:38:161
190007992018,1cyclictest0-21swapper/119:12:151
190007992016,2cyclictest26-21ksoftirqd/120:50:011
19000799200,1cyclictest0-21swapper/122:10:331
19000799200,19cyclictest3056-21PK-Backend20:25:361
19000799200,19cyclictest3056-21PK-Backend19:45:481
19000799200,19cyclictest0-21swapper/122:05:331
19000799200,19cyclictest0-21swapper/122:05:331
19000799200,19cyclictest0-21swapper/120:01:501
19000499202,3cyclictest13-21ksoftirqd/023:55:190
19000499202,3cyclictest13-21ksoftirqd/023:55:190
190004992014,4cyclictest298106-21munin-plugin-st00:15:000
19000499200,1cyclictest0-21swapper/020:03:010
19000499200,19cyclictest0-21swapper/022:58:220
19000499200,19cyclictest0-21swapper/021:58:270
19000499200,19cyclictest0-21swapper/021:54:480
19000499200,19cyclictest0-21swapper/019:51:390
139099200,14rtkit-daemon3056-21PK-Backend19:57:371
2613732190,3chrt0-21swapper/322:30:013
190014991917,1cyclictest0-21swapper/323:40:333
190014991917,1cyclictest0-21swapper/323:15:523
190014991917,1cyclictest0-21swapper/323:12:143
190014991917,1cyclictest0-21swapper/322:59:433
190014991917,1cyclictest0-21swapper/322:50:033
190014991917,1cyclictest0-21swapper/322:45:383
190014991917,1cyclictest0-21swapper/322:17:403
190014991917,1cyclictest0-21swapper/321:56:093
190014991917,1cyclictest0-21swapper/321:29:323
190014991917,1cyclictest0-21swapper/321:09:253
190014991917,1cyclictest0-21swapper/320:10:033
190014991917,1cyclictest0-21swapper/320:06:113
190014991917,1cyclictest0-21swapper/319:22:453
190014991917,1cyclictest0-21swapper/319:22:453
190014991916,2cyclictest3056-21PK-Backend22:24:283
190014991914,3cyclictest263150-21apt22:35:113
190014991913,4cyclictest248699-21latency_hist21:55:003
190014991912,5cyclictest213816-21expr20:15:143
190014991911,6cyclictest264891-21grep22:40:123
190014991911,6cyclictest239773-21apt21:30:123
19001499191,16cyclictest203827-21ntp_states19:45:193
190014991910,7cyclictest279607-21memory23:20:173
19001499190,7cyclictest205826-21/usr/sbin/munin19:50:213
19001499190,1cyclictest0-21swapper/321:17:163
19001499190,1cyclictest0-21swapper/321:17:163
19001499190,1cyclictest0-21swapper/320:45:463
19001499190,1cyclictest0-21swapper/320:25:323
19001499190,1cyclictest0-21swapper/300:15:273
19001499190,18cyclictest0-21swapper/321:41:023
19001499190,18cyclictest0-21swapper/300:21:223
19001499190,17cyclictest273797-21cat23:05:123
190011991911,6cyclictest293385-21/usr/sbin/munin00:00:152
190007991917,1cyclictest3056-21PK-Backend23:03:261
190007991917,1cyclictest3056-21PK-Backend22:45:071
190007991917,1cyclictest3056-21PK-Backend19:31:481
190007991917,1cyclictest3056-21PK-Backend19:19:321
190007991917,1cyclictest3056-21PK-Backend00:21:341
190007991916,2cyclictest3056-21PK-Backend22:58:121
190007991916,2cyclictest3056-21PK-Backend22:50:211
19000799190,1cyclictest3056-21PK-Backend23:29:491
19000799190,1cyclictest3056-21PK-Backend22:43:181
19000799190,1cyclictest3056-21PK-Backend22:27:371
19000799190,1cyclictest3056-21PK-Backend21:47:151
19000799190,1cyclictest3056-21PK-Backend20:21:441
19000799190,1cyclictest3056-21PK-Backend20:21:441
19000799190,1cyclictest3056-21PK-Backend00:11:471
19000799190,18cyclictest3056-21PK-Backend00:06:531
19000799190,18cyclictest3056-21PK-Backend00:06:531
190004991917,1cyclictest0-21swapper/022:32:470
190004991917,1cyclictest0-21swapper/000:25:420
190004991912,6cyclictest284618-21fw_packets23:35:150
190004991911,6cyclictest264003-21sensors_temp22:35:200
19000499190,6cyclictest884-21gmain23:05:230
19000499190,1cyclictest0-21swapper/023:26:000
19000499190,1cyclictest0-21swapper/023:17:560
19000499190,1cyclictest0-21swapper/022:42:270
19000499190,1cyclictest0-21swapper/022:22:190
19000499190,1cyclictest0-21swapper/021:41:300
19000499190,1cyclictest0-21swapper/021:30:010
19000499190,1cyclictest0-21swapper/021:22:580
19000499190,1cyclictest0-21swapper/019:35:370
19000499190,1cyclictest0-21swapper/019:15:090
19000499190,18cyclictest0-21swapper/022:15:300
19000499190,18cyclictest0-21swapper/000:31:370
190014991816,1cyclictest0-21swapper/323:48:523
190014991816,1cyclictest0-21swapper/319:14:273
190014991814,2cyclictest296636-21apt00:10:113
190014991813,3cyclictest227717-21df_inode20:55:143
190014991812,4cyclictest42-21ksoftirqd/322:30:163
190014991811,5cyclictest222074-21date20:40:013
190014991810,6cyclictest200488-21python319:35:193
19001499180,16cyclictest246103-21python321:45:223
190007991816,1cyclictest3056-21PK-Backend21:42:011
190007991816,1cyclictest3056-21PK-Backend21:28:561
190007991815,2cyclictest3056-21PK-Backend23:21:581
19000799180,1cyclictest3056-21PK-Backend20:37:321
19000799180,1cyclictest3056-21PK-Backend19:28:581
19000499188,9cyclictest214954-21idleruntime-cro20:19:590
190004991816,1cyclictest0-21swapper/021:45:090
190004991816,1cyclictest0-21swapper/020:30:050
190004991816,1cyclictest0-21swapper/020:10:450
19000499180,1cyclictest0-21swapper/023:33:170
190014991714,2cyclictest280759-21apt23:25:123
190014991713,2cyclictest294364-21apt-get00:05:033
190014991713,2cyclictest294364-21apt-get00:05:023
190014991711,4cyclictest208288-21apt20:00:123
19001499171,14cyclictest0-21swapper/319:25:153
19001499170,15cyclictest293271-21df00:00:123
190007991715,1cyclictest3056-21PK-Backend23:17:321
190007991715,1cyclictest3056-21PK-Backend20:58:071
19000499179,6cyclictest288558-21python323:45:220
190004991715,1cyclictest3056-21PK-Backend00:17:450
190004991715,1cyclictest0-21swapper/023:43:390
190004991715,1cyclictest0-21swapper/023:10:540
190004991715,1cyclictest0-21swapper/023:02:010
190004991715,1cyclictest0-21swapper/021:09:530
190004991715,1cyclictest0-21swapper/020:21:210
190004991715,1cyclictest0-21swapper/020:21:200
190004991714,2cyclictest218409-21idleruntime-cro20:30:000
190004991712,3cyclictest228012-21grep20:55:180
190004991710,5cyclictest305751-21cat00:35:150
190004991710,5cyclictest226102-21cut20:50:160
190004991614,1cyclictest3056-21PK-Backend19:25:310
190004991614,1cyclictest0-21swapper/023:51:080
190004991614,1cyclictest0-21swapper/022:11:190
190004991614,1cyclictest0-21swapper/020:43:370
190004991614,1cyclictest0-21swapper/019:43:070
190004991614,1cyclictest0-21swapper/000:07:590
190004991614,1cyclictest0-21swapper/000:07:580
190007991512,2cyclictest3056-21PK-Backend22:37:171
190007991511,2cyclictest26-21ksoftirqd/100:20:001
190004991510,4cyclictest2019-21gmain19:35:010
190004991510,3cyclictest259774-21cpuspeed_turbos22:25:120
190004991311,1cyclictest0-21swapper/021:16:570
190004991311,1cyclictest0-21swapper/021:16:560
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional