You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-05 - 23:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Apr 05, 2026 12:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
172124621270,3sleep10-21swapper/108:46:091
177353621110,3sleep03056-21PK-Backend11:15:150
177651221080,2sleep33056-21PK-Backend11:25:093
168412628147,23sleep10-21swapper/107:05:121
139099790,65rtkit-daemon0-21swapper/007:08:560
168549627540,13sleep30-21swapper/307:08:513
168557126545,14sleep20-21swapper/207:09:472
17449732440,4sleep33056-21PK-Backend09:55:113
17832842390,1sleep10-21swapper/111:45:001
17329982340,2sleep30-21swapper/309:20:153
17046842330,1sleep00-21swapper/008:00:120
17291152300,2sleep00-21swapper/009:10:140
168571499271,6cyclictest1749297-21python310:05:202
139099270,4rtkit-daemon1389-21rtkit-daemon07:42:092
17681832260,9sleep234-21ksoftirqd/211:00:142
17443092240,4chrt0-21swapper/309:50:233
139099240,3rtkit-daemon1389-21rtkit-daemon08:22:532
139099240,15rtkit-daemon0-21swapper/212:21:262
17752022230,1sleep11775315-21missed_timers11:20:161
168571999230,21cyclictest3056-21PK-Backend08:57:013
168571499232,19cyclictest1704182-21perf07:59:592
1685714992321,1cyclictest0-21swapper/211:31:252
168571499230,21cyclictest1213-21kerneloops09:05:182
1685711992317,4cyclictest1702446-21perf07:55:001
1685711992315,6cyclictest436-20systemd-journal09:40:521
1685709992318,3cyclictest1751662-21munin-run10:15:010
168570999231,20cyclictest1704196-21awk08:00:000
17860372220,3sleep10-21swapper/111:50:221
17131592220,4sleep00-21swapper/008:25:110
1685719992220,1cyclictest0-21swapper/311:10:453
1685719992220,1cyclictest0-21swapper/308:20:243
1685719992220,1cyclictest0-21swapper/308:18:343
1685719992220,1cyclictest0-21swapper/308:09:573
1685719992219,2cyclictest3056-21PK-Backend10:38:343
1685719992214,6cyclictest1770252-21ntp_states11:05:183
168571999220,20cyclictest3056-21PK-Backend10:41:253
168571999220,20cyclictest3056-21PK-Backend09:12:023
168571499223,17cyclictest1720769-21grep08:45:172
1685714992220,1cyclictest0-21swapper/209:55:272
1685714992220,1cyclictest0-21swapper/209:18:232
1685714992220,1cyclictest0-21swapper/208:39:222
1685714992220,1cyclictest0-21swapper/208:39:222
1685714992213,7cyclictest1738603-21python309:35:192
168571499220,21cyclictest0-21swapper/210:15:552
168571499220,21cyclictest0-21swapper/208:59:172
1685709992215,5cyclictest1689881-21cron07:20:010
1685709992210,9cyclictest913-21gdbus09:55:010
168570999220,21cyclictest0-21swapper/009:27:490
168570999220,20cyclictest0-21swapper/011:56:200
139099220,13rtkit-daemon0-21swapper/210:55:532
17858552210,3sleep2331rcuc/211:50:192
1685719992119,1cyclictest3056-21PK-Backend11:57:573
1685719992119,1cyclictest3056-21PK-Backend11:30:193
1685719992119,1cyclictest3056-21PK-Backend07:26:443
1685719992119,1cyclictest0-21swapper/311:36:003
1685719992119,1cyclictest0-21swapper/308:32:553
1685719992119,1cyclictest0-21swapper/308:29:163
1685719992118,2cyclictest3056-21PK-Backend09:18:033
1685719992118,2cyclictest3056-21PK-Backend09:01:133
1685719992118,2cyclictest3056-21PK-Backend07:15:493
1685719992115,4cyclictest886-21NetworkManager12:05:013
168571999210,20cyclictest3056-21PK-Backend08:13:213
168571999210,20cyclictest0-21swapper/312:35:013
168571999210,20cyclictest0-21swapper/312:10:543
168571999210,20cyclictest0-21swapper/310:26:573
168571999210,20cyclictest0-21swapper/310:22:053
168571999210,19cyclictest3056-21PK-Backend10:16:243
168571999210,19cyclictest3056-21PK-Backend07:52:263
168571999210,19cyclictest0-21swapper/310:45:173
168571499214,11cyclictest1696816-40unattended-upgr07:35:422
1685714992120,1cyclictest0-21swapper/208:06:512
1685714992119,1cyclictest3056-21PK-Backend08:12:052
1685714992119,1cyclictest0-21swapper/212:39:462
1685714992119,1cyclictest0-21swapper/210:24:412
1685714992119,1cyclictest0-21swapper/209:43:162
1685714992119,1cyclictest0-21swapper/207:33:392
1685714992119,1cyclictest0-21swapper/207:18:112
168571499210,20cyclictest0-21swapper/211:20:242
168571499210,20cyclictest0-21swapper/210:49:212
168571499210,20cyclictest0-21swapper/210:37:312
168571499210,20cyclictest0-21swapper/210:30:342
168571499210,20cyclictest0-21swapper/210:25:422
168571499210,20cyclictest0-21swapper/210:11:022
168571499210,20cyclictest0-21swapper/209:11:192
168571499210,20cyclictest0-21swapper/208:00:562
168571499210,20cyclictest0-21swapper/207:23:512
168571199219,10cyclictest1687435-21unin-run07:15:011
1685711992118,2cyclictest0-21swapper/111:39:591
1685711992113,6cyclictest436-20systemd-journal12:20:011
168571199210,20cyclictest0-21swapper/112:05:001
168571199210,1cyclictest3056-21PK-Backend08:07:071
168571199210,19cyclictest3056-21PK-Backend07:31:111
1685709992117,2cyclictest1788583-21/usr/sbin/munin12:00:120
1685709992114,5cyclictest1707668-21cron08:10:000
168570999210,20cyclictest0-21swapper/011:49:230
139099210,3rtkit-daemon1389-21rtkit-daemon07:39:553
139099210,14rtkit-daemon3056-21PK-Backend11:45:293
1685719992018,1cyclictest3056-21PK-Backend12:18:243
1685719992018,1cyclictest3056-21PK-Backend10:13:333
1685719992018,1cyclictest3056-21PK-Backend08:41:133
1685719992018,1cyclictest0-21swapper/311:40:533
1685719992018,1cyclictest0-21swapper/310:32:533
1685719992018,1cyclictest0-21swapper/308:46:403
168571999200,1cyclictest3056-21PK-Backend09:32:563
168571999200,1cyclictest0-21swapper/308:38:223
168571999200,1cyclictest0-21swapper/308:38:213
168571999200,19cyclictest3056-21PK-Backend12:33:593
168571999200,19cyclictest3056-21PK-Backend09:39:133
168571999200,19cyclictest0-21swapper/310:53:003
168571999200,19cyclictest0-21swapper/310:06:033
168571999200,19cyclictest0-21swapper/309:07:483
168571999200,18cyclictest3056-21PK-Backend12:08:043
1685714992018,1cyclictest0-21swapper/212:16:142
1685714992018,1cyclictest0-21swapper/211:57:072
1685714992018,1cyclictest0-21swapper/211:07:192
1685714992018,1cyclictest0-21swapper/209:23:502
1685714992018,1cyclictest0-21swapper/209:01:532
1685714992018,1cyclictest0-21swapper/207:13:312
1685714992017,2cyclictest0-21swapper/210:40:362
168571499200,1cyclictest0-21swapper/211:17:202
168571499200,1cyclictest0-21swapper/207:46:572
168571499200,19cyclictest0-21swapper/212:27:352
168571499200,19cyclictest0-21swapper/212:09:312
168571499200,19cyclictest0-21swapper/211:28:202
168571499200,19cyclictest0-21swapper/210:51:242
168571499200,19cyclictest0-21swapper/210:03:312
168571499200,19cyclictest0-21swapper/209:29:182
168571499200,19cyclictest0-21swapper/208:28:142
168571499200,19cyclictest0-21swapper/208:16:312
1685711992018,1cyclictest3056-21PK-Backend10:47:141
1685711992017,1cyclictest1725078-21systemd-run08:59:591
1685711992015,3cyclictest1767502-21sh11:00:001
1685711992013,5cyclictest1770977-21cron11:10:001
1685711992012,6cyclictest436-20systemd-journal09:05:001
168571199200,1cyclictest3056-21PK-Backend07:44:501
168571199200,19cyclictest3056-21PK-Backend11:02:421
168571199200,19cyclictest0-21swapper/112:10:421
168571199200,19cyclictest0-21swapper/111:28:111
168571199200,19cyclictest0-21swapper/109:05:481
168571199200,18cyclictest3056-21PK-Backend10:28:421
1685709992018,1cyclictest0-21swapper/012:31:150
1685709992018,1cyclictest0-21swapper/008:59:510
1685709992018,1cyclictest0-21swapper/008:49:440
1685709992017,2cyclictest0-21swapper/007:20:140
1685709992014,4cyclictest2156-21gnome-shell11:34:090
168570999201,1cyclictest0-21swapper/008:19:490
1685709992011,7cyclictest1797048-21apt-get12:25:010
1685709992011,6cyclictest436-20systemd-journal11:25:000
168570999200,1cyclictest3056-21PK-Backend10:03:390
168570999200,1cyclictest3056-21PK-Backend07:35:280
168570999200,1cyclictest0-21swapper/012:37:430
168570999200,1cyclictest0-21swapper/011:43:290
168570999200,1cyclictest0-21swapper/009:04:030
168570999200,19cyclictest3056-21PK-Backend08:30:310
168570999200,19cyclictest1772727-21turbostat11:15:000
168570999200,19cyclictest0-21swapper/011:38:360
168570999200,19cyclictest0-21swapper/011:27:480
168570999200,19cyclictest0-21swapper/011:09:500
168570999200,19cyclictest0-21swapper/010:39:140
168570999200,19cyclictest0-21swapper/010:34:210
168570999200,19cyclictest0-21swapper/009:07:280
168570999200,19cyclictest0-21swapper/008:52:340
168570999200,19cyclictest0-21swapper/007:11:560
139099200,13rtkit-daemon3056-21PK-Backend12:25:103
139099200,13rtkit-daemon0-21swapper/212:04:092
139099200,13rtkit-daemon0-21swapper/211:41:332
1685719991917,1cyclictest3056-21PK-Backend09:42:513
1685719991917,1cyclictest3056-21PK-Backend08:54:113
1685719991917,1cyclictest0-21swapper/307:40:233
1685719991912,5cyclictest1786801-21ls11:55:003
1685719991912,5cyclictest1767000-21ls10:55:213
1685719991911,6cyclictest1796620-21sed12:20:213
168571999190,1cyclictest3056-21PK-Backend07:48:193
168571999190,1cyclictest3056-21PK-Backend07:12:093
168571999190,1cyclictest0-21swapper/310:02:243
168571999190,1cyclictest0-21swapper/307:31:243
168571999190,18cyclictest3056-21PK-Backend09:25:543
1685714991917,1cyclictest0-21swapper/212:13:232
1685714991917,1cyclictest0-21swapper/211:10:232
1685714991917,1cyclictest0-21swapper/209:32:082
1685714991917,1cyclictest0-21swapper/208:40:232
1685714991917,1cyclictest0-21swapper/207:52:182
1685714991917,1cyclictest0-21swapper/207:28:452
1685714991912,5cyclictest1783569-21tr11:45:122
168571499190,1cyclictest0-21swapper/209:54:262
168571199192,12cyclictest1732535-21apt09:20:101
1685711991917,1cyclictest3056-21PK-Backend07:35:041
1685711991917,1cyclictest0-21swapper/112:02:111
1685711991917,1cyclictest0-21swapper/111:10:391
1685711991917,1cyclictest0-21swapper/110:11:241
1685711991917,1cyclictest0-21swapper/110:02:191
1685711991917,1cyclictest0-21swapper/109:56:041
1685711991917,1cyclictest0-21swapper/109:48:471
1685711991917,1cyclictest0-21swapper/109:33:131
1685711991917,1cyclictest0-21swapper/109:27:451
1685711991917,1cyclictest0-21swapper/109:16:021
1685711991917,1cyclictest0-21swapper/108:28:031
1685711991914,3cyclictest1753643-21apt10:20:111
1685711991913,4cyclictest1773172-21cpuspeed_turbos11:15:121
1685711991912,5cyclictest1801767-21ntp_kernel_err12:35:161
1685711991912,5cyclictest1787717-21cut11:55:211
1685711991912,5cyclictest1703429-21ntpq07:55:191
1685711991912,5cyclictest1693372-21cron07:30:001
1685711991911,6cyclictest1764045-21awk10:50:101
1685711991911,6cyclictest1710439-21awk08:15:181
168571199190,1cyclictest0-21swapper/112:20:021
168571199190,1cyclictest0-21swapper/111:47:441
168571199190,1cyclictest0-21swapper/110:18:071
168571199190,1cyclictest0-21swapper/109:52:251
168571199190,1cyclictest0-21swapper/108:10:321
168571199190,1cyclictest0-21swapper/107:46:531
168571199190,18cyclictest3056-21PK-Backend08:32:281
168571199190,18cyclictest0-21swapper/111:33:041
168571199190,18cyclictest0-21swapper/109:35:021
168571199190,18cyclictest0-21swapper/108:23:371
168571199190,18cyclictest0-21swapper/108:01:261
168571199190,18cyclictest0-21swapper/107:23:281
168571199190,18cyclictest0-21swapper/107:16:591
1685709991917,1cyclictest0-21swapper/012:28:240
1685709991917,1cyclictest0-21swapper/012:19:520
1685709991917,1cyclictest0-21swapper/010:41:170
1685709991917,1cyclictest0-21swapper/010:21:430
1685709991917,1cyclictest0-21swapper/009:57:220
1685709991917,1cyclictest0-21swapper/007:46:300
1685709991912,5cyclictest1785226-21ls11:50:120
1685709991912,5cyclictest1733087-21cat09:20:160
1685709991910,7cyclictest1718947-21memory08:40:170
168570999190,5cyclictest436-20systemd-journal09:20:010
168570999190,1cyclictest0-21swapper/012:07:430
168570999190,1cyclictest0-21swapper/010:09:060
168570999190,19cyclictest0-21swapper/010:51:030
168570999190,18cyclictest0-21swapper/008:38:340
168570999190,18cyclictest0-21swapper/008:38:340
168570999190,18cyclictest0-21swapper/007:41:220
1685719991816,1cyclictest3056-21PK-Backend11:01:463
1685719991811,5cyclictest1773554-21cat11:15:153
1685714991816,1cyclictest0-21swapper/211:38:222
1685714991816,1cyclictest0-21swapper/208:31:042
1685711991816,1cyclictest3056-21PK-Backend10:08:351
1685711991816,1cyclictest0-21swapper/112:27:321
1685711991813,3cyclictest1799778-21if_enp2s012:30:151
1685711991811,5cyclictest1719860-21NetworkManager08:45:001
1685709991816,1cyclictest0-21swapper/012:13:230
1685709991816,1cyclictest0-21swapper/008:20:510
1685709991812,5cyclictest1738788-21ls09:35:210
1685709991811,6cyclictest1741863-21df09:45:130
1685709991811,5cyclictest1736133-21grep09:30:110
1685709991810,6cyclictest1701345-21df07:50:130
1685709991810,5cyclictest1763823-21latency_hist10:50:000
168570999180,2cyclictest1709425-21turbostat08:15:000
168570999180,1cyclictest0-21swapper/007:29:340
1685714991711,5cyclictest1722899-21sed08:50:212
1685711991710,5cyclictest1759911-21awk10:35:201
168570999179,5cyclictest1741359-21apt-config09:45:010
1685709991712,3cyclictest1095-21runrttasks10:18:030
168571999160,2cyclictest17036882sleep307:55:223
168571499163,12cyclictest1095-21runrttasks12:31:112
1685711991614,1cyclictest0-21swapper/108:36:081
1685711991614,1cyclictest0-21swapper/108:36:081
1685711991611,3cyclictest1761036-21if_err_enp2s010:40:161
1685711991611,3cyclictest1721813-21/usr/sbin/munin08:50:161
168570999163,9cyclictest1693643-21apt07:30:120
1685709991611,4cyclictest1765512-21perf10:59:550
1685709991610,5cyclictest1755851-21awk10:25:170
1685709991610,4cyclictest1769254-21perf11:05:010
168571999150,13cyclictest17754432sleep311:20:163
1685714991513,1cyclictest0-21swapper/209:48:582
1685711991513,1cyclictest0-21swapper/109:13:261
1685711991510,3cyclictest1758581-21turbostat10:35:001
1685719991410,3cyclictest3056-21PK-Backend07:20:263
168571999140,13cyclictest1741865-21chrt09:45:133
1685719991310,2cyclictest3056-21PK-Backend08:01:523
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional