You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-04 - 12:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Jul 04, 2025 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
128839322910,20sleep30-21swapper/319:05:213
128878522630,9sleep234-21ksoftirqd/219:05:272
133100421030,4chrt1329289-21perf21:04:540
12895262990,8sleep10-21swapper/119:10:011
13643152970,25sleep02197014-21PK-Backend22:40:020
13085612910,7sleep30-21swapper/320:00:303
162399760,9rtkit-daemon1622-21rtkit-daemon23:02:203
13317822730,8chrt2197014-21PK-Backend21:05:230
1289538997210,15cyclictest1305570-21latency_hist19:55:013
128903927256,9sleep00-21swapper/019:05:550
162399710,9rtkit-daemon1622-21rtkit-daemon22:59:012
162399710,9rtkit-daemon1622-21rtkit-daemon22:59:012
128953199702,40cyclictest1335565-40lsb_release21:15:241
128953899646,34cyclictest1753-21gnome-shell20:45:393
13653922630,4chrt0-21swapper/322:40:333
1289538996311,47cyclictest2027-21gdbus21:41:563
128953899627,33cyclictest1753-21gnome-shell19:40:323
162399600,9rtkit-daemon1622-21rtkit-daemon22:37:152
1289536996029,26cyclictest1358339-21ntp_states22:20:272
128953899598,27cyclictest859-21NetworkManager22:49:473
14061662560,5sleep30-21swapper/300:35:283
128953199561,37cyclictest855-21dbus-daemon22:49:471
128953899556,45cyclictest875-21polkitd21:20:013
128953899550,50cyclictest1622-21rtkit-daemon23:11:183
1289531995522,28cyclictest1344450-21ntp_kernel_pll_21:40:271
13371662540,4sleep22197014-21PK-Backend21:20:192
12964742540,8sleep10-21swapper/119:25:351
1289531995422,27cyclictest1324330-21expr20:45:221
162399530,8rtkit-daemon1622-21rtkit-daemon21:37:131
128953899536,17cyclictest1360185-21python322:25:263
162399520,8rtkit-daemon1622-21rtkit-daemon20:14:151
1289538995212,16cyclictest1753-21gnome-shell23:49:123
1289538995211,17cyclictest1753-21gnome-shell20:10:193
1289531995220,28cyclictest1323038-21cat20:40:331
128953199521,46cyclictest1334065-21cat21:10:341
12975232510,6chrt0-21swapper/119:30:221
128953899518,12cyclictest3242253-21nautilus21:55:023
1289538995111,15cyclictest1753-21gnome-shell00:10:023
1289538995111,15cyclictest1753-21gnome-shell00:10:023
128953199512,34cyclictest1753-21gnome-shell00:05:001
162399500,36rtkit-daemon2197014-21PK-Backend00:21:310
128953899507,18cyclictest1753-21gnome-shell23:25:263
1289538995012,15cyclictest1753-21gnome-shell20:05:223
1289538995011,16cyclictest1753-21gnome-shell22:10:053
1289538995010,16cyclictest1302278-21taskset19:45:043
1289531995022,24cyclictest1340836-21seq21:30:231
1289531995019,27cyclictest1371595-21cat23:00:141
1289531995018,28cyclictest1402443-21grep00:25:221
1289531995017,28cyclictest1391763-21expr23:55:221
128953199501,44cyclictest1378149-21latency_hist23:20:011
128953199501,44cyclictest1356751-21switchtime22:15:301
128953899499,17cyclictest1753-21gnome-shell23:32:303
128953899497,17cyclictest1753-21gnome-shell00:08:363
128953899495,17cyclictest1403859-21cat00:30:173
128953899494,24cyclictest1349866-21sed21:55:313
128953199492,41cyclictest1309919-21diskstats20:05:181
128953199492,31cyclictest1753-21gnome-shell21:45:281
1289531994919,26cyclictest1389766-21expr23:50:141
1289531994919,25cyclictest1328604-21cut20:55:291
1289531994918,26cyclictest1357789-21cat22:20:141
128953199491,32cyclictest1337320-21latency21:20:241
128953199491,15cyclictest1313609-21ls20:15:271
13059252480,9sleep22197014-21PK-Backend19:55:132
128953899487,17cyclictest1753-21gnome-shell20:16:373
128953199482,42cyclictest1321841-21cat20:40:021
128953199482,41cyclictest1308393-40lsb_release20:00:261
1289531994816,27cyclictest438-20systemd-journal23:43:081
162399470,40rtkit-daemon1753-21gnome-shell23:38:253
162399470,38rtkit-daemon1289524-21cyclictest22:50:323
12986582470,10sleep21298662-21latency_hist19:35:012
128953899479,15cyclictest1753-21gnome-shell20:40:033
128953899478,17cyclictest1753-21gnome-shell21:07:523
128953899477,17cyclictest1292665-21ntp_states19:15:263
128953899477,16cyclictest1329375-21latency_hist21:00:023
128953899476,17cyclictest1753-21gnome-shell22:35:203
128953899474,16cyclictest1353882-21munin-run22:10:003
128953199471,41cyclictest1753-21gnome-shell20:50:321
1289531994713,29cyclictest1380790-21cat23:25:181
128953899465,17cyclictest1390042-21memory23:50:213
128953899464,18cyclictest1307244-21apt-get20:00:013
1289538994628,13cyclictest1394573-21sshd00:03:193
128953199466,36cyclictest1753-21gnome-shell21:00:371
128953199462,39cyclictest1753-21gnome-shell19:20:251
1289531994619,23cyclictest1362669-21latency_hist22:35:021
1289531994619,22cyclictest1331506-21cstates21:05:171
128953199461,41cyclictest1363153-21df22:35:161
128953199461,40cyclictest1753-21gnome-shell23:05:161
128953199461,40cyclictest1753-21gnome-shell23:05:151
128953199461,40cyclictest1394979-21acpi00:05:121
1289531994613,28cyclictest1352756-21sh22:05:161
162399450,6rtkit-daemon2197014-21PK-Backend23:55:410
1289538994512,15cyclictest1753-21gnome-shell20:35:153
128953199459,31cyclictest0-21swapper/100:30:151
128953199453,38cyclictest1351555-21grep22:00:361
128953199452,38cyclictest1365987-21munin-run22:45:001
1289531994519,22cyclictest1387699-21/usr/sbin/munin23:45:241
128953199451,8cyclictest1338617-21ls21:25:131
1289531994517,23cyclictest1347655-21if_err_enp2s021:50:201
128953199451,38cyclictest1375967-21grep23:10:321
128953199451,38cyclictest1302042-21latency_hist19:45:021
1289531994513,28cyclictest1753-21gnome-shell20:21:501
13933442440,6sleep0161rcuc/000:00:160
128953899445,17cyclictest1317176-21hddtemp_smartct20:25:203
128953899444,17cyclictest1296473-21users19:25:343
1289538994413,26cyclictest1379545-21python323:20:303
1289538994411,14cyclictest1370241-21/usr/sbin/munin22:55:253
1289538994411,14cyclictest1370241-21/usr/sbin/munin22:55:243
128953199442,37cyclictest1397175-21/usr/sbin/munin00:10:221
128953199442,37cyclictest1397175-21/usr/sbin/munin00:10:211
1289531994418,22cyclictest1304224-21tr19:50:151
1289531994418,21cyclictest1069-21runrttasks19:16:151
1289531994417,23cyclictest1360329-21sed22:25:291
1289531994414,26cyclictest1354520-21/usr/sbin/munin22:10:211
128953199441,39cyclictest1385225-21wc23:35:301
128953199441,39cyclictest1290694-21cat19:10:311
128953199441,29cyclictest1753-21gnome-shell19:57:341
128953199441,12cyclictest1378884-21cstates23:20:151
128953199441,10cyclictest1370222-21cat22:55:241
128953199441,10cyclictest1370222-21cat22:55:231
13753232430,7chrt2197014-21PK-Backend23:10:182
128953899436,14cyclictest1361244-21tr22:30:153
128953899435,16cyclictest1333002-21/usr/sbin/munin21:10:333
1289538994311,6cyclictest1299219-21df_inode19:35:173
1289536994320,14cyclictest34-21ksoftirqd/223:25:012
128953199432,36cyclictest1302691-21df_inode19:45:191
1289531994318,21cyclictest1398510-21cpuspeed00:15:171
128953199431,4cyclictest0-21swapper/120:33:301
128953199431,37cyclictest1368247-21expr22:50:171
128953199431,28cyclictest1753-21gnome-shell00:35:211
1289530994332,7cyclictest1069-21runrttasks22:09:550
128953099431,4cyclictest141rcu_preempt20:38:440
12994712420,6sleep02197014-21PK-Backend19:35:240
128953899425,15cyclictest1753-21gnome-shell00:21:453
1289538994224,13cyclictest438-20systemd-journal20:50:183
1289538994211,25cyclictest869-21irqbalance21:28:553
1289536994227,11cyclictest1314832-21apt-config20:20:032
128953199428,30cyclictest1383038-21cut23:30:221
1289531994217,21cyclictest1069-21runrttasks20:27:281
128953199421,26cyclictest1400587-21sh00:20:211
128953099421,21cyclictest1362328-21taskset22:32:510
13462122410,6sleep22197014-21PK-Backend21:45:252
128953899414,16cyclictest1345434-21/usr/sbin/munin21:45:313
128953899414,13cyclictest860997-21snmpd22:03:013
1289538994124,13cyclictest1399767-21sshd00:18:183
1289538994123,13cyclictest3242254-21gmain21:40:013
1289538994114,23cyclictest1753-21gnome-shell22:17:333
1289538994111,26cyclictest860997-21snmpd23:10:013
1289538994111,26cyclictest860997-21snmpd23:10:013
1289536994125,11cyclictest438-20systemd-journal20:52:182
128953199418,29cyclictest1753-21gnome-shell21:56:581
1289530994127,10cyclictest13-21ksoftirqd/020:51:070
128953099411,13cyclictest1346811-21taskset21:48:460
128953099411,12cyclictest141rcu_preempt21:15:300
128953899405,10cyclictest1316154-40apt-check20:21:203
1289538994024,12cyclictest1378128-21cron23:20:003
1289538994022,13cyclictest438-20systemd-journal21:23:063
1289538994016,19cyclictest1753-21gnome-shell21:30:173
1289538994012,23cyclictest1753-21gnome-shell00:25:143
1289536994022,13cyclictest34-21ksoftirqd/220:05:212
1289536994019,16cyclictest2179-21gvfsd-metadata20:20:012
1289536994018,12cyclictest1317825-21wc20:25:312
128953199401,19cyclictest231irq_work/119:35:151
128953099409,16cyclictest1753-21gnome-shell22:43:460
128953099407,5cyclictest141rcu_preempt22:15:220
128953099402,7cyclictest1753-21gnome-shell00:36:280
1289530994023,13cyclictest13-21ksoftirqd/019:19:450
1289530994022,13cyclictest1301315-21ntpq19:40:260
162399390,27rtkit-daemon2197014-21PK-Backend00:30:410
162399390,25rtkit-daemon2197014-21PK-Backend00:25:570
1289538993911,24cyclictest1753-21gnome-shell23:40:423
1289538993911,23cyclictest1753-21gnome-shell19:30:123
1289536993926,10cyclictest34-21ksoftirqd/222:18:302
1289536993923,11cyclictest1373214-21apt23:05:142
1289536993923,11cyclictest1373214-21apt23:05:142
1289536993922,13cyclictest1393907-21cat00:00:242
1289536993922,13cyclictest1307743-40apt-check20:00:162
128953699392,12cyclictest141rcu_preempt20:40:172
1289536993920,14cyclictest2179-21gvfsd-metadata20:55:032
1289536993918,7cyclictest1800-21upowerd22:49:472
1289536993912,11cyclictest438-20systemd-journal22:09:542
128953099397,5cyclictest141rcu_preempt20:31:470
128953099395,12cyclictest2197014-21PK-Backend22:13:470
1289530993926,9cyclictest13-21ksoftirqd/022:49:470
1289530993922,13cyclictest1341742-21awk21:35:000
1289530993921,13cyclictest13-21ksoftirqd/019:53:230
162399380,27rtkit-daemon2197014-21PK-Backend22:03:022
162399380,26rtkit-daemon2197014-21PK-Backend23:39:580
128953899387,27cyclictest1753-21gnome-shell23:57:243
1289538993812,21cyclictest1289696-21apt19:10:133
128953699387,10cyclictest34-21ksoftirqd/220:37:042
1289536993827,7cyclictest1335291-21expr21:15:172
1289536993825,8cyclictest1398627-21tr00:15:172
1289536993824,10cyclictest1300010-21sshd19:36:192
1289536993821,13cyclictest855-21dbus-daemon23:28:392
128953099389,25cyclictest1329051-40lsb_release20:57:240
128953099382,8cyclictest1753-21gnome-shell19:12:590
128953099382,6cyclictest141rcu_preempt22:30:010
1289530993822,6cyclictest13-21ksoftirqd/022:05:000
1289530993820,13cyclictest13-21ksoftirqd/023:04:210
128953099381,9cyclictest1313809-21/usr/sbin/munin20:15:320
162399370,30rtkit-daemon0-21swapper/223:31:182
1289538993720,12cyclictest1329964-21fw_packets21:00:233
1289538993711,22cyclictest1753-21gnome-shell22:20:243
1289538993710,22cyclictest1753-21gnome-shell20:30:053
128953699373,16cyclictest1753-21gnome-shell21:42:482
1289536993725,8cyclictest1753-21gnome-shell00:13:412
1289536993725,8cyclictest1753-21gnome-shell00:13:412
1289536993723,10cyclictest34-21ksoftirqd/223:04:042
1289536993723,10cyclictest34-21ksoftirqd/221:26:572
1289536993722,11cyclictest1341351-21taskset21:30:332
1289536993722,11cyclictest1295103-21cron19:25:012
128953699372,15cyclictest3242253-21nautilus23:20:022
1289536993721,12cyclictest1364772-21expr22:40:192
1289536993721,11cyclictest438-20systemd-journal22:53:192
1289536993720,13cyclictest1403409-21pool-org.gnome.00:30:002
1289536993720,12cyclictest438-20systemd-journal23:50:012
128953699371,19cyclictest1343453-21seq21:40:002
128953099372,31cyclictest1753-21gnome-shell00:07:510
1289530993723,10cyclictest13-21ksoftirqd/023:08:370
1289530993723,10cyclictest13-21ksoftirqd/023:08:370
128953099372,11cyclictest2197014-21PK-Backend19:30:020
1289530993720,13cyclictest1333027-21apt21:10:150
1289530993718,14cyclictest13-21ksoftirqd/021:55:050
128953099371,15cyclictest1376159-21chrt23:12:010
162399360,25rtkit-daemon0-21swapper/220:48:112
128953699368,6cyclictest13480402processes21:50:282
128953699368,17cyclictest1392999-40mandb00:00:012
1289536993624,8cyclictest1318930-21ls20:30:212
1289536993618,14cyclictest1291795-21df_inode19:15:072
1289536993617,15cyclictest1632-21dbus-daemon22:00:022
128953099364,8cyclictest2197014-21PK-Backend21:24:120
1289530993622,10cyclictest1325331-21sshd20:47:430
1289530993619,6cyclictest13-21ksoftirqd/019:55:250
128953099361,8cyclictest2197014-21PK-Backend23:33:350
1289536993524,7cyclictest1302453-21tr19:45:172
1289536993523,8cyclictest1354250-21cpuspeed_turbos22:10:162
1289536993520,10cyclictest1334885-21cat21:15:022
1289536993519,12cyclictest1329947-21diskstats21:00:232
1289536993518,12cyclictest438-20systemd-journal00:30:182
1289536993512,5cyclictest141rcu_preempt00:08:392
1289536993510,12cyclictest141rcu_preempt23:35:132
1289530993523,9cyclictest1343739-21grep21:40:160
1289530993521,10cyclictest13-21ksoftirqd/020:40:050
128953099351,8cyclictest2197014-21PK-Backend23:42:260
128953099351,7cyclictest1298586-21turbostat19:35:000
128953099351,13cyclictest2197014-21PK-Backend23:23:100
128953099351,10cyclictest2197014-21PK-Backend23:47:560
162399340,27rtkit-daemon1753-21gnome-shell22:31:572
1289536993425,6cyclictest1296879-21unin-run19:30:012
128953699341,8cyclictest2197014-21PK-Backend20:11:172
1289536993418,12cyclictest1753-21gnome-shell19:14:422
128953099343,12cyclictest1069-21runrttasks19:20:030
128953099342,8cyclictest1380707-21apt23:25:140
1289530993421,9cyclictest1309494-21sed20:05:030
1289530993419,10cyclictest1311000-21grep20:10:000
1289530993416,5cyclictest13-21ksoftirqd/022:56:240
1289530993416,5cyclictest13-21ksoftirqd/022:56:230
1289530993416,13cyclictest1312001-21ntpq20:10:260
1289530993412,5cyclictest141rcu_preempt20:25:200
128953699338,20cyclictest1753-21gnome-shell23:50:122
1289536993322,7cyclictest1401650-21sh00:25:002
1289536993321,9cyclictest1753-21gnome-shell23:44:352
1289536993319,10cyclictest1359520-21cat22:25:132
128953699331,8cyclictest2197014-21PK-Backend19:53:562
1289536993313,6cyclictest141rcu_preempt00:35:162
128953699331,29cyclictest0-21swapper/221:07:542
128953099339,5cyclictest141rcu_preempt22:54:540
128953099339,5cyclictest141rcu_preempt21:36:110
128953099338,5cyclictest141rcu_preempt19:50:010
1289530993320,9cyclictest1753-21gnome-shell00:16:580
128953099331,4cyclictest2197014-21PK-Backend22:23:250
1289538993213,15cyclictest1293806-21tr19:20:163
128953699321,4cyclictest0-21swapper/219:40:032
128953099326,5cyclictest141rcu_preempt21:26:530
128953099321,27cyclictest2197014-21PK-Backend00:10:420
128953099321,27cyclictest2197014-21PK-Backend00:10:420
1289530993121,7cyclictest2197014-21PK-Backend21:52:320
1289530993121,7cyclictest2197014-21PK-Backend20:23:500
128953099311,4cyclictest2197014-21PK-Backend23:18:100
128953099311,25cyclictest2197014-21PK-Backend23:50:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional