You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-11 - 13:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Fri Jul 11, 2025 00:44:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991230,9rtkit-daemon1622-21rtkit-daemon00:04:162
1623991160,22rtkit-daemon2197014-21PK-Backend20:54:393
1623991040,58rtkit-daemon0-21swapper/219:05:442
119761621020,9sleep31753-21gnome-shell23:16:593
1108989991023,13cyclictest3242253-21nautilus22:47:392
12063292990,6sleep01206330-21find23:40:310
12004112950,6chrt42-21ksoftirqd/323:25:253
110861129446,40sleep30-21swapper/319:06:553
110898899912,86cyclictest231irq_work/122:50:331
110898399881,59cyclictest1753-21gnome-shell22:52:450
12140252870,5sleep00-21swapper/000:05:030
12140252870,5sleep00-21swapper/000:05:020
110898399832,11cyclictest1753-21gnome-shell23:09:410
110898899812,13cyclictest1198941-21which23:20:271
1108988998020,50cyclictest26-21ksoftirqd/123:13:521
110898399802,47cyclictest1753-21gnome-shell22:58:230
162399790,5rtkit-daemon54-21kcompactd022:44:432
162399790,5rtkit-daemon54-21kcompactd022:44:422
110898999792,56cyclictest1753-21gnome-shell23:20:522
110898399783,10cyclictest1753-21gnome-shell22:47:560
110898999772,55cyclictest1753-21gnome-shell23:19:032
1108988997711,62cyclictest11921782chrt23:02:451
1108988997711,62cyclictest11921782chrt23:02:451
110898399773,47cyclictest1753-21gnome-shell23:04:030
110898399773,47cyclictest1753-21gnome-shell23:04:020
110898999765,19cyclictest141rcu_preempt23:02:042
110898999765,19cyclictest141rcu_preempt23:02:032
110898399762,47cyclictest1753-21gnome-shell23:10:060
1108983997610,8cyclictest141rcu_preempt23:20:570
1108991997526,9cyclictest1753-21gnome-shell23:01:483
1108991997526,9cyclictest1753-21gnome-shell23:01:473
110898999753,48cyclictest1753-21gnome-shell22:50:152
110898899752,16cyclictest1189154-21acpi22:55:111
162399740,5rtkit-daemon2197014-21PK-Backend20:08:462
1108991997462,8cyclictest1753-21gnome-shell23:09:123
1108988997453,5cyclictest141rcu_preempt23:19:551
110898899742,55cyclictest0-21swapper/122:49:001
110898999736,7cyclictest141rcu_preempt23:10:132
110898899712,10cyclictest1753-21gnome-shell23:05:471
110898999703,10cyclictest1753-21gnome-shell23:09:292
11654352690,6sleep01139567-21kworker/0:221:45:270
110898399692,8cyclictest2197014-21PK-Backend23:17:120
110866026939,24sleep10-21swapper/119:07:281
162399680,9rtkit-daemon1622-21rtkit-daemon20:43:432
110898999682,10cyclictest1753-21gnome-shell22:56:232
162399670,5rtkit-daemon2197014-21PK-Backend19:39:172
110738026751,9sleep00-21swapper/019:05:110
1108991996350,9cyclictest42-21ksoftirqd/322:51:393
162399610,8rtkit-daemon1622-21rtkit-daemon22:09:242
162399610,8rtkit-daemon1622-21rtkit-daemon19:46:462
162399610,8rtkit-daemon1622-21rtkit-daemon19:20:192
110899199612,55cyclictest1753-21gnome-shell22:47:403
1108989996124,15cyclictest1136695-21df_inode20:25:162
162399600,9rtkit-daemon1622-21rtkit-daemon21:39:262
162399600,9rtkit-daemon1622-21rtkit-daemon21:05:422
162399590,9rtkit-daemon1622-21rtkit-daemon23:57:513
1108991995933,10cyclictest42-21ksoftirqd/322:55:173
162399580,8rtkit-daemon1622-21rtkit-daemon21:49:282
162399580,8rtkit-daemon1622-21rtkit-daemon21:01:452
110898999570,53cyclictest1622-21rtkit-daemon23:43:132
1108988995719,11cyclictest1207045-21sed23:45:021
1108988995714,14cyclictest438-20systemd-journal00:23:231
162399560,9rtkit-daemon1622-21rtkit-daemon22:31:443
162399560,8rtkit-daemon1622-21rtkit-daemon23:35:242
162399560,8rtkit-daemon1622-21rtkit-daemon20:56:332
162399560,8rtkit-daemon1622-21rtkit-daemon20:38:122
162399560,8rtkit-daemon1622-21rtkit-daemon20:38:122
1108988995618,10cyclictest1125451-21date19:55:011
162399550,9rtkit-daemon1622-21rtkit-daemon22:12:463
162399550,9rtkit-daemon1622-21rtkit-daemon21:16:443
162399550,9rtkit-daemon1622-21rtkit-daemon21:16:443
162399550,9rtkit-daemon1622-21rtkit-daemon19:11:433
162399550,9rtkit-daemon1622-21rtkit-daemon19:11:423
110899199553,47cyclictest1753-21gnome-shell23:20:283
1108988995520,10cyclictest1130882-21mktemp20:10:011
162399540,9rtkit-daemon1622-21rtkit-daemon23:46:463
162399540,9rtkit-daemon1622-21rtkit-daemon23:46:453
162399540,8rtkit-daemon1622-21rtkit-daemon22:26:563
162399540,8rtkit-daemon1622-21rtkit-daemon20:13:292
110899199543,10cyclictest1753-21gnome-shell23:15:023
1108988995419,11cyclictest1119079-21basename19:35:211
1108988995416,12cyclictest2181-21gdbus20:50:011
162399530,8rtkit-daemon1622-21rtkit-daemon21:25:373
162399530,8rtkit-daemon1622-21rtkit-daemon00:05:552
162399530,8rtkit-daemon1622-21rtkit-daemon00:05:542
1108988995317,11cyclictest1142727-21seq20:41:251
1108988995317,10cyclictest1169602-21cron22:00:011
1108988995313,14cyclictest855-21dbus-daemon21:35:011
1108988995313,13cyclictest1632-21dbus-daemon19:25:031
110898899531,27cyclictest0-21swapper/122:41:121
110898899531,27cyclictest0-21swapper/122:41:121
1108988995221,11cyclictest1202326-21ntp_kernel_pll_23:30:241
1108988995220,7cyclictest1203574-21/usr/sbin/munin23:35:141
1108988995219,6cyclictest1148731-21/usr/sbin/munin21:00:311
1108988995213,12cyclictest1181341-21cat22:30:331
1108988995211,11cyclictest1165017-21df21:45:171
1108988995119,10cyclictest1164103-21cron21:45:001
1108988995114,11cyclictest1170150-21expr22:00:171
1108988995113,12cyclictest1212942-21cat00:00:271
110898899502,24cyclictest1145941-21sed20:50:321
110898899502,22cyclictest0-21swapper/122:35:301
1108988995017,11cyclictest26-21ksoftirqd/100:10:241
110898899501,23cyclictest1122353-21expr19:45:171
1108988995012,15cyclictest1632-21dbus-daemon19:16:501
1108988995012,15cyclictest1632-21dbus-daemon19:16:501
1108988995012,12cyclictest1210865-21expr23:55:131
1108988995012,11cyclictest2962348-21rs:main1
1108988995012,11cyclictest2962348-21rs:main1
110898899501,12cyclictest1147077-21expr20:55:161
110898899492,19cyclictest1223123-21hddtemp_smartct00:30:181
1108988994917,11cyclictest884-21gdbus20:24:471
1108988994916,8cyclictest1173845-21uname22:10:231
1108988994915,7cyclictest1120797-21hddtemp_smartct19:40:201
1108988994914,11cyclictest2962348-21rs:main1
1108988994913,11cyclictest1215278-21seq00:07:041
1108988994913,11cyclictest1215278-21seq00:07:041
1108988994912,9cyclictest1617-10pipewire-media-22:18:011
110898899491,12cyclictest1217757-21cstates00:15:131
110898899491,12cyclictest1217757-21cstates00:15:131
11513792480,8sleep32197014-21PK-Backend21:05:313
110898899481,21cyclictest1161212-21expr21:35:141
110898899481,12cyclictest1200270-21awk23:25:221
110898899481,10cyclictest1208762-21date23:50:031
110898899481,10cyclictest1208762-21date23:50:031
110898899481,10cyclictest1152459-21ls21:10:181
162399470,38rtkit-daemon2197014-21PK-Backend22:25:042
11693592472,7sleep31753-21gnome-shell21:56:593
1108989994723,21cyclictest1167879-21awk21:55:002
110898899472,18cyclictest1221369-21df_inode00:25:191
110898899471,7cyclictest1141100-21systemd-udevd20:38:111
110898899471,7cyclictest1141100-21systemd-udevd20:38:101
1108988994715,8cyclictest1150242-21cron21:09:551
1108988994712,11cyclictest1157783-21cat21:25:211
1108988994710,12cyclictest1125748-21expr19:55:161
1108988994710,12cyclictest1125748-21expr19:55:161
1108988994710,12cyclictest0-21swapper/122:25:341
1108988994710,11cyclictest1176789-21basename22:20:121
1108988994710,11cyclictest1156804-21cat21:20:361
162399460,39rtkit-daemon42-21ksoftirqd/319:31:143
110898899461,7cyclictest1154329-21cstates21:15:131
110898899461,7cyclictest1154329-21cstates21:15:131
1108988994617,7cyclictest1133034-21cstates20:15:171
1108988994616,7cyclictest1172536-21switchtime22:05:301
1108988994615,7cyclictest1117684-21ntp_states19:30:281
1108988994615,7cyclictest1113682-21latency19:20:231
162399450,38rtkit-daemon2197014-21PK-Backend23:31:352
162399450,37rtkit-daemon1775-21gvfs-afc-volume20:49:102
110898899459,16cyclictest1753-21gnome-shell20:06:131
110898899459,15cyclictest1127720-21sshd20:00:221
1108983994534,7cyclictest1069-21runrttasks19:24:050
1108983994532,8cyclictest1212218-21grep00:00:030
162399440,37rtkit-daemon1775-21gvfs-afc-volume19:38:153
1108991994410,16cyclictest1753-21gnome-shell00:19:243
1108991994410,16cyclictest1753-21gnome-shell00:19:243
1108988994415,7cyclictest1069-21runrttasks00:37:311
110898399442,18cyclictest1753-21gnome-shell00:24:030
110898399441,11cyclictest1140501-21ntp_kernel_err20:35:250
110898399441,11cyclictest1140501-21ntp_kernel_err20:35:240
11841162430,8sleep02197014-21PK-Backend22:40:180
11841162430,8sleep02197014-21PK-Backend22:40:170
1108991994318,20cyclictest42-21ksoftirqd/321:45:213
1108983994330,9cyclictest1125337-21perf19:55:000
162399420,33rtkit-daemon1108978-21cyclictest20:12:043
110899199421,37cyclictest1753-21gnome-shell22:05:003
1108983994223,14cyclictest1221359-21df00:25:180
162399410,34rtkit-daemon2197014-21PK-Backend22:35:302
12015492410,8chrt2197014-21PK-Backend23:30:022
11688742410,5sleep00-21swapper/021:55:260
110899199419,6cyclictest1116780-21latency_hist19:30:023
1108991994127,11cyclictest42-21ksoftirqd/319:15:223
1108991994127,11cyclictest42-21ksoftirqd/319:15:223
1108989994123,5cyclictest34-21ksoftirqd/219:29:462
110898399412,34cyclictest1151614-21users21:05:350
1108983994123,13cyclictest438-20systemd-journal21:15:000
1108983994122,6cyclictest13-21ksoftirqd/019:48:080
110898399411,13cyclictest1217889-21/usr/sbin/munin00:15:150
110898399411,13cyclictest1217889-21/usr/sbin/munin00:15:140
162399400,32rtkit-daemon1108978-21cyclictest23:41:563
162399400,32rtkit-daemon1108978-21cyclictest19:56:023
162399400,32rtkit-daemon1108978-21cyclictest19:56:023
162399400,31rtkit-daemon1108978-21cyclictest00:02:003
12248072400,5sleep00-21swapper/000:35:170
11254332400,6sleep234-21ksoftirqd/219:55:022
1108991994030,6cyclictest1125313-21idleruntime-cro19:55:003
1108991994027,9cyclictest1178301-21pool-org.gnome.22:25:013
1108991994015,6cyclictest1142689-21taskset20:40:543
1108989994023,5cyclictest34-21ksoftirqd/222:31:472
1108989994022,6cyclictest34-21ksoftirqd/219:11:232
1108989994022,6cyclictest34-21ksoftirqd/219:11:232
1108989994021,6cyclictest34-21ksoftirqd/222:04:542
1108989994021,15cyclictest1753-21gnome-shell19:32:112
110898999401,8cyclictest2197014-21PK-Backend00:14:152
110898999401,11cyclictest1224380-21cut00:35:012
1108988994019,4cyclictest0-21swapper/123:50:081
110898399402,14cyclictest1146477-21cron20:55:000
1108983994016,18cyclictest855-21dbus-daemon21:00:280
11373102393,11sleep0854-21avahi-daemon20:25:320
110899199392,7cyclictest1753-21gnome-shell22:17:423
1108991993927,8cyclictest1753-21gnome-shell22:44:373
1108991993927,8cyclictest1753-21gnome-shell22:44:363
1108991993927,8cyclictest1753-21gnome-shell20:39:313
1108991993927,8cyclictest1753-21gnome-shell20:39:303
110899199392,33cyclictest1753-21gnome-shell20:33:353
110899199392,32cyclictest1137078-21ntp_states20:25:273
1108991993921,13cyclictest42-21ksoftirqd/320:15:333
110899199391,34cyclictest1753-21gnome-shell20:24:113
110898999392,33cyclictest1753-21gnome-shell21:25:222
1108988993910,7cyclictest0-21swapper/120:30:331
1108983993927,7cyclictest1182594-21memory22:35:240
1108983993923,12cyclictest1160428-21sshd21:32:230
1108983993921,13cyclictest438-20systemd-journal20:16:590
1108983993921,13cyclictest1203604-21seq23:35:120
1108983993920,14cyclictest1216000-21cat00:10:140
162399380,30rtkit-daemon42-21ksoftirqd/319:41:103
1108991993827,7cyclictest1130613-21dpkg20:10:023
1108991993820,14cyclictest1753-21gnome-shell21:11:563
1108991993814,20cyclictest391irq_work/300:33:133
110898999389,17cyclictest1753-21gnome-shell22:15:132
1108989993821,13cyclictest34-21ksoftirqd/200:40:002
1108989993821,13cyclictest1174097-21ntp_states22:10:282
110898899382,17cyclictest1753-21gnome-shell20:29:111
110898399389,11cyclictest1069-21runrttasks21:24:250
1108983993827,7cyclictest1069-21runrttasks22:06:050
1108983993819,14cyclictest1126315-21ntpq19:55:260
1108983993819,14cyclictest1126315-21ntpq19:55:250
12140672370,7sleep32197014-21PK-Backend00:05:133
12140672370,7sleep32197014-21PK-Backend00:05:133
12072012370,5chrt0-21swapper/023:45:150
12072012370,5chrt0-21swapper/023:45:140
1108991993726,7cyclictest1182009-21apt22:35:143
1108991993723,11cyclictest42-21ksoftirqd/300:20:273
1108991993722,11cyclictest1148216-21apt-get21:00:013
1108991993718,14cyclictest1149368-40lsb_release21:00:253
1108991993710,23cyclictest0-21swapper/300:11:013
1108989993717,15cyclictest3242253-21nautilus19:20:022
1108989993717,15cyclictest3242253-21nautilus19:20:012
1108989993716,17cyclictest1138937-21smartctl20:30:322
110898399372,31cyclictest1753-21gnome-shell19:41:140
1108983993722,11cyclictest1135096-21sed20:20:230
1108983993721,11cyclictest1115112-21dpkg19:25:040
1108983993720,13cyclictest1144692-21sshd20:49:230
1108983993720,12cyclictest1119677-21cat19:35:320
1108983993719,13cyclictest438-20systemd-journal21:15:230
1108983993719,13cyclictest438-20systemd-journal21:15:220
1108983993718,14cyclictest2179-21gvfsd-metadata23:35:010
1108983993717,15cyclictest2181-21gdbus22:30:010
1108991993621,10cyclictest1128176-21cut20:00:313
110899199361,15cyclictest1753-21gnome-shell00:29:513
1108989993626,7cyclictest859-21NetworkManager22:23:112
1108989993624,9cyclictest1753-21gnome-shell00:30:022
110898999361,9cyclictest2197014-21PK-Backend19:55:192
110898999361,9cyclictest2197014-21PK-Backend19:55:192
1108989993617,16cyclictest2197014-21PK-Backend23:54:592
110898999361,5cyclictest0-21swapper/220:21:342
110898399368,12cyclictest1069-21runrttasks19:33:140
110898399362,9cyclictest1110690-21munin-run19:15:000
110898399362,9cyclictest1110690-21munin-run19:15:000
110898399362,7cyclictest1753-21gnome-shell22:12:530
110898399362,7cyclictest1200247-21df_inode23:25:200
110898399362,5cyclictest0-21swapper/023:54:410
1108983993622,10cyclictest1164211-21latency_hist21:45:000
1108983993619,13cyclictest1213083-21ntp_states00:00:290
1108983993619,12cyclictest1130512-21apt-get20:10:030
110898399361,13cyclictest2197014-21PK-Backend20:34:170
110899199352,28cyclictest1753-21gnome-shell23:32:243
1108991993519,12cyclictest3242255-21gdbus19:50:013
1108991993519,12cyclictest1172042-21cat22:05:213
110898999357,24cyclictest1069-21runrttasks21:40:412
1108989993523,9cyclictest1753-21gnome-shell21:35:002
1108989993520,11cyclictest1218468-21cat00:15:302
1108989993520,11cyclictest1218468-21cat00:15:302
110898399352,5cyclictest0-21swapper/022:15:050
110898399352,30cyclictest0-21swapper/020:43:010
110898399352,17cyclictest0-21swapper/020:09:330
1108983993519,11cyclictest1177664-21awk22:20:280
1108983993518,12cyclictest438-20systemd-journal20:55:020
1108983993517,13cyclictest1157459-21awk21:25:170
110898399351,31cyclictest0-21swapper/000:31:540
1108983993512,7cyclictest141rcu_preempt21:53:250
162399340,27rtkit-daemon2197014-21PK-Backend23:51:553
1108991993419,10cyclictest1155731-21dpkg21:20:153
1108991993418,12cyclictest1166841-21basename21:50:193
1108991993416,13cyclictest42-21ksoftirqd/320:45:213
110899199341,13cyclictest2197014-21PK-Backend19:21:213
1108989993423,7cyclictest1753-21gnome-shell23:55:312
1108989993423,7cyclictest1753-21gnome-shell19:42:132
110898999342,12cyclictest1753-21gnome-shell00:20:352
1108989993418,12cyclictest1153128-21sensors_temp21:10:312
1108989993418,12cyclictest1129003-21cut20:05:012
1108983993420,10cyclictest13-21ksoftirqd/019:17:250
1108983993420,10cyclictest13-21ksoftirqd/019:17:250
1108983993419,11cyclictest1181014-21cat22:30:270
1108983993418,11cyclictest1161773-21cut21:35:290
1108983993417,13cyclictest1127893-21sed20:00:250
162399330,26rtkit-daemon2197014-21PK-Backend21:36:123
110899199332,13cyclictest1753-21gnome-shell21:40:233
1108991993318,11cyclictest1225039-21cat00:35:243
1108991993311,10cyclictest141rcu_preempt21:30:143
110898999337,22cyclictest1145115-21ls20:50:172
110898999335,5cyclictest141rcu_preempt23:49:532
110898999335,5cyclictest141rcu_preempt23:49:522
1108989993322,7cyclictest1155739-21grep21:20:182
1108989993316,12cyclictest1169213-21cat21:55:322
110898999331,29cyclictest0-21swapper/220:17:312
1108983993317,12cyclictest1170367-21irqstats22:00:240
1108991993217,11cyclictest1205114-21awk23:40:003
1108989993220,8cyclictest1753-21gnome-shell21:17:352
1108989993220,8cyclictest1753-21gnome-shell21:17:352
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional