You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 15:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Nov 28, 2025 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991030,90rtkit-daemon0-21swapper/219:07:132
136307521030,8sleep02197014-21PK-Backend21:35:160
13633042980,8sleep234-21ksoftirqd/221:35:232
162399860,58rtkit-daemon2197014-21PK-Backend22:32:300
13580352810,7sleep02197014-21PK-Backend21:20:140
13923262780,7sleep21753-21gnome-shell22:55:312
162399760,62rtkit-daemon0-21swapper/319:05:103
162399760,5rtkit-daemon0-21swapper/219:10:492
1310381997310,37cyclictest1800-21upowerd23:09:192
131003227340,9sleep00-21swapper/019:07:180
13687652720,17sleep1251rcuc/121:50:221
131038199703,36cyclictest860997-21snmpd00:21:142
162399690,5rtkit-daemon0-21swapper/223:54:242
131038199696,35cyclictest1376425-21awk22:10:332
1310381996925,39cyclictest438-20systemd-journal23:03:072
131038199682,41cyclictest1753-21gnome-shell23:25:172
131038199682,41cyclictest1753-21gnome-shell23:25:172
1310381996816,28cyclictest1069-21runrttasks22:21:062
131008826840,22sleep10-21swapper/119:07:591
13733042660,14chrt26-21ksoftirqd/122:04:561
13733042660,14chrt26-21ksoftirqd/122:04:551
131038199663,37cyclictest1753-21gnome-shell21:20:222
162399650,39rtkit-daemon881143-21gmain23:09:560
131038199658,33cyclictest1069-21runrttasks19:50:522
131038199652,38cyclictest1368439-21/usr/sbin/munin21:50:152
1310381996419,27cyclictest1069-21runrttasks20:43:512
131038199632,37cyclictest1753-21gnome-shell22:41:022
131038199632,36cyclictest1753-21gnome-shell22:54:542
131038199632,34cyclictest1753-21gnome-shell22:31:092
131038199622,37cyclictest1753-21gnome-shell20:52:552
131038199622,36cyclictest1753-21gnome-shell23:56:242
131038199622,36cyclictest1753-21gnome-shell23:30:362
131038199622,36cyclictest1753-21gnome-shell22:39:052
131038199622,13cyclictest1328677-21apt20:00:132
131038199612,36cyclictest1753-21gnome-shell22:09:282
131038199612,36cyclictest1753-21gnome-shell21:14:272
131038199612,36cyclictest1753-21gnome-shell00:08:192
131038199612,35cyclictest1753-21gnome-shell23:42:302
131038199612,35cyclictest1753-21gnome-shell19:44:282
131038199612,35cyclictest1753-21gnome-shell00:32:132
131038199611,9cyclictest860997-21snmpd00:15:532
131038199611,11cyclictest1315746-21taskset19:21:522
162399600,8rtkit-daemon1622-21rtkit-daemon22:35:101
131038199609,47cyclictest1418911-21cat00:15:012
131038199602,35cyclictest1753-21gnome-shell22:03:342
131038199602,35cyclictest1753-21gnome-shell22:03:332
131038199602,35cyclictest1753-21gnome-shell00:36:112
131038199602,34cyclictest1753-21gnome-shell20:19:402
131038199602,34cyclictest1753-21gnome-shell19:56:162
131038199602,34cyclictest1753-21gnome-shell19:48:242
162399590,8rtkit-daemon1622-21rtkit-daemon00:03:391
131038199592,7cyclictest1753-21gnome-shell21:59:032
1310381995925,29cyclictest438-20systemd-journal23:17:192
131038199592,34cyclictest1753-21gnome-shell21:28:102
131038199591,7cyclictest1337673-21taskset20:25:072
131038199591,52cyclictest1320155-21which19:35:202
131038199582,33cyclictest0-21swapper/222:45:332
162399550,9rtkit-daemon1622-21rtkit-daemon23:16:521
162399550,6rtkit-daemon1622-21rtkit-daemon00:14:571
131038199553,48cyclictest1069-21runrttasks23:14:062
1310381995518,32cyclictest1423942-21sort00:25:352
131038199551,7cyclictest1346228-21taskset20:47:542
1310381995516,34cyclictest1409296-21cat23:45:242
162399540,9rtkit-daemon1622-21rtkit-daemon22:48:461
162399540,9rtkit-daemon1622-21rtkit-daemon22:09:371
162399540,9rtkit-daemon1622-21rtkit-daemon19:31:011
162399540,9rtkit-daemon1622-21rtkit-daemon19:25:131
162399540,9rtkit-daemon1622-21rtkit-daemon00:33:211
131038199549,41cyclictest1753-21gnome-shell20:24:572
131038199541,10cyclictest1378857-21date22:20:032
162399530,8rtkit-daemon1622-21rtkit-daemon20:16:171
1310381995318,31cyclictest1753-21gnome-shell21:30:152
131038199531,47cyclictest1753-21gnome-shell23:20:042
1310381995314,34cyclictest1349356-21needreboot20:55:242
162399520,8rtkit-daemon1622-21rtkit-daemon22:41:421
162399520,8rtkit-daemon1622-21rtkit-daemon21:06:361
162399520,8rtkit-daemon1622-21rtkit-daemon20:25:371
162399520,8rtkit-daemon1622-21rtkit-daemon19:41:131
162399520,6rtkit-daemon1622-21rtkit-daemon00:37:051
1310386995214,33cyclictest1358-1kworker/u9:120:00:123
1310381995219,29cyclictest1353162-21sed21:05:322
1310381995218,29cyclictest1414483-21cat00:00:252
131038199521,6cyclictest1319306-21taskset19:32:592
131038199521,47cyclictest1753-21gnome-shell21:49:182
1310381995212,35cyclictest3242255-21gdbus20:35:032
1310381995210,37cyclictest1753-21gnome-shell20:07:262
131038199512,9cyclictest1753-21gnome-shell19:15:062
162399500,25rtkit-daemon2197014-21PK-Backend19:35:491
131038699508,15cyclictest1321012-21taskset19:37:363
1310386995010,12cyclictest1310942-21expr19:10:153
131038199508,38cyclictest1753-21gnome-shell19:28:102
131038199504,41cyclictest1365070-21iwlist21:40:202
1310381995018,27cyclictest1355982-21cpu21:15:132
1310381995012,33cyclictest1340440-21cat20:30:312
1310381995012,33cyclictest1340440-21cat20:30:312
162399490,39rtkit-daemon2197014-21PK-Backend20:37:330
1310386994920,25cyclictest1317877-21sed19:30:023
131038199497,37cyclictest1753-21gnome-shell23:35:282
1310381994917,27cyclictest1332406-21cstates20:10:152
131038699489,10cyclictest1366170-21ls21:45:023
131038699487,14cyclictest1337669-21taskset20:25:043
131038699486,15cyclictest1411787-21sshd23:54:193
1310386994830,14cyclictest1355405-40apt-check21:12:133
1310381994812,31cyclictest1753-21gnome-shell22:25:152
162399470,40rtkit-daemon2007-21evolution-alarm22:50:181
162399470,37rtkit-daemon2197014-21PK-Backend23:03:240
131038699478,12cyclictest1337098-40lsb_release20:21:203
131038699478,11cyclictest1376243-21sed22:10:313
131038699478,11cyclictest1350706-21cstates21:00:203
131038699477,14cyclictest1420364-21taskset00:17:173
131038699477,11cyclictest1371720-21cut22:00:023
131038699474,18cyclictest1414294-21seq00:00:173
1310386994710,18cyclictest1753-21gnome-shell22:21:223
131038699470,30cyclictest1753-21gnome-shell20:47:513
162399460,39rtkit-daemon881143-21gmain21:15:561
14044542460,6chrt0-21swapper/323:30:303
131038699469,19cyclictest1753-21gnome-shell23:29:253
131038699469,19cyclictest1753-21gnome-shell23:29:243
131038699469,12cyclictest1753-21gnome-shell22:15:583
131038699466,14cyclictest1387297-21taskset22:42:383
131038199461,7cyclictest1350442-21basename21:00:162
1310377994624,18cyclictest1413671-40mandb00:00:021
13121162450,5sleep02197014-21PK-Backend19:15:010
131038699459,14cyclictest1330462-21/usr/sbin/munin20:05:123
131038699458,11cyclictest1335015-21users20:15:313
1310386994526,15cyclictest2179-21gvfsd-metadata00:00:003
162399440,37rtkit-daemon881143-21gmain22:15:241
131038699449,10cyclictest854-21avahi-daemon22:10:003
131038699449,10cyclictest1368443-21cat21:50:163
131038699447,14cyclictest1312782-21taskset19:15:093
131038699441,26cyclictest1327266-21memory19:55:213
1310377994425,15cyclictest3242253-21nautilus21:50:011
1310373994422,6cyclictest1373330-21turbostat22:05:000
1310373994422,6cyclictest1373330-21turbostat22:05:000
162399430,33rtkit-daemon1753-21gnome-shell19:45:481
131038699439,11cyclictest1322353-21python319:40:283
131038699438,31cyclictest2179-21gvfsd-metadata21:50:003
131038699437,20cyclictest1315489-21sed19:20:343
131038699437,14cyclictest1409808-21runrttasks23:46:133
131038699434,18cyclictest1406369-21taskset23:36:323
1310386994327,12cyclictest855-21dbus-daemon23:09:183
1310386994326,12cyclictest438-20systemd-journal19:55:013
1310377994314,12cyclictest438-20systemd-journal00:15:191
13293022420,6sleep0161rcuc/020:00:290
131038699428,17cyclictest42-21ksoftirqd/322:58:403
131038699427,11cyclictest1753-21gnome-shell00:09:163
131038699427,11cyclictest1424355-21taskset00:29:323
131038699427,11cyclictest1352297-21/usr/sbin/munin21:05:163
131038699427,11cyclictest1350065-21taskset20:58:423
131038699427,11cyclictest1324351-21taskset19:46:023
131038699425,12cyclictest1422276-40apt-check00:21:153
131038699424,34cyclictest1427567-21sshd00:36:193
1310386994228,10cyclictest1362316-21sshd21:31:203
1310386994225,13cyclictest855-21dbus-daemon00:15:003
1310386994225,12cyclictest438-20systemd-journal22:39:203
1310386994221,8cyclictest42-21ksoftirqd/320:37:013
1310377994225,12cyclictest1339564-21grep20:30:151
1310377994225,12cyclictest1339564-21grep20:30:141
131038699418,11cyclictest1357202-21taskset21:17:263
131038699417,12cyclictest1360419-21cat21:25:333
1310386994124,12cyclictest438-20systemd-journal00:32:193
1310386994123,13cyclictest1390758-21systemd-udevd22:52:073
1310386994110,17cyclictest42-21ksoftirqd/323:16:503
1310377994122,15cyclictest3242253-21nautilus23:50:021
162399400,32rtkit-daemon1310368-21cyclictest00:26:211
162399400,14rtkit-daemon13-21ksoftirqd/000:00:310
13394392400,8sleep02197014-21PK-Backend20:30:130
13394392400,8sleep02197014-21PK-Backend20:30:120
131038699403,15cyclictest1753-21gnome-shell22:01:363
131038699403,15cyclictest1753-21gnome-shell22:01:363
131038699402,34cyclictest1753-21gnome-shell21:25:003
1310386994022,13cyclictest1388515-21ntp_states22:45:273
1310386994016,20cyclictest1332653-21/usr/sbin/munin20:10:213
131037799403,8cyclictest1753-21gnome-shell21:55:321
1310377994025,12cyclictest1359591-21tr21:25:171
131037799402,33cyclictest1407303-21diskstats23:40:181
1310377994023,13cyclictest1632-21dbus-daemon00:25:011
1310373994022,13cyclictest1355405-40apt-check21:12:180
162399390,32rtkit-daemon906-21gmain23:05:021
131038699397,17cyclictest1340416-21sed20:30:313
131038699397,17cyclictest1340416-21sed20:30:313
131038699395,11cyclictest42-21ksoftirqd/322:30:243
1310386993925,10cyclictest1344445-21sshd20:42:153
131038699392,33cyclictest1753-21gnome-shell21:38:273
1310377993923,12cyclictest1-21systemd23:32:471
1310377993915,12cyclictest855-21dbus-daemon21:05:011
1310373993922,13cyclictest1356824-21python321:15:280
162399380,30rtkit-daemon1310368-21cyclictest22:33:391
13678782380,6sleep02197014-21PK-Backend21:50:010
131038699386,10cyclictest854-21avahi-daemon19:32:463
131038699382,12cyclictest1753-21gnome-shell23:12:183
131038699382,11cyclictest1753-21gnome-shell23:00:183
131037799382,32cyclictest1753-21gnome-shell23:25:361
131037799382,32cyclictest1753-21gnome-shell23:25:361
1310377993822,12cyclictest1392772-21systemctl23:00:011
131037399384,30cyclictest1337109-40lsb_release20:21:220
1310373993823,11cyclictest1378165-21seq22:15:370
13326522370,15sleep126-21ksoftirqd/120:10:201
131038699372,7cyclictest1753-21gnome-shell22:26:173
1310386993724,10cyclictest42-21ksoftirqd/320:51:053
1310386993721,12cyclictest1400731-21ntpq23:20:263
131037799372,10cyclictest1350177-21perf21:00:011
1310377993720,7cyclictest1069-21runrttasks19:19:011
1310377993720,13cyclictest1355051-21python321:10:301
1310377993720,12cyclictest438-20systemd-journal22:25:011
1310377993719,6cyclictest26-21ksoftirqd/120:00:221
1310377993719,13cyclictest855-21dbus-daemon20:45:011
131037399372,31cyclictest1753-21gnome-shell22:10:480
131037399372,18cyclictest1753-21gnome-shell21:50:190
131037399371,16cyclictest1358-1kworker/u9:120:43:550
162399360,21rtkit-daemon2197014-21PK-Backend00:34:210
1310377993625,7cyclictest1753-21gnome-shell20:21:221
1310377993625,7cyclictest1405357-21tr23:35:131
131037799361,8cyclictest1417123-21perf00:10:001
1310373993621,11cyclictest1420629-21gpgv00:20:010
1310373993619,12cyclictest1389848-21expr22:50:150
1310373993615,16cyclictest1398015-21unin-run23:15:010
131037399361,14cyclictest1358-1kworker/u9:121:43:240
162399350,28rtkit-daemon26-21ksoftirqd/120:39:241
162399350,27rtkit-daemon1622-21rtkit-daemon19:24:131
131037799359,22cyclictest1331918-21sh20:10:011
131037799354,4cyclictest141rcu_preempt21:35:171
1310377993521,10cyclictest1410723-21expr23:50:171
1310377993520,11cyclictest1358244-21cat21:20:221
1310377993517,13cyclictest1361518-21df21:30:161
131037399357,11cyclictest1069-21runrttasks22:20:310
131037399357,11cyclictest0-21swapper/019:15:130
1310373993520,11cyclictest1421779-21uname00:20:240
131037399351,6cyclictest1753-21gnome-shell19:21:190
1310373993511,6cyclictest141rcu_preempt21:05:300
1310373993511,6cyclictest141rcu_preempt20:05:250
131037399351,16cyclictest113-1kworker/0:1H19:40:280
162399340,27rtkit-daemon26-21ksoftirqd/120:51:131
13271372340,7chrt0-21swapper/119:55:191
1310386993411,20cyclictest391irq_work/323:40:443
1310377993420,10cyclictest26-21ksoftirqd/119:53:311
1310377993419,11cyclictest1365361-21grep21:40:251
1310377993418,12cyclictest1376063-21ntpq22:10:251
131037399343,6cyclictest141rcu_preempt20:53:190
1310373993426,5cyclictest2197014-21PK-Backend00:09:550
1310373993425,6cyclictest2197014-21PK-Backend21:59:140
131037399342,28cyclictest0-21swapper/020:26:450
1310373993421,9cyclictest1753-21gnome-shell20:17:160
131037399341,9cyclictest2197014-21PK-Backend23:29:000
131037399341,9cyclictest2197014-21PK-Backend23:28:590
131037399341,9cyclictest2197014-21PK-Backend23:22:350
1310373993418,12cyclictest1407549-21grep23:40:240
1310373993418,12cyclictest1403452-21apt23:30:100
1310373993412,6cyclictest141rcu_preempt00:00:020
131037399341,14cyclictest2197014-21PK-Backend22:26:530
131037799334,15cyclictest26-21ksoftirqd/123:00:251
1310377993323,6cyclictest26-21ksoftirqd/120:45:141
131037799332,12cyclictest1753-21gnome-shell19:10:341
1310377993318,12cyclictest1069-21runrttasks22:25:301
1310373993325,5cyclictest2197014-21PK-Backend00:37:090
1310373993321,8cyclictest1349330-21needreboot20:55:240
1310373993321,8cyclictest1323911-21netstat19:45:230
131037399331,9cyclictest2197014-21PK-Backend00:10:290
1310373993319,10cyclictest1392728-21cron23:00:010
1310373993317,12cyclictest1326531-21latency_hist19:55:020
131037399331,4cyclictest2197014-21PK-Backend23:37:060
131037399331,13cyclictest2197014-21PK-Backend22:48:340
1310377993212,16cyclictest1397263-21grep23:10:251
131037399328,6cyclictest141rcu_preempt23:15:240
1310373993223,5cyclictest2197014-21PK-Backend22:40:170
1310373993222,6cyclictest1317745-21latency_hist19:30:000
1310373993221,7cyclictest1351470-21timerwakeupswit21:00:360
131037399321,4cyclictest2197014-21PK-Backend20:10:330
1310373993213,7cyclictest141rcu_preempt00:26:510
131037399321,28cyclictest2197014-21PK-Backend23:53:040
131037399321,28cyclictest2197014-21PK-Backend21:28:550
1310373993211,5cyclictest141rcu_preempt21:30:290
131037399321,11cyclictest0-21swapper/022:06:510
1310373993210,18cyclictest1753-21gnome-shell23:45:490
1310373993210,18cyclictest1753-21gnome-shell19:35:490
162399310,24rtkit-daemon2197014-21PK-Backend19:30:480
1310377993112,6cyclictest1753-21gnome-shell23:22:311
1310373993120,8cyclictest1753-21gnome-shell20:48:030
1310373993115,12cyclictest855-21dbus-daemon20:00:010
131037399301,14cyclictest2197014-21PK-Backend22:35:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional