You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 07:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Fri Nov 21, 2025 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
161439421080,4chrt1614007-21/usr/sbin/munin00:30:162
1623991050,9rtkit-daemon1622-21rtkit-daemon23:57:013
15000502970,19sleep20-21swapper/219:05:272
162399960,25rtkit-daemon3242254-21gmain23:13:161
162399930,35rtkit-daemon1753-21gnome-shell00:19:081
15883472890,7sleep32197014-21PK-Backend23:15:223
162399840,9rtkit-daemon1753-21gnome-shell00:18:000
150020928240,14sleep10-21swapper/119:05:431
150032027858,14sleep30-21swapper/319:06:383
150051226839,23sleep00-21swapper/019:09:040
15788532640,5chrt0-21swapper/222:50:022
162399620,10rtkit-daemon1622-21rtkit-daemon23:45:081
162399620,10rtkit-daemon1622-21rtkit-daemon23:43:321
1500733996220,37cyclictest1753-21gnome-shell19:40:013
1500733996220,37cyclictest1753-21gnome-shell19:40:013
162399610,8rtkit-daemon1622-21rtkit-daemon21:52:431
162399600,9rtkit-daemon1622-21rtkit-daemon19:55:261
162399590,9rtkit-daemon1622-21rtkit-daemon22:40:101
162399590,9rtkit-daemon1622-21rtkit-daemon22:00:050
162399590,9rtkit-daemon1622-21rtkit-daemon21:08:450
162399590,9rtkit-daemon1622-21rtkit-daemon21:08:450
162399590,9rtkit-daemon1622-21rtkit-daemon20:43:550
162399590,9rtkit-daemon1622-21rtkit-daemon00:36:160
162399590,9rtkit-daemon1622-21rtkit-daemon00:22:281
162399580,9rtkit-daemon1622-21rtkit-daemon23:51:590
162399580,9rtkit-daemon1622-21rtkit-daemon19:32:010
162399580,9rtkit-daemon1622-21rtkit-daemon00:35:421
162399570,9rtkit-daemon1622-21rtkit-daemon20:45:591
162399570,9rtkit-daemon1622-21rtkit-daemon20:38:291
162399570,8rtkit-daemon1622-21rtkit-daemon19:43:120
1500728995720,32cyclictest1568632-21df22:20:182
1500728995720,32cyclictest1567031-21open_files22:15:302
162399560,9rtkit-daemon1622-21rtkit-daemon21:52:140
162399560,9rtkit-daemon1622-21rtkit-daemon00:12:051
162399560,8rtkit-daemon1622-21rtkit-daemon22:08:381
162399560,8rtkit-daemon1622-21rtkit-daemon21:25:570
1500733995623,12cyclictest438-20systemd-journal21:20:003
1500733995618,12cyclictest438-20systemd-journal19:40:023
1500733995616,11cyclictest1612820-21grep00:25:223
1500728995623,29cyclictest1526951-21sshd20:21:592
162399550,9rtkit-daemon1622-21rtkit-daemon22:58:461
162399550,9rtkit-daemon1622-21rtkit-daemon22:58:461
162399550,9rtkit-daemon1622-21rtkit-daemon21:43:520
162399550,9rtkit-daemon1622-21rtkit-daemon21:39:361
15732912550,7chrt855-21dbus-daemon22:31:570
1500728995520,31cyclictest1605815-21awk00:05:262
162399540,8rtkit-daemon1622-21rtkit-daemon21:00:541
15464552540,7chrt1546411-21systemd-udevd21:17:590
1500733995419,7cyclictest1536124-21sort20:50:123
162399530,9rtkit-daemon1622-21rtkit-daemon20:06:370
150073399532,32cyclictest1753-21gnome-shell23:27:323
1500733995223,25cyclictest1753-21gnome-shell20:37:473
1500733995221,7cyclictest1753-21gnome-shell00:35:243
1500733995219,13cyclictest1569780-21apt-get22:25:003
150073399512,14cyclictest1753-21gnome-shell19:54:433
1500733995119,13cyclictest1514335-21ntpq19:45:233
1500733995117,13cyclictest1607185-21awk00:10:203
1500733995117,10cyclictest1517376-21expr19:55:153
1500728995117,29cyclictest1536036-21ls20:50:012
162399500,6rtkit-daemon1622-21rtkit-daemon20:21:251
150073399509,16cyclictest1753-21gnome-shell21:21:213
150073399508,24cyclictest1753-21gnome-shell00:07:383
150073399504,13cyclictest17531gnome-shell20:45:133
150073399502,28cyclictest1753-21gnome-shell21:10:063
150073399502,27cyclictest1753-21gnome-shell20:03:303
150073399502,27cyclictest1753-21gnome-shell20:03:293
1500733995017,8cyclictest1583213-21tr23:00:243
1500733995015,14cyclictest3242254-21gmain22:15:013
1500733995015,14cyclictest1753-21gnome-shell23:31:563
1500733995015,11cyclictest1611290-21awk00:20:263
1500733995015,10cyclictest1560604-21sed22:00:013
1500733995013,12cyclictest1550864-21uname21:30:243
150072899509,17cyclictest1753-21gnome-shell00:24:042
1500728995019,26cyclictest1069-21runrttasks20:05:452
1500728995017,27cyclictest1519579-21open_inodes20:00:282
1500728995017,27cyclictest1519579-21open_inodes20:00:282
1500728995010,16cyclictest1753-21gnome-shell21:01:292
162399490,6rtkit-daemon1622-21rtkit-daemon21:07:421
162399490,6rtkit-daemon1622-21rtkit-daemon21:07:411
1500733994923,7cyclictest1069-21runrttasks23:45:383
150073399492,25cyclictest1753-21gnome-shell20:05:093
1500733994915,13cyclictest42-21ksoftirqd/322:03:453
1500733994914,11cyclictest1570860-21sort22:25:233
1500728994918,27cyclictest1613877-21latency_hist00:30:002
1500728994915,11cyclictest1595190-21cat23:35:252
1500728994915,10cyclictest1590039-21wc23:20:262
150072899491,25cyclictest1753-21gnome-shell22:55:152
150072899491,25cyclictest1753-21gnome-shell22:55:152
1500725994925,20cyclictest1565106-21python322:10:341
15896982480,14sleep3411rcuc/323:20:173
15649812480,8sleep01753-21gnome-shell22:10:310
150073399489,14cyclictest1753-21gnome-shell21:35:133
150073399482,26cyclictest1753-21gnome-shell23:05:183
150073399482,26cyclictest1753-21gnome-shell20:59:513
150073399482,25cyclictest1753-21gnome-shell21:46:543
1500733994817,7cyclictest1585922-21/usr/sbin/munin23:10:283
1500733994817,6cyclictest1601295-21latency_hist23:55:013
1500733994817,6cyclictest1577212-21latency_hist22:45:023
1500733994816,10cyclictest42-21ksoftirqd/319:23:413
1500733994813,10cyclictest1753-21gnome-shell00:20:013
150073399481,26cyclictest1753-21gnome-shell19:10:223
1500733994811,12cyclictest1577421-21grep22:45:143
150072899488,17cyclictest1753-21gnome-shell00:04:162
150072899482,24cyclictest1753-21gnome-shell20:43:332
1500728994820,25cyclictest1753-21gnome-shell23:18:372
1500728994817,26cyclictest1550943-21load21:30:242
1500728994816,11cyclictest1552135-21cpu21:35:142
150072899481,24cyclictest1753-21gnome-shell22:01:032
1500728994810,18cyclictest1753-21gnome-shell22:28:482
1500728994810,11cyclictest1536382-21cat20:50:152
162399470,13rtkit-daemon881143-21gmain20:52:091
15063912470,7sleep02197014-21PK-Backend19:25:010
150073399478,19cyclictest0-21swapper/319:25:213
150073399472,8cyclictest1753-21gnome-shell23:35:163
150073399472,7cyclictest1753-21gnome-shell21:40:323
1500733994718,8cyclictest1753-21gnome-shell22:15:303
1500733994715,11cyclictest1548769-21grep21:25:133
1500733994714,11cyclictest1575213-21seq22:39:213
1500733994711,11cyclictest1526767-21sort20:20:323
1500733994710,17cyclictest1753-21gnome-shell21:53:093
150072899479,17cyclictest1753-21gnome-shell19:58:092
150072899478,19cyclictest1753-21gnome-shell23:34:212
150072899478,15cyclictest1753-21gnome-shell19:51:402
150072899477,16cyclictest1753-21gnome-shell21:23:132
150072899476,16cyclictest1753-21gnome-shell20:55:332
1500728994721,22cyclictest1753-21gnome-shell22:05:412
1500728994720,23cyclictest1753-21gnome-shell19:37:112
1500728994720,23cyclictest1753-21gnome-shell19:37:112
1500728994719,24cyclictest1753-21gnome-shell00:15:232
1500728994716,26cyclictest2197014-21PK-Backend00:35:392
1500728994714,10cyclictest1607332-21cat00:10:232
1500728994710,8cyclictest0-21swapper/223:00:302
1500728994710,17cyclictest1753-21gnome-shell22:12:552
1500728994710,17cyclictest1753-21gnome-shell21:27:132
1500728994710,17cyclictest1753-21gnome-shell21:11:122
1500728994710,17cyclictest1753-21gnome-shell19:47:522
150073399469,17cyclictest1753-21gnome-shell22:52:363
150073399469,17cyclictest1753-21gnome-shell22:32:513
150073399462,9cyclictest1753-21gnome-shell20:29:463
150073399462,25cyclictest1753-21gnome-shell20:10:363
150073399462,25cyclictest1603912-21memory00:00:243
150072899469,18cyclictest1753-21gnome-shell22:44:402
150072899469,16cyclictest1753-21gnome-shell22:35:082
150072899468,18cyclictest1753-21gnome-shell23:46:212
150072899468,18cyclictest1753-21gnome-shell21:55:092
150072899467,19cyclictest1753-21gnome-shell22:53:132
150072899467,18cyclictest1753-21gnome-shell21:05:032
150072899467,18cyclictest1753-21gnome-shell21:05:032
150072899467,16cyclictest1753-21gnome-shell23:54:192
1500728994619,22cyclictest34-21ksoftirqd/221:50:172
1500728994616,10cyclictest1601619-21cpuspeed23:55:152
1500728994613,10cyclictest1591352-21expr23:25:142
1500728994612,6cyclictest1506503-21latency_hist19:25:012
1500728994610,18cyclictest1753-21gnome-shell21:40:572
1500728994610,17cyclictest1753-21gnome-shell23:06:342
1500728994610,16cyclictest1753-21gnome-shell19:27:502
15135842450,6sleep1251rcuc/119:45:101
150073399459,18cyclictest1753-21gnome-shell22:07:023
150073399459,16cyclictest1753-21gnome-shell22:58:583
150073399459,16cyclictest1753-21gnome-shell22:58:583
150073399459,16cyclictest1753-21gnome-shell19:20:023
150073399459,16cyclictest1753-21gnome-shell19:20:013
150073399452,8cyclictest1753-21gnome-shell00:30:183
150073399451,24cyclictest1753-21gnome-shell21:06:433
150073399451,24cyclictest1753-21gnome-shell21:06:433
1500733994510,16cyclictest1753-21gnome-shell19:33:503
150072899458,18cyclictest1753-21gnome-shell21:47:042
150072899457,19cyclictest1753-21gnome-shell19:11:592
1500728994513,6cyclictest0-21swapper/220:35:002
150072899451,24cyclictest1753-21gnome-shell20:29:272
1500728994510,16cyclictest1753-21gnome-shell21:17:092
1500728994510,16cyclictest1753-21gnome-shell20:39:362
1500728994510,16cyclictest1753-21gnome-shell19:34:382
1500723994527,13cyclictest1603244-21systemd00:00:040
1500733994410,16cyclictest1753-21gnome-shell20:19:443
150072899449,16cyclictest1753-21gnome-shell19:41:342
150072899448,17cyclictest1753-21gnome-shell20:18:292
150072899442,14cyclictest1503125-21/usr/sbin/munin19:15:122
150072899442,14cyclictest1503125-21/usr/sbin/munin19:15:122
1500725994417,23cyclictest1603063-40ogrotate00:00:011
1500725994413,27cyclictest1605433-21/usr/sbin/munin00:05:151
162399430,35rtkit-daemon1500715-21cyclictest23:35:341
150073399431,23cyclictest1753-21gnome-shell21:01:353
1500728994313,7cyclictest0-21swapper/223:11:132
1500728994312,8cyclictest0-21swapper/222:33:322
150072599435,11cyclictest26-21ksoftirqd/100:34:451
1500723994326,12cyclictest438-20systemd-journal22:19:330
162399420,8rtkit-daemon1903-21packagekitd23:26:511
1500733994213,6cyclictest15967582hddtemp_smartct23:40:203
1500728994211,9cyclictest1617-10pipewire-media-20:10:192
1500725994221,7cyclictest1069-21runrttasks23:05:561
150072599421,38cyclictest0-21swapper/123:20:001
150072399425,33cyclictest13-21ksoftirqd/023:48:250
162399410,8rtkit-daemon1785-21gmain00:02:000
162399410,35rtkit-daemon921-21thermald22:27:381
162399410,33rtkit-daemon2197014-21PK-Backend19:27:350
1500723994124,12cyclictest855-21dbus-daemon22:55:010
150072899401,12cyclictest1597571-21seq23:42:042
1500725994028,9cyclictest26-21ksoftirqd/121:17:541
1500725994025,11cyclictest1523721-21cron20:15:001
1500725994019,6cyclictest1580487-21latency_hist22:55:001
1500723994023,12cyclictest1611977-21sshd00:24:330
1500723994022,13cyclictest438-20systemd-journal19:55:000
1500723994022,13cyclictest1608287-21pool-org.gnome.00:15:010
150072399401,35cyclictest1-21systemd21:40:000
1500725993928,8cyclictest1753-21gnome-shell20:34:551
1500725993928,7cyclictest1753-21gnome-shell22:30:261
1500725993921,14cyclictest251rcuc/121:40:261
1500725993920,15cyclictest1069-21runrttasks20:43:051
1500725993910,12cyclictest1559220-21cut21:55:151
1500723993925,10cyclictest13-21ksoftirqd/019:35:240
1500723993925,10cyclictest13-21ksoftirqd/019:35:230
1500723993924,11cyclictest1574576-21/usr/sbin/munin22:35:280
150072399392,33cyclictest1535166-21ntp_states20:45:290
150072399391,10cyclictest1615656-21latency_hist00:35:020
162399380,30rtkit-daemon907-21udisksd22:01:561
150073399382,8cyclictest1753-21gnome-shell20:30:093
1500725993827,7cyclictest1753-21gnome-shell22:38:251
1500725993826,8cyclictest1753-21gnome-shell22:15:251
150072599382,32cyclictest1753-21gnome-shell20:00:331
150072599382,32cyclictest1753-21gnome-shell20:00:331
1500725993820,14cyclictest1543754-21df_inode21:10:151
1500725993816,7cyclictest1589830-21idleruntime23:20:231
150072599381,34cyclictest1578566-21taskset22:47:511
150072399386,10cyclictest1069-21runrttasks21:56:570
1500723993825,8cyclictest1576182-21cat22:40:240
1500723993823,11cyclictest1597805-21cron23:45:010
1500723993821,13cyclictest1563947-21cron22:10:000
1500723993821,13cyclictest1529925-21ntp_kernel_err20:30:300
1500723993819,13cyclictest468-21systemd-udevd22:45:500
162399370,29rtkit-daemon1938-21gsd-housekeepin20:15:260
150072599374,30cyclictest26-21ksoftirqd/121:46:341
1500725993726,7cyclictest1753-21gnome-shell19:30:181
1500725993722,11cyclictest855-21dbus-daemon21:31:561
1500725993712,9cyclictest1510732-21missed_timers19:35:241
1500725993712,9cyclictest1510732-21missed_timers19:35:241
1500725993712,21cyclictest1521288-21seq20:05:261
1500725993711,20cyclictest1827-21JS14:31:441
1500723993720,12cyclictest1587205-21sshd23:12:330
1500723993719,13cyclictest3242255-21gdbus23:20:010
162399360,29rtkit-daemon881143-21gmain20:57:170
162399360,28rtkit-daemon1506-21systemd20:24:060
162399360,28rtkit-daemon1506-21systemd00:05:110
1500725993626,7cyclictest1753-21gnome-shell21:24:341
1500725993625,7cyclictest1753-21gnome-shell19:50:291
1500725993625,7cyclictest1753-21gnome-shell19:25:271
1500725993625,7cyclictest1753-21gnome-shell19:15:261
1500725993625,7cyclictest1753-21gnome-shell19:15:261
1500725993619,12cyclictest26-21ksoftirqd/119:40:271
1500725993618,7cyclictest1582694-21cstates23:00:141
150072399367,25cyclictest1585682-21latency_hist23:10:020
150072399362,16cyclictest1541066-21cron21:05:000
150072399361,11cyclictest1543971-21chrt21:10:220
162399350,29rtkit-daemon0-21swapper/119:20:591
162399350,29rtkit-daemon0-21swapper/020:01:250
162399350,29rtkit-daemon0-21swapper/020:01:240
162399350,27rtkit-daemon1753-21gnome-shell22:55:550
162399350,27rtkit-daemon1753-21gnome-shell22:55:550
162399350,27rtkit-daemon1506-21systemd20:35:070
1500725993525,7cyclictest1753-21gnome-shell21:26:271
1500725993524,7cyclictest1753-21gnome-shell22:20:281
1500725993520,11cyclictest1601274-21cron23:55:001
1500725993517,13cyclictest1501564-21cat19:10:251
1500725993515,5cyclictest26-21ksoftirqd/120:55:181
150072399357,24cyclictest1515150-21wc19:50:010
1500723993524,7cyclictest1753-21gnome-shell19:16:580
1500723993524,7cyclictest1753-21gnome-shell19:16:580
1500723993521,9cyclictest1753-21gnome-shell23:38:180
1500723993519,11cyclictest1555658-21cpuspeed_turbos21:45:150
1500723993519,11cyclictest1500591-21cron19:10:020
162399340,26rtkit-daemon1753-21gnome-shell20:52:280
1500725993424,6cyclictest26-21ksoftirqd/100:00:221
1500725993419,11cyclictest1527838-21sh20:25:171
1500725993417,12cyclictest1613054-21ntpq00:25:271
150072399346,11cyclictest13-21ksoftirqd/022:26:250
1500723993423,7cyclictest1753-21gnome-shell23:03:120
150072399342,29cyclictest0-21swapper/023:33:430
1500723993420,10cyclictest1569831-21chmod22:25:010
1500723993419,11cyclictest1551539-21sshd21:31:330
1500723993417,12cyclictest438-20systemd-journal20:10:020
150072399341,30cyclictest0-21swapper/023:23:050
150073399331,29cyclictest2197014-21PK-Backend20:40:523
150072399332,5cyclictest0-21swapper/020:25:090
1500723993317,11cyclictest1517489-21idleruntime19:55:200
1500723993316,12cyclictest1547299-21which21:20:210
150072399331,28cyclictest2197014-21PK-Backend00:27:500
1500725993216,12cyclictest1593164-21expr23:30:141
150072399311,26cyclictest2197014-21PK-Backend23:28:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional