You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-20 - 00:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Sat Jul 19, 2025 12:44:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
186711021060,6sleep02197014-21PK-Backend11:00:340
186711021060,6sleep02197014-21PK-Backend11:00:340
179122021060,5sleep20-21swapper/207:25:282
18940592970,8sleep31753-21gnome-shell12:17:323
18227542970,6sleep13776176-21snapd08:55:181
162399890,53rtkit-daemon0-21swapper/007:09:350
17957162880,5sleep32197014-21PK-Backend07:40:143
178332528644,33sleep30-21swapper/307:05:213
18173972800,7sleep32197014-21PK-Backend08:40:223
178418527846,23sleep20-21swapper/207:07:452
18968552740,4sleep20-21swapper/212:25:242
18968552740,4sleep20-21swapper/212:25:242
178449999690,65cyclictest1622-21rtkit-daemon09:12:173
178417426954,9sleep10-21swapper/107:07:371
1784495996643,6cyclictest34-21ksoftirqd/207:19:012
18208602630,16sleep342-21ksoftirqd/308:50:163
1784495995917,19cyclictest34-21ksoftirqd/207:15:022
178449299579,15cyclictest1825356-21sed09:00:321
178449299578,10cyclictest3242253-21nautilus07:55:021
178449299578,10cyclictest3242253-21nautilus07:55:011
178449299576,9cyclictest3242254-21gmain08:12:181
178449299576,9cyclictest3242254-21gmain08:12:171
178448999572,14cyclictest1753-21gnome-shell11:38:040
1784495995619,19cyclictest34-21ksoftirqd/210:45:012
1784495995617,19cyclictest34-21ksoftirqd/211:25:022
1784495995616,20cyclictest34-21ksoftirqd/208:20:162
1784495995615,21cyclictest34-21ksoftirqd/211:35:172
178449299569,22cyclictest1881554-21sensors11:40:311
178449299568,29cyclictest1854165-21df_inode10:25:211
178449299566,14cyclictest1861472-21ntp_states10:45:301
178449299565,23cyclictest1801643-21/usr/sbin/munin07:55:241
18884852550,5sleep1251rcuc/112:00:321
1784499995517,8cyclictest42-21ksoftirqd/311:06:173
178449299559,19cyclictest1840808-21python309:45:311
178449299559,10cyclictest1886968-21users11:55:351
1784499995429,8cyclictest42-21ksoftirqd/307:26:373
178449599549,6cyclictest141rcu_preempt11:15:012
1784495995414,20cyclictest34-21ksoftirqd/208:15:022
1784495995414,20cyclictest34-21ksoftirqd/208:15:012
1784495995413,21cyclictest34-21ksoftirqd/212:00:022
178449299549,17cyclictest1753-21gnome-shell12:08:351
178449299549,17cyclictest1753-21gnome-shell12:08:351
178449299548,10cyclictest1753-21gnome-shell07:11:041
178449299544,22cyclictest1845614-21irqstats10:00:251
178449299540,30cyclictest1866793-40lsb_release11:00:261
178449299540,30cyclictest1866793-40lsb_release11:00:261
17937022530,6sleep30-21swapper/307:35:013
1784499995321,9cyclictest42-21ksoftirqd/311:10:353
1784499995318,19cyclictest42-21ksoftirqd/308:55:043
1784495995343,6cyclictest34-21ksoftirqd/207:45:142
1784495995333,6cyclictest859-21NetworkManager08:15:582
1784495995312,22cyclictest34-21ksoftirqd/209:05:242
1784495995312,21cyclictest34-21ksoftirqd/211:15:272
178449299538,16cyclictest1753-21gnome-shell10:31:511
178449299538,12cyclictest1828290-21irqstats09:10:211
178449299538,10cyclictest1798949-21munin-run07:50:011
178449299537,9cyclictest855-21dbus-daemon09:20:011
178449299536,20cyclictest1857570-21expr10:35:151
178449299535,27cyclictest1753-21gnome-shell11:50:121
1784499995219,8cyclictest42-21ksoftirqd/311:45:523
1784499995218,9cyclictest42-21ksoftirqd/311:28:253
1784499995218,8cyclictest42-21ksoftirqd/309:02:393
1784499995212,11cyclictest42-21ksoftirqd/311:43:263
1784499995211,19cyclictest42-21ksoftirqd/308:00:013
1784495995224,24cyclictest1822756-21cpuspeed_turbos08:55:172
1784495995214,19cyclictest34-21ksoftirqd/212:10:022
1784495995214,19cyclictest34-21ksoftirqd/212:10:022
178449299529,20cyclictest1753-21gnome-shell10:59:391
178449299528,23cyclictest1753-21gnome-shell11:26:301
1784489995230,7cyclictest13-21ksoftirqd/008:08:500
178449999518,13cyclictest42-21ksoftirqd/309:40:223
1784499995120,10cyclictest42-21ksoftirqd/310:05:013
1784495995127,20cyclictest1753-21gnome-shell07:20:232
1784495995126,21cyclictest1753-21gnome-shell08:00:292
1784495995110,20cyclictest34-21ksoftirqd/207:45:012
178449299519,19cyclictest1753-21gnome-shell07:25:251
178449299517,17cyclictest1753-21gnome-shell08:28:431
178449299517,10cyclictest1894586-21grep12:20:141
178449299515,22cyclictest1837559-21users09:35:321
178449299515,10cyclictest1753-21gnome-shell09:33:221
178449299515,10cyclictest1753-21gnome-shell09:33:221
178449299514,28cyclictest1753-21gnome-shell11:31:111
178449299511,12cyclictest1753-21gnome-shell08:45:011
178448999512,6cyclictest141rcu_preempt09:25:380
1784489995114,20cyclictest1832399-21sshd09:20:510
178449999504,20cyclictest42-21ksoftirqd/310:05:253
1784499995015,10cyclictest42-21ksoftirqd/312:01:133
1784499995011,19cyclictest42-21ksoftirqd/310:10:333
1784499995011,19cyclictest42-21ksoftirqd/310:10:323
1784495995012,19cyclictest34-21ksoftirqd/212:25:022
1784495995012,19cyclictest34-21ksoftirqd/208:40:302
178449299509,20cyclictest1753-21gnome-shell09:09:021
178449299509,19cyclictest1753-21gnome-shell11:46:181
178449299505,21cyclictest1753-21gnome-shell11:08:301
178449299505,14cyclictest1753-21gnome-shell11:24:531
178449299503,27cyclictest1753-21gnome-shell11:10:531
1784492995010,19cyclictest1753-21gnome-shell11:16:241
178449299500,26cyclictest1808458-21apt08:15:131
178449299500,11cyclictest1753-21gnome-shell09:40:291
178448999502,9cyclictest141rcu_preempt11:48:170
1784489995013,33cyclictest13-21ksoftirqd/011:45:010
178449999498,22cyclictest42-21ksoftirqd/309:55:553
178449999498,22cyclictest42-21ksoftirqd/309:55:553
178449999497,5cyclictest141rcu_preempt08:25:163
178449999497,23cyclictest141rcu_preempt08:00:163
1784499994917,15cyclictest42-21ksoftirqd/312:31:303
1784499994916,10cyclictest42-21ksoftirqd/310:40:293
178449599499,20cyclictest34-21ksoftirqd/208:05:302
178449599499,20cyclictest34-21ksoftirqd/207:50:152
178449599499,20cyclictest34-21ksoftirqd/207:50:142
1784495994936,9cyclictest34-21ksoftirqd/211:10:022
1784495994910,19cyclictest34-21ksoftirqd/212:35:312
178449299499,8cyclictest1819219-21df_inode08:45:171
178449299498,20cyclictest1753-21gnome-shell09:54:571
178449299498,20cyclictest1753-21gnome-shell09:54:571
178449299498,20cyclictest1753-21gnome-shell08:04:531
178449299497,13cyclictest1753-21gnome-shell08:30:271
178449299497,12cyclictest1753-21gnome-shell12:19:171
178449299497,11cyclictest1753-21gnome-shell10:43:321
178449299496,23cyclictest1753-21gnome-shell07:41:141
178449299496,11cyclictest1849342-21ntp_states10:10:271
178449299496,11cyclictest1849342-21ntp_states10:10:271
178449299494,23cyclictest1753-21gnome-shell09:23:311
178449299493,14cyclictest1847731-21python310:05:301
178449299492,14cyclictest1805060-21df_inode08:05:171
1784492994910,19cyclictest1753-21gnome-shell08:36:151
1784492994910,19cyclictest1753-21gnome-shell07:36:471
1784492994910,12cyclictest1753-21gnome-shell12:30:301
1784489994928,8cyclictest13-21ksoftirqd/011:53:440
1784489994924,16cyclictest13-21ksoftirqd/009:05:110
178449999482,20cyclictest42-21ksoftirqd/307:50:253
178449999482,20cyclictest42-21ksoftirqd/307:50:253
1784499994818,8cyclictest42-21ksoftirqd/310:20:023
178449999481,6cyclictest2197014-21PK-Backend08:10:013
1784499994816,8cyclictest42-21ksoftirqd/309:30:343
1784499994816,8cyclictest42-21ksoftirqd/309:30:333
178449599488,20cyclictest34-21ksoftirqd/210:45:252
1784495994814,18cyclictest34-21ksoftirqd/211:00:022
178449299489,18cyclictest1753-21gnome-shell12:29:011
178449299489,18cyclictest1753-21gnome-shell12:29:011
178449299489,18cyclictest1753-21gnome-shell10:17:541
178449299489,10cyclictest1833758-21ntp_states09:25:291
178449299488,19cyclictest1753-21gnome-shell07:32:121
178449299487,17cyclictest1852428-21df_inode10:20:181
178449299487,13cyclictest1811868-21cat08:25:021
178449299486,20cyclictest1891442-21df_inode12:10:201
178449299486,16cyclictest1753-21gnome-shell11:38:161
178449299482,7cyclictest26-21ksoftirqd/107:18:321
1784492994810,17cyclictest1753-21gnome-shell09:55:251
1784492994810,17cyclictest1753-21gnome-shell09:55:241
178448999481,15cyclictest18099002cat08:19:560
1784499994721,8cyclictest42-21ksoftirqd/311:15:583
178449999471,23cyclictest438-20systemd-journal08:15:003
178449999471,23cyclictest438-20systemd-journal08:15:003
178449599477,20cyclictest34-21ksoftirqd/210:50:212
1784495994716,10cyclictest34-21ksoftirqd/209:20:032
1784495994711,19cyclictest34-21ksoftirqd/209:20:192
1784495994710,14cyclictest34-21ksoftirqd/208:45:302
1784495994710,14cyclictest141rcu_preempt12:15:132
1784495994710,13cyclictest34-21ksoftirqd/209:30:242
1784495994710,13cyclictest34-21ksoftirqd/209:30:242
178449299479,18cyclictest1753-21gnome-shell12:37:441
178449299479,17cyclictest1753-21gnome-shell10:51:501
1784492994721,23cyclictest1822217-21sed08:55:021
178449299471,14cyclictest1753-21gnome-shell07:24:271
178448999473,8cyclictest141rcu_preempt07:15:320
1784489994731,6cyclictest13-21ksoftirqd/008:35:260
1784489994724,13cyclictest13-21ksoftirqd/012:00:000
162399470,32rtkit-daemon1753-21gnome-shell08:40:220
178449999462,8cyclictest42-21ksoftirqd/311:00:013
178449999461,6cyclictest2197014-21PK-Backend07:18:323
1784499994611,17cyclictest42-21ksoftirqd/309:40:013
1784499994610,6cyclictest141rcu_preempt10:55:023
1784495994622,19cyclictest1753-21gnome-shell11:25:242
1784495994612,19cyclictest34-21ksoftirqd/211:00:052
1784495994612,19cyclictest34-21ksoftirqd/211:00:042
1784489994620,9cyclictest13-21ksoftirqd/011:10:010
1784489994620,10cyclictest13-21ksoftirqd/011:25:010
18110582450,8chrt2197014-21PK-Backend08:20:313
178449999454,21cyclictest42-21ksoftirqd/312:27:333
178449999454,21cyclictest42-21ksoftirqd/312:27:323
1784499994513,9cyclictest42-21ksoftirqd/312:09:523
1784499994513,9cyclictest42-21ksoftirqd/312:09:513
178449999451,20cyclictest42-21ksoftirqd/311:20:313
1784499994510,18cyclictest42-21ksoftirqd/310:30:213
178449599456,19cyclictest34-21ksoftirqd/212:00:142
1784495994519,9cyclictest34-21ksoftirqd/210:35:012
1784495994517,10cyclictest34-21ksoftirqd/211:40:182
1784495994517,10cyclictest34-21ksoftirqd/210:20:242
1784489994528,6cyclictest13-21ksoftirqd/012:28:170
1784489994528,6cyclictest13-21ksoftirqd/012:28:160
1784489994526,15cyclictest13-21ksoftirqd/010:18:410
178448999452,18cyclictest0-21swapper/007:34:530
1784489994517,13cyclictest141rcu_preempt11:20:000
1784489994516,24cyclictest1803652-40lsb_release08:00:280
162399450,31rtkit-daemon1753-21gnome-shell07:48:080
18414572440,7sleep32197014-21PK-Backend09:50:013
178449999447,16cyclictest42-21ksoftirqd/309:30:023
178449999442,13cyclictest42-21ksoftirqd/308:50:013
1784499994414,11cyclictest141rcu_preempt09:16:543
178449599441,21cyclictest34-21ksoftirqd/212:10:172
1784489994425,14cyclictest13-21ksoftirqd/010:29:420
1784489994425,14cyclictest13-21ksoftirqd/008:45:260
1784489994424,15cyclictest13-21ksoftirqd/009:59:480
1784489994424,15cyclictest13-21ksoftirqd/009:59:480
1784489994416,13cyclictest141rcu_preempt08:50:300
178449999438,13cyclictest1815058-21taskset08:34:263
178449999435,18cyclictest42-21ksoftirqd/309:20:303
178449999434,18cyclictest42-21ksoftirqd/311:39:373
178449999432,25cyclictest2197014-21PK-Backend09:07:083
178449999432,18cyclictest42-21ksoftirqd/307:40:003
1784499994317,16cyclictest42-21ksoftirqd/309:53:203
1784499994317,16cyclictest42-21ksoftirqd/309:53:193
178449999431,21cyclictest42-21ksoftirqd/307:10:313
178449999431,14cyclictest42-21ksoftirqd/310:20:153
178449999431,12cyclictest42-21ksoftirqd/312:39:553
1784499994311,10cyclictest42-21ksoftirqd/312:00:003
1784499994310,10cyclictest42-21ksoftirqd/311:34:003
178449599439,9cyclictest34-21ksoftirqd/209:37:152
178449599437,20cyclictest34-21ksoftirqd/209:04:502
178449599437,18cyclictest1753-21gnome-shell09:10:312
178449599433,22cyclictest34-21ksoftirqd/207:37:452
1784495994323,13cyclictest34-21ksoftirqd/211:48:072
1784495994311,13cyclictest141rcu_preempt07:55:292
178448999434,20cyclictest13-21ksoftirqd/007:50:120
178448999434,20cyclictest13-21ksoftirqd/007:50:120
1784489994335,5cyclictest131ksoftirqd/010:10:020
1784489994327,7cyclictest13-21ksoftirqd/007:57:270
1784489994325,6cyclictest13-21ksoftirqd/009:30:220
1784489994325,6cyclictest13-21ksoftirqd/009:30:220
1784489994324,14cyclictest13-21ksoftirqd/012:30:310
1784489994321,8cyclictest13-21ksoftirqd/008:55:030
1784489994320,6cyclictest13-21ksoftirqd/008:10:310
1784489994320,6cyclictest13-21ksoftirqd/008:10:310
1784489994315,13cyclictest141rcu_preempt12:05:200
1784489994315,13cyclictest141rcu_preempt12:05:200
1784489994314,12cyclictest141rcu_preempt11:34:150
1784489994313,13cyclictest141rcu_preempt07:20:180
162399430,30rtkit-daemon1753-21gnome-shell10:51:250
178449999421,18cyclictest1855100-21chrt10:28:173
1784495994230,8cyclictest34-21ksoftirqd/208:30:022
1784495994214,10cyclictest34-21ksoftirqd/211:55:002
1784495994213,13cyclictest34-21ksoftirqd/207:32:542
1784495994213,10cyclictest34-21ksoftirqd/212:35:012
1784495994213,10cyclictest34-21ksoftirqd/210:05:262
178448999427,17cyclictest13-21ksoftirqd/007:12:090
1784489994224,13cyclictest13-21ksoftirqd/009:45:210
1784489994223,6cyclictest13-21ksoftirqd/007:40:310
1784489994223,14cyclictest13-21ksoftirqd/010:10:220
1784489994223,14cyclictest13-21ksoftirqd/010:10:220
1784489994222,6cyclictest13-21ksoftirqd/011:12:130
1784489994222,15cyclictest13-21ksoftirqd/011:25:110
1784489994221,12cyclictest13-21ksoftirqd/010:55:190
1784489994211,13cyclictest141rcu_preempt08:20:310
1784489994210,14cyclictest141rcu_preempt12:20:180
178449999411,18cyclictest1866150-40apt-check11:00:173
178449999411,18cyclictest1866150-40apt-check11:00:173
178449999411,14cyclictest2197014-21PK-Backend07:45:153
1784499994110,11cyclictest42-21ksoftirqd/308:35:163
178449599417,17cyclictest1839333-20journal-offline09:40:552
178449599414,19cyclictest34-21ksoftirqd/208:39:132
178449599412,22cyclictest34-21ksoftirqd/210:11:572
178449599412,22cyclictest34-21ksoftirqd/210:11:562
178449599411,21cyclictest34-21ksoftirqd/209:57:402
178449599411,21cyclictest34-21ksoftirqd/209:57:392
178449599411,19cyclictest1753-21gnome-shell10:16:472
178448999416,7cyclictest141rcu_preempt12:15:000
178448999415,8cyclictest141rcu_preempt12:15:170
178448999413,20cyclictest1888941-21seq12:03:340
1784489994124,13cyclictest13-21ksoftirqd/008:28:020
1784489994123,13cyclictest13-21ksoftirqd/012:35:150
1784489994123,13cyclictest13-21ksoftirqd/009:40:230
1784489994122,7cyclictest13-21ksoftirqd/008:30:430
1784489994121,15cyclictest13-21ksoftirqd/009:01:200
178449999403,18cyclictest42-21ksoftirqd/311:53:103
178449999402,16cyclictest42-21ksoftirqd/310:46:553
178449999401,15cyclictest1891446-21chrt12:10:203
178449599402,6cyclictest141rcu_preempt10:36:302
178449599401,16cyclictest141rcu_preempt09:48:522
1784489994023,6cyclictest13-21ksoftirqd/010:40:380
1784489994022,6cyclictest13-21ksoftirqd/010:45:180
1784489994021,6cyclictest13-21ksoftirqd/007:35:300
1784489994021,14cyclictest13-21ksoftirqd/010:30:160
1784489994021,14cyclictest13-21ksoftirqd/009:55:020
1784489994021,14cyclictest13-21ksoftirqd/009:55:010
1784489994019,9cyclictest141rcu_preempt09:13:320
1784489994014,6cyclictest141rcu_preempt09:15:160
178448999401,17cyclictest1852778-21ntpq10:20:250
178449999398,10cyclictest42-21ksoftirqd/307:22:133
1784499993920,10cyclictest42-21ksoftirqd/312:22:313
178449599392,32cyclictest1753-21gnome-shell09:50:452
178449599392,32cyclictest1753-21gnome-shell09:50:452
1784495993911,9cyclictest34-21ksoftirqd/208:50:542
178449599391,11cyclictest1814034-21chrt08:30:202
1784495993911,10cyclictest34-21ksoftirqd/211:30:232
1784489993921,6cyclictest13-21ksoftirqd/009:35:240
1784489993920,14cyclictest13-21ksoftirqd/010:40:000
1784489993918,9cyclictest141rcu_preempt10:02:070
1784489993918,10cyclictest13-21ksoftirqd/007:25:160
1784499993819,11cyclictest42-21ksoftirqd/308:17:033
1784495993812,9cyclictest34-21ksoftirqd/210:00:412
178449599381,16cyclictest2197014-21PK-Backend09:25:252
178449999374,18cyclictest42-21ksoftirqd/310:35:263
178449599371,20cyclictest1854754-21/usr/sbin/munin10:25:362
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional