You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-09 - 07:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Tue Dec 09, 2025 00:44:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991210,9rtkit-daemon1622-21rtkit-daemon22:56:190
1623991210,9rtkit-daemon1622-21rtkit-daemon22:56:190
1623991070,8rtkit-daemon1622-21rtkit-daemon19:43:530
33723522970,5sleep1251rcuc/119:50:141
34277762940,5sleep3411rcuc/322:25:403
34277762940,5sleep3411rcuc/322:25:393
34179532900,6sleep02197014-21PK-Backend22:00:120
335736528872,9sleep00-21swapper/019:09:130
34207422740,17sleep234-21ksoftirqd/222:05:312
335756399745,26cyclictest2197014-21PK-Backend22:15:011
335733727444,22sleep30-21swapper/319:08:523
34077802730,5sleep1251rcuc/121:30:151
335756799720,68cyclictest1622-21rtkit-daemon21:24:462
335735727143,22sleep20-21swapper/219:09:072
34390812700,8sleep02197014-21PK-Backend23:00:130
33701292700,4chrt3370130-21sh19:45:013
34255502680,19sleep234-21ksoftirqd/222:20:242
335720026843,15sleep10-21swapper/119:07:091
34584912640,13sleep3411rcuc/323:55:153
34584912640,13sleep3411rcuc/323:55:143
33970382590,6chrt3397035-21latency21:00:240
33970382590,6chrt3397035-21latency21:00:240
335756399593,21cyclictest3470657-21chrt00:28:341
3357560995925,10cyclictest13-21ksoftirqd/021:06:450
3357560995925,10cyclictest13-21ksoftirqd/021:06:440
162399590,8rtkit-daemon1622-21rtkit-daemon00:22:412
162399580,9rtkit-daemon1622-21rtkit-daemon21:59:452
335756399567,23cyclictest3460630-21sed00:00:241
335756399567,14cyclictest1753-21gnome-shell00:06:351
162399560,9rtkit-daemon1622-21rtkit-daemon21:01:482
162399560,9rtkit-daemon1622-21rtkit-daemon21:01:482
3357560995530,8cyclictest13-21ksoftirqd/000:28:330
162399550,9rtkit-daemon1622-21rtkit-daemon00:28:492
34174612530,8chrt1753-21gnome-shell21:57:280
3357567995128,19cyclictest1753-21gnome-shell22:10:222
335756399519,19cyclictest3450957-21apt-config23:35:031
335756399519,19cyclictest3450957-21apt-config23:35:021
335756399518,17cyclictest3364948-21awk19:30:001
335756399518,14cyclictest3396216-21munin-run21:00:001
335756399512,24cyclictest3417841-21latency_hist22:00:011
3357560995138,8cyclictest13-21ksoftirqd/021:19:570
34037542500,5chrt0-21swapper/121:20:141
3357570995021,16cyclictest1753-21gnome-shell00:05:143
335756399501,25cyclictest3399070-21runrttasks21:05:231
335756399501,25cyclictest3399070-21runrttasks21:05:231
335756399501,12cyclictest1753-21gnome-shell22:00:231
3357560995038,9cyclictest13-21ksoftirqd/020:10:270
3357560995012,14cyclictest141rcu_preempt20:50:120
33950392490,13sleep2341ksoftirqd/220:55:192
335756799490,45cyclictest1622-21rtkit-daemon23:41:302
335756799490,45cyclictest1622-21rtkit-daemon22:56:192
335756799490,45cyclictest1622-21rtkit-daemon22:56:182
335756399494,24cyclictest3438159-21/usr/sbin/munin22:55:351
335756399494,24cyclictest3438159-21/usr/sbin/munin22:55:341
335756399494,22cyclictest1753-21gnome-shell21:39:061
335756399492,11cyclictest3471816-21awk00:30:291
335756399491,27cyclictest1753-21gnome-shell20:24:451
335756399491,27cyclictest1753-21gnome-shell20:24:451
335756799480,44cyclictest1622-21rtkit-daemon19:43:542
335756399489,22cyclictest3456894-21fw_packets23:50:201
335756399489,19cyclictest1753-21gnome-shell19:14:451
335756399487,21cyclictest3442156-21cat23:10:021
335756399486,21cyclictest1753-21gnome-shell23:27:561
335756399485,22cyclictest1753-21gnome-shell20:52:471
335756399482,21cyclictest3361164-21taskset19:16:291
3357563994811,18cyclictest1753-21gnome-shell23:56:511
3357563994811,18cyclictest1753-21gnome-shell23:56:501
3357563994810,20cyclictest1753-21gnome-shell20:12:231
335756399480,21cyclictest3453561-21memory23:40:231
335756799477,20cyclictest34-21ksoftirqd/200:15:242
335756399479,21cyclictest1753-21gnome-shell20:37:321
335756399479,20cyclictest1753-21gnome-shell23:00:381
335756399479,19cyclictest1753-21gnome-shell20:31:061
335756399479,18cyclictest1753-21gnome-shell20:47:321
335756399478,13cyclictest1753-21gnome-shell23:10:221
335756399478,13cyclictest1753-21gnome-shell22:45:171
335756399478,10cyclictest1753-21gnome-shell23:22:301
335756399477,15cyclictest3423511-21/usr/sbin/munin22:15:161
335756399471,11cyclictest1753-21gnome-shell22:20:091
3357563994710,19cyclictest1753-21gnome-shell23:35:451
3357563994710,19cyclictest1753-21gnome-shell23:35:451
3357563994710,19cyclictest1753-21gnome-shell19:36:591
3357563994710,19cyclictest1753-21gnome-shell19:36:591
3357563994710,19cyclictest1753-21gnome-shell00:11:201
335756399470,27cyclictest1753-21gnome-shell00:37:361
335756399470,26cyclictest1753-21gnome-shell19:21:411
3357560994722,21cyclictest3443763-21unin-run23:15:010
33765302460,4sleep00-21swapper/020:00:310
33765302460,4sleep00-21swapper/020:00:310
3357567994636,6cyclictest341ksoftirqd/221:45:272
3357567994629,13cyclictest34-21ksoftirqd/223:45:262
3357567994629,13cyclictest34-21ksoftirqd/221:15:162
3357567994615,10cyclictest1753-21gnome-shell21:31:042
3357567994611,11cyclictest34-21ksoftirqd/200:39:262
335756399469,19cyclictest1753-21gnome-shell21:52:231
335756399469,19cyclictest1753-21gnome-shell21:16:291
335756399469,17cyclictest1753-21gnome-shell23:48:211
335756399469,14cyclictest1753-21gnome-shell23:15:271
335756399469,10cyclictest1753-21gnome-shell19:59:051
335756399468,9cyclictest3411375-21df21:40:161
335756399468,13cyclictest1753-21gnome-shell21:45:231
335756399468,10cyclictest1753-21gnome-shell19:30:031
335756399467,18cyclictest1753-21gnome-shell20:02:491
335756399467,18cyclictest1753-21gnome-shell20:02:491
335756399467,12cyclictest1753-21gnome-shell19:40:511
335756399466,16cyclictest3466850-21timerwakeupswit00:15:341
335756399464,16cyclictest3397362-21python321:00:321
335756399464,16cyclictest3397362-21python321:00:321
3357563994616,24cyclictest3405478-21acpi21:25:121
3357560994626,12cyclictest13-21ksoftirqd/020:38:450
3357570994511,7cyclictest141rcu_preempt20:10:293
3357567994528,6cyclictest34-21ksoftirqd/223:02:402
3357567994526,7cyclictest34-21ksoftirqd/221:06:252
3357567994526,7cyclictest34-21ksoftirqd/221:06:252
3357567994526,6cyclictest34-21ksoftirqd/200:10:302
3357567994521,20cyclictest1753-21gnome-shell19:30:302
335756399459,16cyclictest1753-21gnome-shell21:12:501
335756399457,13cyclictest1753-21gnome-shell22:25:091
335756399457,13cyclictest1753-21gnome-shell22:25:081
335756399453,16cyclictest1753-21gnome-shell22:33:571
335756099459,9cyclictest13-21ksoftirqd/020:55:080
335756099455,5cyclictest141rcu_preempt23:45:060
3357560994526,12cyclictest13-21ksoftirqd/023:32:190
3357560994526,12cyclictest13-21ksoftirqd/023:32:190
162399450,38rtkit-daemon2287-21gmain22:33:122
34515172440,19sleep342-21ksoftirqd/323:35:153
34515172440,19sleep342-21ksoftirqd/323:35:143
33578252440,7sleep00-21swapper/019:10:160
335757099441,14cyclictest2197014-21PK-Backend21:20:513
3357567994425,15cyclictest34-21ksoftirqd/200:00:332
3357567994425,14cyclictest3242253-21nautilus22:45:022
3357567994418,9cyclictest34-21ksoftirqd/223:18:212
335756799441,38cyclictest3374054-21tr19:55:142
335756399448,11cyclictest3468312-21/usr/sbin/munin00:20:271
335756399447,10cyclictest3430723-21memory22:35:251
3357563994416,11cyclictest3377565-21cut20:05:161
3357560994426,13cyclictest3409018-21/usr/sbin/munin21:35:010
3357560994424,16cyclictest13-21ksoftirqd/023:15:440
335756099442,19cyclictest3242253-21nautilus21:30:020
3357570994314,10cyclictest2287-21gmain19:35:133
3357570994314,10cyclictest2287-21gmain19:35:133
3357570994310,29cyclictest1753-21gnome-shell23:31:193
3357570994310,29cyclictest1753-21gnome-shell23:31:183
335756799437,20cyclictest1753-21gnome-shell20:51:022
3357567994325,6cyclictest34-21ksoftirqd/221:25:202
3357567994325,6cyclictest34-21ksoftirqd/221:10:262
3357567994325,6cyclictest34-21ksoftirqd/200:30:182
3357567994324,6cyclictest34-21ksoftirqd/220:00:252
3357567994324,6cyclictest34-21ksoftirqd/220:00:252
3357567994324,6cyclictest34-21ksoftirqd/219:25:242
3357567994324,14cyclictest34-21ksoftirqd/222:00:142
3357567994318,10cyclictest34-21ksoftirqd/223:50:292
3357567994316,10cyclictest34-21ksoftirqd/223:28:232
335756399439,16cyclictest26-21ksoftirqd/120:43:561
335756399437,11cyclictest3380855-21/usr/sbin/munin20:15:141
335756399436,13cyclictest3432435-21sed22:40:211
335756099433,5cyclictest141rcu_preempt00:17:260
3357560994333,7cyclictest1069-21runrttasks00:03:080
3357560994332,8cyclictest13-21ksoftirqd/019:52:030
335756099433,19cyclictest141rcu_preempt00:09:010
3357560994327,12cyclictest3454435-21sshd23:44:260
3357560994321,18cyclictest1753-21gnome-shell23:53:010
3357560994320,10cyclictest13-21ksoftirqd/020:31:260
335756099431,6cyclictest141rcu_preempt20:20:240
335756099431,6cyclictest141rcu_preempt20:20:240
3357570994215,23cyclictest1753-21gnome-shell20:41:593
3357570994210,6cyclictest141rcu_preempt22:20:113
3357567994231,7cyclictest1753-21gnome-shell22:35:072
3357567994219,19cyclictest1753-21gnome-shell22:50:342
335756099424,8cyclictest13-21ksoftirqd/000:38:330
3357560994230,9cyclictest13-21ksoftirqd/022:20:130
3357560994223,15cyclictest13-21ksoftirqd/022:10:350
3357560994221,13cyclictest13-21ksoftirqd/022:26:220
3357560994221,13cyclictest13-21ksoftirqd/022:26:210
3357560994217,20cyclictest3459735-40updatedb.mlocat00:00:030
3357560994216,11cyclictest141rcu_preempt23:21:310
3357570994122,13cyclictest855-21dbus-daemon21:45:003
335757099411,22cyclictest1753-21gnome-shell00:23:303
335756799416,13cyclictest34-21ksoftirqd/222:26:252
335756799416,13cyclictest34-21ksoftirqd/222:26:252
3357567994131,7cyclictest34-21ksoftirqd/220:10:302
3357567994126,11cyclictest3410015-21ntp_offset21:35:272
3357567994124,6cyclictest34-21ksoftirqd/222:45:172
3357567994123,6cyclictest34-21ksoftirqd/219:10:182
3357567994123,13cyclictest34-21ksoftirqd/222:15:202
3357567994122,7cyclictest34-21ksoftirqd/223:10:262
3357567994122,6cyclictest34-21ksoftirqd/220:45:152
3357567994122,6cyclictest34-21ksoftirqd/220:30:022
3357567994118,19cyclictest1753-21gnome-shell23:25:002
335756399417,10cyclictest1753-21gnome-shell19:45:501
335756099415,22cyclictest141rcu_preempt19:35:410
335756099415,22cyclictest141rcu_preempt19:35:410
3357560994130,7cyclictest13-21ksoftirqd/019:20:140
335756099412,9cyclictest0-21swapper/023:35:320
335756099412,9cyclictest0-21swapper/023:35:310
3357560994128,10cyclictest13-21ksoftirqd/022:07:190
3357560994123,7cyclictest13-21ksoftirqd/022:48:340
3357560994123,14cyclictest13-21ksoftirqd/022:32:360
3357560994122,7cyclictest13-21ksoftirqd/020:16:110
3357560994120,9cyclictest13-21ksoftirqd/019:35:010
3357560994113,25cyclictest3459735-40.mlocate00:00:020
335756099411,20cyclictest3375009-21taskset19:55:360
335756099411,18cyclictest141rcu_preempt20:42:170
335757099407,14cyclictest1753-21gnome-shell00:36:573
335757099405,17cyclictest1753-21gnome-shell21:55:313
3357570994019,16cyclictest42-21ksoftirqd/322:45:183
335756799405,17cyclictest0-21swapper/220:15:122
3357567994023,6cyclictest34-21ksoftirqd/221:50:302
3357567994022,6cyclictest34-21ksoftirqd/219:35:312
3357567994022,6cyclictest34-21ksoftirqd/219:35:312
3357567994021,14cyclictest34-21ksoftirqd/219:45:282
3357567994020,14cyclictest34-21ksoftirqd/223:40:032
3357567994020,14cyclictest34-21ksoftirqd/223:40:032
3357567994019,7cyclictest1753-21gnome-shell19:55:022
3357567994012,13cyclictest141rcu_preempt19:22:542
335756099408,18cyclictest13-21ksoftirqd/019:26:330
335756099408,16cyclictest3371109-21ntp_kernel_err19:45:230
3357560994029,8cyclictest13-21ksoftirqd/000:34:550
3357560994023,7cyclictest13-21ksoftirqd/021:50:490
335756099402,34cyclictest1753-21gnome-shell00:11:480
3357560994016,8cyclictest13-21ksoftirqd/021:40:150
3357560994011,7cyclictest141rcu_preempt22:50:430
3357560994011,14cyclictest141rcu_preempt00:20:180
3357570993927,8cyclictest42-21ksoftirqd/323:06:443
3357570993924,10cyclictest42-21ksoftirqd/321:00:303
3357570993924,10cyclictest42-21ksoftirqd/321:00:293
3357570993921,14cyclictest3399631-40apt-check21:06:173
3357570993921,14cyclictest3399631-40apt-check21:06:173
3357570993919,17cyclictest1753-21gnome-shell19:50:273
3357570993918,16cyclictest3408204-21cut21:30:263
335757099391,13cyclictest141rcu_preempt00:30:003
335756799394,12cyclictest141rcu_preempt23:33:282
335756799394,12cyclictest141rcu_preempt23:33:282
3357567993923,6cyclictest34-21ksoftirqd/220:35:192
3357567993923,6cyclictest34-21ksoftirqd/220:20:282
3357567993923,6cyclictest34-21ksoftirqd/220:20:282
3357567993922,5cyclictest34-21ksoftirqd/200:10:012
3357567993921,14cyclictest34-21ksoftirqd/220:40:242
335756399396,11cyclictest3435920-21memory22:50:251
335756399396,11cyclictest3385593-21chrt20:27:251
335756099398,11cyclictest141rcu_preempt23:28:220
335756099397,17cyclictest13-21ksoftirqd/020:25:280
3357560993928,7cyclictest3413414-21grep21:45:250
335756099392,5cyclictest141rcu_preempt22:35:310
3357560993922,13cyclictest13-21ksoftirqd/021:20:110
335756099391,17cyclictest141rcu_preempt22:18:040
3357570993826,8cyclictest1753-21gnome-shell19:24:393
3357570993822,13cyclictest1753-21gnome-shell22:15:153
3357570993822,11cyclictest42-21ksoftirqd/323:25:133
335757099381,24cyclictest141rcu_preempt23:17:293
335756799382,19cyclictest1753-21gnome-shell21:44:102
3357567993821,6cyclictest34-21ksoftirqd/219:15:062
3357567993819,6cyclictest34-21ksoftirqd/220:30:152
3357567993815,6cyclictest33788652pluginstate20:10:012
335756799380,20cyclictest141rcu_preempt00:00:032
335756799380,20cyclictest141rcu_preempt00:00:032
3357563993818,16cyclictest1069-21runrttasks22:05:101
335756099388,6cyclictest141rcu_preempt20:06:280
3357560993823,6cyclictest13-21ksoftirqd/023:08:340
3357560993823,12cyclictest3433349-21awk22:45:000
3357560993814,6cyclictest141rcu_preempt21:36:100
335756099381,17cyclictest141rcu_preempt19:16:350
3357570993723,11cyclictest42-21ksoftirqd/321:10:273
3357570993720,12cyclictest438-20systemd-journal20:25:013
3357570993720,12cyclictest438-20systemd-journal20:25:013
3357570993719,13cyclictest468-21systemd-udevd21:19:563
3357570993718,16cyclictest1753-21gnome-shell20:15:283
335757099371,32cyclictest2197014-21PK-Backend19:48:063
335757099371,17cyclictest1753-21gnome-shell19:10:413
3357567993715,6cyclictest141rcu_preempt23:05:172
3357560993721,12cyclictest13-21ksoftirqd/020:49:170
335756099371,5cyclictest141rcu_preempt21:14:200
3357570993623,9cyclictest1753-21gnome-shell20:45:293
335757099362,30cyclictest1753-21gnome-shell23:40:543
3357570993622,11cyclictest42-21ksoftirqd/320:28:263
335757099362,11cyclictest33952502uname20:55:253
3357570993621,11cyclictest3465546-21dpkg00:15:033
3357570993620,10cyclictest42-21ksoftirqd/300:00:053
3357570993619,6cyclictest42-21ksoftirqd/322:40:153
3357570993616,7cyclictest141rcu_preempt20:40:003
335757099352,9cyclictest1753-21gnome-shell00:32:233
335757099352,8cyclictest3430891-21ntp_states22:35:283
3357570993522,10cyclictest42-21ksoftirqd/319:16:233
3357570993521,10cyclictest42-21ksoftirqd/319:35:003
3357570993520,11cyclictest3443355-21vmstat23:10:333
3357570993518,13cyclictest3429138-21ntpq22:30:263
3357570993518,12cyclictest3465870-21sort00:15:173
3357570993518,12cyclictest3457373-21python323:50:313
3357570993517,14cyclictest3418698-21ntp_states22:00:253
3357570993516,5cyclictest141rcu_preempt19:56:103
335757099342,27cyclictest1753-21gnome-shell22:59:263
335757099342,27cyclictest1753-21gnome-shell22:59:263
3357570993419,11cyclictest3415247-21sshd21:50:273
3357570993419,10cyclictest42-21ksoftirqd/322:05:133
3357570993414,8cyclictest141rcu_preempt23:00:293
3357570993411,6cyclictest141rcu_preempt20:00:163
3357570993411,6cyclictest141rcu_preempt20:00:163
335757099341,14cyclictest2197014-21PK-Backend21:49:533
335757099332,27cyclictest1753-21gnome-shell21:37:303
3357570993319,11cyclictest42-21ksoftirqd/322:52:383
3357570993319,10cyclictest42-21ksoftirqd/321:28:523
3357570993313,6cyclictest141rcu_preempt20:51:173
3357570993312,7cyclictest2197014-21PK-Backend23:20:263
33780142320,7sleep32197014-21PK-Backend20:05:253
3357570993219,9cyclictest42-21ksoftirqd/322:10:133
335757099321,27cyclictest2197014-21PK-Backend19:25:553
335757099311,10cyclictest2197014-21PK-Backend23:45:223
335757099301,13cyclictest2197014-21PK-Backend20:34:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional