You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 15:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed Nov 26, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991170,9rtkit-daemon1622-21rtkit-daemon10:06:080
1623991170,9rtkit-daemon1622-21rtkit-daemon10:06:070
1623991170,8rtkit-daemon1622-21rtkit-daemon09:54:330
1623991170,8rtkit-daemon1622-21rtkit-daemon09:54:330
1623991140,8rtkit-daemon1622-21rtkit-daemon08:35:511
56009121000,5sleep30-21swapper/312:05:263
162399950,9rtkit-daemon1622-21rtkit-daemon11:12:040
4895012850,4sleep10-21swapper/108:45:191
4878192850,4chrt487325-21/usr/sbin/munin08:40:233
162399840,31rtkit-daemon2197014-21PK-Backend07:13:450
162399840,31rtkit-daemon2197014-21PK-Backend07:13:440
45443927557,13sleep30-21swapper/307:07:433
45443927557,13sleep30-21swapper/307:07:433
45461027457,10sleep20-21swapper/207:09:512
45461027457,10sleep20-21swapper/207:09:502
45444227240,25sleep10-21swapper/107:07:451
45444227240,25sleep10-21swapper/107:07:451
45477899650,61cyclictest1622-21rtkit-daemon11:17:011
45477899640,60cyclictest1622-21rtkit-daemon07:51:231
454785996312,28cyclictest1753-21gnome-shell11:54:423
454778996329,19cyclictest26-21ksoftirqd/111:25:461
45477899620,58cyclictest1622-21rtkit-daemon09:57:271
45477899620,58cyclictest1622-21rtkit-daemon08:10:531
45458126141,14sleep00-21swapper/007:09:290
45458126141,14sleep00-21swapper/007:09:290
45478599607,27cyclictest465003-21python307:35:283
454785996010,28cyclictest1753-21gnome-shell07:59:363
454778995913,15cyclictest141rcu_preempt08:52:361
162399590,9rtkit-daemon1622-21rtkit-daemon11:11:491
162399590,9rtkit-daemon1622-21rtkit-daemon10:54:171
162399590,9rtkit-daemon1622-21rtkit-daemon09:27:021
454785995814,23cyclictest1753-21gnome-shell09:35:253
162399580,9rtkit-daemon1622-21rtkit-daemon10:44:161
162399580,8rtkit-daemon1622-21rtkit-daemon10:19:511
162399580,8rtkit-daemon1622-21rtkit-daemon07:48:151
454785995715,21cyclictest1753-21gnome-shell10:40:003
454785995712,24cyclictest1753-21gnome-shell09:13:353
454785995710,27cyclictest1753-21gnome-shell10:04:593
454785995710,25cyclictest1753-21gnome-shell07:54:083
4946222560,13sleep1261ksoftirqd/109:00:141
454785995616,17cyclictest474062-21memory08:00:233
454785995614,22cyclictest1753-21gnome-shell08:25:223
454785995613,16cyclictest1753-21gnome-shell07:40:153
454785995611,22cyclictest1753-21gnome-shell09:51:483
454785995611,22cyclictest1753-21gnome-shell09:51:473
162399560,8rtkit-daemon1622-21rtkit-daemon12:14:591
162399560,10rtkit-daemon1622-21rtkit-daemon11:22:171
454785995515,18cyclictest557641-40apt-check12:00:193
454785995512,21cyclictest1753-21gnome-shell11:37:073
454785995512,18cyclictest1753-21gnome-shell07:29:453
45478399556,27cyclictest1753-21gnome-shell12:05:262
45477899550,51cyclictest1622-21rtkit-daemon12:31:301
45477899550,51cyclictest1622-21rtkit-daemon10:28:141
162399550,9rtkit-daemon1622-21rtkit-daemon11:56:221
162399550,9rtkit-daemon1622-21rtkit-daemon08:07:451
5087482540,6sleep2508751-21missed_timers09:40:222
45478599549,25cyclictest1753-21gnome-shell08:55:043
45478599544,30cyclictest517907-21awk10:05:263
45478599544,30cyclictest517907-21awk10:05:263
454785995416,17cyclictest556288-21ntp_states11:55:273
454785995416,17cyclictest552169-21df11:45:143
454785995415,17cyclictest1753-21gnome-shell11:25:263
454785995411,12cyclictest468835-31apt-check07:45:303
45478399543,29cyclictest1753-21gnome-shell08:10:292
45478399542,24cyclictest1753-21gnome-shell10:39:362
162399540,9rtkit-daemon1622-21rtkit-daemon07:58:421
162399540,6rtkit-daemon1622-21rtkit-daemon11:08:041
4797752530,6sleep10-21swapper/108:15:341
454785995316,16cyclictest1753-21gnome-shell10:45:263
454785995316,16cyclictest1753-21gnome-shell10:45:263
454785995311,19cyclictest1753-21gnome-shell11:09:483
454785995310,17cyclictest1753-21gnome-shell10:23:163
45478399535,27cyclictest1753-21gnome-shell07:25:202
45478399532,18cyclictest1753-21gnome-shell09:52:002
45478399532,18cyclictest1753-21gnome-shell09:52:002
454778995324,11cyclictest1753-21gnome-shell07:15:141
454778995314,20cyclictest26-21ksoftirqd/111:45:011
45477899530,49cyclictest1622-21rtkit-daemon10:06:081
45477899530,49cyclictest1622-21rtkit-daemon10:06:071
162399530,9rtkit-daemon1622-21rtkit-daemon12:22:331
162399530,7rtkit-daemon1622-21rtkit-daemon10:20:551
5310252520,17sleep013-21ksoftirqd/010:45:130
5310252520,17sleep013-21ksoftirqd/010:45:130
45478599525,16cyclictest509894-21apt09:45:143
454785995214,11cyclictest1753-21gnome-shell08:35:163
454785995212,11cyclictest1753-21gnome-shell10:25:453
45478399524,26cyclictest1753-21gnome-shell10:35:012
45477899520,48cyclictest1622-21rtkit-daemon09:54:341
45477899520,48cyclictest1622-21rtkit-daemon09:54:341
45477899520,48cyclictest1622-21rtkit-daemon08:21:151
5535902510,5sleep1454765-21cyclictest11:50:141
454785995113,18cyclictest1753-21gnome-shell11:40:163
454785995113,12cyclictest1753-21gnome-shell11:05:013
454785995113,12cyclictest1753-21gnome-shell10:10:153
454785995111,18cyclictest1753-21gnome-shell11:33:273
454785995111,15cyclictest1753-21gnome-shell07:14:553
454785995111,15cyclictest1753-21gnome-shell07:14:553
45478399514,26cyclictest1753-21gnome-shell11:45:282
454783995128,7cyclictest529924-21cut10:40:242
45478399512,27cyclictest1753-21gnome-shell08:16:252
454785995011,15cyclictest1753-21gnome-shell07:24:183
454785995011,12cyclictest1753-21gnome-shell10:15:543
454785995011,11cyclictest1753-21gnome-shell08:48:173
454785995010,16cyclictest1753-21gnome-shell09:31:313
45478399507,22cyclictest1753-21gnome-shell10:05:252
45478399507,22cyclictest1753-21gnome-shell10:05:252
45478399507,22cyclictest1753-21gnome-shell09:55:262
45478399506,25cyclictest1753-21gnome-shell07:40:212
45478399506,23cyclictest1753-21gnome-shell12:21:292
45478399506,23cyclictest1753-21gnome-shell11:55:162
45478399506,22cyclictest1753-21gnome-shell10:45:332
45478399506,22cyclictest1753-21gnome-shell10:45:322
45478399505,24cyclictest1753-21gnome-shell07:40:012
454778995017,13cyclictest26-21ksoftirqd/107:21:561
45477899500,46cyclictest1622-21rtkit-daemon12:03:121
45478599499,16cyclictest496762-21ntp_states09:05:243
45478599494,16cyclictest501451-21cat09:20:143
454785994911,18cyclictest1753-21gnome-shell09:19:173
454785994911,11cyclictest1753-21gnome-shell10:40:453
454785994911,11cyclictest1753-21gnome-shell08:16:423
454785994910,15cyclictest535237-21/usr/sbin/munin10:55:263
45478399495,23cyclictest1753-21gnome-shell07:50:222
45478399492,24cyclictest1753-21gnome-shell08:33:262
45477899499,19cyclictest26-21ksoftirqd/107:30:001
45477899491,13cyclictest1753-21gnome-shell07:30:251
45478599487,15cyclictest0-21swapper/308:32:543
454785994812,13cyclictest0-21swapper/310:50:253
454785994810,12cyclictest563801-21/usr/sbin/munin12:15:313
45478399485,22cyclictest1753-21gnome-shell12:25:212
45478399485,22cyclictest1753-21gnome-shell12:25:202
45478399485,22cyclictest1753-21gnome-shell11:40:202
45478399483,24cyclictest1753-21gnome-shell09:34:132
45478399483,17cyclictest1753-21gnome-shell08:04:032
45478399482,15cyclictest1753-21gnome-shell09:04:452
45477899482,13cyclictest546372-21kworker/1:011:30:201
45477899480,44cyclictest1622-21rtkit-daemon09:12:241
45478599479,15cyclictest1753-21gnome-shell09:00:213
454785994711,12cyclictest1753-21gnome-shell08:53:163
454785994711,11cyclictest1753-21gnome-shell12:35:193
454785994711,11cyclictest1753-21gnome-shell08:24:553
454785994710,17cyclictest1753-21gnome-shell11:13:203
454785994710,11cyclictest1753-21gnome-shell07:19:453
45478399475,22cyclictest1753-21gnome-shell09:45:172
45478399473,27cyclictest1753-21gnome-shell08:59:292
454783994726,8cyclictest34-21ksoftirqd/211:20:262
45478399471,29cyclictest1753-21gnome-shell09:06:412
45478399471,27cyclictest1753-21gnome-shell08:37:192
45478399471,22cyclictest1753-21gnome-shell11:37:142
45477899479,13cyclictest26-21ksoftirqd/112:20:021
45477899477,19cyclictest26-21ksoftirqd/112:40:021
45478599464,16cyclictest854-21avahi-daemon08:06:003
45478599461,24cyclictest0-21swapper/308:10:123
454785994610,11cyclictest1753-21gnome-shell12:11:433
45478399466,21cyclictest438-20systemd-journal07:24:552
45478399465,22cyclictest1753-21gnome-shell10:59:262
45478399463,26cyclictest1753-21gnome-shell10:21:422
45478399463,26cyclictest1753-21gnome-shell08:29:312
45478399463,26cyclictest1753-21gnome-shell07:30:512
454783994626,16cyclictest562784-21sshd12:15:062
45478399462,27cyclictest1753-21gnome-shell09:14:312
45478399462,15cyclictest1753-21gnome-shell10:03:472
45478399461,22cyclictest1753-21gnome-shell12:33:082
454783994612,16cyclictest34-21ksoftirqd/211:10:132
45477899469,6cyclictest141rcu_preempt09:47:281
45477899469,18cyclictest1753-21gnome-shell07:10:331
45477899469,18cyclictest1753-21gnome-shell07:10:331
454778994616,10cyclictest26-21ksoftirqd/109:15:381
45478599457,13cyclictest1753-21gnome-shell11:21:153
45478599454,12cyclictest1753-21gnome-shell09:29:013
45478399459,15cyclictest469492-21apt-config07:50:012
45478399457,18cyclictest34-21ksoftirqd/207:19:102
45478399454,17cyclictest34-21ksoftirqd/211:09:572
45478399453,24cyclictest1753-21gnome-shell08:45:102
45478399453,20cyclictest2197014-21PK-Backend12:00:142
454783994526,14cyclictest3242253-21nautilus11:55:032
45478399452,26cyclictest1753-21gnome-shell10:53:272
45478399452,23cyclictest1753-21gnome-shell09:38:082
45478399451,27cyclictest1753-21gnome-shell12:39:062
45478399451,27cyclictest1753-21gnome-shell12:11:112
45478399451,27cyclictest1753-21gnome-shell10:25:392
45478399451,24cyclictest1753-21gnome-shell11:27:202
45477899458,12cyclictest26-21ksoftirqd/110:11:531
45478599448,15cyclictest567989-21runrttasks12:29:263
45478599448,15cyclictest567989-21runrttasks12:29:263
45478599445,12cyclictest1753-21gnome-shell12:20:193
45478399449,19cyclictest34-21ksoftirqd/211:00:482
45478399446,19cyclictest1753-21gnome-shell08:06:012
45478399445,14cyclictest1753-21gnome-shell09:20:252
45478399444,16cyclictest1753-21gnome-shell10:15:212
45478399443,25cyclictest1753-21gnome-shell07:58:112
45478399443,21cyclictest1753-21gnome-shell09:15:162
45478399442,25cyclictest1753-21gnome-shell11:19:182
45478399442,24cyclictest1753-21gnome-shell11:32:552
45477899443,5cyclictest141rcu_preempt11:01:161
454778994423,8cyclictest26-21ksoftirqd/110:39:261
45477899442,12cyclictest1753-21gnome-shell07:40:301
454778994418,6cyclictest141rcu_preempt10:30:191
45477899441,24cyclictest560367-21timerwakeupswit12:05:311
45477899441,14cyclictest502691-21dump-pmu-power09:25:011
45477499441,9cyclictest1387-21kerneloops10:40:320
45478599439,10cyclictest1753-21gnome-shell10:32:443
45478599434,16cyclictest569712-21taskset12:34:153
45478599434,16cyclictest515007-21taskset09:58:463
45478399432,25cyclictest1753-21gnome-shell08:43:122
45478399431,26cyclictest1753-21gnome-shell10:13:442
45478399431,26cyclictest1753-21gnome-shell08:54:522
45478399431,25cyclictest1753-21gnome-shell09:26:172
45477899438,11cyclictest1753-21gnome-shell08:40:281
45477899437,19cyclictest26-21ksoftirqd/108:00:251
45477899431,20cyclictest483647-21latency_hist08:30:011
45477899431,14cyclictest26-21ksoftirqd/110:00:281
45477899431,13cyclictest26-21ksoftirqd/108:55:291
454778994311,16cyclictest26-21ksoftirqd/111:35:191
5476412420,8sleep02197014-21PK-Backend11:30:290
45477899428,12cyclictest26-21ksoftirqd/108:30:181
454778994226,12cyclictest509635-21/usr/sbin/munin09:45:011
454778994216,5cyclictest141rcu_preempt12:26:321
454778994216,5cyclictest141rcu_preempt12:26:321
45477499421,4cyclictest141rcu_preempt07:15:040
45477499421,11cyclictest508282-21tr09:40:140
162399420,33rtkit-daemon2197014-21PK-Backend08:39:310
4894232410,5sleep02197014-21PK-Backend08:45:150
45477899413,20cyclictest26-21ksoftirqd/107:35:131
454774994123,6cyclictest13-21ksoftirqd/007:45:180
45477499411,35cyclictest506154-21turbostat09:35:000
45478599407,10cyclictest0-21swapper/309:44:493
454785994010,26cyclictest1753-21gnome-shell11:18:203
45477899404,5cyclictest141rcu_preempt11:00:021
45477899402,19cyclictest26-21ksoftirqd/110:45:301
45477899402,19cyclictest26-21ksoftirqd/110:45:291
454778994010,5cyclictest141rcu_preempt11:45:151
454774994023,5cyclictest13-21ksoftirqd/010:05:020
454774994022,6cyclictest13-21ksoftirqd/011:00:140
454774994018,9cyclictest141rcu_preempt10:00:010
454774994011,6cyclictest141rcu_preempt07:40:240
162399400,32rtkit-daemon0-21swapper/011:35:170
162399400,30rtkit-daemon2197014-21PK-Backend10:37:200
162399400,30rtkit-daemon2197014-21PK-Backend08:01:360
454785993910,25cyclictest463418-21cat07:30:303
45477899396,10cyclictest487118-21kworker/1:209:05:301
45477899391,14cyclictest507277-21taskset09:35:271
45477499396,11cyclictest483099-21timerandwakeup08:25:300
454774993923,12cyclictest13-21ksoftirqd/007:50:040
454774993919,8cyclictest141rcu_preempt10:25:140
454774993919,15cyclictest13-21ksoftirqd/011:45:550
454774993918,8cyclictest141rcu_preempt10:10:310
454774993918,8cyclictest141rcu_preempt09:05:160
454774993912,6cyclictest141rcu_preempt12:25:010
45478399386,13cyclictest2197014-21PK-Backend08:20:582
454783993821,12cyclictest455371-21sh07:10:212
454783993821,12cyclictest455371-21sh07:10:212
454774993821,6cyclictest13-21ksoftirqd/008:10:000
454774993821,12cyclictest13-21ksoftirqd/009:10:210
454774993820,13cyclictest13-21ksoftirqd/012:34:590
454774993819,7cyclictest141rcu_preempt12:15:300
454774993819,6cyclictest13-21ksoftirqd/011:30:010
454774993818,8cyclictest141rcu_preempt09:45:380
454774993816,9cyclictest141rcu_preempt08:34:590
454774993816,8cyclictest141rcu_preempt09:20:020
454774993811,10cyclictest502686-21ls09:25:000
45477899377,14cyclictest141rcu_preempt09:30:171
454774993719,13cyclictest13-21ksoftirqd/008:40:190
454774993718,7cyclictest13-21ksoftirqd/009:00:000
454774993716,8cyclictest141rcu_preempt11:10:010
454774993714,9cyclictest13-21ksoftirqd/008:00:010
454774993714,6cyclictest141rcu_preempt09:00:390
162399370,28rtkit-daemon2197014-21PK-Backend12:26:240
162399370,28rtkit-daemon2197014-21PK-Backend12:26:240
454774993619,13cyclictest13-21ksoftirqd/011:20:310
454774993619,12cyclictest13-21ksoftirqd/007:30:140
454774993617,7cyclictest141rcu_preempt08:10:220
454774993614,9cyclictest141rcu_preempt12:39:590
454774993612,5cyclictest141rcu_preempt11:15:160
45477499361,14cyclictest555954-21chrt11:55:210
45477499361,13cyclictest464106-21/usr/sbin/munin07:35:280
162399360,28rtkit-daemon2197014-21PK-Backend10:24:550
45477499354,12cyclictest141rcu_preempt07:25:010
454774993524,7cyclictest461313-21tail07:25:210
45477499352,30cyclictest0-21swapper/010:54:560
454774993518,6cyclictest13-21ksoftirqd/010:55:240
454774993516,7cyclictest141rcu_preempt10:30:330
454774993516,7cyclictest141rcu_preempt08:50:210
454774993515,8cyclictest141rcu_preempt12:00:160
454774993515,7cyclictest141rcu_preempt08:20:150
454774993513,9cyclictest141rcu_preempt11:40:160
454774993513,5cyclictest141rcu_preempt09:27:330
454774993423,6cyclictest13-21ksoftirqd/012:05:180
454774993415,7cyclictest141rcu_preempt12:10:240
454774993415,7cyclictest141rcu_preempt10:15:260
454774993415,7cyclictest141rcu_preempt09:35:300
454774993314,7cyclictest141rcu_preempt11:50:250
454774993215,6cyclictest141rcu_preempt08:15:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional