You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-21 - 07:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Fri Mar 21, 2025 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12947352930,6sleep21294200-21apt-get19:50:082
128068828161,14sleep30-21swapper/319:07:323
162399760,10rtkit-daemon1506-21systemd20:41:311
162399750,62rtkit-daemon0-21swapper/219:05:322
13190222730,7sleep10-21swapper/120:55:271
128087027340,27sleep10-21swapper/119:09:511
162399690,4rtkit-daemon0-21swapper/100:39:171
128103699650,61cyclictest1622-21rtkit-daemon20:47:580
128103699650,61cyclictest1622-21rtkit-daemon20:47:580
128103699623,30cyclictest1753-21gnome-shell22:36:570
127940126040,14sleep00-21swapper/019:05:050
1281044995928,12cyclictest34-21ksoftirqd/219:35:262
1281044995928,11cyclictest34-21ksoftirqd/223:15:162
13237712570,6sleep11323763-21cron21:14:591
13104312570,17sleep342-21ksoftirqd/320:30:303
1281044995611,22cyclictest34-21ksoftirqd/221:40:012
162399550,8rtkit-daemon1622-21rtkit-daemon00:17:421
162399550,11rtkit-daemon1335163-21chrt21:50:323
13620492550,13sleep3411rcuc/323:20:233
1281046995521,24cyclictest42-21ksoftirqd/320:25:033
162399540,8rtkit-daemon1622-21rtkit-daemon19:49:000
12988362540,6sleep3411rcuc/320:00:153
12968002540,5chrt0-21swapper/119:55:141
1281044995424,12cyclictest34-21ksoftirqd/219:12:142
1281044995424,12cyclictest34-21ksoftirqd/200:30:282
1281044995423,12cyclictest34-21ksoftirqd/200:15:142
1281036995427,23cyclictest1326587-21sed21:20:240
128103699536,16cyclictest13-21ksoftirqd/000:00:040
128103699536,16cyclictest13-21ksoftirqd/000:00:030
128103699534,27cyclictest1368348-21chrt23:41:090
13187092520,6chrt0-21swapper/320:55:213
1281044995220,12cyclictest34-21ksoftirqd/220:05:252
1281044995220,12cyclictest34-21ksoftirqd/220:05:242
162399510,39rtkit-daemon1753-21gnome-shell19:23:132
1281044995117,15cyclictest34-21ksoftirqd/200:12:542
128103899511,10cyclictest1357634-21perf23:10:001
128103699516,16cyclictest13-21ksoftirqd/000:25:060
1281036995116,6cyclictest141rcu_preempt23:08:410
13797352500,9chrt1753-21gnome-shell00:20:203
1281044995017,13cyclictest34-21ksoftirqd/220:00:162
1281036995024,8cyclictest13-21ksoftirqd/022:33:500
162399490,14rtkit-daemon1753-21gnome-shell20:06:343
162399490,14rtkit-daemon1753-21gnome-shell20:06:343
128104499494,18cyclictest34-21ksoftirqd/220:30:252
1281044994919,15cyclictest34-21ksoftirqd/200:05:142
1281044994915,26cyclictest34-21ksoftirqd/223:41:342
1281044994914,20cyclictest34-21ksoftirqd/219:15:172
128104499484,6cyclictest141rcu_preempt22:55:182
1281044994836,8cyclictest34-21ksoftirqd/222:35:022
1281044994831,10cyclictest331rcuc/223:11:092
1281044994818,16cyclictest34-21ksoftirqd/223:39:392
128103699487,16cyclictest13-21ksoftirqd/000:35:210
1281036994814,13cyclictest2962348-21rs:main0
1281036994812,18cyclictest13-21ksoftirqd/023:30:080
162399470,39rtkit-daemon1506-21systemd20:05:210
162399470,39rtkit-daemon1506-21systemd20:05:210
13259482470,7sleep1251rcuc/121:20:041
1281044994729,10cyclictest331rcuc/219:32:582
1281044994729,10cyclictest331rcuc/219:32:582
1281044994712,20cyclictest34-21ksoftirqd/220:10:032
1281044994710,12cyclictest34-21ksoftirqd/219:47:272
128103699477,16cyclictest13-21ksoftirqd/020:35:230
1281036994710,12cyclictest1069-21runrttasks21:54:000
1281046994625,16cyclictest42-21ksoftirqd/322:40:173
1281044994616,15cyclictest34-21ksoftirqd/200:37:152
1281044994614,14cyclictest34-21ksoftirqd/223:55:102
1281044994614,14cyclictest34-21ksoftirqd/223:55:102
1281044994612,19cyclictest34-21ksoftirqd/220:54:132
1281038994624,8cyclictest1753-21gnome-shell23:10:091
1281038994619,22cyclictest26-21ksoftirqd/122:10:101
128103699468,17cyclictest13-21ksoftirqd/019:15:010
128103699468,15cyclictest13-21ksoftirqd/022:55:020
128103699468,15cyclictest13-21ksoftirqd/022:55:020
128103699467,19cyclictest1343226-21taskset22:19:090
128103699467,16cyclictest13-21ksoftirqd/000:32:000
128103699467,15cyclictest13-21ksoftirqd/023:20:130
128103699465,18cyclictest1298030-40lsb_release19:57:230
128103699464,10cyclictest13-21ksoftirqd/021:40:010
1281036994622,8cyclictest13-21ksoftirqd/022:27:300
1281036994622,8cyclictest13-21ksoftirqd/000:23:090
128103699462,11cyclictest3242253-21nautilus23:50:010
128103699461,10cyclictest0-21swapper/020:20:260
128103699461,10cyclictest0-21swapper/020:20:260
162399450,32rtkit-daemon0-21swapper/319:36:223
1281046994524,16cyclictest42-21ksoftirqd/319:20:233
1281044994527,10cyclictest331rcuc/223:20:242
1281044994527,10cyclictest331rcuc/219:29:032
1281044994524,6cyclictest141rcu_preempt19:40:272
1281044994515,17cyclictest34-21ksoftirqd/221:20:012
1281044994515,16cyclictest34-21ksoftirqd/223:26:282
1281044994515,15cyclictest34-21ksoftirqd/223:46:002
1281044994515,12cyclictest34-21ksoftirqd/221:20:092
1281044994514,12cyclictest34-21ksoftirqd/220:59:302
128104499451,21cyclictest34-21ksoftirqd/221:55:182
1281038994522,13cyclictest26-21ksoftirqd/120:10:171
128103699457,16cyclictest13-21ksoftirqd/020:12:180
128103699457,15cyclictest13-21ksoftirqd/021:00:330
128103699456,16cyclictest13-21ksoftirqd/000:20:010
128103699455,19cyclictest1299784-21seq20:02:170
128103699453,21cyclictest13-21ksoftirqd/019:36:170
1281036994531,9cyclictest1325506-21turbostat21:20:000
128103699453,17cyclictest1753-21gnome-shell23:17:140
128103699452,18cyclictest1753-21gnome-shell21:14:580
128103699451,25cyclictest438-20systemd-journal22:05:060
128103699451,12cyclictest1753-21gnome-shell22:23:160
128103699451,11cyclictest1-21systemd20:45:010
1281036994511,17cyclictest13-21ksoftirqd/019:18:330
1281036994510,16cyclictest13-21ksoftirqd/022:43:020
162399440,4rtkit-daemon2197014-21PK-Backend21:13:043
128104699442,9cyclictest1753-21gnome-shell22:34:483
128104699442,4cyclictest141rcu_preempt21:40:343
1281044994422,8cyclictest34-21ksoftirqd/222:27:052
1281044994415,10cyclictest34-21ksoftirqd/220:00:002
1281044994413,13cyclictest34-21ksoftirqd/220:47:092
1281044994413,13cyclictest34-21ksoftirqd/220:47:092
1281044994411,8cyclictest34-21ksoftirqd/220:20:172
1281044994411,8cyclictest34-21ksoftirqd/220:20:162
1281038994427,13cyclictest1347439-21pool-org.gnome.22:35:021
1281038994418,11cyclictest141rcu_preempt00:00:041
128103899441,6cyclictest1345704-21unin-run22:30:011
128103699448,16cyclictest13-21ksoftirqd/020:29:590
128103699447,16cyclictest13-21ksoftirqd/023:05:010
128103699447,16cyclictest13-21ksoftirqd/021:06:060
128103699447,15cyclictest13-21ksoftirqd/020:34:050
128103699444,18cyclictest1753-21gnome-shell22:55:210
128103699443,16cyclictest1374793-21chrt00:05:320
128103699443,10cyclictest0-21swapper/019:33:010
128103699443,10cyclictest0-21swapper/019:33:000
1281036994417,7cyclictest141rcu_preempt20:50:250
12901622430,5sleep1251rcuc/119:35:021
12901622430,5sleep1251rcuc/119:35:021
1281044994316,10cyclictest1753-21gnome-shell23:51:122
1281044994315,14cyclictest34-21ksoftirqd/220:27:522
1281044994314,9cyclictest34-21ksoftirqd/222:04:052
128104499431,22cyclictest34-21ksoftirqd/223:06:572
1281038994324,15cyclictest1753-21gnome-shell00:11:281
1281038994322,13cyclictest26-21ksoftirqd/121:15:241
128103699438,16cyclictest13-21ksoftirqd/021:30:290
128103699438,16cyclictest13-21ksoftirqd/021:30:280
128103699438,16cyclictest13-21ksoftirqd/020:58:480
128103699437,20cyclictest13-21ksoftirqd/019:20:110
128103699437,18cyclictest13-21ksoftirqd/000:13:200
128103699437,17cyclictest13-21ksoftirqd/023:35:190
128103699437,14cyclictest13-21ksoftirqd/019:50:100
1281036994334,5cyclictest161rcuc/022:47:140
128103699432,21cyclictest1753-21gnome-shell23:28:080
128103699431,20cyclictest13-21ksoftirqd/021:55:190
13584142420,8sleep30-21swapper/323:10:173
1281046994222,15cyclictest855-21dbus-daemon00:00:033
1281046994222,15cyclictest855-21dbus-daemon00:00:033
1281046994222,15cyclictest42-21ksoftirqd/321:15:203
1281046994221,16cyclictest42-21ksoftirqd/322:00:263
1281046994218,20cyclictest1295465-21ntpq19:50:273
128104499429,18cyclictest34-21ksoftirqd/220:41:212
128104499428,18cyclictest34-21ksoftirqd/222:36:172
128104499427,19cyclictest34-21ksoftirqd/221:49:142
128104499423,19cyclictest34-21ksoftirqd/222:40:172
128104499423,10cyclictest34-21ksoftirqd/221:05:572
128104499422,14cyclictest0-21swapper/223:04:282
1281044994218,5cyclictest141rcu_preempt20:15:392
1281044994217,9cyclictest34-21ksoftirqd/221:40:592
1281044994212,5cyclictest141rcu_preempt22:47:252
128104499421,15cyclictest1340605-21kernelversion22:10:172
128103899427,5cyclictest141rcu_preempt20:20:001
1281038994225,13cyclictest1296277-21pool-org.gnome.19:55:011
1281038994225,12cyclictest438-20systemd-journal23:45:001
1281038994211,8cyclictest141rcu_preempt21:53:141
128103699427,17cyclictest13-21ksoftirqd/022:12:020
128103699427,16cyclictest13-21ksoftirqd/023:10:120
128103699427,16cyclictest13-21ksoftirqd/019:43:060
128103699426,16cyclictest13-21ksoftirqd/019:25:310
128103699425,18cyclictest0-21swapper/000:01:300
128103699423,18cyclictest1753-21gnome-shell21:48:310
128103699423,17cyclictest1753-21gnome-shell21:26:340
162399410,29rtkit-daemon2197014-21PK-Backend20:51:493
128104699412,35cyclictest1753-21gnome-shell20:25:003
128104699412,35cyclictest1753-21gnome-shell20:25:003
128104699412,35cyclictest1753-21gnome-shell00:29:553
128104699412,13cyclictest1753-21gnome-shell21:00:283
128104499419,12cyclictest34-21ksoftirqd/221:30:022
128104499417,13cyclictest34-21ksoftirqd/200:25:012
128104499412,20cyclictest34-21ksoftirqd/221:50:162
128104499412,20cyclictest34-21ksoftirqd/200:27:162
128104499412,20cyclictest1753-21gnome-shell20:37:532
1281044994120,13cyclictest34-21ksoftirqd/222:22:072
1281044994115,6cyclictest141rcu_preempt22:16:472
128104499411,21cyclictest34-21ksoftirqd/221:10:112
1281044994110,13cyclictest34-21ksoftirqd/200:00:292
128103899413,15cyclictest26-21ksoftirqd/119:45:241
1281038994127,9cyclictest26-21ksoftirqd/100:00:021
1281038994127,9cyclictest26-21ksoftirqd/100:00:021
1281038994123,6cyclictest26-21ksoftirqd/121:45:021
1281038994123,14cyclictest26-21ksoftirqd/123:20:291
1281038994122,15cyclictest26-21ksoftirqd/120:50:271
1281038994122,14cyclictest26-21ksoftirqd/119:35:331
1281038994121,5cyclictest26-21ksoftirqd/122:39:461
1281038994110,14cyclictest26-21ksoftirqd/121:28:061
128103699416,16cyclictest13-21ksoftirqd/023:50:110
1281036994110,15cyclictest13-21ksoftirqd/020:17:430
1281036994110,14cyclictest13-21ksoftirqd/021:40:560
162399400,29rtkit-daemon2197014-21PK-Backend22:06:583
1281046994025,11cyclictest42-21ksoftirqd/323:03:083
1281046994023,13cyclictest1371039-21/usr/sbin/munin23:55:013
1281046994020,15cyclictest42-21ksoftirqd/321:20:293
128104499409,15cyclictest34-21ksoftirqd/221:02:262
128104499403,18cyclictest34-21ksoftirqd/221:30:432
128104499403,18cyclictest34-21ksoftirqd/221:30:432
128104499401,14cyclictest34-21ksoftirqd/222:55:022
128104499401,14cyclictest34-21ksoftirqd/222:55:012
1281038994023,6cyclictest26-21ksoftirqd/121:55:151
1281038994023,12cyclictest26-21ksoftirqd/119:15:021
1281038994022,6cyclictest26-21ksoftirqd/123:01:011
1281038994020,6cyclictest26-21ksoftirqd/123:40:021
128103899401,18cyclictest854-21avahi-daemon23:50:331
1281038994011,16cyclictest26-21ksoftirqd/122:01:521
162399390,28rtkit-daemon2197014-21PK-Backend23:16:183
1281046993919,15cyclictest42-21ksoftirqd/321:55:263
1281046993917,10cyclictest1385030-21date00:40:023
1281046993912,23cyclictest0-21swapper/323:37:193
128104499394,19cyclictest34-21ksoftirqd/223:30:252
1281044993916,5cyclictest141rcu_preempt22:05:292
1281038993928,7cyclictest1753-21gnome-shell21:46:491
1281038993924,11cyclictest1365186-21cron23:35:001
1281038993923,5cyclictest26-21ksoftirqd/119:40:391
1281038993922,13cyclictest26-21ksoftirqd/122:40:191
1281038993922,13cyclictest26-21ksoftirqd/119:15:141
1281038993922,13cyclictest26-21ksoftirqd/100:30:251
1281038993921,14cyclictest26-21ksoftirqd/100:23:541
1281038993920,6cyclictest26-21ksoftirqd/120:10:021
1281038993920,6cyclictest26-21ksoftirqd/120:10:011
1281038993919,15cyclictest1352987-21sed22:50:251
1281038993919,15cyclictest1352987-21sed22:50:251
128103899391,8cyclictest0-21swapper/120:24:311
128103899391,8cyclictest0-21swapper/120:24:311
1281038993918,16cyclictest26-21ksoftirqd/122:07:281
1281038993913,15cyclictest141rcu_preempt20:37:061
128103899391,21cyclictest1331490-21chrt21:38:141
162399380,27rtkit-daemon0-21swapper/321:39:473
162399380,25rtkit-daemon0-21swapper/319:41:363
13292012382,30sleep11329207-21ntp_states21:30:341
13292012382,30sleep11329207-21ntp_states21:30:341
1281046993819,14cyclictest42-21ksoftirqd/322:39:423
128103899387,6cyclictest141rcu_preempt20:45:521
128103899387,6cyclictest141rcu_preempt20:45:521
128103899387,12cyclictest0-21swapper/122:22:361
128103899382,32cyclictest1753-21gnome-shell22:46:231
1281038993821,6cyclictest26-21ksoftirqd/120:25:051
1281038993820,14cyclictest26-21ksoftirqd/119:20:251
1281038993812,11cyclictest141rcu_preempt21:02:191
128104699377,25cyclictest1753-21gnome-shell19:25:463
1281046993722,11cyclictest42-21ksoftirqd/300:05:053
128104699372,15cyclictest1753-21gnome-shell19:10:303
1281046993721,12cyclictest1303005-21ntpq20:10:283
128104699371,9cyclictest2197014-21PK-Backend00:15:253
128104699371,21cyclictest175350gnome-shell19:57:143
1281046993710,16cyclictest1753-21gnome-shell23:48:403
1281038993722,11cyclictest1369329-21systemd23:50:011
1281038993722,11cyclictest1354580-21open_inodes22:55:291
1281038993721,5cyclictest26-21ksoftirqd/122:18:051
128103899371,9cyclictest1310805-21chrt20:32:301
1281038993717,6cyclictest26-21ksoftirqd/123:20:021
128104699366,9cyclictest141rcu_preempt22:20:183
1281046993624,8cyclictest1753-21gnome-shell20:40:333
1281046993624,8cyclictest1351806-21sh22:50:003
128104699362,12cyclictest1753-21gnome-shell22:26:243
1281046993621,11cyclictest1354396-21memory22:55:243
1281038993620,13cyclictest251rcuc/123:28:111
1281038993619,6cyclictest26-21ksoftirqd/119:25:261
1281038993618,13cyclictest26-21ksoftirqd/120:01:211
13218392352,6sleep31753-21gnome-shell21:07:393
1281046993525,7cyclictest1753-21gnome-shell19:50:013
1281046993524,7cyclictest1753-21gnome-shell19:30:343
1281046993524,7cyclictest1753-21gnome-shell19:30:343
1281046993524,6cyclictest42-21ksoftirqd/322:12:413
1281046993523,8cyclictest1753-21gnome-shell20:36:103
128104699352,28cyclictest1753-21gnome-shell23:05:063
1281046993519,12cyclictest1367555-21cat23:40:153
128104699351,15cyclictest1753-21gnome-shell21:49:313
128104699351,15cyclictest1284925-21idleruntime-cro19:20:003
128104699351,13cyclictest135341050journal-offline22:53:343
128104699351,13cyclictest135341050journal-offline22:53:343
1281038993517,7cyclictest141rcu_preempt21:06:091
1281046993421,10cyclictest42-21ksoftirqd/320:49:263
1281046993421,10cyclictest42-21ksoftirqd/320:49:263
128104699341,8cyclictest2197014-21PK-Backend23:30:353
1281046993417,12cyclictest1304193-21cut20:15:123
1281046993416,13cyclictest1376097-21cut00:10:193
128104699341,5cyclictest2197014-21PK-Backend21:26:363
128103899341,12cyclictest1381027-21taskset00:25:581
1281046993321,8cyclictest1753-21gnome-shell00:30:253
1281046993317,12cyclictest1342266-21basename22:15:213
128104699331,27cyclictest1372800-40updatedb.mlocat00:00:043
128104699322,26cyclictest1753-21gnome-shell21:30:183
128104699322,26cyclictest1753-21gnome-shell21:30:173
128104699321,4cyclictest2197014-21PK-Backend23:25:203
1281038993221,6cyclictest26-21ksoftirqd/100:05:061
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional