You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-19 - 01:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Fri Jul 18, 2025 12:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
162399950,75rtkit-daemon0-21swapper/107:08:361
162399950,75rtkit-daemon0-21swapper/107:08:361
120925528570,9sleep30-21swapper/307:08:223
120925528570,9sleep30-21swapper/307:08:213
12915142780,4sleep10-21swapper/111:00:221
120882727452,15sleep00-21swapper/007:05:300
120882727452,15sleep00-21swapper/007:05:290
162399730,5rtkit-daemon0-21swapper/009:26:160
162399690,5rtkit-daemon2197014-21PK-Backend09:35:062
120953299689,37cyclictest1753-21gnome-shell09:35:323
120791026744,13sleep20-21swapper/207:05:082
120791026744,13sleep20-21swapper/207:05:072
12565192650,13sleep2341ksoftirqd/209:20:292
1209532996413,25cyclictest1753-21gnome-shell08:35:233
1209532996310,30cyclictest1324252-21chrt12:35:003
1209530996239,9cyclictest34-21ksoftirqd/211:55:032
1209532996111,27cyclictest1753-21gnome-shell07:25:023
1209532996111,27cyclictest1753-21gnome-shell07:25:023
162399600,9rtkit-daemon1622-21rtkit-daemon09:03:012
162399590,9rtkit-daemon1622-21rtkit-daemon09:43:393
120953299599,16cyclictest1753-21gnome-shell11:31:043
120953299594,14cyclictest884-21gdbus08:44:383
1209532995910,27cyclictest1753-21gnome-shell12:36:023
1209532995910,27cyclictest1753-21gnome-shell12:36:023
120953299589,28cyclictest1753-21gnome-shell11:22:493
1209532995811,26cyclictest1753-21gnome-shell11:55:293
1209532995811,16cyclictest1317762-21grep12:15:153
120953099580,54cyclictest1622-21rtkit-daemon07:27:232
162399570,10rtkit-daemon1622-21rtkit-daemon08:09:563
120953299579,27cyclictest1753-21gnome-shell11:00:173
120953299579,27cyclictest1753-21gnome-shell08:59:493
120953299574,16cyclictest1307765-21ps11:45:313
1209532995710,23cyclictest1753-21gnome-shell09:17:573
120953299569,26cyclictest1753-21gnome-shell10:30:233
120953299569,26cyclictest1753-21gnome-shell10:30:223
120953299568,27cyclictest1753-21gnome-shell07:50:163
120953299565,30cyclictest1753-21gnome-shell12:11:563
120953299565,30cyclictest1753-21gnome-shell10:00:293
120953299564,15cyclictest1753-21gnome-shell12:20:303
1209532995611,25cyclictest1753-21gnome-shell10:40:193
1209532995610,25cyclictest1753-21gnome-shell11:40:243
120952999569,11cyclictest1310593-21chrt11:55:041
120953299555,29cyclictest1753-21gnome-shell09:35:023
1209532995511,18cyclictest1257770-21ls09:25:143
1209529995528,8cyclictest26-21ksoftirqd/107:14:051
1209529995526,8cyclictest26-21ksoftirqd/107:51:121
1209529995526,7cyclictest26-21ksoftirqd/110:46:161
1209529995510,41cyclictest1622-21rtkit-daemon10:07:281
120953299549,19cyclictest1753-21gnome-shell10:38:193
1209532995411,23cyclictest1293837-21users11:05:333
1209532995411,18cyclictest1276026-21wc10:15:233
1209532995410,25cyclictest1233470-21df_inode08:15:163
1209529995419,18cyclictest26-21ksoftirqd/108:53:221
120952999540,50cyclictest1622-21rtkit-daemon11:22:261
120953299537,17cyclictest1938-21gsd-housekeepin10:28:273
1209532995311,17cyclictest1753-21gnome-shell07:16:293
1209532995311,12cyclictest1753-21gnome-shell12:00:203
1209532995310,11cyclictest1753-21gnome-shell09:04:163
1209529995321,12cyclictest26-21ksoftirqd/112:03:281
120952999530,49cyclictest1622-21rtkit-daemon12:34:351
120953299529,19cyclictest1294675-21cstates11:10:153
120953299525,19cyclictest1753-21gnome-shell09:59:053
120952999528,13cyclictest26-21ksoftirqd/112:25:341
1209529995226,8cyclictest26-21ksoftirqd/107:50:001
120953299518,25cyclictest1753-21gnome-shell11:54:233
120953299515,20cyclictest1753-21gnome-shell08:50:123
120953299514,28cyclictest1753-21gnome-shell09:07:453
120953299514,20cyclictest1267423-21needreboot09:50:243
120953299514,20cyclictest1267423-21needreboot09:50:243
1209532995111,24cyclictest1753-21gnome-shell07:55:353
1209532995111,11cyclictest1753-21gnome-shell07:30:143
1209530995125,9cyclictest34-21ksoftirqd/207:35:012
1209530995115,17cyclictest34-21ksoftirqd/212:00:252
12374392500,6sleep01753-21gnome-shell08:25:300
120953299509,12cyclictest1753-21gnome-shell08:21:443
120953299508,23cyclictest1753-21gnome-shell08:48:333
120953299508,23cyclictest1753-21gnome-shell08:48:333
120953299506,26cyclictest1296576-21df11:15:183
120953299506,17cyclictest1753-21gnome-shell07:26:283
120953299505,9cyclictest1289119-21cat10:55:033
120953299505,27cyclictest1753-21gnome-shell08:32:283
120953299505,26cyclictest1753-21gnome-shell11:27:393
120953299505,16cyclictest1272720-21ntp_states10:05:283
120953299505,15cyclictest1753-21gnome-shell10:10:293
120953299505,15cyclictest1753-21gnome-shell10:10:283
1209532995010,19cyclictest1753-21gnome-shell09:49:483
1209532995010,15cyclictest1753-21gnome-shell10:20:153
120952999508,19cyclictest26-21ksoftirqd/110:37:151
120952999505,18cyclictest26-21ksoftirqd/108:45:161
120952999505,18cyclictest26-21ksoftirqd/108:45:151
1209529995022,24cyclictest1285169-21cat10:40:331
120952999502,12cyclictest1753-21gnome-shell09:20:011
1209529995017,9cyclictest26-21ksoftirqd/111:15:441
1209529995011,19cyclictest26-21ksoftirqd/110:05:011
1209529995010,20cyclictest26-21ksoftirqd/109:35:161
120953299496,15cyclictest1220060-21python307:35:293
120953299495,16cyclictest1290891-21latency_hist11:00:023
120953299495,16cyclictest1228652-21ntp_states08:00:263
120953299495,16cyclictest1228652-21ntp_states08:00:253
1209532994910,19cyclictest1753-21gnome-shell09:10:423
1209529994916,17cyclictest26-21ksoftirqd/107:26:201
162399480,33rtkit-daemon2197014-21PK-Backend07:44:450
162399480,33rtkit-daemon2197014-21PK-Backend07:44:450
120953299489,24cyclictest1753-21gnome-shell07:46:113
120953299485,18cyclictest1221356-21diskstats07:40:183
120953299485,18cyclictest1221356-21diskstats07:40:173
120953299485,17cyclictest1210280-21cat07:10:223
120953299484,20cyclictest1237206-21sed08:25:243
1209530994813,31cyclictest34-21ksoftirqd/210:30:012
1209529994823,8cyclictest26-21ksoftirqd/108:10:001
1209529994810,14cyclictest26-21ksoftirqd/109:25:021
13148082470,15sleep234-21ksoftirqd/212:05:232
120953299475,16cyclictest1321050-21/usr/sbin/munin12:25:343
120953299475,15cyclictest1286906-21users10:45:303
120953299473,18cyclictest1233054-21date08:15:023
1209532994711,14cyclictest1753-21gnome-shell09:23:323
120953099471,23cyclictest1271681-21chrt10:04:572
120952999475,8cyclictest261ksoftirqd/108:00:011
1209529994737,8cyclictest251rcuc/111:30:221
120952999472,16cyclictest1753-21gnome-shell08:38:081
1209529994717,9cyclictest26-21ksoftirqd/108:40:161
1209529994713,17cyclictest26-21ksoftirqd/109:31:511
1209529994711,17cyclictest26-21ksoftirqd/110:30:241
1209529994711,17cyclictest26-21ksoftirqd/110:30:241
1209529994710,19cyclictest26-21ksoftirqd/108:10:311
120953299465,17cyclictest1304037-21memory11:35:263
1209532994611,17cyclictest1753-21gnome-shell12:09:363
120952999468,20cyclictest26-21ksoftirqd/111:45:311
1209529994627,14cyclictest438-20systemd-journal08:29:331
1209529994617,8cyclictest26-21ksoftirqd/109:50:151
1209529994617,8cyclictest26-21ksoftirqd/109:50:151
1209529994615,10cyclictest26-21ksoftirqd/110:12:471
1209529994615,10cyclictest26-21ksoftirqd/110:12:471
120952999461,22cyclictest26-21ksoftirqd/108:15:301
1209524994627,14cyclictest2027-21gdbus09:11:560
120952999459,8cyclictest26-21ksoftirqd/110:15:321
120952999458,20cyclictest26-21ksoftirqd/107:38:471
120952999457,20cyclictest26-21ksoftirqd/110:54:341
120952999456,17cyclictest26-21ksoftirqd/111:45:001
1209529994534,8cyclictest251rcuc/112:36:301
1209529994534,8cyclictest251rcuc/112:36:301
1209529994510,19cyclictest26-21ksoftirqd/111:00:001
120953099445,4cyclictest141rcu_preempt09:20:002
1209530994426,6cyclictest34-21ksoftirqd/208:26:242
1209530994419,5cyclictest141rcu_preempt09:10:342
1209530994417,23cyclictest34-21ksoftirqd/208:45:022
1209530994414,16cyclictest34-21ksoftirqd/210:55:022
1209530994410,9cyclictest34-21ksoftirqd/208:15:012
1209529994433,7cyclictest251rcuc/109:25:151
1209529994414,10cyclictest26-21ksoftirqd/111:05:241
162399430,25rtkit-daemon2197014-21PK-Backend11:08:422
1209530994320,14cyclictest34-21ksoftirqd/211:42:132
120952999438,16cyclictest26-21ksoftirqd/109:45:181
120952999432,16cyclictest26-21ksoftirqd/108:00:141
120952999432,16cyclictest26-21ksoftirqd/108:00:131
1209529994317,8cyclictest26-21ksoftirqd/110:25:171
120952999431,21cyclictest1753-21gnome-shell08:35:011
120952999431,17cyclictest26-21ksoftirqd/109:40:291
120952499435,15cyclictest17531gnome-shell09:20:010
12564112420,6chrt0-21swapper/009:20:270
120953099421,19cyclictest1210282-21wc07:10:232
120952999422,17cyclictest26-21ksoftirqd/107:15:051
1209529994219,8cyclictest26-21ksoftirqd/109:10:421
120952999421,24cyclictest1753-21gnome-shell07:44:101
120952999421,24cyclictest1753-21gnome-shell07:44:101
120952999421,13cyclictest26-21ksoftirqd/111:15:011
1209524994232,7cyclictest1069-21runrttasks08:22:230
1209524994214,7cyclictest141rcu_preempt10:54:370
120953099414,18cyclictest0-21swapper/207:45:142
1209530994125,12cyclictest1213413-21apt-get07:20:012
120953099412,4cyclictest141rcu_preempt12:40:012
120953099412,4cyclictest141rcu_preempt12:40:012
1209530994122,6cyclictest34-21ksoftirqd/209:40:112
1209530994112,9cyclictest34-21ksoftirqd/210:20:142
120953099411,20cyclictest34-21ksoftirqd/211:21:182
120953099411,11cyclictest1753-21gnome-shell10:55:162
120952999414,13cyclictest26-21ksoftirqd/109:08:491
120952999414,13cyclictest26-21ksoftirqd/108:59:501
1209529994115,8cyclictest26-21ksoftirqd/112:06:271
120952999411,19cyclictest1753-21gnome-shell12:22:301
120952999411,16cyclictest26-21ksoftirqd/110:22:181
120952999411,15cyclictest1218632-21sshd07:31:331
1209524994131,6cyclictest1276516-21users10:15:330
1209524994127,10cyclictest1243192-21colord-sane08:44:380
1209524994126,11cyclictest1298079-21rm11:20:020
1209524994123,14cyclictest1231731-21df08:10:140
120952499412,12cyclictest1753-21gnome-shell11:42:410
1209524994117,7cyclictest141rcu_preempt11:55:160
120952499411,18cyclictest1210994-21chrt07:12:400
162399400,5rtkit-daemon0-21swapper/008:15:560
1209530994029,8cyclictest34-21ksoftirqd/210:39:592
1209530994023,6cyclictest34-21ksoftirqd/210:31:082
1209530994023,6cyclictest34-21ksoftirqd/210:31:072
1209530994023,5cyclictest34-21ksoftirqd/212:27:252
1209530994022,6cyclictest34-21ksoftirqd/211:01:042
1209530994020,16cyclictest34-21ksoftirqd/208:15:132
1209530994016,8cyclictest34-21ksoftirqd/207:50:352
1209530994013,8cyclictest34-21ksoftirqd/208:05:172
1209530994012,8cyclictest34-21ksoftirqd/210:41:022
120953099401,19cyclictest1297163-21/usr/sbin/munin11:15:312
120953099401,16cyclictest438-20systemd-journal08:58:342
1209530994010,14cyclictest141rcu_preempt08:00:262
1209530994010,14cyclictest141rcu_preempt08:00:262
120952999407,18cyclictest26-21ksoftirqd/111:55:001
120952999404,19cyclictest26-21ksoftirqd/112:15:251
120952999403,20cyclictest26-21ksoftirqd/111:25:141
120952999403,12cyclictest26-21ksoftirqd/112:10:081
120952999402,10cyclictest26-21ksoftirqd/109:04:051
1209529994010,11cyclictest26-21ksoftirqd/107:20:331
1209529994010,11cyclictest26-21ksoftirqd/107:20:331
1209524994028,8cyclictest1279703-21grep10:25:290
120952499402,7cyclictest1753-21gnome-shell11:51:550
1209524994026,10cyclictest1266372-21sshd09:48:340
1209524994024,12cyclictest1313862-21cron12:05:000
120952499402,34cyclictest1753-21gnome-shell08:34:450
1209524994023,13cyclictest1269585-21users09:55:290
162399390,27rtkit-daemon2197014-21PK-Backend07:36:330
12121032390,6chrt2197014-21PK-Backend07:15:140
120953099398,9cyclictest34-21ksoftirqd/211:35:002
120953099397,12cyclictest34-21ksoftirqd/212:24:052
120953099393,32cyclictest1827-21JS14:31:442
1209530993929,7cyclictest1753-21gnome-shell07:57:072
1209530993922,13cyclictest34-21ksoftirqd/209:34:382
1209530993922,13cyclictest34-21ksoftirqd/208:33:562
1209530993921,6cyclictest34-21ksoftirqd/211:35:542
1209530993921,6cyclictest34-21ksoftirqd/210:49:042
1209530993921,6cyclictest34-21ksoftirqd/210:17:272
1209530993921,6cyclictest34-21ksoftirqd/208:49:342
1209530993921,6cyclictest34-21ksoftirqd/208:49:342
1209530993916,13cyclictest34-21ksoftirqd/209:25:262
120953099391,5cyclictest141rcu_preempt09:50:012
1209530993913,9cyclictest34-21ksoftirqd/211:55:042
1209530993910,9cyclictest34-21ksoftirqd/208:40:012
120952999395,18cyclictest26-21ksoftirqd/109:55:141
1209529993928,8cyclictest26-21ksoftirqd/108:22:521
120952499397,19cyclictest141rcu_preempt10:25:010
1209524993928,7cyclictest1753-21gnome-shell08:37:190
120952499392,33cyclictest1753-21gnome-shell09:01:200
1209524993923,12cyclictest1245239-21find08:50:090
162399380,26rtkit-daemon2197014-21PK-Backend07:57:120
1209530993828,7cyclictest1753-21gnome-shell08:53:392
1209530993827,7cyclictest1753-21gnome-shell07:35:222
1209530993822,6cyclictest34-21ksoftirqd/208:22:572
1209530993821,6cyclictest34-21ksoftirqd/212:13:292
1209530993821,6cyclictest34-21ksoftirqd/209:50:352
1209530993821,6cyclictest34-21ksoftirqd/209:50:342
1209530993821,6cyclictest34-21ksoftirqd/209:05:142
1209530993821,5cyclictest34-21ksoftirqd/212:31:282
1209530993821,5cyclictest34-21ksoftirqd/212:17:572
1209530993821,5cyclictest34-21ksoftirqd/211:10:452
1209530993820,6cyclictest34-21ksoftirqd/211:45:292
120953099381,18cyclictest1753-21gnome-shell07:20:152
120953099381,18cyclictest1753-21gnome-shell07:20:152
1209524993821,12cyclictest1317508-21date12:15:020
1209524993814,6cyclictest141rcu_preempt07:45:190
1209524993811,6cyclictest141rcu_preempt12:05:320
1209530993726,7cyclictest1753-21gnome-shell10:10:382
1209530993726,7cyclictest1753-21gnome-shell10:10:372
1209530993719,5cyclictest34-21ksoftirqd/211:25:052
1209530993711,6cyclictest141rcu_preempt10:00:002
1209524993728,6cyclictest0-21swapper/011:25:290
1209524993725,8cyclictest1753-21gnome-shell12:15:120
1209524993713,7cyclictest141rcu_preempt12:35:180
1209524993713,7cyclictest141rcu_preempt12:35:180
1209524993713,6cyclictest141rcu_preempt08:00:190
1209524993713,6cyclictest141rcu_preempt08:00:180
1209530993621,5cyclictest34-21ksoftirqd/207:41:572
1209530993621,5cyclictest34-21ksoftirqd/207:41:572
1209529993611,10cyclictest26-21ksoftirqd/111:36:571
120952499369,5cyclictest141rcu_preempt09:10:000
120952499368,6cyclictest141rcu_preempt09:36:500
120952499362,9cyclictest1753-21gnome-shell11:34:100
120952499362,5cyclictest0-21swapper/011:06:560
120952499362,31cyclictest0-21swapper/008:59:460
1209524993622,9cyclictest1753-21gnome-shell07:35:000
120952499362,29cyclictest1284471-21diskstats10:40:180
120952499362,12cyclictest0-21swapper/011:35:520
1209524993619,13cyclictest1295428-21threads11:10:310
1209524993615,5cyclictest141rcu_preempt10:46:190
120952499361,4cyclictest141rcu_preempt12:25:000
120952499352,8cyclictest1753-21gnome-shell12:25:370
120952499352,6cyclictest1262617-21apt-get09:40:060
1209524993525,6cyclictest1244991-21sh08:50:000
1209524993525,6cyclictest1244991-21sh08:50:000
1209524993523,9cyclictest1268257-21awk09:55:000
1209524993523,9cyclictest1268257-21awk09:55:000
1209524993522,9cyclictest1753-21gnome-shell08:08:150
1209524993520,11cyclictest1291887-21python311:00:290
120952499346,7cyclictest141rcu_preempt10:10:230
120952499346,7cyclictest141rcu_preempt10:10:230
1209524993422,8cyclictest1753-21gnome-shell10:05:050
1209524993412,5cyclictest141rcu_preempt11:15:450
1209524993326,4cyclictest2197014-21PK-Backend07:26:530
1209524993322,7cyclictest1308369-21sh11:50:010
1209524993321,8cyclictest1753-21gnome-shell09:35:010
1209524993321,7cyclictest1270987-21ntp_states10:00:260
1209524993318,11cyclictest1282523-21grep10:35:150
1209524993316,12cyclictest2962348-21rs:main0
1209524993316,12cyclictest2962348-21rs:main0
120952499321,8cyclictest2197014-21PK-Backend10:56:110
1209524993218,10cyclictest1323015-21cat12:30:140
1209524993216,12cyclictest1281216-21missed_timers10:30:230
1209524993216,12cyclictest1281216-21missed_timers10:30:230
120952499281,4cyclictest2197014-21PK-Backend07:51:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional