You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 08:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot6s.osadl.org (updated Tue Jul 01, 2025 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
223099880,5rtkit-daemon2229-21rtkit-daemon19:32:461
2673998528,41cyclictest0-21swapper/020:12:420
2673998430,46cyclictest12131-21crond21:12:210
2673998335,29cyclictest1265-21runrttasks20:21:520
2673998232,32cyclictest15334-21taskset21:17:400
2673998231,34cyclictest0-21swapper/000:07:150
2673998120,36cyclictest0-21swapper/023:57:160
2673998030,42cyclictest0-21swapper/021:42:360
2673997936,18cyclictest31061-21taskset19:34:110
2673997928,29cyclictest12464-21taskset20:02:520
2673997834,26cyclictest5155-21head23:12:300
2673997829,41cyclictest0-21swapper/019:22:390
2673997828,43cyclictest0-21swapper/020:37:230
2673997828,34cyclictest0-21swapper/020:37:370
2673997828,32cyclictest0-21swapper/021:27:380
2673997828,32cyclictest0-21swapper/021:27:380
2673997828,30cyclictest0-21swapper/019:42:510
2673997827,34cyclictest0-21swapper/021:02:430
2673997731,39cyclictest0-21swapper/021:12:390
2673997729,30cyclictest31555-21munin-node19:37:520
2673997630,39cyclictest0-21swapper/020:52:510
2673997630,37cyclictest6129-21munin-run22:07:200
2673997528,30cyclictest0-21swapper/020:32:240
2673997527,40cyclictest0-21swapper/022:32:500
2673997526,32cyclictest0-21swapper/023:07:410
2673997429,29cyclictest0-21swapper/021:51:560
2673997427,30cyclictest0-21swapper/018:47:460
2673997426,29cyclictest0-21swapper/018:58:030
2673997337,29cyclictest18429-21strings19:07:570
2673997332,34cyclictest1160-21snmpd23:26:530
2673997328,38cyclictest0-21swapper/021:32:410
2673997328,37cyclictest0-21swapper/023:02:420
2673997327,29cyclictest0-21swapper/021:40:550
2673997325,41cyclictest0-21swapper/022:12:550
2673997325,41cyclictest0-21swapper/022:12:550
2673997325,33cyclictest0-21swapper/022:52:320
2673997235,30cyclictest26783-21head19:27:370
2673997225,40cyclictest0-21swapper/023:47:450
2673997221,43cyclictest0-21swapper/021:22:580
2673997128,25cyclictest17420-21switchtime22:27:520
2673997126,29cyclictest0-21swapper/020:08:450
2673997125,40cyclictest0-21swapper/018:43:270
2673997124,33cyclictest0-21swapper/019:52:580
2673997120,31cyclictest0-21swapper/023:57:300
2673997029,33cyclictest12207-21crond18:57:270
2673996934,19cyclictest1265-21runrttasks19:17:390
2673996934,19cyclictest1265-21runrttasks19:17:390
2673996931,31cyclictest2898-21munin-node18:37:500
2673996931,31cyclictest2898-21munin-node18:37:500
2673996927,36cyclictest14869-21processes22:22:510
2673996926,14cyclictest0-21swapper/020:43:540
2673996925,38cyclictest0-21swapper/022:01:500
2673996925,30cyclictest0-21swapper/019:48:420
2673996922,32cyclictest0-21swapper/023:29:230
2673996921,40cyclictest0-21swapper/019:17:100
2673996848,12cyclictest2660-21munin-run18:37:280
2673996835,26cyclictest17078-21grep23:37:460
2673996835,26cyclictest17078-21grep23:37:460
2673996826,35cyclictest0-21swapper/022:07:340
2673996825,36cyclictest0-21swapper/022:42:190
2673996825,15cyclictest0-21swapper/019:02:510
2673996824,29cyclictest0-21swapper/023:42:440
2673996823,36cyclictest10099-21ntpq19:57:550
2673996819,16cyclictest0-21swapper/020:47:520
2673996819,16cyclictest0-21swapper/020:47:520
268199674,54cyclictest0-21swapper/323:42:493
2678996739,12cyclictest0-21swapper/121:02:441
2678996738,13cyclictest0-21swapper/118:52:591
2673996745,16cyclictest0-21swapper/021:52:420
2673996722,39cyclictest0-21swapper/022:51:000
2678996639,12cyclictest0-21swapper/118:37:591
2678996639,12cyclictest0-21swapper/118:37:591
2673996628,31cyclictest32293-21sshd23:01:030
2673996624,35cyclictest0-21swapper/023:33:290
2673996623,28cyclictest0-21swapper/022:42:470
2673996622,36cyclictest0-21swapper/020:22:450
2673996537,22cyclictest6043-21grep20:57:440
2678996435,13cyclictest0-21swapper/118:47:501
2673996420,37cyclictest0-21swapper/022:20:520
2673996330,27cyclictest0-21swapper/023:17:400
23092623,12sleep30-21swapper/321:57:423
268199614,50cyclictest0-21swapper/319:02:273
268199614,49cyclictest0-21swapper/321:57:213
2678996039,14cyclictest0-21swapper/120:53:101
223099600,8rtkit-daemon0-21swapper/122:33:013
223099600,7rtkit-daemon0-21swapper/122:37:543
223099600,7rtkit-daemon0-21swapper/122:02:552
223099600,7rtkit-daemon0-21swapper/119:47:532
268199595,46cyclictest1265-21runrttasks22:42:483
268199594,48cyclictest0-21swapper/320:32:243
268199594,48cyclictest0-21swapper/320:02:573
268199594,48cyclictest0-21swapper/300:07:163
223099590,7rtkit-daemon0-21swapper/119:28:012
268199585,46cyclictest0-21swapper/321:02:393
268199584,48cyclictest0-21swapper/320:27:243
268199584,47cyclictest0-21swapper/320:37:243
268199584,46cyclictest15515-21taskset20:09:293
162622583,12sleep10-21swapper/119:03:041
268199574,47cyclictest0-21swapper/320:22:253
268199574,46cyclictest0-21swapper/319:47:203
268199574,46cyclictest0-21swapper/319:31:443
268099574,46cyclictest0-21swapper/221:25:212
268099573,47cyclictest2252-21gdm-simple-gree18:46:262
268099573,47cyclictest0-21swapper/221:32:382
2678995738,12cyclictest0-21swapper/121:17:391
268199564,46cyclictest0-21swapper/322:52:463
268099567,44cyclictest0-21swapper/221:16:072
268099564,46cyclictest0-21swapper/222:17:352
268099564,45cyclictest1691-21tail19:42:432
268099563,46cyclictest24138-21latency_hist19:22:272
268099563,46cyclictest24138-21latency_hist19:22:272
268099563,46cyclictest0-21swapper/222:07:562
268099563,46cyclictest0-21swapper/221:12:222
268099562,47cyclictest32371-21sshd20:46:092
2678995634,8cyclictest32054-21diskstats21:52:411
2678995634,10cyclictest0-21swapper/123:22:431
267899563,47cyclictest0-21swapper/120:32:421
268099553,45cyclictest6438-21munin-node19:52:502
267899553,46cyclictest0-21swapper/120:03:221
2678995522,8cyclictest13732-21ntp_states18:57:571
268199543,46cyclictest0-21swapper/323:22:463
268199543,45cyclictest0-21swapper/321:27:503
268199543,45cyclictest0-21swapper/321:27:503
268199542,12cyclictest0-21swapper/323:57:303
268099543,45cyclictest0-21swapper/223:10:532
268099543,44cyclictest0-21swapper/220:02:412
268099543,44cyclictest0-21swapper/219:37:402
268099542,45cyclictest1160-21snmpd22:24:122
267899544,31cyclictest0-21swapper/122:57:391
25762542,9sleep20-21swapper/221:57:452
268199534,42cyclictest1160-21snmpd20:38:533
268199534,42cyclictest0-21swapper/320:12:393
268199533,44cyclictest1160-21snmpd21:14:003
268199533,42cyclictest1160-21snmpd20:50:583
268199533,42cyclictest1160-21snmpd20:50:583
268199532,42cyclictest361ktimersoftd/319:37:493
268099534,42cyclictest9680-21latency_hist23:22:172
2680995337,10cyclictest15618-21head21:17:432
267899535,29cyclictest0-21swapper/123:12:191
267899534,30cyclictest0-21swapper/122:57:181
2678995322,13cyclictest0-21swapper/122:02:061
2678995320,13cyclictest0-21swapper/100:03:581
2678995318,14cyclictest0-21swapper/119:31:441
109352533,42sleep20-21swapper/223:22:412
61582522,9sleep10-21swapper/119:51:571
322022523,11sleep10-21swapper/120:43:571
268199524,41cyclictest16920-21munin-node22:27:423
2681995222,12cyclictest0-21swapper/318:47:543
2680995238,9cyclictest1148-21grep20:47:462
2680995238,9cyclictest1148-21grep20:47:462
2680995233,10cyclictest27073-21cpuspeed_turbos21:42:392
268099523,43cyclictest0-21swapper/223:47:012
267899525,29cyclictest0-21swapper/121:07:491
2678995231,9cyclictest0-21swapper/119:17:311
2678995231,9cyclictest0-21swapper/119:17:311
2678995221,13cyclictest0-21swapper/121:37:221
2678995220,12cyclictest0-21swapper/119:55:121
265452523,41sleep126647-21users23:57:511
264192522,42sleep10-21swapper/121:41:481
71882512,42sleep10-21swapper/118:44:181
268199513,41cyclictest7418-21cat21:02:233
268199513,41cyclictest361ktimersoftd/322:57:433
2681995120,12cyclictest0-21swapper/322:17:503
2680995137,9cyclictest27689-21crond20:37:232
2680995135,9cyclictest21316-21unixbench_multi19:13:002
268099513,41cyclictest9143-21grep19:57:422
268099512,42cyclictest17634-21unixbench-2d20:12:582
267899515,28cyclictest0-21swapper/122:12:571
267899515,28cyclictest0-21swapper/122:12:571
267899515,28cyclictest0-21swapper/120:37:421
267899515,28cyclictest0-21swapper/119:12:511
267899514,30cyclictest0-21swapper/123:37:371
267899514,30cyclictest0-21swapper/123:37:371
267899514,29cyclictest0-21swapper/123:52:171
267899514,26cyclictest0-21swapper/123:42:281
2678995131,9cyclictest0-21swapper/119:23:591
156102512,9sleep10-21swapper/120:10:441
120262512,9sleep30-21swapper/321:11:033
36992502,40sleep10-21swapper/119:46:051
322252502,9sleep20-21swapper/223:00:122
293942503,7sleep229468-21ls00:07:152
288042502,9sleep30-21swapper/321:46:503
268199503,41cyclictest22944-21taskset21:32:463
268199503,39cyclictest14817-21telnet21:17:353
268199503,14cyclictest0-21swapper/319:07:453
2681995021,13cyclictest0-21swapper/323:49:283
2681995020,12cyclictest0-21swapper/322:17:203
2681995020,12cyclictest0-21swapper/322:17:203
268099502,9cyclictest0-21swapper/222:42:192
267899505,38cyclictest0-21swapper/123:12:411
267899505,28cyclictest0-21swapper/123:32:171
267899504,29cyclictest0-21swapper/122:37:371
267899504,28cyclictest0-21swapper/122:47:581
267899504,27cyclictest0-21swapper/119:42:261
267899503,42cyclictest0-21swapper/121:51:011
267899503,28cyclictest0-21swapper/119:07:541
2678995018,14cyclictest0-21swapper/121:27:331
2678995018,14cyclictest0-21swapper/121:27:331
268199494,38cyclictest8645-21memory23:17:433
268199494,27cyclictest0-21swapper/319:48:423
268199493,14cyclictest0-21swapper/320:42:363
268199493,12cyclictest0-21swapper/321:37:533
268099497,25cyclictest0-21swapper/219:32:472
2680994934,9cyclictest23325-21grep23:52:362
268099492,42cyclictest0-21swapper/223:48:322
268099492,42cyclictest0-21swapper/218:48:082
267899495,37cyclictest0-21swapper/122:02:521
267899495,28cyclictest0-21swapper/122:34:141
267899494,28cyclictest0-21swapper/121:22:541
267899494,27cyclictest0-21swapper/122:10:561
267899494,27cyclictest0-21swapper/121:47:211
267899494,24cyclictest0-21swapper/123:52:281
2678994932,10cyclictest23419-21grep22:42:331
2678994930,9cyclictest0-21swapper/122:18:061
2678994919,13cyclictest0-21swapper/122:31:541
2678994918,13cyclictest0-21swapper/122:22:451
202582492,41sleep10-21swapper/120:19:101
190582492,9sleep30-21swapper/321:24:293
178122492,9sleep10-21swapper/120:13:221
268199484,37cyclictest0-21swapper/322:07:473
268199483,8cyclictest0-21swapper/319:32:373
268199483,11cyclictest0-21swapper/322:47:493
2680994833,9cyclictest6056-21sed20:57:442
2680994833,9cyclictest16138-21grep23:37:322
2680994833,9cyclictest16138-21grep23:37:322
268099482,12cyclictest0-21swapper/218:58:282
267899486,13cyclictest0-21swapper/123:02:301
267899484,38cyclictest0-21swapper/120:47:381
267899484,38cyclictest0-21swapper/120:47:381
267899484,28cyclictest0-21swapper/121:14:001
2678994829,9cyclictest0-21swapper/119:57:411
2678994811,11cyclictest24440-21open_inodes20:27:541
173872482,9sleep30-21swapper/323:41:163
173872482,9sleep30-21swapper/323:41:163
153462482,9sleep30-21swapper/322:23:143
143862482,9sleep30-21swapper/323:31:133
268199475,36cyclictest0-21swapper/318:42:583
268199474,37cyclictest0-21swapper/323:56:263
268199474,26cyclictest0-21swapper/322:04:093
2681994732,10cyclictest16600-21unixbench_singl19:03:073
2681994730,12cyclictest3662-21cut23:07:393
268199473,13cyclictest0-21swapper/318:37:563
268199473,13cyclictest0-21swapper/318:37:563
268199472,39cyclictest0-21swapper/320:52:353
268099475,35cyclictest0-21swapper/220:27:372
2680994733,9cyclictest20201-21fschecks_count21:27:402
2680994733,9cyclictest20201-21fschecks_count21:27:402
2680994732,10cyclictest21576-21grep20:22:442
268099473,9cyclictest4337-21mailstats18:37:592
268099473,9cyclictest4337-21mailstats18:37:592
268099472,41cyclictest0-21swapper/221:49:222
268099472,41cyclictest0-21swapper/219:22:552
2680994714,20cyclictest0-21swapper/222:52:312
267899474,27cyclictest0-21swapper/120:23:501
267899474,26cyclictest0-21swapper/123:17:321
24602472,9sleep30-21swapper/323:06:583
192172472,38sleep20-21swapper/219:10:352
268199464,37cyclictest0-21swapper/318:57:273
268199464,35cyclictest0-21swapper/319:57:263
268199464,29cyclictest0-21swapper/323:34:243
268099462,38cyclictest23080-21munin-node22:42:372
267899465,22cyclictest0-21swapper/120:57:371
2678994616,12cyclictest0-21swapper/123:34:241
179242462,9sleep20-21swapper/222:30:262
108532462,9sleep30-21swapper/320:00:543
268199454,11cyclictest20340-21fschecks_time19:12:453
2681994530,10cyclictest6329-21needreboot23:12:443
2680994530,9cyclictest25792-21cat21:37:582
2680994530,10cyclictest7910-21grep21:02:402
268099453,35cyclictest0-21swapper/223:07:192
2680994529,11cyclictest0-21swapper/223:27:472
2680994526,14cyclictest0-21swapper/220:17:482
268099452,39cyclictest0-21swapper/222:12:462
268099452,39cyclictest0-21swapper/222:12:462
268099452,36cyclictest25475-21munin-node22:47:562
2680994516,11cyclictest15746-21latency_hist20:12:252
204262452,9sleep20-21swapper/222:36:592
12262452,9sleep20-21swapper/221:57:212
51282442,35sleep20-21swapper/223:12:302
2681994429,10cyclictest30509-21fgrep21:47:543
268099443,35cyclictest0-21swapper/220:52:402
2680994428,12cyclictest0-21swapper/220:38:002
267899445,13cyclictest0-21swapper/118:37:291
239492442,9sleep30-21swapper/319:20:073
239492442,9sleep30-21swapper/319:20:073
148282442,36sleep20-21swapper/219:02:372
2681994329,9cyclictest26029-21unixbench-2d19:23:003
268502422,9sleep20-21swapper/200:00:142
268099422,36cyclictest0-21swapper/218:52:582
268099418,28cyclictest0-21swapper/223:36:082
7022404,29sleep20-21swapper/218:32:432
16332318,14sleep30-21swapper/318:32:523
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional