You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 18:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Fri Nov 28, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291122238163,24sleep20-21swapper/207:06:542
290722237162,24sleep00-21swapper/007:06:240
291062236161,23sleep10-21swapper/107:06:491
292532204168,24sleep30-21swapper/307:08:443
29468996462,1cyclictest0-21swapper/211:40:252
29468995956,2cyclictest0-21swapper/210:46:212
306322580,4sleep02945799cyclictest09:35:250
29468995652,3cyclictest0-21swapper/208:12:012
80812530,3sleep12946299cyclictest11:10:281
54012490,2sleep20-21swapper/211:05:242
29468994946,2cyclictest0-21swapper/209:23:442
29468994932,2cyclictest10336-21kworker/2:207:37:192
29468994737,2cyclictest10336-21kworker/2:207:56:412
29468994636,2cyclictest4961-21kworker/2:012:30:322
29468994331,3cyclictest4961-21kworker/2:009:33:552
29468994230,2cyclictest4961-21kworker/2:011:37:482
29468993926,3cyclictest4961-21kworker/2:009:36:252
29473993825,3cyclictest0-21swapper/307:56:413
29468993828,2cyclictest4961-21kworker/2:010:16:152
29468993828,2cyclictest10336-21kworker/2:207:24:232
2945799388,14cyclictest26963-21munin-run09:30:010
29457993810,13cyclictest4349-21munin-run07:25:010
29468993726,3cyclictest4961-21kworker/2:009:01:152
29468993725,3cyclictest4961-21kworker/2:012:04:032
29468993724,6cyclictest4961-21kworker/2:012:36:112
2946899371,14cyclictest4961-21kworker/2:011:20:172
29468993626,2cyclictest10336-21kworker/2:207:40:432
29468993625,3cyclictest4961-21kworker/2:010:53:592
29468993625,3cyclictest4961-21kworker/2:010:53:592
29468993624,3cyclictest10336-21kworker/2:208:37:392
29468993624,3cyclictest10336-21kworker/2:208:37:392
2945799367,14cyclictest8742-21munin-run07:35:010
2947399351,22cyclictest0-21swapper/312:36:033
29468993525,2cyclictest10336-21kworker/2:207:30:412
29468993523,2cyclictest4961-21kworker/2:008:56:392
29468993519,2cyclictest10336-21kworker/2:207:25:112
29468993517,3cyclictest4961-21kworker/2:008:52:432
2947399341,27cyclictest0-21swapper/308:12:473
29468993422,5cyclictest4961-21kworker/2:011:16:352
2946899342,22cyclictest4961-21kworker/2:010:21:352
29468993418,3cyclictest4961-21kworker/2:010:25:352
29473993316,8cyclictest0-21swapper/311:02:593
29468993330,2cyclictest4961-21kworker/2:011:02:592
29468993320,3cyclictest4961-21kworker/2:009:25:452
192792330,4sleep02945799cyclictest11:38:040
2947399320,24cyclictest0-21swapper/310:29:473
29468993221,3cyclictest4961-21kworker/2:010:12:352
29468993220,3cyclictest4961-21kworker/2:012:28:582
29468993219,3cyclictest10336-21kworker/2:208:15:412
254622320,4sleep1221ktimersoftd/109:25:141
29468993120,3cyclictest4961-21kworker/2:009:19:072
29468993116,7cyclictest4961-21kworker/2:011:10:052
29468993111,3cyclictest4961-21kworker/2:009:42:572
2947399303,4cyclictest780-21rngd11:50:163
29468993020,2cyclictest10336-21kworker/2:207:46:252
29468993019,2cyclictest4961-21kworker/2:011:33:242
29468993019,2cyclictest4961-21kworker/2:011:26:352
29468993015,3cyclictest4961-21kworker/2:012:22:032
2947399294,5cyclictest780-21rngd11:15:273
29473992916,3cyclictest0-21swapper/310:16:153
29468992924,3cyclictest4961-21kworker/2:010:41:032
2946899292,19cyclictest4961-21kworker/2:010:30:432
29468992920,2cyclictest10336-21kworker/2:207:14:552
29468992919,2cyclictest10336-21kworker/2:208:22:392
29468992912,4cyclictest4961-21kworker/2:009:55:232
29457992911,15cyclictest11282-21munin-run12:35:010
2947399286,8cyclictest780-21rngd12:08:033
29473992821,3cyclictest0-21swapper/312:30:333
29473992815,4cyclictest0-21swapper/307:10:133
2947399281,12cyclictest0-21swapper/309:12:433
29468992818,5cyclictest4961-21kworker/2:010:06:412
29468992818,2cyclictest10336-21kworker/2:208:00:072
2945799285,11cyclictest13154-21perf07:45:010
289342280,4sleep341-21ksoftirqd/308:20:133
29473992717,3cyclictest0-21swapper/307:37:193
29473992713,8cyclictest0-21swapper/309:32:273
29473992711,3cyclictest0-21swapper/308:56:393
2947399271,11cyclictest0-21swapper/309:54:273
2946899271,12cyclictest4961-21kworker/2:009:13:372
2945799271,12cyclictest18193-21perf09:10:010
2947399265,3cyclictest780-21rngd10:56:593
2947399261,12cyclictest0-21swapper/309:57:273
29468992617,2cyclictest4961-21kworker/2:011:50:352
29468992615,2cyclictest4961-21kworker/2:009:05:312
29468992612,3cyclictest4961-21kworker/2:011:49:472
2945799269,14cyclictest29822-21basename10:50:010
2947399256,7cyclictest780-21rngd12:01:453
2947399253,4cyclictest780-21rngd11:40:413
29473992517,7cyclictest86750irq/128-enp1s0-07:40:393
29473992516,3cyclictest0-21swapper/311:37:473
2947399250,16cyclictest0-21swapper/309:45:433
2947399250,16cyclictest0-21swapper/309:45:433
2946899258,4cyclictest86850irq/129-enp1s0-08:40:192
2946899258,4cyclictest86850irq/129-enp1s0-08:40:192
29468992516,2cyclictest10336-21kworker/2:208:07:372
2946899251,15cyclictest4961-21kworker/2:008:47:232
29468992511,2cyclictest4961-21kworker/2:010:58:392
2945799251,12cyclictest13778-21perf09:00:020
2947399246,2cyclictest86550irq/126-enp1s0-10:00:513
2947399245,8cyclictest780-21rngd07:18:333
2947399245,6cyclictest780-21rngd11:25:453
2947399245,5cyclictest780-21rngd10:54:073
2947399245,5cyclictest780-21rngd10:54:073
2947399244,4cyclictest780-21rngd12:21:053
2947399244,3cyclictest3480-21/usr/sbin/munin07:20:233
2947399241,3cyclictest0-21swapper/309:15:213
2947399241,12cyclictest0-21swapper/311:30:513
2947399240,11cyclictest0-21swapper/308:31:353
2946899249,3cyclictest10336-21kworker/2:207:52:032
29468992419,3cyclictest28220-21crond12:00:012
29468992413,3cyclictest4961-21kworker/2:012:15:292
29468992413,3cyclictest4961-21kworker/2:010:04:252
2946899241,12cyclictest4961-21kworker/2:012:07:312
29462992417,4cyclictest24131-21sadc08:10:011
2946299241,5cyclictest0-21swapper/109:57:221
2946299241,18cyclictest0-21swapper/109:31:551
2946299241,15cyclictest18180-21diskmemload09:15:441
2945799242,6cyclictest22552-1kworker/u9:109:55:520
2945799241,12cyclictest26049-21perf11:55:010
2947399239,3cyclictest0-21swapper/310:39:373
2947399239,3cyclictest0-21swapper/310:12:233
2947399237,8cyclictest32656-21ntpq09:40:203
2947399236,8cyclictest86650irq/127-enp1s0-09:05:173
2947399236,8cyclictest780-21rngd12:10:533
2947399235,5cyclictest86550irq/126-enp1s0-11:56:593
2947399235,4cyclictest780-21rngd09:36:403
2947399234,3cyclictest780-21rngd10:48:013
2947399234,3cyclictest780-21rngd09:23:213
2947399232,5cyclictest0-21swapper/308:01:513
2947399231,13cyclictest0-21swapper/310:08:093
29468992312,3cyclictest4961-21kworker/2:012:13:072
29468992310,3cyclictest10336-21kworker/2:208:25:312
29468992310,2cyclictest10336-21kworker/2:208:34:012
2946299238,3cyclictest81962cat07:30:261
2946299237,3cyclictest0-21swapper/108:01:111
2946299231,21cyclictest0-21swapper/109:13:581
2946299231,17cyclictest0-21swapper/112:37:281
2946299231,17cyclictest0-21swapper/112:21:181
2945799231,6cyclictest0-21swapper/011:48:120
29457992310,2cyclictest27322-21kworker/0:011:27:180
2945799230,11cyclictest0-21swapper/011:44:280
5792221,17sleep0803-21sh12:10:130
2947399228,3cyclictest0-21swapper/308:52:513
2947399227,3cyclictest0-21swapper/308:15:413
2947399221,10cyclictest0-21swapper/308:27:433
2946899227,5cyclictest10336-21kworker/2:207:15:072
2946899227,3cyclictest4961-21kworker/2:009:46:392
2946899227,3cyclictest4961-21kworker/2:009:46:392
2946899226,9cyclictest4961-21kworker/2:010:38:512
29462992217,2cyclictest19133-21timerandwakeup07:55:251
2946299221,6cyclictest0-21swapper/112:05:151
29462992216,3cyclictest14634-21snmpd10:55:091
29462992215,2cyclictest0-21swapper/112:13:071
29462992214,3cyclictest0-21swapper/108:57:271
29462992214,2cyclictest18180-21diskmemload10:22:101
2946299221,16cyclictest0-21swapper/111:38:281
2946299221,16cyclictest0-21swapper/111:21:401
2946299221,16cyclictest0-21swapper/109:37:321
2946299221,15cyclictest0-21swapper/111:33:391
2946299221,15cyclictest0-21swapper/109:23:391
2946299221,15cyclictest0-21swapper/108:20:531
2946299221,10cyclictest0-21swapper/110:11:411
29462992210,3cyclictest2509-21ntp_states09:45:231
29462992210,3cyclictest2509-21ntp_states09:45:231
2945799221,18cyclictest12845-21munin-run11:25:010
2947399216,3cyclictest93032chrt08:49:233
2947399211,6cyclictest0-21swapper/310:21:133
2947399211,13cyclictest0-21swapper/312:25:533
2947399211,13cyclictest0-21swapper/311:10:193
2946299219,3cyclictest31806-21kernelversion08:25:171
2946299217,3cyclictest25976-21diskstats10:40:131
29462992114,2cyclictest0-21swapper/107:46:481
2946299211,3cyclictest158512sleep109:03:391
29462992113,3cyclictest0-21swapper/111:44:381
29462992113,3cyclictest0-21swapper/110:48:201
29462992113,3cyclictest0-21swapper/110:19:041
2946299211,2cyclictest18180-21diskmemload11:04:401
2946299211,2cyclictest15053-21perf11:30:011
2946299211,2cyclictest0-21swapper/111:16:071
2946299211,15cyclictest0-21swapper/112:30:531
2946299211,15cyclictest0-21swapper/112:27:171
2946299211,15cyclictest0-21swapper/112:15:071
2946299211,15cyclictest0-21swapper/111:58:471
2946299211,15cyclictest0-21swapper/111:51:551
2946299211,15cyclictest0-21swapper/110:51:591
2946299211,15cyclictest0-21swapper/110:51:591
2946299211,15cyclictest0-21swapper/110:39:101
2946299211,15cyclictest0-21swapper/110:32:451
2946299211,15cyclictest0-21swapper/110:25:181
2946299211,15cyclictest0-21swapper/108:47:161
2946299211,15cyclictest0-21swapper/107:25:521
2946299211,15cyclictest0-21swapper/107:12:391
2946299211,14cyclictest0-21swapper/108:30:041
2946299211,11cyclictest5604-21latency_hist09:55:011
2945799217,5cyclictest0-21swapper/010:21:120
2945799213,10cyclictest22162-21/usr/sbin/munin08:05:240
2945799211,3cyclictest5510-21gltestperf12:20:160
2945799210,9cyclictest0-21swapper/009:10:130
55802201,2sleep10-21swapper/108:40:151
55802201,2sleep10-21swapper/108:40:151
2947399207,3cyclictest0-21swapper/311:06:013
2947399206,7cyclictest7128-21sshd08:44:393
2947399206,7cyclictest7128-21sshd08:44:393
2947399206,4cyclictest780-21rngd08:05:573
2947399204,2cyclictest86850irq/129-enp1s0-11:22:333
2947399203,6cyclictest0-21swapper/307:34:353
2947399202,3cyclictest0-21swapper/310:40:173
29473992014,4cyclictest22786-21grep10:30:273
2946899200,10cyclictest4961-21kworker/2:009:51:372
2946299201,3cyclictest0-21swapper/110:07:211
29462992013,3cyclictest0-21swapper/108:14:271
29462992013,2cyclictest0-21swapper/108:39:401
29462992013,2cyclictest0-21swapper/108:39:401
29462992013,2cyclictest0-21swapper/108:15:551
2946299201,2cyclictest0-21swapper/112:04:521
2946299201,18cyclictest0-21swapper/110:03:381
2946299201,15cyclictest0-21swapper/107:40:061
2946299201,14cyclictest0-21swapper/111:48:171
2946299201,11cyclictest18183-21perf09:10:011
2946299201,11cyclictest11559-21latency_hist08:55:011
2946299201,10cyclictest10937-21dump-pmu-power07:40:011
2946299200,15cyclictest0-21swapper/109:41:141
2945799209,3cyclictest4570-21/usr/sbin/munin07:25:250
2945799204,3cyclictest1191-21munin-run09:45:010
2945799203,7cyclictest32380-21kworker/u8:112:36:560
2945799201,5cyclictest0-21swapper/007:46:380
2945799201,11cyclictest15564-21/usr/sbin/munin07:50:170
2945799201,11cyclictest0-21swapper/012:17:320
2945799201,11cyclictest0-21swapper/010:59:220
2945799201,11cyclictest0-21swapper/010:37:590
2945799201,10cyclictest0-21swapper/011:33:000
2945799200,10cyclictest0-21swapper/010:18:460
2947399198,6cyclictest0-21swapper/308:37:393
2947399198,6cyclictest0-21swapper/308:37:393
2947399198,2cyclictest86850irq/129-enp1s0-07:52:393
2947399195,7cyclictest0-21swapper/311:45:373
2947399194,8cyclictest780-21rngd09:02:053
2947399190,5cyclictest0-21swapper/312:16:253
2947399190,2cyclictest86650irq/127-enp1s0-07:29:413
2946299191,6cyclictest0-21swapper/111:07:361
2946299191,14cyclictest0-21swapper/107:21:081
2946299191,14cyclictest0-21swapper/107:17:071
2945799199,3cyclictest18180-21diskmemload12:00:320
2945799198,3cyclictest86550irq/126-enp1s0-09:45:170
2945799198,3cyclictest86550irq/126-enp1s0-09:45:170
2945799198,3cyclictest24361-21/usr/sbin/munin08:10:220
2945799194,6cyclictest0-21swapper/010:31:270
2945799191,9cyclictest31100-21ntp_states10:50:220
2945799191,9cyclictest31100-21ntp_states10:50:220
29457991913,3cyclictest5224-21missed_timers11:05:200
2945799191,15cyclictest0-21swapper/010:41:360
2945799191,11cyclictest0-21swapper/010:12:220
2945799191,10cyclictest8031-21/usr/sbin/munin10:00:190
2945799191,10cyclictest32696-21sed12:10:010
2945799191,10cyclictest30962-21/usr/sbin/munin08:25:110
2945799191,10cyclictest14634-21snmpd09:53:580
2945799191,10cyclictest10366-21expr10:05:120
2945799191,10cyclictest0-21swapper/011:56:370
2945799191,10cyclictest0-21swapper/011:12:480
2945799191,10cyclictest0-21swapper/011:01:230
2945799191,10cyclictest0-21swapper/009:24:280
2945799191,10cyclictest0-21swapper/009:16:580
2947399181,3cyclictest0-21swapper/309:26:253
29462991810,3cyclictest0-21swapper/107:50:471
2945799188,3cyclictest0-21swapper/009:31:570
2945799181,9cyclictest8674-21/usr/sbin/munin11:15:120
2945799181,9cyclictest32535-21/usr/sbin/munin07:15:150
2945799181,9cyclictest14634-21snmpd08:01:380
2945799181,8cyclictest0-21swapper/012:27:520
29457991813,3cyclictest5201-21/usr/sbin/munin08:40:200
29457991813,3cyclictest5201-21/usr/sbin/munin08:40:190
2945799181,10cyclictest0-21swapper/008:36:400
2945799181,10cyclictest0-21swapper/008:36:390
2947399173,12cyclictest14874-21tail07:45:263
2945799172,13cyclictest720-21/usr/sbin/munin08:30:150
2945799171,8cyclictest784-21NetworkManager09:04:500
29457991712,3cyclictest30311-21head08:20:260
29457991712,2cyclictest29646-21/usr/sbin/munin07:10:240
29457991711,3cyclictest1-21systemd08:50:090
2945799165,3cyclictest86750irq/128-enp1s0-08:45:250
2945799162,2cyclictest22552-1kworker/u9:110:25:110
29457991610,3cyclictest26789-21expr08:15:120
29457991610,3cyclictest10487-21grep07:35:250
2945799160,9cyclictest0-21swapper/007:55:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional