You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 13:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Dec 10, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291412240164,24sleep30-21swapper/319:09:293
288392232161,58sleep00-21swapper/019:05:370
289422231163,54sleep10-21swapper/119:06:541
290712228155,23sleep20-21swapper/219:08:362
2178021500,4sleep22928599cyclictest22:30:202
13042560,3sleep32928999cyclictest21:45:123
313162520,1sleep10-21swapper/120:25:181
30112510,1sleep00-21swapper/019:20:170
101782460,1sleep30-21swapper/319:35:273
294512420,2sleep129456-21pmu-power00:00:221
246062420,3sleep30-21swapper/323:50:173
29272993511,11cyclictest20845-21awk22:30:010
2928599331,22cyclictest3459-21kworker/2:200:10:282
2927299335,13cyclictest25241-21munin-run22:40:010
29285993219,3cyclictest18843-21kworker/2:020:19:372
2928599321,26cyclictest18843-21kworker/2:021:49:522
2928599320,26cyclictest3459-21kworker/2:223:16:572
2927299328,11cyclictest21780-21awk20:05:000
2927299328,11cyclictest0-21swapper/022:00:010
2927299327,12cyclictest29007-21perf21:35:010
2927299323,13cyclictest21467-21munin-run23:45:010
8382310,2sleep20-21swapper/220:30:122
2928599312,20cyclictest3459-21kworker/2:223:22:152
2928599301,27cyclictest3459-21kworker/2:200:22:042
2928599301,20cyclictest18843-21kworker/2:021:42:002
2928599299,3cyclictest18843-21kworker/2:020:38:492
2928599291,19cyclictest3459-21kworker/2:200:31:432
2928599291,18cyclictest3459-21kworker/2:221:51:082
29272992911,15cyclictest29648-21awk22:50:010
262242290,5sleep02927299cyclictest22:40:170
2928599281,22cyclictest3459-21kworker/2:223:33:062
2928599281,12cyclictest3459-21kworker/2:223:35:462
2928599271,11cyclictest3459-21kworker/2:223:45:552
2928599270,11cyclictest18843-21kworker/2:021:33:522
2928999261,10cyclictest41-21ksoftirqd/320:14:373
29285992616,3cyclictest18843-21kworker/2:020:22:032
29285992616,3cyclictest18843-21kworker/2:020:22:032
29285992611,3cyclictest18843-21kworker/2:020:06:512
2928599261,11cyclictest18843-21kworker/2:021:16:342
2927899261,16cyclictest14634-21snmpd23:54:001
2927899260,11cyclictest0-21swapper/121:18:191
2927299262,12cyclictest25870-21munin-run23:55:010
2927299262,11cyclictest21066-21/usr/sbin/munin22:30:160
2927299261,11cyclictest32061-21/usr/sbin/munin22:55:230
2927299261,10cyclictest9426-21/usr/sbin/munin20:50:230
2928999252,11cyclictest0-21swapper/322:21:193
2928999251,6cyclictest18864-21/usr/sbin/munin22:25:173
2928999251,6cyclictest0-21swapper/322:35:293
2928999251,12cyclictest0-21swapper/323:40:123
2928599251,18cyclictest3459-21kworker/2:223:59:122
2928599251,13cyclictest3459-21kworker/2:222:00:222
29285992511,3cyclictest3459-21kworker/2:200:15:202
2928599251,10cyclictest3459-21kworker/2:223:04:032
2928599251,10cyclictest3459-21kworker/2:222:52:072
29285992510,3cyclictest3459-21kworker/2:223:40:382
2927899251,16cyclictest409-21irqstats22:55:201
29278992511,12cyclictest26688-21kworker/u8:021:37:111
2927899250,20cyclictest0-21swapper/122:21:051
2927299252,11cyclictest32727-21/usr/sbin/munin00:10:270
2927299252,10cyclictest1898-21/usr/sbin/munin23:00:230
2927299251,11cyclictest7858-21/usr/sbin/munin22:00:210
29289992417,4cyclictest25440-21head21:25:193
29289992417,4cyclictest25440-21head21:25:193
2928999241,5cyclictest0-21swapper/322:10:183
2928999240,11cyclictest0-21swapper/320:55:213
2928599248,3cyclictest18843-21kworker/2:019:45:212
2928599241,11cyclictest3459-21kworker/2:222:42:582
29285992410,8cyclictest18843-21kworker/2:020:10:332
29285992410,2cyclictest3459-21kworker/2:222:57:402
2927899242,3cyclictest24754-21perl20:10:191
2927899241,7cyclictest2898-21df00:15:131
29278992413,3cyclictest6144-21ls23:10:021
2927899241,20cyclictest19550-21crond20:00:011
2927299242,10cyclictest29476-21/usr/sbin/munin19:10:130
2927299241,4cyclictest0-21swapper/000:31:560
2928999236,5cyclictest0-21swapper/300:02:513
2928999231,6cyclictest0-21swapper/323:25:193
2928999231,6cyclictest0-21swapper/322:51:343
2928999231,4cyclictest21771-21timerandwakeup21:15:263
2928599239,8cyclictest86850irq/129-enp1s0-22:19:392
2928599232,11cyclictest3459-21kworker/2:222:24:582
2928599231,7cyclictest0-21swapper/222:25:532
2928599231,10cyclictest3459-21kworker/2:221:59:242
29285992310,3cyclictest3459-21kworker/2:200:05:142
29285992310,2cyclictest3459-21kworker/2:223:14:052
2927899232,3cyclictest18106-21irqstats23:35:191
2927899231,9cyclictest28572-21memory22:45:191
2927899231,7cyclictest0-21swapper/123:59:501
29278992313,3cyclictest18363-21cut21:10:141
2927899231,16cyclictest31252-21sshd19:12:421
2927899231,14cyclictest28998-21munin-run21:35:011
2927299238,5cyclictest0-21swapper/000:05:270
2927299233,3cyclictest86750irq/128-enp1s0-21:46:470
2927299231,10cyclictest14463-21/usr/sbin/munin22:15:190
5352221,2sleep0537-21acpi20:30:110
2928999224,9cyclictest12049-21ntpq19:40:203
2928999221,6cyclictest0-21swapper/322:50:013
2928999221,6cyclictest0-21swapper/300:30:553
2928999221,3cyclictest86550irq/126-enp1s0-21:35:013
2928999221,14cyclictest39-21rcuc/323:12:413
2928999221,11cyclictest18008-21diskmemload22:00:253
2928599229,7cyclictest3459-21kworker/2:222:10:342
2928599228,3cyclictest3459-21kworker/2:200:25:402
2928599227,3cyclictest3459-21kworker/2:222:36:202
2928599226,10cyclictest32-21ksoftirqd/220:01:092
2928599223,8cyclictest32-21ksoftirqd/220:43:412
2928599222,12cyclictest18843-21kworker/2:021:30:022
2928599221,11cyclictest18843-21kworker/2:020:57:352
29285992210,7cyclictest86750irq/128-enp1s0-19:23:252
29285992210,3cyclictest18843-21kworker/2:021:40:022
2928599220,12cyclictest18843-21kworker/2:021:20:162
2927899222,4cyclictest32298-21sshd00:07:371
2927899221,3cyclictest86550irq/126-enp1s0-23:20:181
2927899221,3cyclictest21618-21taskset20:03:141
2927899221,3cyclictest11909-21memory19:40:181
2927899221,19cyclictest0-21swapper/100:25:251
2927899221,14cyclictest11175-21sed00:35:011
2927899221,14cyclictest0-21swapper/123:15:201
2927899221,13cyclictest8779-21grep22:00:201
2927899221,10cyclictest21977-21memory22:30:211
2927899220,2cyclictest0-21swapper/120:42:411
2927899220,16cyclictest0-21swapper/121:41:561
2927299223,2cyclictest86750irq/128-enp1s0-23:11:200
2927299221,5cyclictest0-21swapper/022:24:160
2927299221,3cyclictest0-21swapper/000:29:450
23872220,1sleep30-21swapper/319:20:133
2928999216,4cyclictest0-21swapper/300:15:433
2928999212,6cyclictest10472-1kworker/u9:021:53:383
2928999211,6cyclictest0-21swapper/323:36:023
2928999211,6cyclictest0-21swapper/323:15:073
2928999211,6cyclictest0-21swapper/321:37:373
2928999211,6cyclictest0-21swapper/321:20:503
2928999211,6cyclictest0-21swapper/300:07:063
2928999211,5cyclictest0-21swapper/323:00:093
2928999211,5cyclictest0-21swapper/322:31:123
2928999211,5cyclictest0-21swapper/321:58:463
2928999211,5cyclictest0-21swapper/321:44:373
2928999211,4cyclictest0-21swapper/322:06:463
2928599219,2cyclictest3459-21kworker/2:200:02:022
2928599215,3cyclictest18843-21kworker/2:019:28:152
2928599215,10cyclictest18843-21kworker/2:020:27:572
2928599214,8cyclictest32-21ksoftirqd/219:32:532
2928599211,5cyclictest26688-21kworker/u8:021:29:572
2928599211,4cyclictest0-21swapper/223:54:322
2928599211,4cyclictest0-21swapper/223:07:322
2928599211,12cyclictest11029-21kworker/2:100:35:282
2928599211,10cyclictest18843-21kworker/2:021:11:392
29285992110,3cyclictest86750irq/128-enp1s0-19:36:332
2928599210,15cyclictest32-21ksoftirqd/221:02:282
2927899212,14cyclictest0-21swapper/122:37:541
29278992111,2cyclictest6307-21/usr/sbin/munin23:10:141
2927299218,3cyclictest31371-21unixbench_multi22:50:260
2927299215,13cyclictest7830-21head20:45:170
2927299212,6cyclictest18001-21munin-run21:10:000
2927299212,2cyclictest86750irq/128-enp1s0-22:05:150
2927299211,7cyclictest0-21swapper/023:21:310
118062211,16sleep011964-21df_inode20:55:160
2928999201,9cyclictest705-21awk22:55:243
2928999201,4cyclictest0-21swapper/323:58:273
2928999201,4cyclictest0-21swapper/323:45:063
2928999201,4cyclictest0-21swapper/321:10:463
2928999201,12cyclictest0-21swapper/300:35:283
2928999201,11cyclictest0-21swapper/322:17:533
2928999201,11cyclictest0-21swapper/300:20:353
2928999200,14cyclictest0-21swapper/323:21:523
2928599207,2cyclictest3459-21kworker/2:222:07:362
2928599205,8cyclictest32-21ksoftirqd/219:55:572
2928599204,9cyclictest18843-21kworker/2:019:11:532
2928599204,2cyclictest32-21ksoftirqd/219:42:132
2928599203,10cyclictest32-21ksoftirqd/220:52:412
2927899206,3cyclictest0-21swapper/122:25:551
2927899206,3cyclictest0-21swapper/121:24:411
2927899203,14cyclictest1632-21cut21:45:151
2927899203,14cyclictest15750-21grep23:30:161
2927899201,8cyclictest23892wc19:20:121
29278992017,2cyclictest0-21swapper/122:52:571
2927899201,5cyclictest0-21swapper/122:10:521
2927899201,5cyclictest0-21swapper/120:16:581
2927899201,2cyclictest0-21swapper/119:54:011
2927899201,18cyclictest0-21swapper/119:27:111
2927899201,17cyclictest0-21swapper/120:56:131
2927899201,17cyclictest0-21swapper/100:21:531
2927899201,14cyclictest0-21swapper/121:55:321
2927899201,13cyclictest0-21swapper/123:02:271
2927899201,12cyclictest2414-21cat00:15:021
2927899201,10cyclictest32548-21cstates19:15:141
2927899201,10cyclictest32548-21cstates19:15:141
29278992010,2cyclictest0-21swapper/123:26:091
2927899200,8cyclictest0-21swapper/123:40:411
2927899200,15cyclictest0-21swapper/100:35:421
2927899200,14cyclictest0-21swapper/121:27:591
2927899200,14cyclictest0-21swapper/121:27:591
2927299209,2cyclictest15393-21/usr/sbin/munin19:50:230
2927299208,3cyclictest22017-21users21:15:270
2927299201,9cyclictest30446-21awk21:35:250
2927299201,9cyclictest0-21swapper/021:25:190
2927299201,9cyclictest0-21swapper/021:25:190
2927299201,4cyclictest0-21swapper/022:12:260
2927299200,5cyclictest0-21swapper/023:06:110
2928999199,3cyclictest14634-21snmpd21:00:423
2928999196,4cyclictest0-21swapper/300:25:433
2928999191,7cyclictest0-21swapper/320:46:373
2928999191,11cyclictest14634-21snmpd20:35:383
2928999191,10cyclictest0-21swapper/323:30:403
2928999191,10cyclictest0-21swapper/323:05:103
2928999191,10cyclictest0-21swapper/322:40:093
2928599197,2cyclictest86750irq/128-enp1s0-20:49:332
2928599196,2cyclictest32-21ksoftirqd/222:48:242
2928599195,3cyclictest18843-21kworker/2:019:50:232
2928599194,3cyclictest18843-21kworker/2:019:15:322
2928599194,3cyclictest18843-21kworker/2:019:15:312
2928599193,5cyclictest17997-21sadc21:10:012
29285991910,2cyclictest86650irq/127-enp1s0-23:26:222
2927899197,4cyclictest0-21swapper/122:06:271
29278991916,2cyclictest0-21swapper/119:47:081
2927899191,3cyclictest0-21swapper/120:47:351
29278991913,3cyclictest7510-21memory19:30:191
29278991913,3cyclictest10453-21/usr/sbin/munin20:50:231
2927899191,16cyclictest0-21swapper/120:32:131
2927899191,15cyclictest2369050sshd20:06:411
2927899191,12cyclictest22274-21fw_packets23:45:161
2927899191,10cyclictest17216-21processes21:05:241
2927899190,12cyclictest0-21swapper/121:51:421
2927299196,3cyclictest5018-21grep21:50:260
2927299195,3cyclictest0-21swapper/021:23:190
2927299193,3cyclictest0-21swapper/000:04:050
2927299191,7cyclictest0-21swapper/000:21:570
2927299191,3cyclictest0-21swapper/021:11:160
29272991913,3cyclictest27984-21users20:15:310
9912181,2sleep00-21swapper/021:44:540
2928999185,6cyclictest780-21rngd20:50:243
2928999182,15cyclictest0-21swapper/319:11:023
2928999181,6cyclictest0-21swapper/300:12:113
2928999181,2cyclictest17506-21sed21:05:263
29289991811,4cyclictest7954-21sed19:30:263
2928999181,14cyclictest14634-21snmpd19:26:333
2928999181,11cyclictest0-21swapper/319:45:203
2928999181,10cyclictest842-21lldpd20:06:113
29289991810,2cyclictest86550irq/126-enp1s0-19:51:583
29278991815,2cyclictest0-21swapper/122:41:141
2927899181,16cyclictest0-21swapper/121:00:051
2927899181,16cyclictest0-21swapper/120:35:311
2927899181,16cyclictest0-21swapper/120:22:161
2927899181,16cyclictest0-21swapper/120:22:161
29278991810,3cyclictest0-21swapper/119:35:141
29278991810,2cyclictest0-21swapper/122:15:121
2927299186,3cyclictest86750irq/128-enp1s0-23:35:270
2927299186,3cyclictest17588-21/usr/sbin/munin19:55:240
2927299182,13cyclictest21692-21/usr/sbin/munin23:45:140
2927299181,9cyclictest0-21swapper/020:20:190
2927299181,9cyclictest0-21swapper/020:20:190
2927299181,9cyclictest0-21swapper/019:35:230
2927299181,8cyclictest0-21swapper/023:55:210
2927299181,7cyclictest13033-21tr23:25:130
2927299181,6cyclictest0-21swapper/000:17:210
2927299181,5cyclictest5766-21/usr/sbin/munin19:25:250
29272991813,3cyclictest15647-21seq21:03:100
2927299181,14cyclictest6597-21/usr/sbin/munin19:30:140
2927299181,14cyclictest10198-21sshd23:16:410
2928999171,14cyclictest0-21swapper/320:25:083
2928999171,14cyclictest0-21swapper/320:01:033
2928999171,12cyclictest32132-21grep19:15:043
2928999171,12cyclictest32132-21grep19:15:033
2927299171,9cyclictest0-21swapper/023:31:440
2927299171,7cyclictest31516-21basename20:25:190
2927299171,6cyclictest3953-21pmu-power20:35:210
2927299171,6cyclictest18008-21diskmemload00:35:130
2927299171,15cyclictest0-21swapper/020:12:420
2927299171,14cyclictest0-21swapper/019:40:450
29272991711,3cyclictest836-21runrttasks20:42:000
281002171,2sleep30-21swapper/320:16:503
10552171,13sleep01155-21ps19:15:250
10552171,13sleep01155-21ps19:15:250
9532160,2sleep30-21swapper/320:30:143
2928999161,6cyclictest86550irq/126-enp1s0-20:20:143
2928999161,6cyclictest86550irq/126-enp1s0-20:20:143
29289991611,3cyclictest18635-21ntp_kernel_pll_19:55:213
29289991610,3cyclictest6459-21timerwakeupswit20:40:263
2927299161,12cyclictest13193-21/usr/sbin/munin19:45:240
29272991610,3cyclictest22083-21ls20:05:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional