You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-16 - 04:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Jul 16, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
323492237161,24sleep00-21swapper/019:08:330
322722237165,24sleep10-21swapper/119:07:321
322062231164,54sleep20-21swapper/219:06:412
322072203165,25sleep30-21swapper/319:06:423
1406121810,4sleep23255699cyclictest22:05:152
2508521450,4sleep33256399cyclictest21:15:243
32556991091,106cyclictest0-21swapper/221:35:592
24782970,5sleep03254299cyclictest21:40:130
24782970,5sleep03254299cyclictest21:40:130
32556996760,4cyclictest8511-21fschecks_time00:20:172
32556996359,3cyclictest0-21swapper/219:44:192
199202430,1sleep30-21swapper/319:50:243
3254299368,13cyclictest18115-21idleruntime-cro23:30:010
3254299315,12cyclictest4920-21perf23:00:010
3254299311,23cyclictest87350irq/126-enp1s0-22:45:120
3255099291,18cyclictest0-21swapper/121:17:211
3255099284,21cyclictest221992chrt23:36:241
3255099281,6cyclictest0-21swapper/122:21:471
3256399271,6cyclictest0-21swapper/323:25:583
3255099271,2cyclictest0-21swapper/121:10:221
3255099271,15cyclictest87650irq/129-enp1s0-00:15:161
3254299271,21cyclictest0-21swapper/021:28:150
3254299270,3cyclictest0-21swapper/023:33:090
278992270,4sleep2311ktimersoftd/222:35:252
175212272,3sleep117533-21ntp_states19:45:211
3255699261,4cyclictest0-21swapper/221:57:462
3255099261,2cyclictest0-21swapper/120:58:021
3255099261,2cyclictest0-21swapper/119:28:281
3255099261,11cyclictest10607-21tr20:45:131
3255099261,11cyclictest10221-21diskstats19:30:141
51072250,4sleep23255699cyclictest23:00:012
3255699257,9cyclictest18107-21awk23:30:012
3255699251,4cyclictest0-21swapper/200:03:362
3255699251,17cyclictest0-21swapper/219:35:172
3255099252,7cyclictest87650irq/129-enp1s0-22:50:201
3255099252,7cyclictest87650irq/129-enp1s0-22:50:201
3255099251,4cyclictest4621-21/usr/sbin/munin19:15:221
3255099251,21cyclictest15668-21grep22:10:131
3255099251,16cyclictest0-21swapper/121:28:531
3255099251,11cyclictest0-21swapper/123:25:031
3255099250,6cyclictest0-21swapper/123:24:191
3255099250,3cyclictest0-21swapper/123:53:521
258112250,2sleep011-21rcuc/022:30:260
123722250,2sleep10-21swapper/122:00:251
3256399240,10cyclictest0-21swapper/319:40:113
3255699249,3cyclictest87650irq/129-enp1s0-22:10:172
3255699242,8cyclictest32-21ksoftirqd/221:43:122
3255699242,8cyclictest32-21ksoftirqd/221:43:122
3255699241,7cyclictest25281-21vmstat21:15:252
3255699241,6cyclictest0-21swapper/221:53:462
3255699241,5cyclictest0-21swapper/221:25:392
3255699241,5cyclictest0-21swapper/200:16:532
3255699241,3cyclictest0-21swapper/222:54:462
3255699241,3cyclictest0-21swapper/222:54:462
3255699241,3cyclictest0-21swapper/221:20:022
3255099241,5cyclictest0-21swapper/122:44:121
3255099241,4cyclictest14048-21awk19:40:011
3255099241,3cyclictest9828-21memory21:55:191
3255099241,3cyclictest26328-21ntp_states20:05:211
3254299249,2cyclictest7-21ksoftirqd/000:05:250
3254299241,21cyclictest0-21swapper/020:20:200
3254299241,12cyclictest0-21swapper/019:15:010
3254299240,22cyclictest0-21swapper/000:17:210
3256399231,15cyclictest0-21swapper/300:29:563
3255699238,4cyclictest14665-21awk20:55:012
3255699234,9cyclictest3092-21load19:15:042
3255699231,6cyclictest0-21swapper/221:14:432
3255699231,6cyclictest0-21swapper/200:10:182
3255699231,4cyclictest18033-21kworker/u8:023:33:092
3255699231,3cyclictest0-21swapper/221:32:332
3255099232,3cyclictest0-21swapper/121:05:581
3255099232,2cyclictest87650irq/129-enp1s0-23:42:051
3255099232,2cyclictest87650irq/129-enp1s0-21:41:351
3255099232,2cyclictest87650irq/129-enp1s0-21:41:341
3255099231,8cyclictest226892grep22:25:151
3255099231,5cyclictest0-21swapper/123:32:271
3255099231,3cyclictest31752-21sed20:20:011
3255099231,3cyclictest13533-21latency20:50:191
3255099231,2cyclictest0-21swapper/100:38:541
3255099231,19cyclictest1461-21perf20:25:011
3255099230,6cyclictest0-21swapper/100:02:191
3254299239,4cyclictest0-21swapper/021:45:150
3254299239,4cyclictest0-21swapper/021:10:260
3254299231,5cyclictest0-21swapper/022:51:540
3254299231,5cyclictest0-21swapper/022:51:540
3254299231,16cyclictest0-21swapper/022:08:530
32542992310,4cyclictest0-21swapper/023:15:190
3254299230,3cyclictest28851-21kworker/u8:100:03:360
32563992212,3cyclictest87650irq/129-enp1s0-00:06:463
3255699227,5cyclictest14667-21cpuspeed_turbos00:35:142
3255699226,9cyclictest0-21swapper/222:19:432
3255699222,11cyclictest26277-21munin-run22:35:012
3255699221,4cyclictest0-21swapper/222:01:402
3255699221,10cyclictest28851-21kworker/u8:123:55:592
3255699221,10cyclictest0-21swapper/223:48:382
3255099229,3cyclictest87550irq/128-enp1s0-19:55:231
3255099223,3cyclictest89150irq/134-enp2s0-21:50:271
3255099222,5cyclictest0-21swapper/123:47:331
3255099221,5cyclictest0-21swapper/123:06:001
3255099221,3cyclictest0-21swapper/100:12:411
3255099221,2cyclictest0-21swapper/122:38:521
3255099220,14cyclictest0-21swapper/121:48:581
3254299229,3cyclictest0-21swapper/021:55:130
3254299227,9cyclictest0-21swapper/000:25:200
3254299222,17cyclictest26868-21cut22:35:150
3254299221,7cyclictest0-21swapper/021:06:240
3254299221,2cyclictest87350irq/126-enp1s0-23:22:440
3254299221,16cyclictest0-21swapper/021:20:180
3254299221,14cyclictest0-21swapper/021:52:160
3254299220,3cyclictest0-21swapper/000:14:300
3254299220,18cyclictest0-21swapper/023:01:430
3254299220,11cyclictest11-21rcuc/020:55:010
255042220,2sleep30-21swapper/320:05:133
3256399211,6cyclictest0-21swapper/321:11:553
3255699214,9cyclictest21261-21diskmemload23:00:112
3255699214,8cyclictest87350irq/126-enp1s0-21:46:392
3255699214,4cyclictest13711-21perf23:20:012
3255699214,10cyclictest21261-21diskmemload22:41:512
3255699213,6cyclictest0-21swapper/220:12:452
3255699212,5cyclictest24693-21munin-run23:45:012
3255699211,7cyclictest0-21swapper/220:55:142
3255699211,6cyclictest579-21systemd-journal19:30:012
3255699211,15cyclictest0-21swapper/220:04:152
3255699211,12cyclictest15912-21awk23:25:012
3255699211,11cyclictest22101-21/usr/sbin/munin22:25:132
3255699211,11cyclictest1904-21expr20:25:122
3255699211,11cyclictest17076-21/usr/sbin/munin21:00:182
3255699211,10cyclictest0-21swapper/200:32:592
3255099216,2cyclictest89050irq/133-enp2s0-19:10:241
3255099212,6cyclictest0-21swapper/122:33:231
3255099212,17cyclictest0-21swapper/121:05:001
3255099211,8cyclictest29891-21perl23:55:131
32550992117,3cyclictest0-21swapper/122:56:231
3255099211,5cyclictest8003-21grep00:20:121
3255099211,5cyclictest0-21swapper/122:06:011
3255099211,3cyclictest0-21swapper/122:16:571
3255099211,3cyclictest0-21swapper/120:01:221
3255099211,18cyclictest0-21swapper/120:27:061
3255099211,18cyclictest0-21swapper/120:11:171
3255099211,12cyclictest0-21swapper/121:30:491
3255099211,11cyclictest0-21swapper/123:04:441
3255099210,7cyclictest0-21swapper/100:25:161
3255099210,10cyclictest0-21swapper/123:16:511
3254299219,8cyclictest0-21swapper/023:11:560
3254299219,3cyclictest0-21swapper/023:06:460
3254299217,3cyclictest20868-21diskstats23:35:140
3254299215,8cyclictest1700-21/usr/sbin/munin20:25:190
3254299213,10cyclictest0-21swapper/022:30:010
3254299212,12cyclictest0-21swapper/021:15:450
3254299211,3cyclictest16066-21cut22:10:160
3254299211,13cyclictest0-21swapper/023:55:110
3254299211,12cyclictest0-21swapper/023:51:300
3254299210,19cyclictest0-21swapper/019:30:230
3256399209,3cyclictest12074-21/usr/sbin/munin19:35:193
3256399204,4cyclictest9-21rcu_preempt23:30:183
3256399202,5cyclictest18245-1kworker/u9:222:48:413
3256399201,8cyclictest0-21swapper/320:30:173
3256399201,8cyclictest0-21swapper/320:30:173
3256399201,7cyclictest0-21swapper/323:54:043
3256399201,5cyclictest10264-21munin-run20:45:023
3256399201,4cyclictest18033-21kworker/u8:023:42:013
3256399201,11cyclictest5480-21/usr/sbin/munin19:20:233
3256399201,11cyclictest5480-21/usr/sbin/munin19:20:233
3256399201,11cyclictest13930-21/usr/sbin/munin23:20:183
32563992010,3cyclictest87650irq/129-enp1s0-23:11:123
3255699204,9cyclictest30-21rcuc/220:40:062
3255699204,4cyclictest20630-21sh19:55:012
3255699202,9cyclictest87350irq/126-enp1s0-23:13:052
3255699202,5cyclictest763-21dbus-daemon19:20:202
3255699202,5cyclictest763-21dbus-daemon19:20:202
3255699201,7cyclictest26204-21missed_timers20:05:192
3255699201,7cyclictest17205-21if_enp1s019:45:172
3255699201,7cyclictest0-21swapper/222:46:432
3255699201,6cyclictest0-21swapper/222:20:362
3255699201,5cyclictest20917-21fschecks_count23:35:152
3255699201,3cyclictest0-21swapper/223:08:562
3255699201,11cyclictest808-21snmpd20:38:542
3255699201,10cyclictest32-21ksoftirqd/200:06:082
3255699201,10cyclictest11250-21ntp_states00:25:222
3255699200,15cyclictest0-21swapper/220:45:412
3255099209,9cyclictest11506-21latency_hist23:15:011
3255099207,3cyclictest13429-21ntpq00:30:191
3255099207,12cyclictest0-21swapper/121:20:551
3255099205,4cyclictest0-21swapper/122:48:411
3255099204,7cyclictest87550irq/128-enp1s0-20:40:141
32550992016,3cyclictest0-21swapper/120:31:151
32550992016,3cyclictest0-21swapper/120:31:151
32550992016,3cyclictest0-21swapper/100:05:181
3255099201,3cyclictest0-21swapper/121:35:531
3255099201,3cyclictest0-21swapper/119:21:011
3255099201,3cyclictest0-21swapper/119:21:011
3255099201,18cyclictest0-21swapper/119:54:581
3255099201,17cyclictest0-21swapper/119:41:071
3254299209,2cyclictest7-21ksoftirqd/000:30:120
3254299209,2cyclictest0-21swapper/020:15:120
3254299207,4cyclictest0-21swapper/022:00:190
3254299207,4cyclictest0-21swapper/021:37:450
3254299206,4cyclictest15747-21ps00:35:230
3254299205,7cyclictest5661-21cstates19:20:120
3254299205,7cyclictest5661-21cstates19:20:120
3254299201,3cyclictest11-21rcuc/023:45:160
32542992012,6cyclictest7-21ksoftirqd/000:25:010
3254299201,16cyclictest23790-21ntpq23:40:210
3254299201,10cyclictest0-21swapper/022:20:590
3256399199,3cyclictest29312-21/usr/sbin/munin23:55:173
3256399194,6cyclictest11109-21/usr/sbin/munin22:00:183
3256399191,5cyclictest1799-21ntp_states22:50:203
3256399191,5cyclictest1799-21ntp_states22:50:203
3256399191,4cyclictest0-21swapper/322:39:023
3256399191,4cyclictest0-21swapper/321:53:173
32563991914,3cyclictest20574-21hddtemp_smartct22:20:153
3256399191,3cyclictest7919-21perl23:05:153
3256399191,3cyclictest0-21swapper/300:37:553
32563991913,3cyclictest6441-21cut00:15:153
3256399191,10cyclictest839-21lldpd00:33:063
3256399191,10cyclictest19291-21/usr/sbin/munin21:05:233
3256399191,10cyclictest12687-21/usr/sbin/munin20:50:183
32563991910,7cyclictest26084-21sshd22:32:463
3256399190,6cyclictest18245-1kworker/u9:200:12:533
3255699191,9cyclictest5866-21latency_hist20:35:012
3255699191,9cyclictest5866-21latency_hist20:35:012
3255699191,6cyclictest32750-21/usr/sbin/munin19:10:262
3255699191,6cyclictest21988-21pmu-power19:55:222
3255699191,10cyclictest32394-21grep20:20:162
3255699191,10cyclictest28046-21missed_timers23:50:192
3255699190,7cyclictest20705-21sed21:05:282
3254299199,3cyclictest0-21swapper/021:00:240
3254299198,2cyclictest0-21swapper/022:15:190
3254299192,14cyclictest8295-21gltestperf19:25:150
3254299191,7cyclictest0-21swapper/022:40:200
32542991913,3cyclictest12882-21interrupts19:35:180
3254299191,15cyclictest3899-21/usr/sbin/munin20:30:160
3254299191,15cyclictest3899-21/usr/sbin/munin20:30:160
3254299191,15cyclictest27461-21/usr/sbin/munin20:10:230
3254299191,15cyclictest14884-21/usr/sbin/munin20:55:170
3256399188,3cyclictest808-21snmpd21:03:583
3256399186,4cyclictest21261-21diskmemload21:56:293
3256399181,9cyclictest808-21snmpd19:18:393
3256399181,8cyclictest0-21swapper/321:45:593
3256399181,7cyclictest13698-21sh23:20:013
3256399181,7cyclictest0-21swapper/322:19:203
3256399181,7cyclictest0-21swapper/321:35:433
3256399181,7cyclictest0-21swapper/321:27:593
3256399181,6cyclictest15512-21/usr/sbin/munin22:10:193
3256399181,6cyclictest0-21swapper/321:42:503
3256399181,6cyclictest0-21swapper/321:42:503
3256399181,5cyclictest18245-1kworker/u9:223:47:253
32563991813,3cyclictest30942-21grep21:30:163
3256399181,10cyclictest7985-21/usr/sbin/munin00:20:183
3256399181,10cyclictest0-21swapper/320:45:273
3256399181,10cyclictest0-21swapper/319:33:423
3255699185,3cyclictest87350irq/126-enp1s0-19:34:052
3255699181,5cyclictest29660-21/usr/sbin/munin20:15:262
3255099181,16cyclictest0-21swapper/120:35:571
3254299188,2cyclictest0-21swapper/021:30:120
3254299188,2cyclictest0-21swapper/020:05:190
3254299185,7cyclictest0-21swapper/020:35:130
3254299181,5cyclictest18662-21/usr/sbin/munin19:50:180
3254299181,5cyclictest10724-21expr20:45:140
32542991812,4cyclictest24304-21sendmail_mailtr20:00:240
3254299180,3cyclictest0-21swapper/019:15:180
291202181,3sleep30-21swapper/320:11:083
164932181,3sleep30-21swapper/320:55:343
3256399175,3cyclictest0-21swapper/300:00:123
3256399174,4cyclictest5190-21egrep23:00:103
3256399174,3cyclictest29036-21df_inode22:40:133
3256399172,8cyclictest16466-21/usr/sbin/munin19:45:283
3256399171,7cyclictest30637-21needreboot20:15:203
3256399171,7cyclictest22446-21/usr/sbin/munin19:55:263
3256399171,7cyclictest0-21swapper/322:56:533
3256399171,6cyclictest9632-21chrt19:29:573
3256399171,6cyclictest0-21swapper/321:24:353
3256399171,4cyclictest0-21swapper/322:07:313
32563991713,3cyclictest0-21swapper/320:20:453
32563991710,4cyclictest20916-21fschecks_count23:35:153
3254299171,5cyclictest16466-21/usr/sbin/munin19:45:120
32542991710,4cyclictest15422-21proc_pri19:40:210
3254299170,8cyclictest0-21swapper/019:55:200
3256399165,3cyclictest87350irq/126-enp1s0-22:25:143
3256399165,3cyclictest32750-21/usr/sbin/munin19:10:213
3256399161,14cyclictest0-21swapper/320:37:543
3256399161,14cyclictest0-21swapper/320:25:203
32563991611,3cyclictest24643-21users20:00:263
3254299161,14cyclictest0-21swapper/020:41:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional